img
Home > Candidate Patterns > Reference Data Distribution

Reference Data Distribution

How can business reference data be automatically distributed to consumers, while being extensible to support more consumers, without changing the distribution logic for every new system connected to the distribution mechanism?

Problem

When reference data is centrally managed in a SOA. A common approach to distribute this is to propagate all changes to all surrounding interested consumers by application of a dedicated distribution service which knows where-to to distribute the data. Problem with this approach is that for every new target the data must be delivered to, the service must be changed. This does not allow for a future extensible mechanism where it is irrelevant to which system the data is distributed.

Solution

Publish the reference data in the service bus. Create a separate consumer for every event-target. This dedicated service subscribes to relevant events and is used to update one specific system with new reference data.

Application

For every target, create an intermediate service which sole responsibility is to provide data forwarding to that specific target. The source data for propagating are managed event data as published by the Reference Data Observer pattern.

Impacts

Because a new consumer service is created for every target system interested in a particular reference data, the amount of design-time effort as well as cost of implementation is considered high.

The implementation of the message exchange infrastructure is considered somewhat more elaborate.

Architecture

Service Inventory, Service

Status

Under Review

Contributors

Roger Stoffers

Table: Profile summary for the Reference Data Distribution pattern.

Problem

img

Figure 1 - A central composition coordinator service publishes received data to a number of known target systems. These must be known up-front and every new target system which is to be supported requires a core logic change in this service.

Solution

img

Figure 2 - The message infrastructure duplicates the received reference data update and publishes onto dedicated queues.

Application

Create a topic to which reference data is published (the reference data observer pattern can be used to achieve this). Create a 'durable' subscription for every target system (inherently this requires a queue for every target system which holds a permanent copy of the data published on the topic. This helps resolving availability issues of the target system. Create a dedicated consumer for every target system and subscribe it to the queue intended for that system. The consumer service is now responsible for successfully propagating the reference data into the target system. Some target systems only require updates to reference data, some require full reference data sets upon every change event issued to this pattern. The Reference Data Centralization pattern describes a reference data management pattern and also provides service access to allow for retrieving those full datasets. Ie. if a country code changes, the centrally managed reference data is updated by that pattern and the service to retrieve the full set of country codes reference data can be accessed in order to enrich the update to the underlying target system.

img

Figure 3 - The messaging infrastructure duplicates the data into dedicated queues, one for each target consumer. Every underlying system is assigned a dedicated service to read a specific queue and publish the data into that system. Some targets may require a full reference data set whereas for others, the update of changes suffices.

Impacts

Cost of implementation may increase as there is a consumer for every dataset->target combination. The operational cost and effort to manage this environment may increase because of the messaging infrastructure being more elaborate for the benefit of more modularized reference data distribution.

Relationships

Because every target system gets its own distributing service, the service is not depending on as many back end systems, resulting in better performance and availability. This supports the service autonomy and service loose coupling principles as well as the service normalization pattern. The reference data observer pattern can be used to provide the reference data distribution pattern with reference data updates.

img

Figure 4 - Reference Data Distribution pattern increases service autonomy and service loose coupling and enhances service normalization.