The Merge Utility combines two independent real-time data streams into a single unified stream. It synchronizes events from both sources using configurable join keys and a configurable grace period, allowing related records that arrive at different times to be merged into a single event. Unlike the Enrich Utility, which enriches a stream using reference data, the Merge Utility treats both input streams as active data sources and joins them in real time before publishing the merged output.

Key Capabilities

The Merge Utility enables you to:

Merge two real-time Kafka streams into a single output stream.
Configure independent source topics for both streams.
Define custom join keys for matching related events.
Handle out-of-order events using a configurable grace period.
Configure how unmatched events are processed.
Resolve field name conflicts during merging.
Preview the merged event structure before deployment.

How It Works

The Merge Utility follows a two-step configuration workflow.

Step 1: Stream Connection

Configure the two input streams and define how records should be matched. During this step you configure:

Source Stream A
Source Stream B
Join Key for Source A
Join Key for Source B
Maximum Grace Period
Timeout Strategy

After configuration, the utility continuously listens to both Kafka topics and buffers incoming events while waiting for their corresponding records.

Step 2: Dynamic Rule Engine

Once both streams are configured, define how the merged event should be generated. In this step you configure:

Output Kafka Topic
Field mapping
Key collision strategy
Final merged event structure

A live preview displays the resulting merged JSON before deployment.

Stream Configuration

Stream Source A

Select the primary Kafka topic that contains the first stream of events. Example:

Vehicle telemetry
Device status
Orders

Stream Source B

Select the secondary Kafka topic that will be merged with Source A. Example:

Driver information
GPS location
Shipment updates

Join Keys

Join keys determine which records belong together. For example:

Source A	Source B
vin	vehicle_id
order_id	order_id
device_id	asset_id

The utility waits until matching records from both streams become available before creating a merged event.

Grace Period

The Grace Period defines how long the Merge Utility waits for the matching record from the second stream. Because events from different systems rarely arrive at exactly the same time, the grace period allows temporary buffering before the timeout strategy is applied. Choose a value based on the expected latency between the two streams.

Timeout Strategy

If a matching record does not arrive before the grace period expires, the configured timeout strategy determines how the event is handled. Available strategies include:

Drop unmatched records
Emit partial records
Send unmatched records to a Dead Letter Queue (DLQ)

Choose the strategy that best fits your processing requirements.

Output Configuration

Merged Topic

Select the Kafka topic where merged events will be published. The output topic becomes the source for downstream pipelines, transforms, connectors, and applications.

Key Collision Strategy

When both streams contain fields with the same name, the Merge Utility can resolve conflicts using one of the following strategies:

Prefer Stream A
Prefer Stream B
Append source-specific suffixes to conflicting fields

This allows both values to be preserved or prioritized according to your business requirements.

Preview

Before deployment, the Merge Utility displays a live preview of the merged event. The preview helps verify:

Field mappings
Join results
Output schema
Collision handling

This allows configuration issues to be identified before the utility is deployed.

Example Output

{
  "order_id": "ORD-992",
  "status_source_a": "PICKED_UP",
  "location_source_b": "40.7128,-74.0060",
  "merged_at": "2023-10-27T10:00:01Z"
}

Deploying the Utility

After completing the configuration:

Configure both input streams.
Define the join keys.
Set the grace period and timeout strategy.
Configure the output topic.
Review the merged event preview.
Click Deploy Utility.

Once deployed, the Merge Utility continuously processes both input streams and publishes merged events to the configured output topic.

Best Practices

Choose join keys that uniquely identify related events across both streams.
Configure the grace period based on expected upstream latency.
Use dedicated output topics for merged data.
Select an appropriate collision strategy when field names overlap.
Review the merged event preview before deploying the utility.
Monitor the utility to identify unmatched events and optimize timeout settings.

Spotlight

Understanding Condense

Build Pipeline

Vapr AI

Security and Compliance

Help and Support

AI Agents

Fully Managed Kafka

Merge Utility

Key Capabilities

How It Works

Step 1: Stream Connection

Step 2: Dynamic Rule Engine

Stream Configuration

Stream Source A

Stream Source B

Join Keys

Grace Period

Timeout Strategy

Output Configuration

Merged Topic

Key Collision Strategy

Preview

Example Output

Deploying the Utility

Best Practices

​Key Capabilities

​How It Works

​Step 1: Stream Connection

​Step 2: Dynamic Rule Engine

​Stream Configuration

​Stream Source A

​Stream Source B

​Join Keys

​Grace Period

​Timeout Strategy

​Output Configuration

​Merged Topic

​Key Collision Strategy

​Preview

​Example Output

​Deploying the Utility

​Best Practices

Key Capabilities

How It Works

Step 1: Stream Connection

Step 2: Dynamic Rule Engine

Stream Configuration

Stream Source A

Stream Source B

Join Keys

Grace Period

Timeout Strategy

Output Configuration

Merged Topic

Key Collision Strategy

Preview

Example Output

Deploying the Utility

Best Practices