Explaining how duplicate Kafka events get removed from a streaming pipeline
Kafka's at-least-once delivery guarantee means a single producer retry, consumer rebalance, or connector restart can put the same event into a topic more than once. This isn't a rare incident, it happens repeatedly over the life of any production pipeline. A duplicated transaction record double-charges a customer. A duplicated inventory update overcounts stock. So which combination of unique keys, state stores, time windows, and idempotent writes actually stops that before a reconciliation report finds it first?
Start by assigning a stable identifier to each event. For transactional data this is often a transaction ID generated upstream. For CDC pipelines built on tools like Debezium, each change event carries a log sequence number or binlog offset. The Aerospike CDC guide notes that consumers track the last seen position to catch and ignore duplicates as a backup to Kafka's own idempotent producer setting. Once the pipeline has that identifier, the next question is what it does with it.
Next comes state management. Frameworks like Apache Flink or Kafka Streams maintain a keyed state store holding every ID the pipeline has already seen. A new record arrives, the pipeline checks the state store first. ID already there? Drop it. Not there? Process it and write the ID to state. Zilliz's engineering guidance describes this check-then-write pattern as a core mechanism for streaming deduplication, and it works whether the state store runs on RocksDB, an embedded key-value store, or an external cache. But a state store that never forgets anything grows without bound. So how does the pipeline decide when an ID is safe to drop?
That's where time comes in, and it's really the crux of handling out-of-order arrival: bounding the state by a window. Keeping every ID forever doesn't scale, so pipelines use time-windowed deduplication instead. Flink SQL implements this by assigning a processing-time column via PROCTIME(), partitioning by the event's unique ID, and ranking arrivals with ROW_NUMBER() ordered by that processing time. The query then filters to row_num = 1, keeping only the earliest arrival within the partition:
SELECT id, payload, proc_time
FROM (
SELECT *,
ROW_NUMBER() OVER (
PARTITION BY id
ORDER BY proc_time ASC
) AS row_num
FROM (
SELECT *, PROCTIME() AS proc_time
FROM raw_events
)
)
WHERE row_num = 1;
This handles the out-of-order case directly. Two events with the same ID can arrive in either order, but PARTITION BY id groups them together regardless of sequence, and ROW_NUMBER() picks a deterministic winner. The window is also what keeps the state store bounded: once an event falls outside the configured time range, Flink expires its state entry, keeping memory usage proportional to event rate times window size rather than total event volume since the pipeline started. Catching duplicates after they're produced is one approach. Can some of them be stopped before they ever reach the topic?
Yes, and that's the fourth piece: upstream prevention, which cuts down how much deduplication logic has to run downstream. Kafka's idempotent producer setting stops retries of the same message from creating duplicate records at the broker level. Transactional writes group a sequence of operations, say, a CDC snapshot followed by a stream of changes, into a single atomic unit, so downstream consumers never see partial or duplicated fragments of that sequence. Topic partitioning matters here too: events sharing a key need to land on the same partition, since Kafka only guarantees ordering within a partition, not across an entire topic. These four pieces describe the mechanism. What a developer actually implements for a given system is a separate call.
Deciding whether your pipeline needs this now, and what it costs if you skip it
The techniques above describe how deduplication works mechanically. The real question is which of them a given pipeline needs, and that comes down to how expensive an unhandled duplicate turns out to be downstream. Streamkap's engineering guidance frames duplicates as a design consequence of distributed systems, not a bug to patch once: a CDC connector restart replaying the tail of its last checkpoint, or a consumer group rebalance reprocessing a batch, will keep happening for as long as the pipeline runs. What that costs depends entirely on what the duplicate row touches downstream, as in the billing and inventory examples above.
Teams running raw Kafka consumers without a deduplication layer tend to discover that cost only after it shows up in a reconciliation report, which is a far more expensive place to find it than a code review. Once a team accepts that cost is coming one way or another, the next decision is which technique to actually build.
The choice is a tradeoff between infrastructure complexity and correctness guarantees. Checking unique IDs against a lightweight cache is cheapest to implement, but the cache has to be sized correctly, an undersized one evicts IDs before late duplicates show up and lets them through anyway. Full state management in Flink or Kafka Streams gives stronger guarantees but adds an operational dependency: RocksDB state stores need disk space proportional to window size and key cardinality, and someone has to estimate that sizing before production load hits, not after. Kafka's own producer configuration is a different kind of tradeoff entirely compared to either of those.
Kafka's idempotent producer setting is close to a free win by comparison. It's a configuration flag, not a new system to operate, and it wipes out one entire class of duplicate: the retry-after-timeout case where the write actually succeeded on the broker but the producer never got the acknowledgment. Turning it on doesn't remove the need for downstream deduplication, rebalances and connector restarts still generate duplicates that idempotent producers can't catch, but it shrinks the volume everything else has to absorb. That still leaves the question of which combination fits the budget of a specific pipeline.
The answer isn't one technique, it's a layered setup: a stable identifier, idempotent producer settings to shrink the volume of duplicates reaching the topic, and a time-bounded state store to catch whatever's left, sized to the latency and cost budget of the system being built. Developers who treat this as a required component at design time spend an afternoon configuring it. Developers who treat it as optional hardening spend a lot longer than that explaining a reconciliation report.