Deduplication

Deduplication - the deduplication streaming pipelines run to drop the duplicate copies that at-least-once delivery created, so an event redelivered after a crash is counted once. The mechanism is a memory: keep a record of what has already been seen, by unique key or by sequence number, and drop any arrival that matches. The hard part is the memory's size. Remember every key forever and state grows without bound; remember a window of them and a duplicate older than the window slips through. Every real dedup is a bet on how late a duplicate can arrive, the same bet a watermark makes about late data.

Sequence numbers make the bet cheap: a broker only needs the last number per producer, which is exactly how Kafka's idempotent producer dedups on the server side.

Do you need it? Somewhere in the pipeline, yes: this is where "exactly-once" quietly happens. The choice is between buying it as machinery and getting it as a property. Ubik's output is an upserted changelog keyed by window and group, so a replayed window overwrites itself and there is nothing left to dedup.