Idempotent producer

Idempotent producer - the idempotent producer Kafka ships is a producer that tags every write so a retry cannot become a duplicate, turning at-least-once sending into exactly-once appends within a partition. The mechanism is small: the broker assigns the producer an id, each batch carries a per-partition sequence number, and the broker drops any batch whose sequence it has already appended. That is the whole thing, one config flag (enable.idempotence, on by default since Kafka 3.0), for a few bytes per batch.

The fine print is the scope. It protects one producer session writing to a partition. It does not span topics, does not cover the consumer side, and does not catch an application resending the same event, which arrives with a fresh sequence number and looks new. Crossing those lines needs transactions.

Do you need it? It is the cheap half of exactly-once, so take it; it is already on. Ubik's Kafka sink leans on the expensive half too: each closed window is emitted in one transaction, keyed for upsert, so a replayed window overwrites rather than duplicates.