Upsert / retract stream

Upsert / retract stream - the retract stream Flink produces and its upsert cousin are the two ways a stream carries corrections when a result it already emitted stops being true. The retract form sends a minus row withdrawing the old value, then a plus row asserting the new one; an upsert stream skips the minus and simply re-emits the latest row per key, with deletion travelling as a null. You need one of them the moment an aggregate's output feeds another computation, because a running sum re-emits every time its input grows, and whatever sits downstream must not add the old total to the new one.

The correctness is real. So is the headache: an append-only sink (a file, a metrics store, most warehouse ingest paths) has no idea what a minus row means, which is why sinking a retract stream is where streaming SQL projects go to acquire a second engineer.

Do you need it? The concept, yes. The retract algebra, usually not: ubik emits only upsert changelogs, keyed by the group tuple (plus the window, when there is one), so the sink contract is latest-per-key wins and a plain Kafka topic with log compaction can hold it.