Don't PanicWindows & triggers

Emit / output mode

A streaming output mode decides which rows the sink sees: append each final result, update the changed ones, or resend the lot.

Also called: output mode, append mode

TRANSACTIONALdefaulttimewindow sealsKafka transactioncommitwhole window visible at oncevisibility waits for the commitIDEMPOTENT--emit-mode idempotenttimevisible as produced(window_start, group) upsert keydurability barrierevery delivery report verifiedcheckpoint advances offsets
Both modes hold exactly-once. Transactional (the default) wraps each window in a Kafka transaction: nothing is visible before the commit, then the window appears atomically. Idempotent drops the per-window transaction: rows land under a (window_start, group) upsert key and a durability barrier verifies every delivery report before a checkpoint advances offsets. Measured p50 result-emission latency 10.2 ms vs 33.1 ms at 10k groups on the pinned bench box. The trade: after a crash a consumer can observe a partially emitted window, which converges on resume.

Emit / output mode - which rows the sink actually sees: a streaming output mode of append delivers each result once, when it is final; update delivers only the rows that changed; complete redelivers the whole result table every time. The names vary by engine, the three shapes do not.

Append only works when a result will never change again, which for a windowed aggregate means waiting for the watermark to close the window. Update is a changelog keyed by the result key, the same stream-of-updates-to-a-table idea Kafka Streams is built on, and the consumer reads latest-per-key. Complete re-emits everything on every refresh, which is fine for ten groups and absurd for ten million.

Do you need it? Don't panic: the scary dropdown is chosen for you by the shape of the query. In ubik a windowed query appends, one final row per closed window, and a continuous view updates, an upsert changelog keyed by the group that republishes only the groups that changed. There is no complete mode, because a consumer holding latest-per-key already has the whole table.

Sources

Related