Mostly Harmless
A field guide to stream processing jargon
The scary words of stream processing, in plain English. Verdict included.
Mostly HarmlessLearn it, free.Don't PanicSounds hard, isn’t.Beware of the LeopardCluster tax, skip.
Time & progress
Allowed latenessMostly Harmless
Allowed lateness is the grace period after the watermark before a window is truly, finally closed.Event time vs processing timeDon't Panic
The event time vs processing time distinction: when a thing happened versus when your machine finally noticed.Event-time skewMostly Harmless
Event time skew is one source running minutes ahead of another in event time, which every join between them has to respect.Idle partitionBeware of the Leopard
The idle partition watermark stall: one quiet Kafka partition freezes the watermark and no window ever fires.Ingestion timeMostly Harmless
Ingestion time stamps each event with the clock of the machine that first received it, and hopes the doorway was quick.Late dataDon't Panic
Late data in stream processing is any event that arrives after the watermark has already closed its window.Out-of-order eventsMostly Harmless
Out-of-order events are the rule, not the exception: every real stream arrives shuffled, and event time is how you cope.Reprocessing and backfillDon't Panic
Stream reprocessing backfill is replaying history through the same event-time pipeline that handles the live feed.WatermarkDon't Panic
What is a watermark in stream processing: the engine's moving claim that everything up to time T has now arrived.Watermark alignmentBeware of the Leopard
Watermark alignment throttles fast sources so a slow one does not force the cluster to buffer unbounded state.Windows & triggers
Accumulation modeBeware of the Leopard
The accumulating vs discarding mode choice decides whether a refired window replaces its last output or adds to it.Early and late firings (panes)Beware of the Leopard
With early and late firings, one window emits its result several times as more data lands, each pane a snapshot so far.Emit / output modeDon't Panic
A streaming output mode decides which rows the sink sees: append each final result, update the changed ones, or resend the lot.Global windowDon't Panic
A global window puts every event in one bucket that never closes until something else says so.Hopping and sliding windowMostly Harmless
Hopping window vs sliding window is one construct with two names: overlapping windows of fixed size that step forward by a slide.Session windowDon't Panic
A session window closes after a gap of silence, grouping events by activity instead of by the clock.TriggerBeware of the Leopard
A window trigger decides when a window emits its result, separately from which events it contains.Tumbling windowMostly Harmless
A tumbling window chops the stream into fixed, non-overlapping buckets of time, and every event lands in exactly one.Watermark triggerMostly Harmless
The watermark trigger fires a window once, when the watermark passes its end, and it is the trigger you almost always want.Windowed vs continuousDon't Panic
Windowed or continuous is the choice between finished snapshots per bucket of time and one running answer that keeps updating.Delivery & correctness
At-least-onceMostly Harmless
At least once delivery retries until everything arrives, so nothing is lost and some of it arrives twice.At-most-onceMostly Harmless
Fire and forget: at most once delivery sends each message once, never retries, and loses whatever was in flight when something fails.Dead-letter queueMostly Harmless
A dead-letter queue is where a pipeline parks unprocessable messages, so one poison pill does not stop the job.DeduplicationMostly Harmless
Deduplication drops the duplicate copies that at-least-once delivery created.Deterministic replayMostly Harmless
Deterministic replay is what recovery leans on: the same input re-run yields the same output.Effectively-onceMostly Harmless
Effectively once is the honest name for exactly-once: deliver at least once, then make the duplicates change nothing.Exactly-onceDon't Panic
Exactly-once semantics means every event counts once, even when a machine dies mid-flight and the event is redelivered.Idempotent producerMostly Harmless
An idempotent producer tags each batch with a sequence number, so a retried send cannot become a duplicate.Transactional messagingBeware of the Leopard
Kafka transactions group writes across partitions into one atomic unit that consumers see entirely or not at all.Two-phase commit sinkBeware of the Leopard
A two phase commit sink coordinates an external write with the pipeline's checkpoint so both commit together or not at all.State & fault tolerance
Changelog streamDon't Panic
A changelog stream expresses a table as the sequence of changes that produced it: replay it and the table comes back.CheckpointDon't Panic
A streaming checkpoint is a periodic snapshot of state and source offsets, so a crash resumes instead of restarting from zero.Checkpoint barrierBeware of the Leopard
A checkpoint barrier is the marker a job floats through its graph so every operator snapshots the same logical instant.Incremental checkpointBeware of the Leopard
An incremental checkpoint snapshots only what changed since the last one: cheaper to write, more complex to recover.Keyed stateMostly Harmless
Keyed state is state partitioned by key, so each key's aggregate updates on its own.RocksDB (as state store)Beware of the Leopard
RocksDB is the embedded LSM store behind Flink's big-state backend, plus the tuning tax that rides along.SavepointMostly Harmless
A savepoint is a snapshot you take on purpose: manual and self-contained, for upgrades, rescales and migrations.Spill to diskMostly Harmless
Spill to disk is what an engine does when state outgrows RAM: the job slows down and keeps running.State backendBeware of the Leopard
A state backend is where a streaming job keeps its in-flight state, a choice Flink forces on you before you know the workload.State TTLMostly Harmless
State TTL expires idle keys, so state does not pile up until the 3am OOM.Streams, tables & the cluster
ASOF joinDon't Panic
An ASOF join matches each row to the most recent matching row as of its timestamp, per key.BackpressureDon't Panic
Backpressure is a system slowing its own intake because a downstream stage cannot keep up.Consumer group and offsetsMostly Harmless
A Kafka consumer group offset records where each member left off as the group splits partitions between them.Interval joinDon't Panic
An interval join pairs rows from two live streams that share a key and fall within a time band of each other.JobManager / TaskManagerBeware of the Leopard
JobManager and TaskManager are the two processes a Flink cluster needs before it runs a line of SQL.Log compactionMostly Harmless
Kafka log compaction keeps only the latest record per key, turning a log into a table you can replay.Materialized view (streaming)Mostly Harmless
A streaming materialized view is a query result kept continuously up to date as events arrive.RebalanceBeware of the Leopard
A Kafka consumer rebalance reassigns partitions when a member joins or dies, pausing the group while it settles.Stream-table dualityDon't Panic
A table is a stream of changes and a stream is a table over time: stream table duality is the one model that unlocks the rest.Upsert / retract streamBeware of the Leopard
A retract stream emits a minus row then a plus row, so downstream totals stay right as values change.