Don't PanicState & fault tolerance

Changelog stream

A changelog stream expresses a table as the sequence of changes that produced it: replay it and the table comes back.

Also called: change stream

Changelog stream - a table expressed as the stream of changes that produced it: replay the changelog stream from the start and you rebuild the table, exactly. Every insert, update and delete becomes an event, and the idea runs both ways. A table is a stream compacted down to the latest value per key; a stream is a table's history laid out flat. Kleppmann calls it turning the database inside out, Kafka's storage trick for it is log compaction, and CDC tools sell it back to you as a product.

The name intimidates because it arrives wrapped in vocabulary (CDC, materialised views, stream-table duality), but the mechanism is a ledger: keep the changes, derive the current state. Accountants have run on it for five hundred years.

Do you need it? The concept, yes; it is the idea under half this glossary, keyed state included. The tooling, less than you would think. Ubik's output already speaks it natively: a windowed aggregate written to Kafka is an upsert changelog keyed by window and group, so a consumer reading latest-per-key always holds the current answer, with no extra machinery to stand up.

Sources

Related