Log compaction

Log compaction - Kafka log compaction keeps, for each key, only the latest record in a topic, deleting older versions in the background. A tombstone (a key with a null value) eventually removes the key entirely. The topic stops being a full history and becomes something more useful: a table you can replay from the start in bounded space, always ending up with the current value per key. That is the whole trick, and it is what lets a changelog topic live forever without infinite disk.

The fine print matters. Compaction keeps the latest value per key, not the last N, and it runs lazily, so a fresh reader can still meet a few superseded versions before the cleaner reaches them. A correct consumer treats the topic as latest-per-key wins, always.

Do you need it? Yes, if a Kafka topic is your table. Ubik leans on it from both sides: it writes aggregates as an upsert changelog whose latest-per-key state is the current result, and it follows a compacted topic as a live dimension, resuming ASOF join picks correctly even after compaction has dropped the superseded versions.