Watermark alignment

Watermark alignment - the practice of keeping many parallel sources' watermarks in step, because when one source races ahead in event time, everything it emits must be buffered until the slowest source's watermark catches up. Picture a backfill topic being read at full speed next to a live topic ticking along in real time: the fast side is suddenly hours ahead, the join cannot complete without the slow side, and the buffered state grows until something falls over.

The cluster answer is to throttle: Flink lets you put sources in an alignment group with a maximum allowed drift, and pauses the ones that run too far ahead. It works. It is also genuinely fiddly, per-source configuration in service of keeping a distributed job's state under control, which is a problem the cluster brought with it.

Do you need it? Only if you operate the cluster. In a single process the partitions already share one coordinated watermark, the minimum across all of them, so nothing closes early; and state that outgrows RAM spills to NVMe instead of taking the job down. See event-time skew for the underlying condition. The meeting about alignment groups is one you can skip.