Idle partition

Idle partition - the cause of the classic idle partition watermark stall: one Kafka partition goes quiet, and because the watermark is the minimum across all partitions, the minimum of a set containing "no news" is "no progress". Every window in the job holds open. Nothing emits, nothing errors, every log looks healthy, and it is 2am.

The stall is the engine being correct: a silent partition might still be about to deliver an old event, and closing windows without it would make that event late. The standard fix is an idleness timeout that excludes a quiet partition from the watermark, and the standard way to get it wrong is a timeout that also excludes a partition which was merely slow, silently turning its unread records into dropped ones.

Do you need it? You need to know the failure mode, because it will find you. The knob itself is leopard territory. Ubik's version is --idle-partition-timeout, and it only excludes a partition after the broker proves the consumer is at the end of that partition's log and it has stayed quiet for the interval; a partition still holding unread records is never excluded. The 2am mystery becomes one flag with a proof obligation attached.