Backpressure

Backpressure - backpressure streaming is a system slowing its intake because a downstream stage cannot keep up, so data queues at the source instead of overflowing somewhere in the middle. The scary word describes a safety feature. At the edge, Kafka makes it nearly free: consumers pull, so a slow reader is never flooded, it just lags, and the lag is a number you can watch. Inside a distributed engine it is more involved: stages exchange data over the network through bounded buffers, a full buffer stalls its sender, and the stall propagates stage by stage back to the source.

The alternative to backpressure is unbounded queueing, which is an out-of-memory kill on a delay timer. When a dashboard shows backpressure, it is pointing at the bottleneck, not confessing a failure.

Do you need it? The concept, yes; the machinery, mostly not. In ubik there is no network hop to pressure across: one process pulls from Kafka at the pace it folds rows, falling behind shows up as lag in the progress output, and state that outgrows the memory budget spills to disk instead of dying. Catching up is what replay is for.