Spill to disk

Spill to disk - letting state that will not fit in RAM overflow to local storage instead of killing the job, the spill to disk streaming engines borrowed from decades of database sorts and hash joins. Keep the hot part in memory, write cold blocks out, read them back when their key turns up again.

On modern hardware the penalty is smaller than the folklore says. An NVMe drive reads at gigabytes per second, so an aggregate over more keys than fit in RAM slows down and keeps running, which quietly changes the capacity question from "how much RAM does the cluster have" to "how big is the disk on this machine". That question is why one box goes further than the sizing spreadsheet assumed.

Do you need it? Yes, in the sense that you want it to exist and never think about it. In ubik it is on by default: keyed state stays resident up to UBIK_STATE_MEMORY_MB (1024 by default), blocks past that go to the spill directory, and the spilled_blocks counter in the progress stats tells you when it happened. That is the whole feature. Boring, which is the point.