Beware of the LeopardState & fault tolerance

State backend

A state backend is where a streaming job keeps its in-flight state, a choice Flink forces on you before you know the workload.

Also called: state store

State backend - the storage layer where a streaming job keeps its in-flight state, and the thing a Flink state backend setting decides for you: on the JVM heap as live objects, or in RocksDB as serialised bytes on local disk. Every running aggregate, every open window, every join buffer lives in one of these, and the choice moves your throughput, your maximum state size and how checkpoints get written.

Now the fine print. Heap is fast until state outgrows memory or garbage collection starts eating the latency budget. RocksDB holds state far bigger than RAM but pays a serialisation round-trip on every read and write, and brings its own tuning surface with it (see RocksDB as a state store). So the backend decision is really a capacity plan, made in config, before the workload has told you anything.

Do you need it? The concept, yes: state has to live somewhere. The decision surface, mostly no. The knob exists because a cluster cannot know your machine. Ubik keeps state in memory up to a budget and spills to disk past it, in one process. The backend is a file.

Sources

Related