Mostly Harmless!FeaturesEmbedCloudOEMLive shellExactly-onceBenchmarksAlternativesDocsPricing
Don't PanicState & fault tolerance

Checkpoint

A streaming checkpoint is a periodic snapshot of state and source offsets, so a crash resumes instead of restarting from zero.

INPUTevents consumedtimeSTATEdurable at a checkpointin memorylost at the killCKPTCKPToffsets + stateKILL -9RESUMEfrom the last checkpointrewind to the checkpointed offsetsREPLAYED FROM THE LOGthen forwardbyte-identical output
A checkpoint is one atomic write: every source offset and every operator's state, or nothing. A kill -9 takes only what accumulated after the last one. Resume reads the unit, rewinds the log to the offsets it recorded, replays the tail and carries on, so the run writes the bytes an uninterrupted run would have written.

Checkpoint - a periodic snapshot of state and source offsets that lets a crashed job resume where it stopped: the streaming checkpoint is the entire difference between a blip and a restart from zero. It sounds like black magic. It is a consistent photograph: every operator's running aggregate and every source position, captured at the same logical point in the stream, written somewhere durable.

Recovery is the inverse, and just as dull. Load the last complete snapshot, rewind the sources to the recorded offsets, replay what came after. In a cluster the photograph has to be coordinated across machines with a checkpoint barrier, which is where most of the perceived difficulty lives. The idea itself fits in a sentence.

Do you need it? Yes, and you can stop panicking about it. Ubik writes the whole thing as one local file, state and offsets together, fsynced and renamed atomically over the old one, so a partial checkpoint cannot exist on disk. kill -9 mid-window is a case in its test suite, and the resumed run is checked byte-identical against a batch oracle. A snapshot and a rename. That is the machinery.

Sources

Related

SavepointIncremental checkpointCheckpoint barrierState backend
← All 50 terms