Two-phase commit sink

Two-phase commit sink - a sink built on the two phase commit sink pattern, which makes an external system commit in step with the pipeline's checkpoint: prepare the write everywhere first, then commit it everywhere only once the checkpoint is durable. Flink's implementation is the canonical one. The sink opens a transaction per checkpoint interval, pre-commits it when the checkpoint barrier arrives, and finalises it after the job manager confirms the checkpoint completed. If anything crashes between those phases, recovery must roll the transaction forward or abort it, and the external system has to hold the pre-committed transaction open, sometimes for minutes, without timing it out. That last clause is where the operational pain lives: transaction timeouts, orphaned transactions, sinks that cannot really hold a prepare.

It is real machinery, correctly designed, for a real problem: an external store that cannot be written idempotently.

Do you need it? Almost certainly not at your scale. If the sink can take an idempotent upsert, deterministic replay plus overwrite reaches the same end state with no coordinator. That is ubik's route: replay from one local checkpoint into keyed, upserted output, and the phrase "two-phase commit" stays unlearned.