Mostly Harmless!FeaturesEmbedCloudOEMLive shellExactly-onceBenchmarksAlternativesDocsPricing
Don't PanicTime & progress

Late data

Late data in stream processing is any event that arrives after the watermark has already closed its window.

Also called: late events, stragglers

EVENT TIMEone cell per row, placed where it arrivedARRIVAL ORDERWINDOWstartendWATERMARKwindow closes · emits oncearrives behind the watermarkcounted as late_dropped, then droppedRECEIVEDCOUNTED IN THE WINDOWLATE · REPORTED, THEN DROPPED
The watermark is the claim that no row older than it will still arrive. Out-of-order rows still land in their event-time window; the window closes and emits once when the watermark passes its end. A row that arrives behind the watermark cannot be folded into a closed window, so it is counted as late_dropped and reported, never dropped in silence. --watermark-delay buys the grace before that line.

Late data - the name for late data in stream processing: an event that arrives after the watermark passed the end of its window, so the result it belonged to has already shipped. The watermark was a bet that the stragglers had stopped coming, and this row is the bet losing.

What happens next is a policy, not a catastrophe. An engine can drop the row, or accept it within a grace period and re-emit the window as an update (see allowed lateness), or route it to a side output for you to deal with by hand. The one thing a correct engine never does is fold it into a total it already published and say nothing.

Do you need it? You need to know your policy, which takes a minute, and that is the extent of it. Ubik's policy is short: a late row is dropped, counted, and reported as late_dropped in the run's own stats line, so drift shows up as a number in your output rather than as quietly wrong totals. If the number rises, widen the grace with --watermark-delay. The scary word turns out to be a counter.

Sources

Related

WatermarkAllowed latenessOut-of-order events
← All 50 terms