Don't PanicStreams, tables & the cluster

Interval join

An interval join pairs rows from two live streams that share a key and fall within a time band of each other.

Also called: stream-stream join

a JOIN b ON a.k = b.k AND b.ts BETWEEN a.ts + lo AND a.ts + hiSTREAM Aowns emissionSTREAM Bprobed in bandevent timea.k · a.tsemits oncematchmatchout of bandout of band[ a.ts + lo , a.ts + hi ]F = min(wm_a, wm_b - hi)
Each a-row owns its emission: when the frontier F = min(wm_a, wm_b - hi) proves its band complete, it probes b once and emits its matches (INNER, or one null-pad for LEFT), exactly once. Retention is time-bucketed to the band, so state stays bounded; an unbanded two-stream join is refused at plan time (UNBOUNDED_JOIN_STATE).

Interval join - an interval join streaming two live sources pairs rows that share a key and fall within a time band of each other: orders to the clicks within thirty minutes of them, payments to the alerts within five. Each side buffers rows only as long as the band requires, and a pair is emitted once the watermark proves no earlier row can still arrive to change it. The band is not decoration; it is what bounds the state, which is why a stream-to-stream join with no band has nowhere to stop growing.

This is the join people assume needs a warehouse, because two moving datasets feels like a distributed-systems problem. It is a buffer and a clock.

Do you need it? If you correlate two event streams, yes. Ubik ships it as plain SQL: declare the second stream with --stream, write the band as a BETWEEN on event time, and an unbanded join is refused with UNBOUNDED_JOIN_STATE rather than eating your RAM. Both streams' offsets and buffers land in one checkpoint, proven byte-identical to a batch join across kill -9 cycles. For the most-recent-version cousin, see ASOF join.

Sources

Related