Tiles

A tile is the serialized partial state of a combinable aggregate for one entity and one time bucket, published as the value of a single Kafka message. One job writes tiles; another reads them and re-combines them, across a different or larger horizon, without recomputing from the raw events. The tile topic is the reuse mechanism: a lifetime count or a rolling window is a merge over tiles, not a replay of years of events nobody keeps.

Three function shapes exist for every combinable aggregate (count, sum, avg, min, max, approx_count_distinct, approx_quantile):

• <agg>_state(x) returns a tile BLOB: the partial state, for the producing job.
• <agg>_merge(tile) returns a tile BLOB: combines tiles into a coarser tile.
• <agg>_final(tile) returns the finalized value.

Writing tiles

A _state job groups by an entity and a TUMBLE window and writes one tile per (entity, bucket) to a compacted topic. The topic must be created with cleanup.policy=compact.

ubik --from kafka://broker/events --to kafka://broker/daily_tiles \
  "SELECT user_id, approx_count_distinct_state(page) AS tile
   FROM events
   GROUP BY user_id, TUMBLE(ts, INTERVAL 1 DAY)"

The message key is [user_id, bucket], the bucket last; the value is the tile BLOB. Each tile is published once, when the window closes, and a crash re-emits it byte-identical.

Reading and merging tiles

A consumer reads the topic with --tile-source. The source presents three columns: entity (the key's grouping prefix as a JSON array), bucket (a TIMESTAMP), and tile (the BLOB). Merge the daily tiles into a 7-day sliding distinct count:

ubik --from kafka://broker/daily_tiles --tile-source --follow \
  --to kafka://broker/uniques_7d \
  "SELECT entity, approx_count_distinct_final(tile) AS uniques_7d
   FROM daily_tiles
   GROUP BY entity, HOP(bucket, INTERVAL 1 DAY, INTERVAL 7 DAY)"

approx_count_distinct_final(tile) unions the HLL sketches in the window and returns the cardinality. Use _merge instead of _final to write a coarser tile (a weekly tile from daily tiles) for a further merge downstream. avg is sum_state plus count_state under one tile; avg_final finalizes it through an exact division.

The source deduplicates per entity: a compacted topic can replay an already-merged tile, so a bucket at or below an entity's highest merged bucket is dropped. The result is the same whether the topic is freshly written or long compacted.

Rules

• _state is TUMBLE-only: its tiles are keyed on disjoint buckets, so HOP or SESSION would emit overlapping tiles a merge would double-count. _merge and _final may re-bucket with HOP.
• A tile topic's partition count is fixed for its lifetime; entity routing is by the entity key, so a re-partitioned topic is refused on resume.
• Tiles are append-only and non-decrementable. A tile topic and allowed-lateness window re-close are mutually exclusive: a correction has no way to reach a downstream that already merged the earlier tile.
• The wire format is frozen and versioned, so a tile written by one binary version merges in another, on any of linux amd64, linux arm64, or darwin arm64. An incompatible merge (two different sketch parameters) is refused, never silently combined.

Run ubik describe for the full per-aggregate tile surface.