Landing the changelog online

Land a ubik changelog into a millisecond-latency key-value store (Redis, DynamoDB) for low-latency reads. The key is a frozen contract; ubik emits it, the host upserts it.

A windowed or continuous query publishes a changelog: rows keyed by their (group, window) tuple, upserted (source and sink). To answer a read on the millisecond path it has to reach a key-value store (Redis, DynamoDB). ubik emits the changelog; the host consumes it and upserts into its store. ubik owns no store and no read path.

The key

The Kafka message key is a JSON array of the GROUP BY columns, in GROUP BY order. For GROUP BY merchant, TUMBLE(event_time, INTERVAL '1 MINUTE') the key is:

["acme", "2026-07-18 10:00:00"]

This encoding is a frozen contract. A host maps array positions to its store's primary-key columns and can rely on the bytes:

position i is the i-th GROUP BY column; a compound key is a multi-element array, in GROUP BY order;
each cell is rendered by the same cast the value uses, so the key and the value agree byte-for-byte on a shared column;
a string is a JSON string; a number is a JSON number; a TIMESTAMP is DuckDB text (YYYY-MM-DD HH:MM:SS, with .ffffff when sub-second);
a NULL cell and a non-finite DOUBLE/FLOAT (NaN, Infinity) both render as JSON null.

The value is the projected row as one JSON object. The key is taken before projection, so it carries the full (group, window) tuple even when the SELECT drops a group column.

The upsert rule

The message key is the identity; upsert it, keeping the latest. How the store's primary key relates to the message key depends on the query shape:

A continuous view (a GROUP BY with no window) keys on the group tuple alone. Upsert by it directly: the latest value per key is the current aggregate, and the sequence of a key's published values never decreases.
A windowed query keys on (group, window). The common read is the latest window per group, so upsert by the group (the group columns without the window) and accept a row only when its window_start is at or past the stored one. ["acme", w1] and ["acme", w2] are different message keys and can land on different partitions, so ordering across windows is not guaranteed; the window_start guard is what makes the landing monotone.

Compact the serving topic (cleanup.policy=compact) so a new consumer rebuilds the latest state without replaying history.

No tombstones

ubik never emits a tombstone (a null-value delete). A key that stops being produced keeps its last value in the store; ubik removes nothing. Deletes, TTL and eviction are the store's policy, not the changelog's.

One consequence to design around: a continuous view with a HAVING clause stops publishing a group once it falls below the threshold, and with no tombstone the store keeps that group's last above-threshold value indefinitely. If a HAVING group can drop out, either tombstone it on the host side, apply the threshold at read time, or avoid HAVING on a served view.

In process, the same key

An embedded host writing the store directly (no broker) gets the same key from ubik_next_keyed / ubik.stream(..., keys=True): the key bytes are byte-identical to the topic's, from one encoder, so the in-process and topic landings key the same (the C ABI). The consistency of the two paths is proven in test/e2e/dual_landing_broker.sh.

What ubik does not do

ubik does not own the store. It writes no store, holds no store client, and serves no point read. The changelog (and, in process, the key beside the batch) is the seam; the host consumes it, upserts into a KV store, and answers reads. Computing a value at request time from fresh inputs the host supplies is a separate surface, not a read of ubik-held state.