What actually runs today

SQL stream processing

The test suite includes 66 end-to-end scripts, including checks against a real broker. See limits and unsupported workloads.

DuckDB SQL

DuckDB parsing, binding, types and scalar functions. Group by TUMBLE, HOP or SESSION for event-time windows.

Three window kinds

TUMBLE fixed, HOP sliding, SESSION gap-based and merging. Event-time, watermark-closed, emitted mid-stream. Late events are counted and reported before they are dropped.

EVENT TIMEone cell per row, placed where it arrivedARRIVAL ORDERWINDOWstartendWATERMARKwindow closes · emits oncearrives behind the watermarkcounted as late_dropped, then droppedRECEIVEDCOUNTED IN THE WINDOWLATE · REPORTED, THEN DROPPED
The watermark is the claim that no row older than it will still arrive. Out-of-order rows still land in their event-time window; the window closes and emits once when the watermark passes its end. A row that arrives behind the watermark cannot be folded into a closed window, so it is counted as late_dropped and reported, never dropped in silence. --watermark-delay buys the grace before that line.

Aggregates that survive a crash

count, sum, avg, min, max, first, last, arg_min, arg_max, string_agg, list, exact quantile and median, approx_count_distinct, approx_quantile. Each one checkpointable and bounded in memory, or refused at plan time rather than mis-saved.

Lookup joins, including ASOF

Enrich a stream against a bounded file, a host Arrow or pandas table, or a live Kafka changelog, with the versioned ASOF pick. Checkpoint-safe and parallel-safe, proven against a DuckDB JOIN oracle across SIGKILL cycles.

Stream-stream joins

INNER and LEFT joins on an equality key and bounded time interval. Join state spills to disk and shares a checkpoint with downstream aggregates and lookups. Unbounded stream-stream joins are rejected.

Idle partitions

A configurable timeout excludes idle partitions from the watermark minimum, allowing windows to close. Exclusions are checkpointed and reported.

Disk spill

Aggregate state spills to NVMe past a memory budget, so state larger than memory does not OOM. Even a single group larger than RAM sorts, through a bounded external merge, for an exact quantile.

Kafka native, no migration

Source and sink over the Kafka protocol, against the Kafka, Redpanda, WarpStream or MSK you already run. Avro and JSON Schema read from a Confluent Schema Registry, or a local .avsc.

Credentials stay in the environment

TLS and SASL (SCRAM, PLAIN, OAUTHBEARER, mTLS) configured from env vars, proven against a real broker. Nothing sensitive reaches pipeline.json, logs, state or checkpoints. Broker settings the engine owns for correctness are refused, so config cannot silently break exactly-once.

Pipeline supervision

ubik pipeline create, start, stop, list, logs. One detached process per pipeline, resumed from its last checkpoint across a restart. The OS is the supervisor. Every error carries a code, a message and what to do next.

Read next
Live shellThe released binary in your browser, on a live stream. No install.Exactly-onceWhat the guarantee is, how it is built, and the four places it stops.Downloadbrew, pip, or one checksum-verified tarball, and a durable pipeline in one command.