CLI
One-shot query
ubik --from <source> "<SQL>", the primary invocation, its sources and sinks, output formats, and every flag the CLI forwards to the engine.
The primary way to run ubik is one command: a source, the SQL, and the result on stdout.
ubik --from file://events.ndjson \
"SELECT merchant, count(*) AS n FROM events GROUP BY merchant"The SQL's FROM names the source: a Kafka topic, or a file's basename. The query
runs to the source's current end and stops, unless --follow tails it live.
Sources
| Source | --from | Behaviour |
|---|---|---|
| File | file://events.ndjson | Replayed once, to end of file, then stops. |
| Kafka | kafka://broker[,broker]/topic | Replayed to the current head, then stops; --follow tails it live. |
A file source is queried by its basename: file://data/events.ndjson is the
table events.
Output
Without --to, results are written to stdout: an aligned table at a terminal,
raw JSON on a pipe. -f table|json|csv forces one either way.
With --to kafka://broker/topic, the aggregation changelog is written to Kafka
as a keyed upsert (keyed by the window and group), so a consumer reading
latest-per-key always has the current aggregate. See
Kafka source and sink.
Flags
Every flag the one-shot query accepts, forwarded to the engine:
| Flag | Type | Default | Meaning |
|---|---|---|---|
--broker-timeout | string | — | fail a Kafka replay with BROKER_LOST after this much sustained MID-RUN broker unreachability (default '5 minutes'; '0 seconds' waits forever); a broker unreachable at startup always fails fast, ~5s, nothing is invested yet |
--checkpoint | string | — | write exactly-once checkpoints to this path (a sibling <path>.lock file guards it against a second engine; it stays behind, leave it) |
--checkpoint-every | int | 0 | checkpoint every N chunks (~2048 rows each) and keep running (requires --checkpoint) |
--dimension | stringArray | [] | lookup-join build side, name=file://path.ndjson or name=kafka://broker/topic (repeatable) |
--emit-every | string | — | refresh cadence of a continuous view. A live --follow run whose GROUP BY has no window is a materialized view: at each refresh it republishes the groups that changed, as an upsert changelog keyed by the group tuple, so a consumer's latest-per-key is always the true aggregate. Default '2 seconds'. A view that has caught up refreshes immediately whatever this says, so '0 seconds' leaves only the caught-up and pre-checkpoint refreshes, never no emission |
--emit-mode | string | — | how the Kafka sink emits each closed window: 'transactional' (default) wraps each window in a Kafka transaction so a read_committed consumer sees it atomically, or 'idempotent' drops the transaction for lower per-window latency, keeping exactly-once through the (window, group) upsert key and a durability barrier that flushes and verifies before advancing. Pick transactional when a downstream needs window atomicity, idempotent for a latest-per-key or changelog consumer that wants the latency (Kafka sink only) |
--follow | bool | false | tail the topic live instead of replaying to its current head and stopping (Kafka only); Ctrl-C to stop. Window semantics differ: a replay closes every remaining window when the topic ends, follow closes windows only as the event-time watermark advances, so a partition with no fresh events holds every window open (list/logs show 'wm held by'). A one-shot validation can emit rows a follow never will |
-f, --format | string | — | result output format: table|json|csv. Default follows the destination: the aligned table at a terminal, raw JSON on a pipe. The flag forces one either way (csv to a file, json at a TTY, table down a pipe) |
--from | string | — | source, e.g. kafka://broker/topic |
--idle-partition-timeout | string | — | stop counting a source partition in the coordinated watermark once the broker proves it is at the end of its log and it has stayed quiet that long (e.g. '60 seconds'), so one permanently idle partition stops holding every window open. A partition that still holds unread records is never excluded, whatever the engine is doing. Empty (the default) and '0 seconds' disable it; a run without it is byte-identical to one built before the feature existed (windowed queries, Kafka source) |
--pipeline | string | — | durable auto-resume pipeline: checkpoint to this path and resume it automatically if it already exists, so ONE command serves first run and restart (for systemd/k8s: one ExecStart + Restart=on-failure gives kill-9 recovery). Needs --follow (live) or --checkpoint-every (periodic). The auto form of --checkpoint/--resume; don't combine them |
--progress-interval | string | — | heartbeat a JSON progress line on stderr every interval: rows in/out and ev/s (per-run counters, reset on restart), watermark, checkpoint seq/age (null until one commits), late-dropped/spilled counters, Kafka lag (null until first measured), plus windows closed / rows folded on a windowed query (default '10 seconds', or 500ms when a terminal is watching; '0 seconds' disables). A run shorter than one interval (a quick one-shot) emits none; the final stats line carries the run's totals and is its summary |
--resume | string | — | restore state and offsets from a checkpoint, then continue; re-supply --from and the original SQL (a mismatch is refused) |
--schema | string | — | local Avro schema (.avsc) for a raw-Avro topic with no registry |
--schema-registry | string | — | Confluent Schema Registry URL for an Avro topic, e.g. https://sr.example.com |
--stop-after | int | 0 | checkpoint after N chunks (~2048 rows each) and stop (requires --checkpoint) |
--to | string | — | sink for the aggregation changelog, e.g. kafka://broker/topic (default: stdout) |
-v, --version | bool | false | print the ubik CLI and engine versions and exit |
--view | string | — | running-view density at a terminal: 'simple' (one status line) or 'extended' (adds the stats box). Press v while a run is live to switch; that choice becomes your default, this flag forces one run without changing it. Piped output is unaffected, it stays raw JSON |
--watermark-delay | string | — | bounded out-of-orderness: close a window at end + interval (e.g. '5 seconds'), so a late row within the grace still counts (windowed queries) |
The exactly-once flags (--checkpoint, --checkpoint-every, --resume,
--pipeline, --stop-after) are the manual form of a durable run; a named,
supervised job is usually simpler with ubik pipeline.
See exactly-once for what a checkpoint holds.
Environment
Tuning knobs are environment variables, inherited by the engine and by a pipeline's engine. See the configuration reference.