TLS and SASL

Authenticate to a Kafka broker with TLS and SASL through UBIK_KAFKA_ environment variables, which map one to one onto librdkafka properties.

Ubik reaches a secured broker through environment variables, not flags and not a config file. Any variable named UBIK_KAFKA_<PROP> sets the librdkafka property <prop>: drop the prefix, lowercase, turn each _ into a ..

Environment variable	librdkafka property
`UBIK_KAFKA_SECURITY_PROTOCOL`	`security.protocol`
`UBIK_KAFKA_SASL_MECHANISM`	`sasl.mechanism`
`UBIK_KAFKA_SASL_USERNAME`	`sasl.username`
`UBIK_KAFKA_SASL_PASSWORD`	`sasl.password`
`UBIK_KAFKA_SSL_CA_LOCATION`	`ssl.ca.location`
`UBIK_KAFKA_SSL_CERTIFICATE_LOCATION`	`ssl.certificate.location`
`UBIK_KAFKA_SSL_KEY_LOCATION`	`ssl.key.location`

The mapping is total: any librdkafka property is reachable this way, so any mechanism librdkafka supports works without a new flag. Credentials live in the environment and the engine never writes them to disk.

Confluent Cloud

SASL_SSL with the PLAIN mechanism, the API key as the username and the API secret as the password.

export UBIK_KAFKA_SECURITY_PROTOCOL=SASL_SSL
export UBIK_KAFKA_SASL_MECHANISM=PLAIN
export UBIK_KAFKA_SASL_USERNAME=<API key>
export UBIK_KAFKA_SASL_PASSWORD=<API secret>

ubik --from kafka://pkc-xxxxx.region.aws.confluent.cloud:9092/orders "$SQL"

Redpanda Cloud

SASL_SSL with SCRAM-SHA-256 (or SCRAM-SHA-512), the service account user and its password.

export UBIK_KAFKA_SECURITY_PROTOCOL=SASL_SSL
export UBIK_KAFKA_SASL_MECHANISM=SCRAM-SHA-256
export UBIK_KAFKA_SASL_USERNAME=<user>
export UBIK_KAFKA_SASL_PASSWORD=<password>

Mutual TLS

SSL with a CA to verify the broker and a client certificate to present.

export UBIK_KAFKA_SECURITY_PROTOCOL=SSL
export UBIK_KAFKA_SSL_CA_LOCATION=/etc/ubik/ca.pem
export UBIK_KAFKA_SSL_CERTIFICATE_LOCATION=/etc/ubik/client.pem
export UBIK_KAFKA_SSL_KEY_LOCATION=/etc/ubik/client.key

Secrets never reach an error message

A password or a private key is never echoed. If librdkafka rejects a value, the error names the property and hides the value: Kafka property sasl.password was rejected (value hidden). librdkafka's own logging is silenced, so a credential cannot leak through a broker log line either.

Schema Registry credentials

A secured Confluent Schema Registry (used by --schema-registry, see formats) has its own three variables, separate from the broker's, because a registry is an HTTP endpoint, not a Kafka connection.

Variable	Sets
`UBIK_SCHEMA_REGISTRY_USER`	HTTP basic-auth user.
`UBIK_SCHEMA_REGISTRY_PASSWORD`	HTTP basic-auth password.
`UBIK_SCHEMA_REGISTRY_CA`	CA bundle to verify the registry's TLS.

Properties ubik owns

A few properties are set by the engine and cannot be overridden, because they are load-bearing for exactly-once. Setting one is refused rather than silently ignored.

{"error":{"code":"SOURCE_ERROR","message":"ubik owns the Kafka property bootstrap.servers and sets it itself; unset UBIK_KAFKA_BOOTSTRAP_SERVERS"}}

The reserved set is bootstrap.servers and group.id on the consumer, and bootstrap.servers, transactional.id and enable.idempotence on the producer. The bootstrap address comes from the kafka:// URL, and the transactional identity is what fences a crashed producer on resume.

When the handshake fails

A protocol mismatch, a bad certificate or a wrong password fails the connection rather than falling back to plaintext. Pointing an SSL client at a plaintext listener, for instance, cannot complete the transport.

{"error":{"code":"BROKER_LOST","message":"cannot reach any Kafka broker at broker:9092 (topic orders): Local: Broker transport failure","hint":"Check the broker address and that it is up. Nothing was consumed past the last checkpoint: --resume it if you checkpointed, otherwise re-run."}}