Consumer group and offsets

Consumer group and offsets - a Kafka consumer group offset is how a member remembers its place: each partition of a topic is read by exactly one consumer in the group, and each consumer keeps an offset per partition, the position of the next record to read. Commit the offset to the broker and a restart resumes from there. All of Kafka's delivery semantics hide in the timing of that commit: commit before processing and a crash skips records, at-most-once; process before committing and a crash repeats them, at-least-once. The group is the unit of scale-out, the offset is the unit of progress, and one topic can carry many groups, each with its own offsets, reading independently.

Do you need it? You need to understand it, because every Kafka reader does this bookkeeping whether it shows you or not. Ubik does it differently: it reads every partition itself and stores the per-partition offsets inside its own checkpoint, atomically with the aggregate state, instead of committing them to the broker. State and position cannot disagree after a crash, and there is no group membership to rebalance.