JobManager / TaskManager

JobManager / TaskManager - JobManager TaskManager Flink names the two process types a cluster runs before it executes any of your SQL: the JobManager coordinates, meaning it schedules tasks, triggers checkpoints and reacts to failures, while the TaskManagers do the work, each offering task slots that pieces of your job land in. That is the minimum. A production deployment also wants a highly available JobManager, which means external metadata storage and leader election (ZooKeeper or Kubernetes), plus durable storage for checkpoints, plus whatever supervises all of the above. You are operating a distributed system in order to run a query.

To be fair, this is the honest cost of coordinating work across machines. The question worth asking first is whether your job spans machines at all, or whether the cluster is there because the tool assumes one.

Do you need it? Only if the job genuinely needs a cluster: petabyte fan-out exists, and this is what it costs. Ubik's answer to the control plane is to not have one: a single process holds the plan, the state and the checkpoint, and the supervisor is systemd with Restart=on-failure. The JobManager for that is the kernel.