Kubernetes workload aware scheduling fixes gang, not waste

Kubernetes v1.37 graduated the Workload and PodGroup APIs to beta, along with workload-aware preemption and shared Dynamic Resource Allocation claims for PodGroups. It also introduced CompositePodGroup in alpha.

The release post contains no utilisation percentages, no cost reduction figures, and no bin-packing improvements. Kubernetes workload aware scheduling is described as architectural capability, not quantified savings — and that absence is the most useful thing in the announcement, because the coverage you read next will not have it.

What actually shipped

Four changes, from the v1.37 release post:

• Workload and PodGroup APIs, plus gang scheduling → beta. All-or-nothing scheduling, so a set of pods is placed together or not at all.
• Workload-aware preemption → beta. Preemption that understands workload semantics, so evicting one pod does not partially destroy a job whose remaining pods then hold resources for nothing.
• Shared DRA ResourceClaims for PodGroups → beta. Resource pooling across the pods of one coordinated workload.
• CompositePodGroup → alpha. Hierarchical scheduling with multi-level topology constraints, giving JobSet and LeaderWorkerSet native support rather than extension APIs.

There is also a workloadbuilder Go library and integration APIs for out-of-tree controllers, plus the native Job controller consuming the expanded APIs. That last part matters more than it sounds: it means ordinary batch Jobs get topology-aware scheduling without adopting a third-party scheduler.

The problem this solves

Gang scheduling exists because distributed training and large batch jobs have a property most web workloads do not: partial placement is worthless and actively harmful.

Schedule six of eight workers for a training job and you have not achieved 75% of the work. You have achieved none of it, while holding six workers' worth of GPU that nothing else can use, until the remaining two are placed or the job times out. Two such jobs competing can deadlock indefinitely, each holding what the other needs.

That is a real and expensive failure, and until now the standard answer was a third-party scheduler. Having it in-tree, in beta, with the native Job controller aware of it, is a significant piece of work.

Worth noting what gang scheduling does not promise even within its own remit: it does not make capacity appear. If the cluster genuinely cannot fit eight workers, all-or-nothing scheduling means the job waits rather than starting badly. That is the correct behaviour and it will still read as a regression to whoever was previously watching six workers spin up.

The problem this does not solve

Here is where the coverage will mislead you. The waste in most clusters is not a placement problem. It is a request problem.

A typical cluster is expensive because engineers set CPU and memory requests by guessing, guessed high because a throttled pod generates a ticket and an oversized one does not, and never revised the guess. The scheduler then honours those requests exactly as instructed, packing nodes against numbers that bear no relation to what the workloads consume. The scheduler is not making a mistake. It is doing precisely what it was told, with bad inputs.

Gang scheduling changes none of that. If your pods each request 2 CPUs and use 200 millicores, they will still request 2 CPUs, and scheduling them as a coordinated group will not recover a single core. Workload-aware scheduling makes correct placement possible for workloads that need coordination. It has nothing to say about workloads that are simply asking for too much.

The GPU version of the same mistake

The pattern repeats one level up. A CNCF write-up on multi-tenant GPU metrics opens with a GPU that had sat at zero percent utilisation for eleven straight days — allocated, powered, and invisible to the team that owned it because the metrics lived in a store no tenant was allowed to query.

No scheduler improvement addresses that. The GPU was scheduled correctly. It was scheduled to somebody who had stopped using it and could not see that they had. Placement was never the problem; visibility was.

That is the same shape as the request problem, and it is why we are wary of reading a scheduling release as a cost release. Both stories end with resources committed to workloads that do not need them, and in both cases the fix is measurement rather than mechanism.

How to tell which problem you have

Compare requested against actual usage across the cluster, at the ninety-fifth percentile, over a fortnight.

If requests are broadly two to five times observed usage, you have a request problem, and the fix is right-sizing — a vertical autoscaler in recommendation mode, or an afternoon with your metrics and the twenty largest deployments. Nothing in v1.37 helps.

If requests roughly track usage but you still see pending pods, held resources and jobs that never quite start, you have a coordination problem, and this release is aimed directly at you.

Most teams we look at are firmly in the first category and are being sold the second. Knowing which one you are is a metrics query, not a procurement decision.

What right-sizing actually involves

Since that is where the money is, it is worth saying what the work looks like rather than leaving it as an instruction.

Start with requests, not limits. Requests decide placement and therefore cost; limits decide throttling and eviction. Teams frequently tune the one that does not drive the bill.

Set CPU requests near observed p95 and leave CPU limits off or generous, because CPU is compressible and throttling a bursty service to protect a node nobody is contending for is a self-inflicted latency problem. Memory is the opposite: it is not compressible, so requests and limits should be close together and set above your real peak, since being wrong there means an eviction rather than a slowdown.

Then do it continuously. A right-sizing exercise done once decays within two quarters as traffic and code change, which is why the recommendation mode of a vertical autoscaler is worth more than a spreadsheet, even if you never let it apply changes automatically.

Why the absence of numbers is a good sign

It would have been easy to publish a benchmark. Pick a workload where coordinated placement helps, measure before and after, and print a percentage. The release notes did not do that, and the restraint is appropriate — the gains depend entirely on how much of your workload actually needs gang semantics, which for most clusters is very little.

When a vendor blog attaches a utilisation figure to this release over the coming weeks, the question to ask is what workload mix produced it. If the answer is a GPU training cluster and you run web services and cron jobs, the number is real and irrelevant to you.

What beta actually means here

Beta in Kubernetes means enabled by default and API-stable enough to build against, but not covered by the same deprecation guarantees as GA. For a scheduling primitive that distinction has teeth: if you build a platform abstraction on the PodGroup API now and the shape changes before GA, the migration is yours, and it touches every workload that adopted it.

The reasonable posture for most teams is to let the workloads that genuinely need gang semantics use the APIs directly, and to avoid wrapping them in your own platform layer until GA. The alpha CompositePodGroup should be treated as a preview to read rather than a feature to adopt — alpha APIs can change or disappear between minor versions.

Should you upgrade for this

If you run distributed training, large coordinated batch, or anything using JobSet or LeaderWorkerSet, this is a meaningful release and the alpha CompositePodGroup is worth reading properly.

If you run services, APIs and scheduled jobs — which is most people — upgrade on your normal cadence and change nothing. The features are beta, they are opt-in, and adopting them for workloads that do not need coordination adds concepts to your platform in exchange for nothing.

The uncomfortable version: if cluster cost is what prompted you to read about this release, you are reading about the wrong thing. The money is in the requests, it has always been in the requests, and it is unglamorous work that no release note will do for you.

We run Kubernetes platforms for clients and the first thing we look at is the ratio above, before any discussion of schedulers. It answers in an hour whether a cost conversation is about placement or about numbers somebody typed in a YAML file two years ago. If you would like a second pair of eyes on yours, that is a short conversation.