The GPU utilization Kubernetes clusters can reach: one bank went from 35% to 60%

The GPU utilization Kubernetes clusters actually achieve is the most expensive number in self-hosted AI, and the least visible. A cluster can look healthy — pods scheduled, nodes busy — while most accelerator capacity sits idle. China Merchants Bank has just shown how large the gap can be: by unifying training and inference on one Kubernetes control plane, it lifted average utilisation across nearly 10,000 accelerator cards from 35% to more than 60%, and cut the cost of processing a million tokens by more than 60%.

What China Merchants Bank built

The bank won the CNCF End User Case Study Contest for KubeCon China 2026, announced in September. Its AI infrastructure team combined Kubernetes with five CNCF-ecosystem projects so that model training, fine-tuning and online inference could share one pool of heterogeneous accelerators:

• Kueue for queueing and admitting batch training jobs.
• KEDA for scaling inference with traffic.
• HAMi for sharing accelerator capacity in fine-grained units across workloads.
• Fluid for faster access to data and model files.
• Prometheus for observability across the pool.

Bringing 99% of its accelerator compute under that framework is what moved utilisation from 35% to over 60%. Separately, the bank's in-house training framework lets five LoRA fine-tuning tenants share one base-model instance by default, which it says cuts accelerator use for that setup by 80% and raises training density fivefold.

Why GPU utilization Kubernetes clusters report stays low

The bank's own explanation of the problem is the useful part. Training, inference and fine-tuning want different things from the same hardware:

• Distributed training needs stable, predictable capacity, and sits idle while waiting for the remaining workers or cards to become available.
• Online inference needs to scale quickly with unpredictable traffic, so teams over-provision for peaks.
• Fine-tuning for many tenants tends to replicate the same base model again and again.

Run those as separate pools, each sized for its own peak, and utilisation around a third is unsurprising. Kubernetes' default device model makes it worse: a pod requests whole GPUs, so a small inference service can hold a card it uses a fraction of. Fractional sharing, a common queue and elastic inference are what recover the idle capacity.

We covered the scheduling side — gang scheduling for training jobs — when Kubernetes 1.37 added workload-aware scheduling. Utilisation is the other half: getting the jobs scheduled is not the same as keeping the cards busy.

How do you measure AI inference cost?

In cost per token, not in pods or nodes. That is the argument Val Bercovici, chief AI officer at WEKA, made to The New Stack this week: inference cost "depends on state Kubernetes was never designed to manage: request mix, KV cache occupancy, the balance of prefill and decode, and how memory and bandwidth are consumed inside the accelerator after a pod is already running." His view is that Kubernetes stays, but "unless its resource model evolves, it becomes a tax on inference economics".

China Merchants Bank's reported result is framed exactly that way — cost per million tokens, input and output combined — and its stated next step is capacity management based on unit cost, combining utilisation, queue state and latency signals. For any team self-hosting models, that is the metric to adopt: divide the cost of the accelerator pool by the tokens it served. If the number is not falling as you add optimisations, the optimisations are not working.

How to increase GPU utilization in Kubernetes

1. Measure first. Export per-card utilisation and memory use to Prometheus, and calculate cost per million tokens for each inference service. Most teams have never seen the number.
2. Stop giving small services whole cards. Use fractional sharing — HAMi, or the time-slicing and partitioning options your hardware supports — for inference services that do not saturate a GPU.
3. Put training behind a queue. A queue such as Kueue admits jobs when capacity is available instead of reserving it in advance.
4. Scale inference on demand. Autoscale on request rate or queue depth, and consider scaling to zero between demand spikes, which the bank lists as a next step for KEDA.
5. Share base models. For multi-tenant fine-tuning, serve many LoRA adapters from one base model instead of one copy per tenant.
6. Pool, then govern. One shared pool beats several team-owned pools, but it needs quotas and priorities so critical inference is never starved by a training run.

Is Kubernetes good for LLM inference?

For teams already running Kubernetes, yes — with the caveats above. The bank's result shows the platform can reach high utilisation in a regulated, large-scale setting, using open, vendor-neutral components. But its default resource model sees GPUs as indivisible devices and knows nothing about tokens, so the tooling around it does the real work.

For smaller teams, the first question is still whether to self-host at all. At low volume, hosted APIs usually win on cost, as we found when we checked the claims about self-hosted LLM cost. Self-hosting starts to pay when volume is steady and high enough to keep cards busy — which is exactly what utilisation measures.

If your GPU bill is growing faster than your traffic, our Kubernetes implementation team can instrument cost per token on your cluster and set up sharing, queueing and autoscaling to raise utilisation.