Kubernetes native histograms end the bucket guessing

Every team running Prometheus has, at some point, argued about bucket boundaries. Someone picks a set, they turn out to be wrong for the traffic, and the p99 you are reporting to a customer is an interpolation across a gap between 1 second and 2.5 seconds. Kubernetes native histograms graduate to Beta in v1.37 and are enabled by default, and they remove that argument rather than settling it.

They also remove a cost most teams pay without attributing it to the right cause.

Worth saying plainly who this affects. If you run Kubernetes and scrape component metrics — which is nearly everyone running it in production — this changes data you already collect, on upgrade, without you asking. That is unusual for a Beta feature and it is the reason it deserves half an hour of attention rather than a note in the backlog.

Three problems, one change

The announcement is precise about what is wrong with classic histograms, and it is worth taking them one at a time because they fail differently.

The bucket guessing game. A classic histogram needs boundaries defined in advance — the familiar 0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10. You choose them before you know the latency distribution, and they are wrong in one of two directions: too coarse where the traffic actually lives, or fine detail in a range nothing ever hits. Native histograms adjust automatically to the observed distribution.

Cardinality and storage cost. This is the expensive one. Each bucket boundary creates a separate time series, because _bucket{le="..."} is its own series. A ten-bucket histogram multiplies your time series count by ten. Native histograms use a single compact representation instead.

Interpolation error. histogram_quantile() interpolates linearly between bucket boundaries. If your p99 falls between the 1 and 2.5 second buckets, the number you get is a straight-line guess across a range where real latency distributions are anything but straight. Native histograms are materially more accurate here.

The third is the one that should worry you most, because it is silent. A wrong percentile does not look wrong. It looks like a number.

There is a fourth benefit that follows from the third and is easy to miss. Because the resolution adapts, you stop having to re-tune buckets when traffic changes. A service that got ten times faster after an optimisation used to need its histogram reconfigured to keep reporting anything useful, and in practice nobody did — so the metric silently became less informative exactly when the system changed most.

The cardinality maths

The tenfold multiplication is worth doing arithmetic on, because it explains a surprising share of monitoring bills.

Take one latency metric on one component. With classic histograms and ten buckets, that is ten time series before you add a single label. Now add the labels people actually use — verb, resource, response code, instance — and each one multiplies. A metric with four labels averaging five values each is 625 combinations; with ten buckets it is 6,250 series. From one metric.

This is why Prometheus memory grows in ways nobody can account for, and why the usual response is to add RAM or shorten retention, neither of which addresses the cause. Native histograms collapse the bucket dimension entirely. The label dimensions remain, so this is not a licence to add labels freely — it removes one multiplier, not all of them.

It is the same class of problem we described wherever high-cardinality identifiers reach metric labels — a session or request ID in a label creates enough series to take down a metrics server. Cardinality is the recurring tax on observability, and almost every unexplained monitoring cost traces back to it.

What Kubernetes native histograms change operationally

Being on by default in Beta is the important detail. Upgrading to 1.37 changes how components expose latency and duration metrics, including API server request latencies and scheduling durations, without anyone opting in.

Which makes the compatibility question the one to answer before the upgrade, not after. Native histograms are a Prometheus format, and everything downstream has to understand it — your Prometheus version, your remote write target if you use one, your managed metrics vendor, your Grafana dashboards, and any alerting rule that queries the _bucket series by name. A dashboard built on classic bucket series does not gracefully degrade when the shape of the data changes; it shows nothing, and it does so quietly.

The checks worth running before you upgrade a production cluster:

1. Confirm your Prometheus, or your vendor, supports native histograms and that the support is enabled rather than merely available.
2. Search your alert rules for _bucket and for histogram_quantile. Those are the queries whose behaviour changes.
3. Check the dashboards nobody owns. The ones that matter during an incident are usually the ones nobody has edited in a year.
4. Upgrade a non-production cluster first and compare a percentile against the old value. If your interpolation error was large, the number will move — and the new one is the correct one, which is an awkward conversation worth having before a customer has it with you.

That last point is the practical sting. Improving accuracy means some of your published latency figures were optimistic, and fixing them looks like a regression on a graph. Better to discover that in staging.

One further note for anyone running a platform on behalf of other teams. If you charge internal customers for observability, or expose usage figures to them, this change will move those numbers and the shift has nothing to do with their behaviour. Explaining it before the invoice arrives is easier than explaining it afterwards, and it is a good opportunity to make the label-cardinality conversation concrete for people who have never had to think about it — the remaining multiplier is still entirely theirs.

Why this arrived now

Classic histograms were a reasonable design for a world where metrics were cheap and dimensionality was low. What changed is not Prometheus but what people ask of it: more services, more labels, higher retention, and multi-tenant platforms where every tenant multiplies the series count again.

There is also a straightforward economic pressure. Managed metrics vendors price on active series, so cardinality has a line item now. A tenfold reduction on latency metrics is a number a platform team can take to a budget conversation, which is a more persuasive argument for upgrading than accuracy is.

Both pressures point the same way, which is why the feature landed on by default rather than behind a gate that nobody would have turned on for two years.

The accuracy improvement is the part that matters more and gets discussed less, because nobody has ever been paged by a percentile that was quietly wrong.

It is also a reminder that observability has quietly become one of the larger line items in a platform budget, frequently rivalling the compute it watches. Teams scrutinise instance types monthly and have never once audited how many active series they are paying for. The bucket dimension disappearing is a real saving; the label dimension, which is entirely under your control and usually the larger multiplier, is the one worth looking at next.

A short exercise that tends to surprise people: count the distinct values of every label on your three highest-volume metrics and multiply them together. That product is what you are paying for, and it is almost always larger than anyone in the room guesses. It is the first thing we look at when a client asks why monitoring costs have grown faster than their infrastructure.

Where it sits in a 1.37 upgrade

If you are planning the upgrade, this belongs on the list with the scheduling changes rather than as a separate project — we covered the other half in workload-aware scheduling in 1.37, and the two have opposite risk profiles. Scheduling changes are visible: pods land differently and you notice. Metrics format changes are invisible until an incident, when a dashboard is empty and nobody remembers why.

So the sequencing we would suggest is to treat the metrics compatibility check as a prerequisite of the upgrade rather than a follow-up. It is an hour of grepping alert rules against the alternative of discovering the gap at the worst possible moment.

Beyond that, the honest assessment is that this is a good change with almost no downside for anyone whose tooling supports it. Better percentiles, smaller bills, no configuration to maintain, and one fewer thing to argue about in review. The only teams who will find it painful are those running old Prometheus versions or managed backends lagging on the format — which is itself useful information about a dependency worth checking, and exactly the sort of thing we look at in a Kubernetes engagement before an upgrade rather than during one.