Self hosted LLM cost: the $8 a month claim checked

A guide doing the rounds this week promises Llama 3.3 70B on an eight-dollar-a-month GPU droplet at one hundred and fifty-fifth of Claude Opus pricing. It is worth working through, not because the conclusion is wrong in principle — sometimes self-hosting does win — but because every number in it is checkable and three of them do not survive the check. Anyone quoting a self hosted LLM cost should be able to reproduce the arithmetic in about thirty seconds.

Three numbers that do not hold

The headline contradicts the article. Further down, the author concedes the eight-dollar figure assumes intermittent rather than continuous use, and puts realistic running somewhere between $150 and $365 a month. A monthly cost that only applies if you do not run the thing most of the time is not a monthly cost, it is a rounding of your idle time.

The hourly rate is wrong. The piece quotes $0.50 per hour for an L40S. DigitalOcean's L40S specification and DigitalOcean's published GPU Droplet pricing lists the L40S at $1.57 per GPU per hour. Run continuously, that is roughly $1,146 a month, not $365 and certainly not eight dollars.

The card is described wrongly, and that one matters most. The article calls the L40S a 24GB card. DigitalOcean lists 48GB. And the model in question does not fit in either.

There is a fourth problem that is harder to see and worth naming, because it recurs in every comparison of this kind. The piece was published by an account describing itself as autonomous and publishing continuously, and the code sample stops mid-function. Content generated at volume tends to be internally inconsistent in exactly this way — a headline optimised for sharing, a body that quietly contradicts it, and numbers that were never checked against the vendor page they came from. The defence is not scepticism about the source. It is running the arithmetic yourself, which takes less time than reading the article.

The arithmetic anyone can run

Model weights occupy a predictable amount of memory, and the calculation is multiplication.

Parameters multiplied by bytes per parameter gives you the weights. At float16 that is two bytes each, so 70 billion parameters is about 140GB before anything else. Then add the KV cache, which grows with context length and with how many requests you serve at once, plus activations and framework overhead. A working rule is weights plus twenty to thirty per cent, and more if you want real concurrency.

So 70B at float16 needs roughly 170GB of VRAM in practice. That is not a 48GB card. It is not two of them. On DigitalOcean's published list it means an H200 at 141GB or an MI300X at 192GB, both several dollars an hour, or multiple cards with the tensor-parallelism setup the article never mentions.

Quantisation changes the picture and is the obvious answer — eight-bit halves it to about 70GB, four-bit halves it again to about 35GB, which genuinely fits on a single 48GB card with room for a cache. But quantisation is a quality decision, not a free win, and an article comparing output quality against a frontier model while silently quantising to four bits is comparing two different things. The piece gives no throughput figures at all, which means its cost-per-token comparison has no denominator.

What self hosted LLM cost actually includes

The GPU hour is the number everyone quotes and it is not the number that decides this.

Utilisation is the whole game. An API bills per token, so idle costs nothing. A GPU bills per hour whether or not a request arrives. A workload running eight hours a day pays for twenty-four unless someone builds the automation to stop and start it, and that automation is itself work with its own failure modes. Most internal tools have deeply spiky traffic, which is the worst possible shape for a fixed hourly cost.

Somebody has to operate it. Model updates, driver and CUDA versions, inference server upgrades, capacity when a second team wants access, an on-call path when it stops responding at ten at night. A conservative half-day a month of senior engineering time is comfortably more than the token bill of a small API workload.

Redundancy doubles the honest figure. The single-instance price is not a production price. If the service matters, you need a second instance or a documented fallback to a hosted API — and if the fallback exists, part of the argument for self-hosting has already gone.

Latency and concurrency are capacity decisions. With an API, ten simultaneous users is the provider's problem. On your own GPU it is your queue, and the answer is either more cards or a slower service.

Batch size changes everything and is usually ignored. A GPU serving one request at a time wastes most of its capacity, because inference is memory-bandwidth bound rather than compute bound at low batch sizes. Continuous batching recovers a large share of that, which is why serving frameworks exist — but it also means the cost per token depends on how many concurrent requests you actually have, and a single-user benchmark tells you almost nothing about the economics at ten users.

When it does win

None of that is an argument against self-hosting. It is an argument against deciding on a headline. There are four situations where the economics genuinely favour running the model yourself.

1. High, steady throughput. Continuous batch work — classification, extraction, enrichment over a queue that is never empty — is exactly the shape that suits a fixed hourly cost, because utilisation approaches the thing you are paying for.
2. Data that cannot leave. A regulatory or contractual constraint is a reason on its own, and then the comparison is not against an API at all. It is against not shipping.
3. A small model doing a narrow job. This is the case most often overlooked. A 7B or 13B model fine-tuned for one task fits comfortably on a modest card, and comparing it to a frontier model is the wrong comparison — the question is whether it is good enough at the one thing you need.
4. Predictable billing as a requirement. Some finance teams genuinely prefer a fixed monthly number to a variable one, even a larger fixed number. That is a legitimate preference and worth naming as such rather than dressing it up as a saving.

What does not appear on that list is general-purpose chat or coding assistance for a team of twenty. The traffic is spiky, the quality bar is a frontier model, and the arithmetic almost never works.

One more consideration that rarely makes the spreadsheet: which model you are comparing against. A great deal of self-hosting enthusiasm is really dissatisfaction with a frontier model's list price for work a much smaller model would do. If the task is classification, extraction or routing, the honest comparison is a small hosted model against a small self-hosted one — and once you make that comparison, the hosted option often wins on operational simplicity rather than losing on price.

How to actually decide

Measure before you model. Take a fortnight of real API usage and pull out three numbers: tokens per day, the distribution across the day, and the peak concurrent requests. Then price the GPU instance that could serve that peak, at the quantisation you would actually accept, running the hours you would actually need. Compare that against the API bill you already have — not against a headline rate, and not against a frontier model's list price if a smaller model would do.

The pattern here is the same one we keep meeting in infrastructure decisions: the sticker price is the cheapest part, and utilisation plus operational burden is where the money is. It is the same reason running across three clouds costs more than the invoice suggests, and why the interesting question with model routing turned out to be access control rather than price.

If someone sends you a self-hosting guide with a number in the headline, the fastest test is the one above: parameters times bytes, plus overhead, against the VRAM of the card named. It took under a minute here and it disqualified the premise. That kind of check is the cheap part of any cloud engineering decision, and we would rather run it with a client before the hardware is provisioned than after — the same discipline we apply to AI engineering work generally.