Batch processing memory optimisation: 500 files to 50,000

A pipeline that had processed a few hundred documents a day for months started failing with MemoryError when the volume grew. No deployment, no code change, no new dependency. It simply crossed a threshold somewhere between 500 and 5,000 files, and every assumption the code had made about memory came due at once. Batch processing memory optimisation is nearly always this story, and the trigger is growth rather than a release.

Which is what makes it hard to catch. There is no bad commit to revert and no obvious culprit in the diff, because the code was always like this — it just used to be small enough not to matter.

The three causes, and why they hide

In Simon Briggs's account the failure decomposed into three, and each is individually reasonable.

The whole file in memory for the whole job. Every file's full byte content sat in memory for the entire duration of its processing. Reading a file into a buffer is the obvious way to write the code, and with 200 KB documents it is invisible. With scanned PDFs of 40 to 80 MB, holding even a handful concurrently is most of a container's allocation.

Results accumulating in a list. The results list grew while processing continued, so the job's memory rose monotonically until the batch finished. This is the one that turns a per-item problem into a per-batch one: even if each file were cheap, keeping every outcome means the job's footprint is a function of batch size.

Scale multiplying both. Neither is a problem at a few hundred files. Together at tens of thousands, they compound.

Note what is absent: no leak, no bug, nothing a profiler would flag as anomalous. Every byte held was held deliberately by code doing exactly what it was written to do. That is why the instinct is to add RAM, which works until the next growth step and teaches the team nothing.

It is also why the problem survives code review. A reviewer reading the function sees correct, idiomatic code that handles one file properly, because it does. The defect is not in the function, it is in the relationship between the function and the volume it will eventually be given, and that relationship is nowhere in the diff. The only reliable way to see it is to ask, of every loop that collects something, what bounds the size of what it collects.

The fixes are ordinary, which is the point

Four changes, none of them clever.

• Stream file reads rather than loading whole files into buffers.
• Yield results from a generator instead of accumulating them in a list.
• Write output immediately after each file finishes rather than at the end.
• Chunk the batch — around 200 files — with explicit garbage collection checkpoints.

The reported outcome is peak memory down roughly 80% and the same footprint whether the job handles 500 files or 50,000. Those figures are self-reported and the post carries a promotional mention of a PDF tool, so treat the numbers as indicative. The structural change is what matters and it stands on its own: memory stopped being a function of batch size.

That is the property to design for. Not lower memory — constant memory. A job whose footprint does not depend on how much work you give it can be scheduled, sized and reasoned about. A job whose footprint scales with the batch is a capacity incident waiting for a busy month.

Chunking with explicit garbage collection deserves a caveat, because it is the one of the four that can be cargo-culted. Forcing collection is not normally something application code should do, and reaching for it is usually a sign that something is still holding references it should not. In a batch loop processing large buffers it is a reasonable pragmatic checkpoint; as a general habit it hides the real problem. Fix the retention first and treat the collection call as a belt-and-braces measure rather than the fix.

Generators are the load-bearing idea

The single change with the widest application is returning a generator instead of a list, and it is worth being clear about why.

A function that builds a list decides, on your behalf, that every result will exist simultaneously. A generator produces each result when asked and forgets it afterwards, so the caller decides how much to hold. Same interface to a for loop, entirely different memory profile, and the change is usually two lines.

The equivalents exist everywhere and are equally underused: streams in Node, IEnumerable with deferred execution in .NET, cursors rather than full result sets in database drivers, and pagination on any API returning a collection. The same principle applies to the database layer, where fetching a whole row to update one column is the identical mistake in a different costume — the same instinct behind exhausting cheaper options before sharding a database, where the first fix is almost always a query reading far more than it uses.

There is a corollary worth stating for anyone writing library or service code: return the generator, let the caller materialise it if they want to. A caller can always wrap a generator in a list when they genuinely need everything at once. A caller handed a list has no way back to laziness, and every consumer inherits the decision made by whoever wrote the function first.

Where this shows up in real systems

We are called in for this often enough to recognise it by symptom. A few places it reliably hides.

Report generation. Building the full result set in memory to render a CSV or PDF. Fine for a month's data, fatal for a year's, and it fails at the year end when someone runs the annual report for the first time.

Document and invoice pipelines. Exactly the case above. Scanned documents are large, volumes are seasonal, and the pipeline was written against a sample of clean files.

Data migrations. The script that worked beautifully on staging with ten thousand rows and dies on production's four million, usually at two in the morning during a cutover window.

Bulk API integrations. Fetching all pages into a list before processing any of them, which also means one failure at page 40 discards the work of the first 39.

That last consequence is underrated. Streaming is not only cheaper in memory, it is better behaved on failure — work already written stays written, so a crash costs you the current item rather than the whole run.

Batch processing memory optimisation before it bites

The useful move is not to optimise everything. It is to know which jobs have the problem, which takes an afternoon.

1. List every batch job and write down what bounds its memory. If the answer is the size of the input, you have found one.
2. Run one at ten times the current volume. Not a load test — one job, ten times the data, watch the memory graph. A flat line is fine. A rising line tells you exactly where the ceiling is, and roughly when you will reach it at current growth.
3. Fix the ones that rise, starting with the largest inputs. Streaming and generators, in that order.
4. Alert on job memory as a proportion of the limit, not on the crash. A job sitting at 70% of its container's memory is a warning with weeks of notice in it; the MemoryError is the outcome you were trying to avoid, arriving with none.

The reason to do this before you need to is that these failures arrive at the worst possible time by construction. A job breaks when volume grows, and volume grows when the business is doing well — the annual report, the seasonal peak, the new client four times larger than the last one. The system fails precisely when it is being asked to prove itself.

Designing for constant memory is not premature optimisation. It is choosing the version of the code that does not have a ceiling, usually at no extra cost, and it is one of the more reliable places where a small amount of care up front avoids the cost of architecture that has to be reworked later. If you have a document or reporting pipeline you have never tested above its current volume, that is the one to look at, and it is the sort of thing we check early on any custom software engagement.