Workflow orchestration at scale: what Netflix deleted

Netflix rebuilt Conductor, its workflow engine, and the headline change is a deletion rather than an addition: it stopped loading the entire running workflow into memory to evaluate it. That single decision took maximum workflow size from about 2,500 tasks to 30,000 and cut p99 latency by roughly 40%. For anyone thinking about workflow orchestration at scale, it is the clearest recent illustration of where these systems actually break.

The scale is worth stating so the rest has context: roughly 200,000 workflow definitions across 150 applications, executing about 420 million workflows per month.

The change that mattered

Conductor 4.0 separates workflow metadata from execution data. Previously the engine loaded the whole running workflow into memory for evaluation. Now tasks are stored independently and the evaluator works from a lightweight workflow blueprint, loading only the task data it needs.

Notice the shape of the original problem. It was not that any individual workflow was enormous or that traffic was too high. It was that the cost of evaluating a workflow scaled with the size of the workflow, when the work of any single evaluation step does not. Every decision about what to run next dragged the entire history along with it.

That is a common shape and it is usually invisible early. A workflow with twelve steps loads twelve steps' worth of state; nobody notices. The same engine meets a workflow with two thousand steps and the per-evaluation cost has grown by a factor nobody designed for. The system did not degrade gradually — it hit a ceiling, which is why the maximum size was around 2,500 tasks rather than a performance curve.

The general principle is one we apply on much smaller systems: stop loading state you are not going to read. It sounds obvious and it is routinely violated by ORMs fetching whole aggregates, by handlers that hydrate an entire order to update one line, by an ERP screen that loads every historical revision to show the current one, and by exactly this kind of engine that rebuilds full context to make a small decision.

It also explains why the fix delivered a latency improvement as well as a size increase. Once evaluation stopped dragging the full workflow into memory, every evaluation got cheaper, not only the large ones — which is how a change framed as raising a ceiling ended up moving the p99 for ordinary traffic too. Capacity problems and latency problems are often the same problem wearing different clothes.

They designed the lock away rather than tuning it

The second change is the one we would steal first. Conductor eliminated locking by storing pending and terminal task states separately, with terminal state taking precedence during reconciliation. Lock acquisition failures went from about 2,700 per interval under contention to essentially zero.

Read that as a modelling decision, not a concurrency trick. The lock existed because two writers might disagree about a task's state. Storing pending and terminal separately means they cannot conflict — a task that has finished is recorded in a different place from a task that is waiting, and if both exist, finished wins. There is nothing left to serialise.

Most teams meet lock contention and reach for the knobs: shorter hold times, finer-grained locks, optimistic concurrency with retries. Those work until they do not, and each adds a failure mode. The more durable fix is usually to find the piece of shared mutable state causing the contention and restructure it so concurrent writers are no longer writing the same thing. It is more design work up front and it removes the problem rather than deferring it.

The 2,700-to-zero figure is worth holding onto for the argument you will eventually have with someone who wants to tune instead.

There is a diagnostic worth running on your own system while the idea is fresh. Take the query your job or workflow table serves most often, and ask how many bytes it returns versus how many the caller uses. If the answer is off by an order of magnitude, you have the same problem Netflix had, several zeroes down. It will not stop you today and it sets the ceiling you will eventually hit.

Where the data went

The storage changes read as a list of migrations and are better read as a set of decisions about what each store is for.

• Execution data moved from Dynomite to Cassandra.
• Large task inputs and outputs moved to Amazon S3.
• DynoQueues was replaced by Timestone.
• Kafka now decouples indexing from execution.
• Elasticsearch and Apache Iceberg handle indexing and long-term storage.

Two of those generalise to systems a thousand times smaller. Moving large payloads out of the operational store and into object storage, keeping only a reference, is the single most effective thing you can do to a workflow or job table that has become slow — those columns are read constantly and their contents almost never are. And decoupling indexing from execution means a slow or unavailable search index stops being able to slow down the thing that actually matters.

We have made both changes on client systems running a rounding error of Netflix's volume, and in both cases the payload column was the problem.

The Kafka decision deserves a sentence of its own, because it is the one most often skipped in smaller systems. Indexing exists so people can search and report; execution exists so work gets done. Coupling them means a reporting problem becomes an operational outage. Separating them costs a queue and buys the guarantee that nobody's dashboard can stall the pipeline.

What this is not

One thing to be clear about before anyone opens a repository: Netflix discontinued public maintenance of Conductor in December 2023 and works on an internal fork. Community contributions live in a separate project. So this is an architecture case study, not a library you can adopt on the strength of these numbers.

It is also worth resisting the obvious conclusion. Almost nobody reading this needs 30,000-task workflows or 420 million monthly executions, and a business that copies Netflix's architecture at a hundredth of the scale ends up with the operational burden and none of the reason for it. The transferable content is the two principles — do not load what you will not read, and design the contention away — not the component list.

Workflow orchestration at scale, or just a status column?

The question we are usually asked is not how to scale one, but whether to introduce one. The honest answer is that most teams need one later than they think and build one earlier than they should, usually by accident.

The accidental version is recognisable. A status column on a table, a cron job that moves rows between statuses, some retry logic, a dead-letter table, and eventually a second cron job that fixes rows the first one left stuck. That is a workflow engine. It is just one nobody designed, with no visibility into why a given item is where it is.

A real engine is worth it when you need durable execution across process restarts, visibility into in-flight work, and retries with defined semantics — most often in order processing, document pipelines, onboarding flows and anything involving an external party with its own timing. It is not worth it when the work is a handful of steps that complete in a second and a queue would do.

The failure we are most often called in to fix sits between those, and it is always the same one: the payload column. Someone stored the full document, the full API response or the full generated file in the row that tracks the job. Every status query now reads megabytes it does not want. Netflix's answer — the reference in the operational store, the content in object storage — is the fix at every scale, and it is usually a day's work.

That is the useful frame for this whole rework. The engineering is remarkable at Netflix's volume, and the two ideas underneath it are the ones we would apply to a custom system processing a few thousand items a day. Deciding which of the two problems you actually have is the part that matters, and it is the same judgement as deciding when to shard a database or weighing what poor architecture actually costs: the principle scales down, the architecture does not.