Cloudflare Serves Stale Assets When Tiered Cache Promotion Outruns Purge
Cloudflare's tiered cache is a hierarchy: lower-tier edge nodes serve users, upper-tier nodes serve the lower tiers. That design cuts origin traffic hard. It also creates a race that a purge alone will not close. When a promotion lands after an invalidation has already passed through, the upper tier can reassert a stale object. This piece explains the mechanism, why TTL does not save you, and the versioned-key ordering that does.
Two Paths Through the Edge
Every request in a tiered topology can travel two paths. On a hit, the lower tier answers directly and the origin never hears about it. On a miss, the lower tier consults the upper tier, and only if that also misses does the request reach the origin. The upper tier exists to absorb the fan-in from many lower-tier locations, which is where the bandwidth savings come from.
Purge has to walk that same hierarchy to be effective. A purge issued for one object is not a single global event; it is a sequence of invalidations applied per location and per tier. Each hop takes measurable time, and the ordering of those hops relative to normal traffic is not guaranteed. That is the whole problem in one sentence.
The related piece on anycast and DNS failover covers a similar theme: distributed routing decisions that look atomic from a dashboard are actually a sequence of local state changes. Cache invalidation has the same shape, and the same failure mode when the sequence inverts.
What Promotion Actually Moves
Promotion is the mechanism that populates the upper tier. When a lower tier misses and goes upstream, the upper tier fetches from origin, serves the lower tier, and retains a copy. A later request from a different lower tier can then hit that upper-tier copy instead of paying the origin round trip again. Promotion copies objects upward, toward the tier with the widest fan-in.
The upper tier becomes the authority for that object across every lower tier beneath it. That is efficient and fragile at the same time. If the upper-tier copy is wrong, every lower tier that consults it inherits the error. A single stale promotion can outlive the request that created it by the full TTL window.
Promotion is usually asynchronous. It runs on its own schedule, decoupled from the request path that triggered the fetch. That decoupling is what buys throughput, and it is also what lets a promotion land after a purge has already declared the object invalid.
Purge Is Not a Broadcast
It helps to stop thinking of purge as a broadcast and start thinking of it as a message that propagates. Each location applies the invalidation when it receives it. Propagation across a large edge footprint takes time, and during that window some locations serve the old object while others serve the new one. That is normal and usually acceptable.
The dangerous case is narrower. An in-flight promotion can escape the purge window entirely if the promotion was already underway when the invalidation passed through. The purge marks the object invalid at time T. The promotion, started at T minus a few hundred milliseconds, completes at T plus a few hundred milliseconds and writes its copy into the upper tier. The invalidation has already moved on.
This is textbook cache coherence, just at a different scale. The Wikipedia article on cache coherence describes the requirement precisely: if multiple clients hold a copy of the same region, all copies must agree. Without that agreement, a change made by one client is not seen by others, and mismatched data produces errors. The edge is a coherence problem wearing a CDN costume.
Where the Race Goes Wrong
The failure sequence is short. A purge arrives at the upper tier and removes the object. Before that removal, a lower-tier miss triggered a promotion fetch. The promotion completes after the purge and writes the object back. The upper tier now holds a copy that the purge already declared dead.
TTL does not rescue this. A shorter TTL shrinks the window but never closes it, because the race is about ordering, not duration. Setting TTL to a few seconds just means the stale copy expires sooner; it still serves stale responses for the duration. Teams that respond to stale-serve incidents by cutting TTL are treating a symptom.
The real fix is versioned keys. Stamp each object with a monotonic version or generation number at write time, carry that version through promotion, and have the upper tier reject any promotion whose version is older than the version it already holds. Promotion stops being an unconditional write and becomes a conditional one.
Designing for Promotion Ordering
Monotonic versions need a source of truth. A counter per object, incremented on every origin write, is enough. The version travels in a header alongside the object, and the upper tier compares it before accepting a promotion. Older versions lose. Equal versions are idempotent and can be dropped.
Logging matters as much as the comparison. Record purge timestamps and promotion timestamps separately, with the object key and the tier that handled each event. When a stale-serve incident happens, the skew between those two timestamps tells you whether ordering or propagation latency was the cause. Without separate logs, the two are indistinguishable after the fact.
Testing has to use skewed traffic. Uniform request patterns hide the race because promotions and purges rarely overlap. Skewed patterns, where one region drives most of the traffic and another drives almost none, surface the interleavings that uniform load never produces. The piece on Git bisect and build caches makes a parallel point about caches that quietly diverge from their source.
Practical Hardening Steps
Add a version or generation header to cache keys and to the promotion path. The upper tier must compare versions before accepting a write, and reject anything older than what it holds. This is the single highest-leverage change.
Instrument promotion latency as its own metric, separate from cache hit ratio. A rising promotion latency is an early warning that the purge window is widening. Alert on the skew between purge and promotion timestamps, not just on stale-serve counts.
Rehearse the stale-serve failure mode deliberately. Force a purge during an active promotion in a staging environment and confirm the version comparison rejects the late write. Document the tier ordering assumptions in the runbook, including which tier is authoritative and how promotion is sequenced relative to invalidation. A related piece on firmware rollback windows argues that ordering constraints deserve the same documentation discipline as the happy path.
How Real Systems Handle This
Cloudflare's tiered cache is one implementation of a pattern that shows up wherever a hierarchy of caches sits between clients and origin. The same race appears in CDN architectures that use a mid-tier, in database buffer pools that replicate reads, and in build systems that share a remote cache across CI runners. The names change, the ordering problem does not.
In practice, teams that operate large edge footprints report that purge propagation across all locations typically completes within a few seconds, though the tail can stretch longer during traffic spikes or control-plane churn. Promotion, by contrast, is triggered by miss traffic and can happen at any moment. The overlap window is small but nonzero, and over millions of requests per day, small windows produce incidents. Some operators set purge propagation budgets in the low single-digit seconds and treat any promotion that lands inside that window as suspect.
The trade-off is between simplicity and correctness. Conditional promotion adds a version comparison on the write path, which costs a few microseconds and one extra header. Unconditional promotion is simpler and faster, but it leaves the race open. For most teams, the version check is cheap insurance against a class of bug that is hard to reproduce and expensive to debug. The alternative, accepting occasional stale serves, is defensible only if the data is truly non-critical and the business can tolerate it.
There is also a middle ground: promotion can be delayed or batched. Instead of writing the upper tier immediately on a miss, the system could queue promotions for a short interval and apply them in order. That reduces the chance of a late write overtaking a purge, but it adds latency to the first request that would have benefited from promotion. The right choice depends on whether your workload values freshness or origin offload more.
Finally, consider the failure domain. A versioned key scheme assumes the version counter is monotonic and globally consistent. If the origin is multi-region and writes can happen in different locations, you need a version source that does not go backward. A logical clock per object, or a timestamp with a tiebreaker, can work, but it requires discipline. Without that, the version comparison can itself become a source of inconsistency.
None of this is unique to Cloudflare. Any tiered cache that promotes asynchronously and purges asynchronously will have a window where the two operations can interleave incorrectly. The fix is always ordering: make the write conditional on the version, and make the purge carry enough information to invalidate future promotions of the same version. That is the core lesson, and it generalizes.
