AEM Dispatcher Caching Strategies for High-Traffic Sites
When a campaign goes viral or a retail promotion lands on the evening news, the pressure on an AEM publish tier spikes within minutes. Dispatcher sits in front of Adobe Experience Manager as a reverse proxy and cache, absorbing the bulk of that read traffic so the underlying JCR and servlets stay calm. For Australian brands serving audiences from Sydney to Perth and across into South-East Asia, the gap between a configured Dispatcher and a tuned one can be the difference between a one-second page and a timeout.
Most teams treat Dispatcher as something you install and forget, then watch the cache hit rate drift back to sixty or seventy percent once marketing starts publishing daily. Real resilience comes from treating caching as a layered system, with deliberate invalidation, predictable TTLs, and a load-balanced farm that knows how to fail over. The same patterns that protect a bank during a rate change also keep an AEM site responding through a Boxing Day rush.
Reading the Dispatcher Cache Hierarchy
Dispatcher runs two principal caches: a document cache that stores rendered responses keyed by URL, and a fragment cache that holds partial output for reuse across pages. The document cache is the largest and most forgiving, suitable for HTML, JSON endpoints, and asset responses. Fragment cache is shorter-lived and works best when templates are stable, such as navigation, headers, and shared product panels.
A practical rule for high-traffic Australian publishers is to push as much as possible into the document cache with conservative TTLs, then carve out narrow paths for fragment caching where personalised or short-lived content lives. Configuration lives in dispatcher.any under /cache, with rules for /statfiles, /invalidate, /allowedClients, and the per-section directives. Treat each /cache section as a contract: who may cache it, how long, and how it gets cleared.
Document cache entries are validated against stat files, which is where freshness actually lives. By default, a statfile is touched on every replication, signalling Dispatcher to refetch the affected paths. Teams that rely solely on default statfiles often see stale content or over-invalidation; both can be solved with explicit /statfileslevel definitions and by isolating dynamic paths into their own cache sections with shorter TTLs.
Cache Invalidation Patterns That Scale
The default flush replication agent publishes a single invalidation request that lands on every Dispatcher in the farm. For a two-server farm in a Sydney data centre, that works fine. For a global brand with edges in Singapore, Tokyo, and Frankfurt, the default model turns into a thundering herd at the worst possible moment.
The pattern that holds up under load is hierarchical flushing combined with selective invalidation. Configure a flush agent that targets only the Dispatchers serving the affected region, and pair it with a content-tree-based invalidation rule so child pages flush when a parent changes. /invalidate { /0001 { /glob "" /type "deny" } /0002 { /glob "/jcr:content/*" /type "allow" } } gives you a baseline that flushes only what actually changed.
For news sites and retailers, schedule-based invalidation on a short loop catches anything that slipped through manual flushes. Pair that with versioned URLs for marketing landing pages, and the cache never has to invalidate at all during a campaign. The technique dovetails with content fragments for structured content management, since fragments can be referenced by hash and stay cached across edits.
Stat File Strategy and Freshness Discipline
Statfiles are the unsung heroes of Dispatcher behaviour. Every request asks the cache whether the underlying path has been touched since the document was stored; a single statfile entry can protect thousands of cached responses, or invalidate them all on the next miss.
For Australian sites with a heavy editorial cadence, set /statfileslevel to a value that matches the publish frequency. A level of three or four means that a touch on a parent page invalidates everything beneath it, which is the right behaviour for an editorial site but disastrous for a product catalogue. Split such sites into multiple dispatcher.any profiles, each with its own statfileslevel, document root, and flush agent, and load them on different Dispatcher instances behind the same load balancer.
Logging statfile hits and misses gives you a freshness dashboard without much extra work. Grep the access log for stat-file entries and chart them weekly; a sudden rise in misses after a deployment often points to an over-eager invalidation rule somewhere in the section hierarchy.
CDN Integration and Edge Caching
A modern AEM stack should treat the CDN as the outermost cache layer, with Dispatcher acting as the warm origin and AEM publish as the cold one. Cloudflare, Akamai, Fastly, and the local providers serving the Australian market all support surrogate keys or cache tags, which let you purge by logical group rather than by URL.
Configure Dispatcher to emit Cache-Control and Surrogate-Control headers so the edge knows what to cache and for how long. A typical pattern is a one-hour browser TTL, a 24-hour edge TTL, and a seven-day shared cache TTL for static assets. /enableTTL "1" in dispatcher.any, paired with a response header set on the publish side, makes this configuration stick.
When a major campaign lands, the edge absorbs the surge and only authenticated or personalised requests touch Dispatcher. Australian audiences spike hard during events like the AFL Grand Final, Melbourne Cup, and Boxing Day sales, so plan capacity around those dates rather than average traffic. Origin shielding on the CDN, where edge nodes fan out through a small set of regional peers, keeps a Sydney edge from hammering a Melbourne origin on a Friday night.
Load Balancing and Failover Across Data Centres
For sites with presence across Australia and the wider Asia-Pacific, run at least two Dispatcher farms in different data centres and use DNS-based load balancing with health checks. AWS Sydney, AWS Melbourne, and the local Azure regions each give you a stable platform; the question is how Dispatcher talks to AEM publish.
Stick to active-passive replication between publish instances if your content team works in one region, or active-active if editorial spans Sydney and Singapore. The Dispatcher farm should always treat the closest healthy publish as its origin and fail over within seconds. Health checks on the load balancer should probe a cache-friendly endpoint that returns 200 only when AEM and its bundles are healthy, not just when the servlet container is up.
Distance matters. A round trip from Perth to Singapore adds latency that no amount of caching fully hides, so put a Dispatcher in Perth serving Western Australian traffic and let it cache aggressively. The same applies to traffic heading north to Jakarta or Manila; a regional edge reduces the load on the primary origin and keeps the user experience consistent regardless of where the request originates.
Tuning for Peak Demand and Observability
Most performance regressions in Dispatcher are configuration regressions. A new template ships, a designer adds a personalised header, and suddenly the cache hit rate drops ten points. Build a regression suite that re-runs the canonical URLs through the staging Dispatcher after every release and reports the hit rate, p95 response time, and number of origin requests.
Instrument the Dispatcher with extended logging and a small exporter that pushes counters into Prometheus or a similar observability stack. Track cache hit rate by section, request rate by status code, and average response size. The event agenda from past CIRCUIT conferences has included sessions on this exact discipline, with speakers walking through dashboards that surface configuration drift before customers feel it.
| Cache Layer | Typical TTL | Hit Rate Target | Best Use Case |
|---|---|---|---|
| Browser cache | 1 hour | N/A | Returning visitors, static assets |
| CDN edge | 24 hours | 95%+ | Marketing pages, media, JSON APIs |
| Dispatcher document cache | 6-24 hours | 85-95% | HTML pages, templates, asset responses |
| Dispatcher fragment cache | 5-30 minutes | 70-85% | Headers, footers, navigation, shared panels |
| Memcached or JCR | Variable | Variable | Personalised content, session data |
Finally, rehearse the worst case. Run a load test that simulates a marketing campaign spike, an editor publishing a thousand pages in a minute, and a CDN purge. Measure recovery time, not just peak throughput. A Dispatcher farm that hits a million requests an hour but takes fifteen minutes to recover after a flush is fragile; one that handles half that load and recovers in thirty seconds is production-ready.
Reach out to the CIRCUIT team if you want to keep exploring Dispatcher patterns alongside other AEM practitioners. The conference sessions from Chicago cover caching architectures, microservices, and IoT integrations in depth, and recordings remain available for teams who want to bring their architects along. Join the next gathering, share your own farm configuration, and trade war stories about the day the cache fell over.