Optimising AEM Dispatcher Cache For High-Traffic Sites
High-traffic Adobe Experience Manager sites depend on a cache strategy that is fast, predictable and safe. AEM Dispatcher can absorb a large share of anonymous requests, reduce pressure on publish instances and improve page delivery through a CDN. When its rules are too broad, too conservative or poorly aligned with publishing workflows, the cache can become a source of stale content, unnecessary origin traffic and difficult incidents.
Australian organisations face a particularly varied audience profile. A retail site may serve shoppers in Sydney and Melbourne during a campaign, customers on slower regional connections, and mobile users moving between Brisbane, Perth and Adelaide. Optimising Dispatcher therefore requires more than increasing cache duration: it means understanding content volatility, invalidation behaviour, request patterns and the distance between visitors and the origin.
Map The Request And Publishing Paths
Begin by documenting the complete delivery chain: browser, CDN, load balancer, Dispatcher, publish tier and authoring environment. Identify which responses should be cached at each layer, where cache-control headers are changed, and which publishing events trigger invalidation. This map often reveals that a CDN is bypassing Dispatcher for some paths or that a load balancer is distributing requests unevenly across publish nodes.
Separate cacheable anonymous traffic from personalised or authenticated requests. Public HTML, client libraries, images and content fragments usually have different lifetimes and invalidation needs. Requests containing session cookies, authorisation headers or user-specific parameters should be excluded unless the application has been deliberately designed for safe caching. A cached response that includes account information is a security incident, not a performance optimisation.
Review URL shapes before writing Dispatcher rules. Query strings used for tracking, sorting or filtering can create a large number of cache variations. Where those parameters do not change the response, remove them at the edge or configure a consistent normalisation policy. Where they do change the response, define a bounded strategy rather than allowing unlimited variants to occupy the filesystem.
Design Cache Rules Around Content Behaviour
A strong Dispatcher configuration generally uses allowlists for cacheable paths, extensions and methods. GET and HEAD requests for public content are common candidates, while POST, PUT and DELETE requests should reach the application unless a specific design requires otherwise. Deny rules must cover internal paths, repository endpoints, system information and authoring resources before a request can reach the publish environment.
Use file extensions, selectors and suffixes carefully. A rule that caches every response under a content path may accidentally store JSON, error pages or dynamically generated variants. Define permitted selectors and extensions, then test unexpected combinations such as additional selectors, trailing slashes and encoded characters. This is especially important for multilingual sites serving Australian English alongside other regional content.
Cache headers should express the intended lifetime rather than leaving every decision to Dispatcher defaults. For relatively stable assets, long-lived browser and CDN caching combined with fingerprinted filenames is effective. For HTML, a shorter edge lifetime with reliable purge or revalidation can reduce the risk of stale campaigns. Dispatcher settings such as statfile levels and invalidation rules should reflect the structure of the content tree, not simply be copied from another project.
Make Invalidation Fast And Deliberate
Publishing a page does not automatically mean every cached representation disappears in the way an editor expects. Flush agents, replication events, statfiles and CDN purge APIs must work together. Test activation, deactivation, page moves, asset replacement and language-copy updates, because each operation can affect a different set of URLs.
The statfile level controls how broadly Dispatcher considers content invalidated. A low level can clear a large section of the cache after one publication, creating an origin spike. A higher level preserves more cached content but requires a more precise understanding of dependencies. For sites with large navigation structures, shared content fragments or experience fragments, measure the impact of each setting instead of assuming the most aggressive flush is safest.
Purge failures should be visible to operations teams. Log the request, response status, affected paths and retry outcome for every flush or CDN invalidation. A queue with controlled retries is preferable to silently dropping purge events. During a major sale or public campaign, Australian teams may be operating across AEST, ACST and AWST, so clear timestamps and alert ownership matter when an issue spans Sydney and Perth support teams.
Measure Hit Rate And Origin Pressure
Cache hit ratio is useful, but it should never be the only performance metric. Track Dispatcher hits and misses, publish response time, request volume by path, filesystem usage, purge frequency, backend connection counts and the percentage of responses served with errors. Break results down by URL family and geography so a healthy global average does not hide poor performance for visitors in regional areas.
Application performance monitoring can connect cache behaviour with Java and AEM activity. A practical performance monitoring guide can help teams relate slow origin transactions to Dispatcher misses, repository queries and external service calls. Correlating these signals makes it easier to distinguish a cache configuration problem from an overloaded publish instance or a slow integration.
Test with realistic traffic patterns rather than a single constant request rate. Model a product launch, a news event, a ticket release and a burst of mobile traffic. Include cache-warming behaviour, purge storms, origin failures and simultaneous requests for the same uncached URL. In Melbourne or Sydney, a campaign can generate intense demand within minutes; the test should measure whether the first requests collapse into one origin fetch or trigger a thundering herd.
Plan For Resilience And Safe Change
Caching is part of availability engineering. Configure suitable connection and response timeouts, health checks and load-balancing behaviour so a failed publish node is removed promptly. Decide whether stale content may be served during a short origin outage, and document which pages are acceptable to deliver in that mode. A controlled stale response is often preferable to a blank page for public editorial content, but it is unsuitable for inventory, payment or account data.
Protect the cache filesystem from exhaustion. Monitor inode usage as well as storage capacity, because a large number of small objects can prevent new files from being created even when disk space remains. Use separate storage or retention policies for large assets where appropriate, and ensure permissions prevent the web server from writing outside the intended cache directory.
Treat Dispatcher changes as production code. Keep configurations in version control, review security rules separately from performance changes, and deploy through repeatable pipelines. Verify behaviour after every Dispatcher or CDN upgrade, since supported directives and cache-header handling can change. The technical sessions listed in the CIRCUIT agenda reflect the kind of architecture, integration and AEM thinking useful when building this operational discipline.
Practical Optimisation Priorities
Use the following priorities when reviewing an existing AEM delivery stack:
- Establish an allowlist for public cacheable paths, methods, extensions and selectors.
- Remove irrelevant tracking parameters and control meaningful query-string variants.
- Align Dispatcher invalidation, flush agents and CDN purges with publishing workflows.
- Monitor hit ratio alongside origin latency, backend load, storage, errors and purge failures.
- Load-test cache misses, simultaneous requests, campaign spikes and publish-node failure.
- Keep security exclusions and personalisation boundaries explicit and regularly tested.
The best configuration is one that remains understandable during an incident. Record expected cache lifetimes, purge ownership, rollback steps and exceptions for business-critical content. Give content authors clear guidance on when a publication should be visible and provide support teams with a reliable way to verify whether a problem is in the browser, CDN, Dispatcher or AEM.
A disciplined review can turn Dispatcher from a basic reverse proxy into a dependable performance layer for a high-volume AEM platform. Teams preparing an architecture review or technical uplift can use the conference registration details to follow the wider AEM engineering discussions preserved by CIRCUIT, then apply the same evidence-driven approach to their own cache, monitoring and release practices.
Audit the request flow, measure the current baseline and test each invalidation path before changing cache durations. With clear ownership and repeatable deployment, Australian sites can handle sharp demand across major cities and regional networks while keeping content fresh, secure and available.