AEM and Redis: Offloading Dispatcher Caching for Scale

When an Adobe Experience Manager site serves visitors spread across Sydney, Perth, Brisbane, and remote mining towns in the Pilbara, every millisecond of latency feels punishing. AEM's built-in Dispatcher is excellent at its designed job: sitting in front of publish instances, caching rendered HTML, absorbing steady anonymous traffic. Once the audience grows past a few million page views a day, or content has to be served from Adelaide to Darwin with regional failover, the single-box file cache becomes the bottleneck. Redis enters as a distributed tier handling response caching, micro-caching, and invalidation across the fleet.

For Australian teams running retail checkouts or government portals bound by the Australian Privacy Principles, an in-memory store offers predictable performance. Redis runs on commodity hardware, replicates across availability zones in Sydney or Melbourne, and gives architects a deterministic hit ratio. The aim of moving Dispatcher responses into Redis is rarely raw speed. It is consistency across geographically separated read replicas, surviving publish instance failure without a thundering herd, and scaling horizontally without redesigning the content tree every quarter.

How the Dispatcher Cache Actually Works

The Dispatcher module for Apache HTTP Server or Microsoft IIS inspects each request, checks its internal rules for caching eligibility, and either serves a stored file from its document root or forwards the request upstream. Cached files are keyed on the request path plus a small set of headers defined in dispatcher.any, written to disk as plain files that are easy to inspect and trivial to invalidate with a recursive delete. The elegance of this approach is its transparency; operators can browse the cache directory and reason about behaviour using standard Unix tooling.

The fragility shows up at scale. When a single Dispatcher host must handle thousands of requests per second, when the cache directory grows past hundreds of gigabytes, or when content authors publish aggressively and the filesystem becomes a contention point, throughput collapses once the working set no longer fits in the OS page cache. Two Dispatchers behind a load balancer cannot share state without help, so each independently re-fetches the warm-up set after a restart, doubling origin load during the cold-start window. As Australian teams often deploy across multiple data centres to satisfy latency requirements for users on opposite coasts, this replication gap becomes a planning problem.

Where Traditional Cache Layers Fall Short

Teams using Varnish or a CDN like Akamai know the value of edge caching for public, anonymous, geographically concentrated traffic, covering most marketing pages and content fragments. These layers struggle with personalised experiences, A/B variants, authenticated sessions, and the long tail of dynamic components that AEM produces through Sling Models and Sightly scripts. Pushing everything through the same cache leads to a cache key explosion: variants, query parameters, locale, and personalisation tokens multiply the working set until the hit ratio drops below the threshold where the cache pays for itself.

The Australian retail calendar, with click-frenzies around EOFY sales and Boxing Day, sharpens this further because the working set must absorb sudden traffic shifts without warm-up time. A second-tier in-memory cache sitting behind the CDN but in front of the JCR repository has a different job: catching personalised fragments, cart previews, authenticated JSON responses, and the small percentage of long-tail URLs edge caches always miss. This is the niche Redis fills, and where design effort should focus.

Redis as a Distributed Cache Backend

Redis is a single-threaded, in-memory key-value store with optional persistence, pub/sub messaging, and a small set of data types. For caching AEM Dispatcher responses, the relevant features are GET, SET with expiry, atomic counters, and the ability to publish invalidation events across a cluster. Cluster handles sharding automatically; Sentinel or a managed service handles failover. The integration point is a small servlet filter or OSGi component on the publish tier that computes a cache key from method, host, path, and a header whitelist, looks up Redis, and returns the stored payload if present. For binary assets, the same pattern often sits in front of object storage, as described in the write-up on scalable asset delivery.

On a miss, the filter forwards the request to the Sling pipeline, captures the response, writes it to Redis with a TTL, and returns it. From the user's perspective the response is identical, but the origin is shielded from repeat work. For high availability, Redis should be deployed in at least two availability zones within an Australian region. AWS Sydney, Azure Australia East, and Google Cloud Sydney all provide the low-latency inter-zone links Redis replication depends on. Teams wanting geographic redundancy can run a primary cluster in Sydney with a read-only replica in Melbourne and accept a few milliseconds of replication lag.

Designing the Cache Key and TTL Strategy

A cache key that is too coarse serves the wrong content to the wrong user; a key that is too fine fragments the cache into millions of one-off entries that almost never hit. The art is in the middle, starting with a clear definition of what is cacheable. Anything that varies by authenticated user, depends on session state, or performs a write should be excluded by default. For what remains, the key encodes method, path, host, and a header whitelist like Accept-Language and X-Forwarded-Proto, with query parameters sorted and only included if they appear on an allow-list.

TTL choice is where many implementations over- or under-shoot: editorial pages might cache for ten minutes; product listing JSON for thirty seconds; personalised fragments for a single second, relying on the millisecond advantage of Redis over a database round trip. The stored value should be the response body, content type, and cache-control headers worth preserving, serialised as a byte string or small hash with a versioned schema.

Cache Invalidation and Consistency Patterns

Caching without a plan for invalidation is storing stale data on purpose. AEM provides replication and event hooks that Redis caches can subscribe to: a JCR observation listener on the publish instance can fire on every node change, compute the affected cache keys, and publish them to a Redis channel that every Dispatcher-side filter listens on. Workflows touching content publishing approvals, like the patterns in using AEM workflows to automate content publishing approvals, can drive this listener directly and ensure cache freshness from the moment content goes live.

The simplest invalidation primitive is a key delete; each filter receives the command, removes the local entry, and the next request rebuilds it. For content trees that change wholesale, tag every cache entry with the content paths it depends on, then scan a small in-memory index for matching tags and drop the entries. Eventual consistency is acceptable for most marketing content because AEM's own replication queue is already eventually consistent. For commerce flows touching stock levels or pricing, the right answer is a short TTL plus an application-layer write-through pattern.

Deployment, Monitoring, and Australian Compliance

Running Redis in production is straightforward, but running it well requires attention to memory headroom, eviction policy, replication lag, and security. The default eviction policy, noeviction, is rarely what teams want in a cache role; allkeys-lru is a safe default that drops the least recently used entry when memory fills, with maxmemory sized at seventy percent of host RAM to leave headroom for forks and replication buffers. Monitoring should track hit ratio, eviction rate, command latency, replication offset, and per-key TTL distribution.

Australian teams operating under the Notifiable Data Breaches scheme should treat Redis as a sensitive store. Authentication should use ACLs, transport should be TLS, and cache values containing personal information should be excluded or encrypted at rest with a key managed in a local HSM. The Privacy Act 1988 does not distinguish between disk and memory; the obligation sits with the data controller regardless. For teams on AEM as a Cloud Service, much of this is offered as managed capability through CDN configuration and caching headers. For self-hosted AEM 6.5, the implementation is a few hundred lines of code in an OSGi bundle plus a Redis cluster sized for peak concurrency.

If your team is sizing a Redis cluster for an upcoming campaign or migrating off a single Dispatcher instance, the CIRCUIT session recordings from 2015 and 2016 include deep-dive talks on AEM integrations, Sightly, and architecture that complement this article. Subscribe to the CIRCUIT newsletter for the call for papers, follow the channel for early-bird pricing, and reserve a seat when registrations open for the next Chicago gathering.