AEM and Apache Kafka: event-driven content invalidation

Most teams running Adobe Experience Manager eventually hit the same wall: a content author publishes a page in Sydney, a marketer in Perth tweaks a campaign component, and somewhere in between, a cached version lingers on a publish farm, frustrating both editors and customers. The fix has historically been a polling dispatcher flush, a cron-based cache cleanup, or a heavy-handed manual replication. Each approach works, but none of them scales gracefully across multi-region deployments where every minute of stale content erodes trust.

Apache Kafka changes the picture by turning invalidation into a streaming concern rather than a batch job. Instead of asking the dispatcher to "check again in a few minutes," your application code can announce change events as soon as a node is touched, and downstream consumers react in near real time. For organisations juggling AEM alongside other systems such as search indexes, mobile apps, or personalisation engines, that shift matters: one event can fan out to many subscribers without coupling them to the CMS itself.

This piece walks through how teams are wiring AEM into Kafka, what the pipeline looks like in practice, and where Australian implementations have run into quirks worth knowing about before you start.

Why traditional invalidation falls short

The default AEM replication mechanism was designed when most deployments were a single authoring instance pointing at one or two publish farms. Replication agents push content from author to publish, and the dispatcher cache is purged by comparing paths or by relying on TTLs. That model still works, but it assumes the cache, the CMS, and the consumers all live in the same data centre.

Once you introduce a CDN, a second region, or a headless front end that consumes AEM content over APIs, the assumptions begin to crack. TTLs become a trade-off: too long and editors complain about stale campaign banners; too short and the dispatcher revalidates constantly, defeating the purpose of caching. Manual purges do not scale when content changes hundreds of times a day, which is common for retailers running flash sales or news outlets pushing breaking stories.

Australian teams feel this acutely because distance amplifies latency. A publish farm in Sydney and a CDN edge node in Singapore already have noticeable round-trip times, and a TTL-based strategy compounds that delay by design. Event-driven invalidation sidesteps the issue by pushing notifications immediately, regardless of where the cache sits.

Fitting Kafka between authoring and publish

The cleanest way to introduce Kafka into an AEM topology is to treat it as a transport layer rather than a replacement for replication. The author instance still owns the workflow that activates pages, but before replication completes, an event listener publishes a record to a Kafka topic describing what changed. Common payloads include the path, the node type, the user who triggered the change, and any tags or metadata that downstream consumers might filter on.

From there, Kafka acts as a buffer and a router. The publish farm consumes the topic and invalidates its dispatcher cache locally. A separate consumer can rebuild a search index. Another can push a delta to a mobile API gateway. Each subscriber reads at its own pace, and a slow consumer will not back-pressure the CMS.

Teams often ask whether this means abandoning AEM's built-in replication entirely. The pragmatic answer is no; replication still handles the heavy lifting of moving binaries and renditions, while Kafka handles the lightweight, high-frequency invalidation signals. The two complement each other rather than compete, and that hybrid pattern now shows up across production AEM deployments across the country.

Building the pipeline: producers, topics, consumers

In practice, the producer side tends to live inside a custom OSGi service or an AEM workflow step that fires after activation. A small Java class listens for repository events, filters out internal-only nodes, and serialises the payload as JSON or Avro before handing it to a Kafka producer. Schema management matters here: a registry such as Confluent Schema Registry keeps producers and consumers from drifting out of sync when the payload evolves.

Topic design follows the consumer's needs more than the producer's convenience. A single cms.events topic with a JSON payload is the easiest starting point, but partitioning quickly becomes important. Hashing by path ensures that all events for a given page land on the same partition, which preserves ordering for cache invalidation. A consumer group per region then lets a Sydney farm and a Melbourne farm each process the events that matter to them without stepping on each other.

Error handling deserves attention early. A consumer that fails to invalidate a cache entry should not silently drop the message; it should retry, dead-letter the record, or surface the failure to an operations dashboard. Skipping this step is how teams end up debugging "why is this banner still old" tickets three weeks after launch.

Cache coherence across separated publish farms

Multi-region AEM setups are increasingly common in Australia, partly because organisations want failover resilience and partly because content delivery expectations are uniform across capital cities. A bank headquartered in Melbourne might serve customers from Sydney, Brisbane, and Adelaide with similar latency budgets, and the CDN edge nodes scattered around the country need to honour invalidations quickly.

With Kafka in place, each publish farm subscribes to its own consumer group, processes the events relevant to its local dispatcher, and confirms back via Kafka offsets. That confirmation is what gives operations teams confidence: if a record has been committed, the cache has been touched. Replication lag, in this model, is no longer measured by the replication agent's queue depth but by the consumer offset lag visible in Kafka tooling.

Some teams also publish events to a separate topic for read-only consumers, such as analytics platforms that want to log every content change. Keeping that traffic on its own topic prevents analytics back-pressure from slowing down cache invalidation, which is the latency-sensitive path. The TarMK and DocumentMK comparison covered at a past CIRCUIT session is worth revisiting here, because the persistence choice influences how quickly repository events surface in the first place.

Operational realities: lag, partitioning, replay

Once the pipeline is live, the conversation shifts from architecture to operations. Consumer lag is the metric that tells you whether invalidations are keeping up. A lag of a few seconds is fine; a lag that climbs into minutes during a marketing push is a sign that the consumer is under-provisioned or that partition counts need revisiting.

Replay is another operational lever. Because Kafka retains events for a configurable window, a new consumer can be spun up to backfill a search index or to rebuild a cache after a regional outage. That capability is genuinely useful in a country where undersea cable incidents and data centre maintenance windows are not hypothetical risks; both Sydney-to-Singapore and Sydney-to-Los Angeles routes have had outages that affected Australian AEM operators in recent years.

Partitioning strategy deserves a second look as traffic grows. A topic with three partitions might be enough for a small team, but a retailer running weekend campaigns can easily overwhelm that. Increasing partition counts later is possible but requires care around key affinity. Planning for headroom early avoids a migration that risks consumer downtime.

Patterns from Australian AEM deployments

Conversations with teams running AEM in Australia tend to circle back to three patterns. The first is consolidation around Sydney or Melbourne as the primary authoring region, with everything else treated as edge. The second is a strong preference for managed Kafka services, whether Confluent Cloud, MSK, or Aiven hosted in ap-southeast-2, because running your own brokers adds operational weight that smaller AEM teams cannot justify.

The third pattern is the pragmatic use of Australian English spelling and locale handling inside AEM itself. It sounds trivial until a campaign references "colour" and "favour" and the site mysteriously serves American English variants because the i18n dictionary was never reviewed. Engineering teams often partner with content teams to maintain locale dictionaries that reflect how Australians actually talk, including idioms that slip into marketing copy.

Large financial services groups, government departments, and retail brands are the heaviest AEM users in the country. Commonwealth Bank, NAB, and a handful of state government portals run multi-farm topologies where Kafka-based invalidation is no longer a research project but a production workload. The lessons from those deployments tend to be less about Kafka itself and more about how AEM's event model surfaces change notifications reliably under load.

Event-driven invalidation has matured from experimental to routine across Australian AEM deployments, and the tooling around Kafka has kept pace. Recordings from the CIRCUIT conference in Chicago remain a useful starting point, with session videos and slides available through the conference app. Pairing those talks with hands-on prototyping usually surfaces the design considerations that matter before a production rollout.