AEM and Kafka: streaming AEM asset events to microservices
Adobe Experience Manager is often the centre of an organisation’s digital asset workflow. Images, videos, documents and metadata enter AEM, move through approval stages, and eventually appear across websites, apps, commerce platforms and campaign tools. That flow becomes difficult to manage when each downstream system relies on its own polling job or custom integration.
Apache Kafka provides a durable event stream between AEM and those services. Instead of asking AEM repeatedly whether an asset has changed, a consumer can receive a structured notification when an asset is created, updated, published, moved or removed. This approach supports responsive processing while keeping each microservice focused on a clear responsibility.
The pattern is especially useful for Australian organisations operating across Sydney, Melbourne, Brisbane, Perth and Adelaide. A distributed team may publish content during different business hours, while customers expect fast mobile experiences over variable network conditions. A shared event backbone gives engineering teams a consistent way to coordinate asset processing without tightly coupling every platform to AEM.
The CIRCUIT conference’s focus on AEM architecture, Java development and practical integrations provides useful context for this design. Teams reviewing historic sessions and event material can also check registration details to understand how the conference brought together developers, architects and systems engineers around production-grade implementation decisions.
Why an asset event stream matters
AEM can emit repository and workflow events when content changes. A listener in an AEM bundle can inspect the event, validate the affected asset path and publish a compact message to Kafka. The message should describe what happened rather than replicate the complete binary file. A typical payload might include the asset identifier, action, version, MIME type, checksum, author, timestamp and publication state.
Kafka then acts as a buffer between the content platform and downstream consumers. An image optimisation service can subscribe to asset publication events, while a search indexer listens for metadata changes and a rights-management service handles licensing information. Each consumer progresses at its own pace, and a temporary outage in one service does not necessarily stop editors from working in AEM.
This separation is valuable when AEM sits alongside Salesforce, commerce, analytics and mobile applications. The principles discussed in AEM and Salesforce integration apply here as well: establish a clear contract between systems, keep personalisation data governed, and avoid making one platform responsible for every delivery decision.
Designing the AEM-to-Kafka pipeline
A practical implementation usually starts with an AEM event handler or workflow step. The handler should filter noisy repository activity and select business-relevant events, such as an approved asset becoming available for publication. A Sling Job or equivalent asynchronous mechanism can provide retry handling inside AEM before the message reaches Kafka, reducing the risk that a brief broker or network failure blocks editorial operations.
The Kafka topic structure should reflect the event domain rather than individual consumers. Topics such as asset.lifecycle, asset.metadata and asset.publication can be useful when the organisation has clear ownership boundaries. Partition keys should preserve ordering for a related asset, while avoiding a single highly active folder becoming a bottleneck. A schema registry can enforce compatibility as Java services and other consumers evolve.
Message delivery is commonly at least once, so consumers must be idempotent. If an asset publication event is delivered twice, the image processor should recognise the asset ID and version rather than create duplicate work. Consumers should also record offsets only after successful processing, send unrecoverable messages to a dead-letter topic, and expose lag metrics for operations teams. These controls matter more than attempting to promise exactly-once behaviour across AEM, Kafka and external services.
Security, privacy and Australian delivery needs
The event should contain the minimum information required to perform its job. An asset path, repository ID and version may be enough; editor names, customer identifiers or embedded campaign notes may be unnecessary. Australian organisations should consider the Privacy Act 1988 and the Australian Privacy Principles when metadata can identify a person or reveal sensitive business activity. A Kafka topic is still a data store, so access controls, retention rules and audit trails need to apply.
Data residency can also influence broker placement. A retailer serving customers in New South Wales and Victoria may choose Australian cloud regions for operational data, while a national organisation may require disaster recovery in a second local region. Encryption in transit, encryption at rest and restricted service accounts should be standard. Network design must account for private connectivity between AEM hosting, Kafka brokers and microservices rather than assuming every component belongs on the public internet.
Australian digital teams also need to consider the Spam Act 2003 when asset events trigger marketing messages. A new product image should not automatically cause a promotional email unless consent, subscription status and campaign rules have been checked by a dedicated service. Kafka can distribute the signal, but it should not become an uncontrolled mechanism for sending communications.
Checks for a production-ready stream
- Define asset actions, payload fields and schema ownership
- Use keys that preserve ordering for each asset
- Apply least-privilege access to topics and consumer groups
- Set retention and deletion rules for operational data
Reliability from editor action to customer experience
A useful design distinguishes between the source event and the processing result. When an editor publishes an asset, AEM can emit an AssetPublished event. The image service may then produce an AssetTransformed event, followed by a delivery service recording that the rendition is available. This chain gives teams visibility into where work is delayed and prevents every service from interpreting the original AEM event differently.
Observability should cover AEM listener failures, Kafka producer errors, broker health, consumer lag, retry counts and dead-letter volume. Correlation IDs should travel through every message and appear in logs and traces. When a Melbourne content team reports that a campaign image is missing from a mobile application, support staff should be able to trace the asset from AEM publication through transformation, CDN delivery and application retrieval.
Load testing should represent realistic publishing behaviour rather than only steady traffic. A major retailer may release thousands of product assets before a seasonal campaign, while a media organisation may publish bursts around a sporting event. Test partition limits, consumer recovery, schema changes and replay procedures. Replay is one of Kafka’s strengths, but it must be safe: consumers need version-aware logic and clear rules for whether an old event should rebuild or merely validate an existing rendition.
For a public-facing prototype, keep credentials and customer data out of the stream. A small destination such as Chili Chillin can be used as a low-risk example of where content might appear, while production experiments remain isolated behind test topics, synthetic assets and non-production delivery endpoints.
Choosing services and operating the platform
Microservices should subscribe according to business capability, not according to the AEM internal class structure. A search service needs searchable metadata and lifecycle events. An asset transformation service needs binaries or secure references to binaries. A rights service may need licence expiry information but should not receive every editorial detail. This bounded approach keeps contracts stable if the AEM implementation changes.
Kafka is powerful, but it carries operational overhead. Teams need broker monitoring, partition planning, upgrades, backup strategy and incident runbooks. Organisations without that capability may use a managed Kafka service, provided its networking, Australian region availability and support model meet requirements. Java developers familiar with AEM can build producers and consumers, while platform engineers should own capacity, security and reliability standards.
The wider developer community can help teams compare patterns across ecosystems. A conference audience that includes Java, front-end and systems engineers benefits from seeing how event-driven AEM integrations relate to broader enterprise development practices; the C# developer community offers a useful adjacent perspective on services, messaging and application architecture.
Start with one measurable workflow, such as generating web renditions after approved publication. Define the event contract, implement one consumer, measure processing time and failure recovery, then add search, commerce or analytics integrations. This gradual approach provides evidence before the organisation commits every AEM asset operation to a larger event platform.
A practical implementation sequence
Begin by mapping the asset lifecycle from upload to publication and identifying which state changes genuinely matter. Decide whether the event is emitted by a workflow, repository listener or application service, then document how retries behave. Keep the first payload small and versioned, with a secure reference for large binaries rather than placing files directly in Kafka.
Next, create a development topic and a consumer that performs one deterministic task. Add idempotency using the asset ID and version, capture processing status, and test duplicate, delayed and out-of-order events. Include a dead-letter path from the beginning so failed messages become visible instead of disappearing into application logs.
Finally, establish operational ownership. AEM administrators, application developers, security specialists and platform engineers should agree on topic permissions, retention, alert thresholds and incident response. Teams can draw on the technical culture represented by CIRCUIT’s workshops and recordings while adapting the pattern to local privacy, hosting and delivery requirements.
AEM and Kafka work best together when the event contract is deliberate and the surrounding platform is observable. Build the smallest useful stream, verify it with real asset workflows, and then expand carefully into search, personalisation, commerce and mobile delivery. Explore the available conference material and register through the event site to continue developing the architecture with practical AEM integration knowledge.