AEM and Kafka for real-time event streaming
CIRCUIT brought Java developers, AEM architects, front-end engineers and systems specialists together in Chicago for hands-on work around Adobe Experience Manager. Its sessions covered integrations, microservices, mobile delivery, analytics and architecture, providing a useful foundation for connected publishing platforms rather than treating AEM as an isolated content repository.
Kafka adds a durable event backbone to that foundation. When AEM publishes page views, asset changes, approvals or personalised interactions as structured events, downstream teams can react without waiting for batch jobs or tight request chains. Content and customer signals can then move at the speed of the business.
That pattern works only when delivery is treated as a product, not a side effect. Producers need stable contracts, consumers need replay and idempotency, and operators need visibility from the browser action to the processed record. A Kafka topic cannot repair an ambiguous event model or compensate for an AEM request thread waiting on an unhealthy broker.
Australian teams also face practical constraints that shape the design. Sydney and Melbourne host major cloud regions and large enterprise estates, while Brisbane, Adelaide and Perth create distinct operating windows. The Privacy Act 1988 and Australian Privacy Principles govern many customer records, and the Spam Act 2003 matters whenever an event feeds commercial messaging. Local time zones, daylight-saving changes and hybrid identity environments must therefore be designed into the architecture.
Why AEM needs an event backbone
Integrating AEM with Kafka for real-time event streaming starts by assigning clear responsibilities. AEM owns authored content, asset metadata, permissions and the request context required to assemble an experience. Kafka owns the durable, ordered transport that carries those facts beyond the publishing tier.
A useful division prevents two common failures: coupling AEM directly to every consumer, and sending unstructured notifications that no downstream service can interpret. An internal AEM observation event may be enough for a local workflow; a cross-platform business event deserves a versioned envelope, a named topic and an explicit retention period.
Architecture decisions are easier to test when the people responsible for content, platform operations and data consumption can inspect the same examples. Recordings of specialist CIRCUIT sessions remain valuable for that purpose, while broader developer community events can help teams compare AEM practices outside a single product team.
Designing reliable event contracts
Begin with a contract that describes meaning, not implementation. Include an event identifier, type, schema version, occurrence time, producer, tenant, correlation identifier, subject key and trace context. Add business fields only where a consumer needs them; raw form submissions, session transcripts and full identity profiles rarely belong in a broadly shared stream.
Partition by the key that defines business order, such as a pseudonymous customer identifier or content path. Kafka then preserves order within that partition, while separate partitions allow parallel work. Store schemas in a governed registry, prefer backward-compatible changes and use Avro or JSON Schema according to the skills available to both producers and consumers.
Assume at-least-once delivery. Consumers should deduplicate by event identifier, and producers should be idempotent where possible. Retention enables replay, but it also widens the period during which a privacy mistake remains recoverable. Classify fields, tokenise identifiers and define how long each topic can safely retain its data.
Publishing from AEM without blocking requests
Keep Kafka off the author request path. A page request that waits for broker acknowledgement adds latency and creates a direct failure dependency. Instead, write an outbox record with the business change in the same JCR transaction where practical, then let a small relay publish it asynchronously. The relay may run as a controlled sidecar, a scheduled AEM job or a dedicated service, depending on the hosting model.
Configure the producer for reliability rather than fire-and-forget delivery. Use all required acknowledgements, enable idempotence, bound retries with delivery timeouts and attach correlation metadata to every record. AEM Cloud Service teams must also allowlist broker endpoints and provide the identity, certificate or OAuth credentials required for outbound TLS.
Monitor the gap between committed outbox records and acknowledged Kafka records. Alert on oldest unpublished record age, publish failures, retries and changes in consumer lag. Poison events should enter a quarantined topic or dead-letter flow with the original payload protected according to policy; repeatedly retrying an invalid record can otherwise block operational work.
Connecting Kafka to downstream systems
Downstream consumers should be independent services with their own deployment cadence and failure domains. Search indexing, analytics enrichment, notification dispatch and CRM synchronisation can subscribe to the same facts without sharing runtime code. This creates useful back-pressure: a slow consumer catches up from Kafka rather than exhausting AEM threads.
Kafka Connect can bridge supported databases, queues and SaaS platforms, but a custom AEM source usually needs an HTTP producer or a connector tailored to the outbox schema. Keep connectors stateless where possible, version their mappings and test them against replayed records. A bespoke consumer may be clearer when the transformation, rate limit or error policy is domain-specific.
Mobile delivery deserves particular care. A Kafka event can trigger a notification service or update a read model, yet phones regularly lose connectivity and enter low-power modes. Push systems must tolerate delayed receipts, and back-end consumers should not assume that every published interaction represents an immediately available device.
Security, privacy, and Australian operations
Secure the route from AEM to the broker and from consumers back to source systems. Use TLS, short-lived credentials, topic-level authorisation and least-privilege service identities; a network allowlist is useful defence in depth, not a replacement for authentication. Treat Kafka ACLs as production policy and test certificate rotation before expiry becomes an outage.
Privacy controls must follow the data, including cached events and derived indexes. Under the Privacy Act 1988, the Australian Privacy Principles can require entities to limit collection, use and disclosure, keep accurate records and protect personal information. If a stream powers email or SMS marketing, consent, purpose and unsubscribe signals must reach the relevant processor so Spam Act 2003 obligations are not lost between systems.
Location is an architectural choice, not a checkbox. For Australian banks, insurers and retailers, a Sydney or Melbourne region can reduce latency for many users, but failover, replication cost and support coverage still need testing. Brisbane remains on Australian Eastern Standard Time, Sydney and Melbourne shift during daylight saving, Adelaide uses a half-hour offset and Perth is another two hours behind; store and calculate in UTC, then render in the user’s zone.
From events to search and identity
Search is a strong first consumer because its effects can be measured. When an asset or page event updates an index, a Solr or Elasticsearch implementation can maintain current results and near-real-time custom search facets, including counts, availability and campaign labels. Use an idempotent indexer and compare catalogue totals against AEM to detect missed or duplicate updates.
Identity provides another useful consumer path. A directory or customer-profile service can resolve the event’s pseudonymous subject key, while AEM continues to enforce publishing permissions. For organisations integrating an existing corporate directory, CIRCUIT guidance on custom AEM authentication offers a practical starting point; that identity layer should remain separate from Kafka’s service-to-service authentication.
Roll out the pattern with one event, one consumer and one measurable outcome. Start in a non-production environment, replay representative records, document ownership, then test broker loss, duplicate delivery and schema evolution. Move to production only when alerts identify the event’s journey, consumers can rebuild their state, and legal and security owners have approved the payload and retention policy.
Take the next step with a concrete event from your own AEM estate. Define its business purpose, owner, schema, destination and success measure, then validate it against the failure modes in this architecture. A small, observable stream that survives replay is more valuable than a broad catalogue of events nobody trusts.
Share session notes, implementation findings and operating metrics with fellow CIRCUIT participants, and bring the chosen event to the next technical working session. The target is a responsive AEM publishing layer connected to dependable services, with every customer signal traceable from origin to outcome.