AEM and Apache Spark: Turning Logs into Insight

Adobe Experience Manager (AEM) creates a detailed operational trail of publishing, delivery and user interaction. CIRCUIT’s two-day Chicago conferences brought the people who build and operate that trail into the same room: Java developers, AEM architects, front-end engineers and systems specialists. Their discussions of integrations, microservices, analytics and architecture established the practical context for combining AEM with Apache Spark.

Apache Spark is suited to this work because log streams are continuous, high-volume and rarely clean. AEM publish instances, dispatchers, CDNs and reverse proxies can each emit different fields, while the same request may cross several systems. Spark can parse those variants, join them into a coherent event stream and calculate both operational metrics and longer-term behaviour without placing heavy analytical workloads on the content delivery tier.

For Australian teams, the technical challenge is paired with a regulatory one. A useful AEM log pipeline must account for different city and daylight-saving patterns, domestic cloud choices, consumer expectations for fast mobile experiences and the Australian Privacy Principles. Done well, it reveals content and delivery friction without turning every visitor into an identifiable marketing record.

What AEM Logs Can Reveal

Start by separating AEM’s log families. Publisher and dispatcher access logs show requests, status codes, response times and cache decisions; CDN or load-balancer logs add edge information; error logs reveal component failures. Author and audit trails serve governance rather than web analytics and should not be mixed into the same dataset. A practical schema includes site, tenant, timestamp, path, method, status, latency, bytes, edge location and anonymised session identifier.

Not every row deserves equal trust. Synthetic checks, crawlers and security scanners can distort traffic, while duplicated events from multiple tiers can inflate page views. Apply bot classification, request deduplication and server-side sampling rules before producing dashboards. Search terms, identifiers, reset tokens and authorisation values can appear in URLs, cookies or headers, while request bodies may contain form fields; allow-list useful parameters and redact everything sensitive at the collector.

Historical insight from CIRCUIT’s 2015 and 2016 events remains valuable. The speaker archive preserves technical explanations from AEM practitioners and shows why domain expertise matters. Ask engineers which status codes matter, which fields are dependable and which events can be joined. That context prevents a technically correct Spark job from answering the wrong business question.

Designing the Spark Pipeline

Keep the analytical workload outside AEM. A durable pattern uses lightweight collectors at each server or edge, an append-only landing zone for raw records, an optional event bus, and Spark Structured Streaming for parsing, enrichment and aggregation. Dispatchers should continue serving content during traffic spikes, while the stream handles bursts asynchronously. Store immutable raw data separately from curated, privacy-reduced outputs so audits and replays remain possible.

Event-time processing is essential. Parse timestamps consistently, retain the original offset and partition data by Australian date and hour. Configure watermarks for late arrivals, especially when westbound events cross time-zone boundaries or a publisher clock drifts. Checkpoint streaming state, make sink writes idempotent and monitor schema changes; a new AEM field should trigger deliberate validation rather than silently corrupt a monthly report.

Reproducibility also includes the code and libraries behind each result. A CIRCUIT article on AEM dependency management shows why controlled artefacts matter in Java ecosystems, and the same principle applies to Spark jobs, connectors and Scala packages. Pin versions, isolate secrets and record the configuration used for every run. A clean rerun six months later is evidence that the pipeline is an operational system, not a one-off demonstration.

Privacy by Design in Australia

In Australia, privacy governance starts with the Privacy Act 1988 and the Australian Privacy Principles, including purpose limitation, security, transparency and controls around disclosure. AEM deployments in retail, banking and government often sit within those obligations, and contracts can add stricter requirements. Define who owns the log data, why each field is collected and how long it is retained before Spark receives it.

IP addresses can be personal information when linked to an individual or device, and usernames, document paths or form values may be more revealing. A one-way hash alone does not guarantee anonymisation; use a keyed transformation, separate the key from the data store and rotate it under a documented policy. Restrict access by role, encrypt data in transit and at rest, log access to sensitive tables, and treat pseudonymised records as personal information where re-identification remains possible.

Mobile interaction makes consent and retention even more visible. A practical mobile example is the CIRCUIT event app, which creates its own interaction path alongside the website. Combining paths does not erase the need for purpose-specific controls. For commercial email or SMS, assess the Spam Act 2003 and obtain the required consent; maintain an audit trail for data breaches under the Notifiable Data Breaches scheme and assess Australian Privacy Principle 8 controls before disclosing personal information overseas.

Operating Across Australian Time Zones

Australian geography adds genuinely tricky time semantics. Sydney and Melbourne usually run on UTC+10 and shift to UTC+11 during daylight saving, while Brisbane remains on UTC+10 without changing. That creates repeated local hours and late events around the autumn transition. Configure Spark session windows with time-zone-aware event timestamps, not server-local strings, and test jobs across both daylight-saving boundaries before publishing hourly or daily figures.

Connectivity patterns matter too. Many households rely on NBN broadband, while many commuters and shoppers move between 4G and 5G coverage, causing mobile sessions to resume or repeat requests. Apply session stitching conservatively and use the client network type, not the server’s origin, to describe user conditions. Australian retailers and banks also expect AEM to support public holidays, promotional campaigns and branch networks, so baselines should separate calendar effects from genuine platform regressions.

Operational discipline turns those realities into a service. Track collector lag, Spark processing latency, dead-letter volume, cost per million events and data-quality failures as service-level indicators. A DataOps layer built around a platform such as the Rhizo4 data platform can coordinate metadata, lineage and access controls, while an internal runbook should identify owners, replay procedures and escalation paths. Keep raw and curated retention periods aligned with Australian legal requirements and contractual commitments.

Turning Analysis into Action

The first useful deliverable should be narrow. A dispatcher dashboard showing cache-hit rate, 5xx errors and latency by site can reveal delivery problems that content analytics miss. Another candidate is a daily report pairing landing-page failures with affected asset paths. These operational views are usually easier to trust than ambitious profiles assembled from incomplete logs, and they create a clean foundation for later experimentation.

Spark becomes more valuable as curated aggregates move beyond monitoring. Join request data with approved content metadata, campaign timing and Adobe Analytics measures only where identifiers and definitions align. AEM logs can reveal broken journeys, unavailable assets and technical abandonment, while analytics tools describe what people officially measured. Comparing the two can expose gaps without pretending request-level logs are a complete replacement for consented behavioural tracking.

Broader DataOps practices help teams sustain that comparison. Material from DataOps Summit explores similar ideas around quality, lineage, governance and reliable data products. Publish metric definitions beside each dashboard, version transforms in source control and attach a data owner to every production output. Then route actionable exceptions to the engineers who can resolve them, rather than burying them in a report nobody owns.

Begin with one measurable AEM delivery problem and one clearly bounded dataset. Choose the fields, retention period, data owner and success measure before writing a Spark job, then run a two-week pilot against a known traffic period. Compare the result with existing analytics and incident records, quantify false positives and document every privacy decision. That creates evidence an Australian team can review with legal, security and engineering peers.

Once the pilot meets its accuracy and governance thresholds, productionise the collectors, schema checks, replay path and alerts as one managed service. Publish the dashboard, name its operational owner and schedule a regular review against business outcomes. Then repeat the pattern for the next use case, reusing the landing zone, definitions and controls instead of rebuilding another disconnected data pipeline.