Powering AEM Search With Elasticsearch for Rich Full-Text Queries

AEM ships with a capable query engine built on Oak and Lucene, but production teams running large-scale content estates often find that the platform's stock search features struggle once traffic climbs, content fragments multiply, and business users expect faceted filtering on top of relevance ranking. Elasticsearch steps in as a parallel index, absorbing documents from the JCR repository and exposing them through its own REST API for lightning-fast retrieval, typo-tolerant matching, and analytics-grade aggregations.

Developers who attended the CIRCUIT developer conference in Chicago have spent the past several years refining these hybrid architectures. The pattern that keeps surfacing is straightforward: let AEM stay the system of record for authoring, governance, and workflow, while Elasticsearch becomes the read-optimised search layer powering site search, content recommendations, and reporting dashboards. The two systems exchange data through event-driven pipelines rather than direct database calls, which preserves the integrity of the source content while giving end users sub-second responses.

Why Elasticsearch Complements AEM's Built-In Search

Oak indexes in AEM do an admirable job of supporting XPath and JCR-SQL2 queries against the repository, yet they were designed for traversal rather than analytic exploration. Aggregation requests — the kind that power category facets on a retail site, count-by-author reports in a newsroom, or geo-clustered listings on a property portal — quickly expose their limits. Elasticsearch was built from day one for exactly this workload, with inverted indexes that handle fuzzy matching and bucketed roll-ups in a single round trip.

Australian enterprise teams have noticed this gap most acutely in regulated industries such as banking and healthcare. ANZ-style deployments in Sydney and Melbourne often combine AEM Sites with compliance review workflows that tag every asset before publication. Once the tags land in the JCR, replicating them into an Elasticsearch cluster lets compliance officers and content authors slice the corpus by region, product line, or risk classification without hammering the repository. The cluster can be sized independently of AEM, which matters when search demand spikes during a marketing campaign but authoring load stays steady.

For teams new to the conference recordings, the CIRCUIT conference archive offers a clear baseline of the integration patterns discussed here, including session videos that walk through Oak index definitions and Lucene query parsing before pivoting to Elasticsearch DSL examples.

Mapping Content for Full-Text Search

The first engineering decision is the document model. AEM pages, assets, and Experience Fragments each have their own node structure, and the mapping layer needs to flatten those trees into JSON documents that Elasticsearch can index efficiently. Most teams settle on a transformer service that listens for JCR observation events, walks the affected subtree, and emits a denormalised payload containing the page path, title, body text, tags, last-modified timestamp, and any custom properties required by the front-end search UI.

Schema choices ripple through the rest of the system. Picking the right analyser matters enormously for English content with Australian spellings, product names, and acronym-heavy industries. A custom analyser that combines a standard tokenizer with lowercase, asciifolding, and a synonym graph covering common regional terms ("footpath" alongside "sidewalk", "lift" alongside "elevator") dramatically improves recall for local readers. Date fields benefit from explicit formats; numeric fields should be coerced at index time to avoid Lucene's default string sorting, which produces results like "10" appearing before "2".

The mapping document also defines fields used purely for aggregations. Tag lists, author identifiers, and category paths are stored as keyword fields with doc_values enabled, while full-text body fields are configured with index: true but doc_values: false to save heap space. This division of labour keeps the cluster healthy when content volume grows and lets ops teams tune memory budgets separately for search versus analytics workloads.

Building Aggregation Pipelines for Content Insights

Aggregations are where Elasticsearch earns its keep on top of ordinary text retrieval. A terms aggregation on the author field, nested inside a date histogram on the published month, surfaces editorial output trends that would otherwise require a separate data warehouse. Metric aggregations compute average read time per category, while pipeline aggregations such as derivative and cumulative_sum track week-over-week changes in content production.

In practice, content teams in Brisbane and Perth have used these capabilities to build internal dashboards that monitor publishing velocity across business units. One common query combines a filter on a workflow state, a nested aggregation on tag co-occurrence, and a geo-distance aggregation that bins assets by the office location of their owning team. The result renders as a heatmap in Kibana and feeds into the weekly editorial standup, replacing the brittle spreadsheet reports that used to circulate by email.

Performance considerations shape how queries are written. Deeply nested aggregations can balloon heap usage, so engineers usually cap the cardinality of terms aggregations with a size parameter and pair them with composite aggregations for paginated buckets. Runtime fields let teams derive new dimensions on the fly without reindexing, which is handy when the business invents a new taxonomy category mid-cycle. Keeping all of this in version-controlled query templates, rather than hard-coding them in JSP or HTL, makes the system auditable when a regulator such as the ACCC asks how a particular content recommendation was generated.

Integration Patterns and Microservices

Most production deployments treat Elasticsearch as a sidecar service rather than an embedded library. Sling jobs or an external scheduler running on Amazon EC2 instances in the ap-southeast-2 region push deltas into the cluster, while an OSGi bundle inside AEM exposes a thin proxy that translates search requests from Sightly components into Elasticsearch DSL. The proxy enforces authentication, applies tenant-specific filters, and caches hot queries with a short TTL to absorb traffic bursts.

Resilience comes from queueing. Apache Kafka sits between the AEM publisher and the indexer, so a brief Elasticsearch outage does not block content publication. The indexer drains the topic on recovery, replays missed events, and reconciles the cluster state against a periodic full reindex pulled from the JCR. This pattern, sometimes called the outbox pattern, has become standard advice from integration specialists who have worked on large AEM estates.

For teams reviewing architecture decisions, the ICF Olson team background page outlines how a partner consultancy structures these engagements, including which deliverables are handed back to the client and which responsibilities stay with internal platform owners. The page doubles as a useful checklist for procurement teams drafting statements of work.

Performance, Scaling, and Operations in Production

Once the search layer is live, attention turns to operations. Cluster sizing in Australia typically starts with three master-eligible nodes and at least two data nodes spread across availability zones, with shard counts set to roughly one-and-a-half times the number of data nodes to allow rolling restarts. Heap sizes between 16 and 31 gigabytes are common, and circuit breakers are tuned aggressively to prevent a single runaway aggregation from starving the JVM.

Monitoring leans heavily on Elastic's own stack. Beats ship cluster metrics and slow log entries into a separate monitoring cluster, where Kibana dashboards flag p99 query latency, JVM pressure, and merge backlogs. Alerts fan out to PagerDuty or Opsgenie, and runbooks reference specific CIRCUIT talks on Lucene segment merging that explain the underlying mechanics. The same dashboards inform quarterly capacity reviews where finance teams want to know whether the next marketing campaign justifies another data node or a move to a managed Elastic Cloud subscription on AWS Sydney.

Cost optimisation rarely gets discussed alongside relevance tuning, but it deserves space. Frozen indices, searchable snapshots backed by S3, and tiered storage between hot and warm nodes can cut infrastructure spend by double digits without affecting end-user latency. Australian organisations operating across state borders must also weigh data residency, particularly when serving NSW Government audiences whose records must remain onshore. Configuring snapshot repositories to point at Sydney-region buckets satisfies the requirement and keeps restore times within recovery objectives. With these operational guardrails in place, the search layer becomes a quiet piece of plumbing rather than a recurring incident, freeing engineers to focus on the next wave of content features.

Ready to dive deeper into the architecture decisions that shape production AEM search? Watch the recorded sessions from CIRCUIT 2015 and 2016, download the slide decks, and bring your own integration questions to the next gathering of AEM engineers across the Asia-Pacific community.