Exporting large AEM asset collections with the Asset Download API

Using AEM's Asset Download API to export large sets of assets is a practical exercise in retrieval, throttling and verification. The API exposes AEM Assets over HTTP, giving Java and front-end teams a consistent way to obtain originals and renditions. A dependable export needs a defined scope, predictable storage, retry rules and evidence that every file arrived intact.

AEM asset collections can contain much more than the visible marketing library: campaign variations, product cut-outs, videos, metadata-only assets and renditions for specific screens. A migration might require approved images, while an audit might need originals, thumbnails, metadata and every source path. Those goals deserve separate manifests rather than one vague download folder.

CIRCUIT's two-day Chicago conference brought together AEM architects, Java developers, front-end engineers and systems specialists. Its 2015 and 2016 recordings cover integrations, microservices, analytics, mobile delivery and open source, providing useful context for this work. They help connect an asset pipeline with broader decisions about staging, delivery and governance.

Start with a small, representative collection and measure what the instance actually delivers. Separate discovery from downloading, stream binaries rather than buffering them, and preserve source identifiers for reconciliation. Treat authentication, authorisation and audit records as part of the export contract, not details to add after the first failure.

Why bulk exports need deliberate design

Large exports compete with editorial work, preview rendering and delivery traffic on the same AEM environment. A download that seems harmless in testing can consume file descriptors, worker threads, CPU and outbound bandwidth. Author instances are poor bulk-transfer targets: their role is authoring and interactive preview, not serving files continuously for hours.

Use bounded concurrency and back-pressure. Give each worker a timeout, a retry budget and a destination, then stop feeding new jobs when storage or the origin slows. Keep temporary files on the destination volume so the final rename does not become a cross-filesystem copy. A steady transfer rate is easier to sustain than an aggressive burst.

Alongside AEM documentation, the WCSQ reference material can help teams examine related content workflows. Internal owners should still define naming, metadata completeness and accepted rendition formats. These rules prevent a technically successful download from becoming an unusable collection.

Selecting the right assets

Begin with a query that expresses business meaning, not every node under /content/dam. AEM QueryBuilder can filter dam:Asset records through /bin/querybuilder.json; parameters such as path, type=dam:Asset and property=dam:assetState separate usable assets from metadata or unprocessed uploads. Save the query and selected identifiers in a reproducible manifest.

Page results deliberately and avoid treating deep pagination as free. A request with p.limit=200 and an incrementing p.offset is easy to prototype, but offsets become costly as the result set grows. For a broad migration, use server-side traversal or an indexed query returning identifiers in bounded pages. Decide how assets created or modified during the run will be handled.

Before transferring bytes, decide whether the destination needs original binaries, selected renditions or both. Preserve the DAM hierarchy in the manifest rather than flattening filenames, since two folders can contain hero.jpg. Record asset paths, identifiers, rendition names, media types, byte counts and checksums. For excluded assets, record the reason.

Building a reliable download worker

An asset representation can be fetched through a path such as /api/assets/{asset-path}.asset, exposing metadata and rendition references. Follow the supported download route for the binary; do not mistake the JSON representation for image or video content. Use the appropriate service identity, and keep credentials out of logs and manifests.

Java clients should stream responses to disk instead of buffering entire videos in memory. Apache HttpClient remains practical on older AEM deployments, while newer Java runtimes can use java.net.http.HttpClient. Write each response to a temporary .part file, verify it, then rename it into place. Configure timeouts, certificate validation and restrict redirects to trusted hosts. The core copy loop stays deliberately small:

try (InputStream in = response.getEntity().getContent();
     OutputStream out = Files.newOutputStream(partial)) {
    byte[] buffer = new byte[64 * 1024];
    int read;
    while ((read = in.read(buffer)) != -1) {
        out.write(buffer, 0, read);
    }
}

Test the worker against pre-production content before a full run. A repeatable content staging workflow lets the team validate permissions, rendition availability and reconciliation rules without risking live campaign material. Exercise the same endpoints, manifest format and acceptance checks used in production.

Design choice	Direct API exporter	Queued export worker
Best fit	Thousands of assets and short-lived jobs	Very large collections or scheduled migrations
Execution	A controlled process with a small worker pool	Durable jobs consumed with limited concurrency
Recovery	Restarts from a manifest checkpoint	Leases work, retries transient errors and resumes after restarts
Governance	Simple logs and a single run identifier	Per-asset status, operator actions and audit history

Handling concurrency and failure recovery

Keep simultaneous transfers low enough for normal AEM use. A fixed worker pool, per-host limits and a shared progress rate provide predictable load. If the origin already serves editors and preview traffic, reduce export concurrency instead of scaling AEM around a temporary spike. Read replicas or an approved delivery service may be better sources, provided they expose the required fidelity and access controls.

Treat HTTP status codes as signals, not automatic instructions. Retry timeouts, connection resets, 429 Too Many Requests and selected 5xx responses with exponential backoff and jitter, and honour Retry-After. A 404 usually requires path reconciliation, while an authentication failure should pause that asset. Record the final reason for every exception.

For recurring exports, make work resumable and observable. Track queued, downloading, verified and failed states, and let another worker reclaim abandoned jobs through a lease. Repeating a GET is safe; repeating a partially written file is not. Personalised delivery adds another dependency: the pattern in Salesforce and AEM integration shows why campaign, region and consent context can determine which assets belong in an export.

Verifying assets, metadata and versions

Completion is not correctness. Compare expected assets with downloaded originals and renditions, then verify byte counts and checksums where available. Inspect samples manually, including transparent PNGs, colour-sensitive photography, video poster frames and filenames containing spaces. A zero-byte response that passed a nominal HTTP check must still fail acceptance.

For version comparisons, AEM content diffing is a practical companion to the export manifest. Use structural comparison to spot changed metadata, references or component content, and file hashes to identify binary changes. Keep the source version identifier beside each asset so auditors can identify the retrieved campaign revision.

Access and rights checks belong in final review. Remove temporary files, restrict download URLs and keep service credentials out of manifests. Exclude material subject to legal hold or an approved retention rule, and document who authorised each exception. Explicit omissions are easier to audit than a successful-looking download with unexplained gaps.

Planning for Australian delivery conditions

Australian teams often hold large DAMs in Sydney or Melbourne, while editors and agency partners work across the country. Distance to the AEM origin can make long transfers more sensitive to latency than bandwidth alone, particularly through corporate VPNs. Schedule bulk runs during quieter AEST or AEDT periods, monitor the origin as well as the client, and agree expectations before campaign deadlines.

Data residency is a practical requirement for many Australian clients. Prefer approved Australian cloud regions when policy calls for local processing, and record where temporary files and manifests live. A temporary disk overseas can still contain personal information even if the final archive sits in Sydney. Include regional deletion and backup settings in the runbook.

The Privacy Act 1988 and Australian Privacy Principles matter when images or metadata contain customer information. Screenshots of banking applications, delivery addresses and identity documents need redaction or approved exclusion. Australian retail, banking and public-sector campaigns also need consistent handling of GST-inclusive Australian-dollar prices and state-specific promotional variants, with business approval recorded separately from technical success.

Customs rules affect physical campaign materials rather than HTTP downloads, but they can still shape an asset collection. Australian Customs and Border Protection processes for imported props or merchandise may impose conditions on photographs and records. Label restricted clearance material and prevent it from being indexed as a general campaign asset, keeping a local production requirement from becoming an unintended global disclosure.

Define the export contract: asset population, accepted formats, destination, access model, audit evidence and maximum runtime. Test it against a representative collection, then turn observed behaviour into limits for workers, retries and storage. This discipline supports both a one-off migration and a recurring campaign feed.

Build the first export from a thousand processed assets, record throughput and failures, and scale only after the audit passes. Use the CIRCUIT session archive and the linked AEM practices to support the design, then run the validated worker under monitoring with an accountable owner.