Content Collections and Data Layer

Relevant source files

• packages/astro/e2e/fixtures/view-transitions/src/pages/form-five.astro
• packages/astro/src/assets/utils/resolveImports.ts
• packages/astro/src/cli/add/index.ts
• packages/astro/src/cli/check/index.ts
• packages/astro/src/cli/index.ts
• packages/astro/src/cli/install-package.ts
• packages/astro/src/cli/sync/index.ts
• packages/astro/src/config/index.ts
• packages/astro/src/content/config.ts
• packages/astro/src/content/consts.ts
• packages/astro/src/content/content-layer.ts
• packages/astro/src/content/data-store-source.ts
• packages/astro/src/content/data-store-writer.ts
• packages/astro/src/content/data-store.ts
• packages/astro/src/content/index.ts
• packages/astro/src/content/loaders/errors.ts
• packages/astro/src/content/loaders/file.ts
• packages/astro/src/content/loaders/glob.ts
• packages/astro/src/content/loaders/types.ts
• packages/astro/src/content/mutable-data-store.ts
• packages/astro/src/content/paths.ts
• packages/astro/src/content/runtime.ts
• packages/astro/src/content/server-listeners.ts
• packages/astro/src/content/types-generator.ts
• packages/astro/src/content/utils.ts
• packages/astro/src/content/vite-plugin-content-imports.ts
• packages/astro/src/content/vite-plugin-content-virtual-mod.ts
• packages/astro/src/core/build/index.ts
• packages/astro/src/core/build/static-build.ts
• packages/astro/src/core/config/config.ts
• packages/astro/src/core/config/settings.ts
• packages/astro/src/core/config/vite-load.ts
• packages/astro/src/core/create-vite.ts
• packages/astro/src/core/createMinimalViteDevServer.ts
• packages/astro/src/core/dev/container.ts
• packages/astro/src/core/dev/dev.ts
• packages/astro/src/core/dev/index.ts
• packages/astro/src/core/dev/restart.ts
• packages/astro/src/core/messages/runtime.ts
• packages/astro/src/core/preview/index.ts
• packages/astro/src/core/sync/index.ts
• packages/astro/src/types/public/content.ts
• packages/astro/src/virtual-modules/live-config.ts
• packages/astro/src/vite-plugin-load-fallback/index.ts
• packages/astro/src/vite-plugin-markdown/content-entry-type.ts
• packages/astro/src/vite-plugin-markdown/images.ts
• packages/astro/templates/content/module.mjs
• packages/astro/templates/content/types.d.ts
• packages/astro/test/fixtures/content-layer/images/lunar-module.jpg
• packages/astro/test/fixtures/content-layer/public/buzz.jpg
• packages/astro/test/fixtures/content-layer/src/content.config.ts
• packages/astro/test/fixtures/content-layer/src/content/space/atlantis.JPG
• packages/astro/test/fixtures/content-layer/src/content/space/atlantis.md
• packages/astro/test/fixtures/content-layer/src/data/dogs.json
• packages/astro/test/fixtures/content-layer/src/pages/collections.json.js
• packages/astro/test/fixtures/core-image/src/assets/c++.png
• packages/astro/test/fixtures/core-image/src/assets/penguin with apostrophe'.jpg
• packages/astro/test/fixtures/core-image/src/assets/penguin with percent%.jpg
• packages/astro/test/fixtures/core-image/src/assets/penguin with space.jpg
• packages/astro/test/fixtures/core-image/src/pages/specialChars.md
• packages/astro/test/fixtures/live-loaders/src/live.config.ts
• packages/astro/test/types/define-config.ts
• packages/astro/test/types/define-live-collection.ts
• packages/astro/test/units/cli/install-package.test.ts
• packages/astro/test/units/config/config-vite-load.test.ts
• packages/integrations/markdoc/components/Renderer.astro
• packages/integrations/markdoc/components/TreeNode.ts
• packages/integrations/markdoc/test/syntax-highlighting.test.ts
• packages/integrations/markdoc/tsconfig.test.json
• packages/integrations/mdx/template/content-module-types.d.ts
• packages/markdown/remark/src/rehype-images.ts
• packages/markdown/remark/src/remark-collect-images.ts

Purpose and Scope

This document describes Astro's Content Collections API and the underlying Data Layer. The system provides a unified, type-safe architecture for loading, processing, and querying content from diverse sources. It supports arbitrary data sources including local files (Markdown, MDX, JSON, YAML, TOML), remote APIs, databases, and CMS platforms through a loader-based architecture.

The Data Layer orchestrates a sync process that loads content, maintains an in-memory and on-disk data store, and generates TypeScript types to enable end-to-end type safety in content usage.

Sources: packages/astro/src/content/index.ts1-14 packages/astro/src/content/utils.ts56-121 packages/astro/src/content/consts.ts1-26

Architecture and Data Flow

The Data Layer is organized around a "push" model where Content Loaders retrieve data and push entries into a centralized DataStore. The main coordinating class is ContentLayer, which manages the loading lifecycle, caching, and syncing to disk.

System Components and Code Entities

This diagram links domain concepts to specific implementations:

• content.config.ts defines collection configurations using defineCollection.
• ContentLayer manages content syncing using loaders packages/astro/src/content/content-layer.ts46-75
• MutableDataStore holds data in-memory and persists to disk packages/astro/src/content/mutable-data-store.ts18-40
• The astroContentImportPlugin transforms content files into JS modules packages/astro/src/content/index.ts12
• The astroContentVirtualModPlugin provides the virtual module interface (astro:content) packages/astro/src/content/vite-plugin-content-virtual-mod.ts24-26
• createContentTypesGenerator() produces type definitions used at dev/build time packages/astro/src/content/types-generator.ts79-100

Sources: packages/astro/src/content/content-layer.ts46-75 packages/astro/src/content/mutable-data-store.ts18-40 packages/astro/src/content/types-generator.ts79-100 packages/astro/src/content/vite-plugin-content-virtual-mod.ts24-191

Data Sync Pipeline

The ContentLayer class orchestrates syncing content during astro sync, astro dev, or astro build.

• Sync jobs are queued with PQueue to avoid concurrent data mutation packages/astro/src/content/content-layer.ts73
• Loaders load collection data asynchronously and push entries to the store.
• The syncInternal function handles clearing the cache if the --force flag is used packages/astro/src/core/sync/index.ts134-136
• Once all loaders complete, the store writes out changes to disk atomically packages/astro/src/content/mutable-data-store.ts209-214

Sources: packages/astro/src/content/content-layer.ts185-260 packages/astro/src/content/mutable-data-store.ts41-46 packages/astro/src/core/sync/index.ts123-184

Content Loaders

Loaders abstract the logic for loading data from different sources into the store. They conform to a common Loader interface accepting a LoaderContext.

Loader Interface and Context

Loaders receive a context providing:

• store: a scoped API to set entries for a collection (store.set(key, entry)) packages/astro/src/content/content-layer.ts134
• parseData: function to validate raw data against Zod schemas packages/astro/src/content/content-layer.ts138
• generateDigest: function to hash source content for change detection, using xxhash-wasm packages/astro/src/content/content-layer.ts140
• watcher: Vite FSWatcher instance for file watching and incremental reloads packages/astro/src/content/content-layer.ts141
• Additional meta: logger, config, markdown rendering functions packages/astro/src/content/content-layer.ts136-139

Notable Built-in Loaders

• Glob Loader: Matches files by glob patterns and loads content entries. Supports Markdown, JSON, and other formats. It handles caching of render functions per content type packages/astro/src/content/loaders/glob.ts77-129
• File Loader: Loads a single JSON, YAML, or TOML file. Supports array and keyed object formats packages/astro/src/content/loaders/file.ts25-129
• Live Loader: Supports dynamic content collections backed by functions returning data dynamically (e.g., remote API). It bypasses the normal sync process in favor of runtime fetching packages/astro/src/content/runtime.ts55-92

Sources: packages/astro/src/content/loaders/types.ts1-20 packages/astro/src/content/loaders/glob.ts77-129 packages/astro/src/content/loaders/file.ts25-129 packages/astro/src/content/runtime.ts55-92

Data Store Architecture

The Data Store is the single source of truth for content during dev and build.

Storage and Serialization

• In-Memory Store: Uses a Map<string, Map<string, DataEntry>> to organize data by collection and entry ID packages/astro/src/content/data-store.ts30
• Mutable vs Immutable: MutableDataStore extends ImmutableDataStore with mutation APIs (set, delete, clear) and atomic writes packages/astro/src/content/mutable-data-store.ts18-40
• Persisted Store: The store writes to .astro/data-store.json (or a chunked directory if experimental) with serialized content ensuring fast reload packages/astro/src/content/mutable-data-store.ts41-55 packages/astro/src/core/sync/index.ts99-114
• Threshold for Large Data: In dev, large data stores (>5MB) are emitted as JSON strings to avoid oxc-parser AST limits during Vite SSR transforms packages/astro/src/content/vite-plugin-content-virtual-mod.ts31-44

Virtual Module Integration

• The virtual module astro:content is resolved via the astroContentVirtualModPlugin. This module re-exports content entries for runtime consumption packages/astro/src/content/vite-plugin-content-virtual-mod.ts87-191

Invalidation and HMR

• invalidateDataStore signals the SSR runner to clear its route cache so that getStaticPaths() is re-evaluated with updated content packages/astro/src/content/vite-plugin-content-virtual-mod.ts92-94
• It also triggers a full-reload for the client environment to reflect changes packages/astro/src/content/vite-plugin-content-virtual-mod.ts99-104

Sources: packages/astro/src/content/mutable-data-store.ts18-67 packages/astro/src/content/vite-plugin-content-virtual-mod.ts31-104 packages/astro/src/core/sync/index.ts99-114

Type Generation

TypeScript types are automatically generated to provide intellisense for collections.

createContentTypesGenerator() Workflow

1. Initialization: Scans collections and content config packages/astro/src/content/types-generator.ts101-104
2. Event Handling: Listens for file system events. If the content.config.ts changes, it reloads the config observer packages/astro/src/content/types-generator.ts139-148
3. Collection Mapping: Builds a CollectionEntryMap containing all recognized collections and their entry IDs/slugs packages/astro/src/content/types-generator.ts165-220
4. Writing Types: Injects data into content-types.d.ts inside the .astro directory packages/astro/src/content/types-generator.ts99-100

Sources: packages/astro/src/content/types-generator.ts79-220 packages/astro/src/core/sync/index.ts202-203

Runtime API

Querying content at runtime is handled by functions exported from astro:content.

• getCollection(collection, filter?): Retrieves all entries. If a live collection is requested via this method, it throws an error directing the user to getLiveCollection() packages/astro/src/content/runtime.ts99-108
• getEntry(collection, idOrSlug): Retrieves a single entry. It handles image reference updates via updateImageReferencesInData packages/astro/src/content/runtime.ts161-213
• updateImageReferencesInData: Recursively walks the data object to replace serialized image strings with optimized asset metadata packages/astro/src/content/runtime.ts273-306

Sources: packages/astro/src/content/runtime.ts94-306

Implementation Summary

Code Entity	Responsibility	File Path
ContentLayer	Orchestrates loaders and sync process	packages/astro/src/content/content-layer.ts
MutableDataStore	In-memory and on-disk data management	packages/astro/src/content/mutable-data-store.ts
syncInternal	Core logic for the astro sync command	packages/astro/src/core/sync/index.ts
createContentTypesGenerator	Generates .astro/types.d.ts	packages/astro/src/content/types-generator.ts
astroContentVirtualModPlugin	Provides astro:content virtual module	packages/astro/src/content/vite-plugin-content-virtual-mod.ts

Sources: packages/astro/src/content/content-layer.ts46-75 packages/astro/src/content/mutable-data-store.ts18-40 packages/astro/src/core/sync/index.ts123-184 packages/astro/src/content/types-generator.ts79-100 packages/astro/src/content/vite-plugin-content-virtual-mod.ts107-116