Unflag content layer (#11911)

* Unflag content layer

* Lint

* More detailed changeset

* Update .changeset/heavy-seahorses-poke.md

Co-authored-by: Sarah Rainsberger <sarah@rainsberger.ca>

---------

Co-authored-by: Sarah Rainsberger <sarah@rainsberger.ca>

.changeset/heavy-seahorses-poke.md

@@ -0,0 +1,102 @@
+---
+'astro': minor
+---
+
+The Content Layer API introduced behind a flag in [4.14.0](https://github.com/withastro/astro/blob/main/packages/astro/CHANGELOG.md#4140) is now stable and ready for use in Astro v5.0.
+
+The new Content Layer API builds upon content collections, taking them beyond local files in `src/content/` and allowing you to fetch content from anywhere, including remote APIs. These new collections work alongside your existing content collections, and you can migrate them to the new API at your own pace. There are significant improvements to performance with large collections of local files. For more details, see [the Content Layer RFC](https://github.com/withastro/roadmap/blob/content-layer/proposals/0050-content-layer.md).
+
+If you previously used this feature, you can now remove the `experimental.contentLayer` flag from your Astro config:
+
+```diff
+// astro.config.mjs
+import { defineConfig } from 'astro'
+
+export default defineConfig({
+-  experimental: {
+-    contentLayer: true
+-  }
+})
+```
+
+### Loading your content
+
+The core of the new Content Layer API is the loader, a function that fetches content from a source and caches it in a local data store. Astro 4.14 ships with built-in `glob()` and `file()` loaders to handle your local Markdown, MDX, Markdoc, and JSON files:
+
+```ts {3,7}
+// src/content/config.ts
+import { defineCollection, z } from 'astro:content';
+import { glob } from 'astro/loaders';
+		 
+const blog = defineCollection({
+  // The ID is a slug generated from the path of the file relative to `base`
+  loader: glob({ pattern: "**/*.md", base: "./src/data/blog" }),
+  schema: z.object({
+    title: z.string(),
+    description: z.string(),
+    publishDate: z.coerce.date(),
+  })
+});
+
+export const collections = { blog };
+```
+
+You can then query using the existing content collections functions, and use a simplified `render()` function to display your content:
+
+```astro
+---
+import { getEntry, render } from 'astro:content';
+
+const post = await getEntry('blog', Astro.params.slug);
+
+const { Content } = await render(entry);
+---
+
+<Content />
+```
+
+### Creating a loader
+
+You're not restricted to the built-in loaders – we hope you'll try building your own. You can fetch content from anywhere and return an array of entries:
+
+```ts
+// src/content/config.ts
+const countries = defineCollection({
+  loader: async () => {
+    const response = await fetch("https://restcountries.com/v3.1/all");
+    const data = await response.json();
+    // Must return an array of entries with an id property,
+    // or an object with IDs as keys and entries as values
+    return data.map((country) => ({
+      id: country.cca3,
+      ...country,
+    }));
+  },
+  // optionally add a schema to validate the data and make it type-safe for users
+  // schema: z.object...
+});
+
+export const collections = { countries };
+```
+
+For more advanced loading logic, you can define an object loader. This allows incremental updates and conditional loading, and gives full access to the data store. It also allows a loader to define its own schema, including generating it dynamically based on the source API. See the [the Content Layer API RFC](https://github.com/withastro/roadmap/blob/content-layer/proposals/0050-content-layer.md#loaders) for more details.
+
+### Sharing your loaders
+
+Loaders are better when they're shared. You can create a package that exports a loader and publish it to npm, and then anyone can use it on their site. We're excited to see what the community comes up with! To get started, [take a look at some examples](https://github.com/ascorbic/astro-loaders/). Here's how to load content using an RSS/Atom feed loader:
+
+```ts
+// src/content/config.ts
+import { defineCollection } from "astro:content";
+import { feedLoader } from "@ascorbic/feed-loader";
+
+const podcasts = defineCollection({
+  loader: feedLoader({
+    url: "https://feeds.99percentinvisible.org/99percentinvisible",
+  }),
+});
+
+export const collections = { podcasts };
+```
+
+To learn more, see [the Content Layer RFC](https://github.com/withastro/roadmap/blob/content-layer/proposals/0050-content-layer.md).

benchmark/make-project/markdown-cc2.js

@@ -70,11 +70,7 @@ const { Content } = await render(entry);
 		`\
 import { defineConfig } from 'astro/config';
 
-export default defineConfig({
+export default defineConfig({});`,
-		experimental: {
-			contentLayer: true
-		}
-});`,
 		'utf-8'
 	);
 }

benchmark/make-project/mdx-cc2.js

@@ -74,9 +74,6 @@ import mdx from '@astrojs/mdx';
 
 export default defineConfig({
   integrations: [mdx()],
-		experimental: {
-			contentLayer: true
-		}
 });`,
 		'utf-8'
 	);

packages/astro/src/content/content-layer.ts

@@ -2,7 +2,6 @@ import { promises as fs, existsSync } from 'node:fs';
 import * as fastq from 'fastq';
 import type { FSWatcher } from 'vite';
 import xxhash from 'xxhash-wasm';
-import { AstroUserError } from '../core/errors/errors.js';
 import type { Logger } from '../core/logger/core.js';
 import type { AstroSettings } from '../types/astro.js';
 import type { ContentEntryType, RefreshContentOptions } from '../types/public/content.js';
@@ -140,18 +139,6 @@ export class ContentLayer {
 			logger.debug('Content config not loaded, skipping sync');
 			return;
 		}
-		if (!this.#settings.config.experimental.contentLayer) {
-			const contentLayerCollections = Object.entries(contentConfig.config.collections).filter(
-				([_, collection]) => collection.type === CONTENT_LAYER_TYPE,
-			);
-			if (contentLayerCollections.length > 0) {
-				throw new AstroUserError(
-					`The following collections have a loader defined, but the content layer is not enabled: ${contentLayerCollections.map(([title]) => title).join(', ')}.`,
-					'To enable the Content Layer API, set `experimental: { contentLayer: true }` in your Astro config file.',
-				);
-			}
-			return;
-		}
 
 		logger.info('Syncing content');
 		const { digest: currentConfigDigest } = contentConfig.config;

packages/astro/src/core/config/schema.ts

@@ -93,7 +93,6 @@ export const ASTRO_CONFIG_DEFAULTS = {
 		clientPrerender: false,
 		serverIslands: false,
 		contentIntellisense: false,
-		contentLayer: false,
 	},
 } satisfies AstroUserConfig & { server: { open: boolean } };
 
@@ -530,7 +529,6 @@ export const AstroConfigSchema = z.object({
 				.boolean()
 				.optional()
 				.default(ASTRO_CONFIG_DEFAULTS.experimental.contentIntellisense),
-			contentLayer: z.boolean().optional().default(ASTRO_CONFIG_DEFAULTS.experimental.contentLayer),
 		})
 		.strict(
 			`Invalid or outdated experimental feature.\nCheck for incorrect spelling or outdated Astro version.\nSee https://docs.astro.build/en/reference/configuration-reference/#experimental-flags for a list of all current experiments.`,

packages/astro/src/core/dev/restart.ts

@@ -181,15 +181,13 @@ export async function createContainerWithAutomaticRestart({
 			{ key: 'c', description: '' },
 		];
 
-		if (restart.container.settings.config.experimental.contentLayer) {
+		customShortcuts.push({
-			customShortcuts.push({
+			key: 's',
-				key: 's',
+			description: 'sync content layer',
-				description: 'sync content layer',
+			action: () => {
-				action: () => {
+				globalContentLayer.get()?.sync();
-					globalContentLayer.get()?.sync();
+			},
-				},
+		});
-			});
-		}
 		restart.container.viteServer.bindCLIShortcuts({
 			customShortcuts,
 		});

packages/astro/src/integrations/hooks.ts

@@ -375,22 +375,20 @@ export async function runHookServerSetup({
 	logger: Logger;
 }) {
 	let refreshContent: undefined | ((options: RefreshContentOptions) => Promise<void>);
-	if (config.experimental?.contentLayer) {
+	refreshContent = async (options: RefreshContentOptions) => {
-		refreshContent = async (options: RefreshContentOptions) => {
+		const contentConfig = globalContentConfigObserver.get();
-			const contentConfig = globalContentConfigObserver.get();
+		if (
-			if (
+			contentConfig.status !== 'loaded' ||
-				contentConfig.status !== 'loaded' ||
+			!Object.values(contentConfig.config.collections).some(
-				!Object.values(contentConfig.config.collections).some(
+				(collection) => collection.type === CONTENT_LAYER_TYPE,
-					(collection) => collection.type === CONTENT_LAYER_TYPE,
+			)
-				)
+		) {
-			) {
+			return;
-				return;
+		}
-			}
 
-			const contentLayer = await globalContentLayer.get();
+		const contentLayer = await globalContentLayer.get();
-			await contentLayer?.sync(options);
+		await contentLayer?.sync(options);
-		};
+	};
-	}
 
 	for (const integration of config.integrations) {
 		if (integration?.hooks?.['astro:server:setup']) {

packages/astro/src/types/public/config.ts

@@ -1648,205 +1648,6 @@ export interface AstroUserConfig {
 		 * To use this feature with the Astro VS Code extension, you must also enable the `astro.content-intellisense` option in your VS Code settings. For editors using the Astro language server directly, pass the `contentIntellisense: true` initialization parameter to enable this feature.
 		 */
 		contentIntellisense?: boolean;
-
-		/**
-		 * @docs
-		 * @name experimental.contentLayer
-		 * @type {boolean}
-		 * @default `false`
-		 * @version 4.14.0
-		 * @description
-		 *
-		 * The Content Layer API is a new way to handle content and data in Astro. It is similar to and builds upon [content collections](/en/guides/content-collections/), taking them beyond local files in `src/content/` and allowing you to fetch content from anywhere, including remote APIs, by adding a `loader` to your collection.
-		 *
-		 * Your existing content collections can be [migrated to the Content Layer API](#migrating-an-existing-content-collection-to-use-the-content-layer-api) with a few small changes. However, it is not necessary to update all your collections at once to add a new collection powered by the Content Layer API. You may have collections using both the existing and new APIs defined in `src/content/config.ts` at the same time.
-		 *
-		 * The Content Layer API is designed to be more powerful and more performant, helping sites scale to thousands of pages. Data is cached between builds and updated incrementally. Markdown parsing is also 5-10 times faster, with similar scale reductions in memory, and MDX is 2-3 times faster.
-		 *
-		 * To enable, add the `contentLayer` flag to the `experimental` object in your Astro config:
-		 *
-		 * ```js
-		 * // astro.config.mjs
-		 * {
-		 * 	experimental: {
-		 * 		contentLayer: true,
-		 * 	}
-		 * }
-		 * ```
-		 *
-		 * #### Fetching data with a `loader`
-		 *
-		 * The Content Layer API allows you to fetch your content from outside of the `src/content/` folder (whether stored locally in your project or remotely) and uses a `loader` property to retrieve your data.
-		 *
-		 * The `loader` is defined in the collection's schema and returns an array of entries. Astro provides two built-in loader functions (`glob()` and `file()`) for fetching your local content, as well as access to the API to [construct your own loader and fetch remote data](#creating-a-loader).
-		 *
-		 * The `glob()` loader creates entries from directories of Markdown, MDX, Markdoc, or JSON files from anywhere on the filesystem. It accepts a `pattern` of entry files to match, and a `base` file path of where your files are located. Use this when you have one file per entry.
-		 *
-		 * The `file()` loader creates multiple entries from a single local file. Use this when all your entries are stored in an array of objects.
-		 *
-		 * ```ts  {3,8,19}
-		 * // src/content/config.ts
-		 * import { defineCollection, z } from 'astro:content';
-		 * import { glob, file } from 'astro/loaders';
-		 *
-		 * const blog = defineCollection({
-		 *   // By default the ID is a slug generated from
-		 *   // the path of the file relative to `base`
-		 *   loader: glob({ pattern: "**\/*.md", base: "./src/data/blog" }),
-		 *   schema: z.object({
-		 *     title: z.string(),
-		 *     description: z.string(),
-		 *     pubDate: z.coerce.date(),
-		 *     updatedDate: z.coerce.date().optional(),
-		 *   })
-		 * });
-		 *
-		 * const dogs = defineCollection({
-		 *   // The path is relative to the project root, or an absolute path.
-		 *   loader: file("src/data/dogs.json"),
-		 *   schema: z.object({
-		 *     id: z.string(),
-		 *     breed: z.string(),
-		 *     temperament: z.array(z.string()),
-		 *   }),
-		 * });
-		 *
-		 * export const collections = { blog, dogs };
-		 * ```
-		 *
-		 * :::note
-		 * Loaders will not automatically [exclude files prefaced with an `_`](/en/guides/routing/#excluding-pages). Use a regular expression such as `pattern: '**\/[^_]*.md'` in your loader to ignore these files.
-		 * :::
-		 *
-		 * #### Querying and rendering with the Content Layer API
-		 *
-		 * The collection can be [queried in the same way as content collections](/en/guides/content-collections/#querying-collections):
-		 *
-		 * ```ts
-		 * // src/pages/index.astro
-		 * import { getCollection, getEntry } from 'astro:content';
-		 *
-		 * // Get all entries from a collection.
-		 * // Requires the name of the collection as an argument.
-		 * const allBlogPosts = await getCollection('blog');
-		 *
-		 * // Get a single entry from a collection.
-		 * // Requires the name of the collection and ID
-		 * const labradorData = await getEntry('dogs', 'labrador-retriever');
-		 * ```
-		 *
-		 * Entries generated from Markdown, MDX, or Markdoc can be rendered directly to a page using the `render()` function.
-		 *
-		 * :::note
-		 * The syntax for rendering collection entries is different from the current content collections syntax.
-		 * :::
-		 *
-		 * ```astro title="src/pages/[slug].astro"
-		 * ---
-		 * import { getEntry, render } from 'astro:content';
-		 *
-		 * const post = await getEntry('blog', Astro.params.slug);
-		 *
-		 * const { Content, headings } = await render(post);
-		 * ---
-		 *
-		 * <Content />
-		 * ```
-		 *
-		 * #### Creating a loader
-		 *
-		 * With the Content Layer API, you can build loaders to load or generate content from anywhere.
-		 *
-		 * For example, you can create a loader that fetches collection entries from a remote API.
-		 *
-		 * ```ts
-		 * // src/content/config.ts
-		 * const countries = defineCollection({
-		 *   loader: async () => {
-		 *     const response = await fetch("https://restcountries.com/v3.1/all");
-		 *     const data = await response.json();
-		 *     // Must return an array of entries with an id property,
-		 *     // or an object with IDs as keys and entries as values
-		 *     return data.map((country) => ({
-		 *       id: country.cca3,
-		 *       ...country,
-		 *     }));
-		 *   },
-		 *   // optionally add a schema
-		 *   // schema: z.object...
-		 * });
-		 *
-		 * export const collections = { countries };
-		 * ```
-		 *
-		 * For more advanced loading logic, you can define an object loader. This allows incremental updates and conditional loading while also giving full access to the data store. See the API in [the Content Layer API RFC](https://github.com/withastro/roadmap/blob/content-layer/proposals/0047-content-layer.md#loaders).
-		 *
-		 * #### Migrating an existing content collection to use the Content Layer API
-		 *
-		 * You can convert an existing content collection with Markdown, MDX, Markdoc, or JSON entries to use the Content Layer API.
-		 *
-		 * 1. **Move the collection folder out of `src/content/`** (e.g. to `src/data/`). All collections located in the `src/content/` folder will use the existing Content Collections API.
-		 *
-		 *     **Do not move the existing `src/content/config.ts` file**. This file will define all collections, using either API.
-		 *
-		 * 2. **Edit the collection definition**. Your updated collection requires a `loader`, and the option to select a collection `type` is no longer available.
-		 *
-		 *     ```ts ins={3,8} del={7}
-		 *     // src/content/config.ts
-		 *     import { defineCollection, z } from 'astro:content';
-		 *     import { glob } from 'astro/loaders';
-		 *
-		 *     const blog = defineCollection({
-		 *       // For content layer you no longer define a `type`
-		 *       type: 'content',
-		 *       loader: glob({ pattern: '**\/[^_]*.md', base: "./src/data/blog" }),
-		 *       schema: z.object({
-		 *         title: z.string(),
-		 *         description: z.string(),
-		 *         pubDate: z.coerce.date(),
-		 *         updatedDate: z.coerce.date().optional(),
-		 *       }),
-		 *     });
-		 *     ```
-		 *
-		 * 3. **Change references from `slug` to `id`**. Content layer collections do not have a `slug` field. Instead, all updated collections will have an `id`.
-		 *
-		 *     ```astro ins={7} del={6}
-		 *     // src/pages/index.astro
-		 *     ---
-		 *     export async function getStaticPaths() {
-		 *       const posts = await getCollection('blog');
-		 *       return posts.map((post) => ({
-		 *         params: { slug: post.slug },
-		 *         params: { slug: post.id },
-		 *         props: post,
-		 *       }));
-		 *     }
-		 *     ---
-		 *     ```
-		 *
-		 * 4. **Switch to the new `render()` function**. Entries no longer have a `render()` method, as they are now serializable plain objects. Instead, import the `render()` function from `astro:content`.
-		 *
-		 *     ```astro ins={4,9} del={3,8}
-		 *     // src/pages/index.astro
-		 *     ---
-		 *     import { getEntry } from 'astro:content';
-		 *     import { getEntry, render } from 'astro:content';
-		 *
-		 *     const post = await getEntry('blog', params.slug);
-		 *
-		 *     const { Content, headings } = await post.render();
-		 *     const { Content, headings } = await render(post);
-		 *     ---
-		 *
-		 *     <Content />
-		 *     ```
-		 *
-		 * #### Learn more
-		 *
-		 * For a complete overview and the full API reference, see [the Content Layer API RFC](https://github.com/withastro/roadmap/blob/content-layer/proposals/0047-content-layer.md) and [share your feedback](https://github.com/withastro/roadmap/pull/982).
-		 */
-		contentLayer?: boolean;
 	};
 }
 

packages/astro/test/fixtures/content-intellisense/astro.config.mjs

@@ -6,7 +6,6 @@ import { defineConfig } from 'astro/config';
 export default defineConfig({
 	integrations: [mdx(), markdoc()],
 	experimental: {
-		contentLayer: true,
 		contentIntellisense: true
 	}
 });

packages/astro/test/fixtures/content-layer-markdoc/astro.config.mjs

@@ -4,6 +4,5 @@ import { defineConfig } from 'astro/config';
 
 // https://astro.build/config
 export default defineConfig({
-	integrations: [markdoc(), preact()],
+	integrations: [markdoc(), preact()]
-	experimental: { contentLayer: true }
 });

packages/astro/test/fixtures/content-layer-rendering/astro.config.mjs

@@ -11,7 +11,4 @@ export default defineConfig({
 			}
 		},
 	},
-	experimental: {
-		contentLayer: true,
-	},
 });

packages/astro/test/fixtures/content-layer/astro.config.mjs

@@ -43,7 +43,6 @@ export default defineConfig({
 		},
 	},
 	experimental: {
-		contentLayer: true,
 		contentIntellisense: true,
 	},
 });