Infrastructure/astro
0
Fork 0
mirror of https://github.com/withastro/astro.git synced 2025-03-31 23:31:30 -05:00
Code Issues Activity

Content Layer (#11360)

Browse source 
* Empty commit

* Changeset

* feat: add Content Layer loader (#11334)

* wip

* wip

* wip

* Update demo

* Add meta

* wip

* Add file loader

* Add schema validation

* Remove log

* Changeset

* Format

* Lockfile

* Fix type

* Handle loading for data store JSON

* Use rollup util to import JSON

* Fix types

* Format

* Add tests

* Changes from review

* fix: sync content layer in dev (#11365)

* wip

* wip

* wip

* Update demo

* Add meta

* wip

* Add file loader

* Add schema validation

* Remove log

* Changeset

* Format

* Lockfile

* Fix type

* Handle loading for data store JSON

* Use rollup util to import JSON

* Fix types

* Format

* Add tests

* Changes from review

* Sync content layer in dev

* feat: add typegen for loaders (#11358)

* fix: watch for content layer changes (#11371)

* fix: watch for content layer changes

* Add test

* feat: adds simple loader (#11386)

* wip

* Add simple loader

* Fix type guard

* Tighten loader schema

* Add loader function to type

* Reinstall vitest

* feat: add glob loader (#11398)

* feat: add glob loader

* Enable watching and fix paths

* Store the full entry object, not just data

* Add generateId support

* Fix test

* Rename loaders to sync

* Refacctor imports

* Use getEntry

* Format

* Fix import

* Remove type from output

* Windows path

* Add test for absolute path

* Update lockfile

* Debugging windows

* Allow file URL for base dir

* Reset time limit

* feat: add markdown rendering to content layer (#11440)

* feat: add glob loader

* Enable watching and fix paths

* Store the full entry object, not just data

* Add generateId support

* Fix test

* Rename loaders to sync

* Refacctor imports

* Use getEntry

* Format

* Fix import

* Remove type from output

* Windows path

* Add test for absolute path

* Update lockfile

* Debugging windows

* Allow file URL for base dir

* Reset time limit

* wip: add markdown rendering to content layer

* use cached entries

* CLean up types

* Instrument more of the build

* Add digest helper

* Add comments

* Make image extraction work

* feat: image support for content layer (#11469)

* wip

* wip

* Add image to benchmark

* Stub assets if missing

* Resolve assets in data

* Ignore virtual module

* Format

* rm log

* Handle images when using cached data

* Fix CCC

* Add a comment

* Changes from review

* Format

* Use relative paths for asset files

* Pass all md props to getImage

* Ensure dotastro dir exists

* Fix tests

* Changes from review

* Don't use temp array in getcollection

* Add error handling

* Format

* Handle paths that are already relative

* Dedupe sync runs

* Fix syncing in dev

* Changes from review

* Windows paths ftw

* feat(content-layer): support references in content layer (#11494)

* Support references in content layer

* Fix utf8 rendering

* Warn for invalid entries

* Fix test

* lol windows paths

* Remove assertion

* chore: fix content layer types (#11527)

* Add experimental_content type

* Fix import

* Make data store methods generic

* fix loader types

* Lockfile

* Clean content layer with `--force` (#11541)

* Clearn content layer with `--force`

* Add tests

* Document --force flag

* Fixes to content layer render types (#11558)

* Lockfile

* feat: use devalue to serialize content layer data (#11562)

* feat: use devalue to serialize content layer data

* Fix import

* Use devalue stringify

* Unused import

* Propagate error messages correctly

* Support --force flag in sync and dev (#11581)

* Support --force flag in sync and dev

* Fix test

* Separate render function and merge content layer types (#11579)

* Separate render function and merge content layer types

* Changes from review

* fix: clear content layer cache if config has changed (#11591)

* fix: clear content layer cache if config has changed

* Add test

* Watch config

* Change from review

* fix: skip glob files in content dir (#11622)

* fix: skip glob files in content dir

* Changes from review

* Log pattern

* Refactor content layer into shared instance (#11625)

* Refactor content layer into shared instance

* Clean up when testing

* Handle cleanup

* fix: support filters in content layer getCollection (#11631)

* Throw when using deprecated getEntryByX functions with content layer (#11637)

* Updates to content layer types and jsdocs (#11643)

* Add hot key to reload content layer (#11626)

* Add hot key to reload content layer

* Fix filename

* Remove cli message

* Update example

* Change key to "s"

* feat: handle simple mdx rendering (#11633)

* feat: handle simple mdx rendering

* cleanup

* feedback

* fix regression

* remove log

* flip condition

* update tests

* log collections to understand the error

* let's try this alternative

* try parallel test to understand the issue

* chore: use a new fixture to fix tests

* rebase and docs

* fix regressions

* remove old code

* address feedback

* rename param

* log error

* rebase

* chore: try a different cache dir to solve the error test

* fix invalidation of the module when there's no store available

* address suggestion

* run formatter

* update lock file

* Lint

* Add experimental content layer flag (#11652)

* Add experimental content layer flag

* Syntax and format

* Aside

* Format

* Reset content config between runs

* Update fixture

* Update terminology

* Lint

* wut

* Normalize render function return value (#11663)

* Add markdoc support to content layer (#11664)

* Add markdoc support to content layer

* Switch test to cheerio

* Update benchmarks

* update lock file

* Update content layer flag docs (#11682)

* Update content layer flag docs

* Apply suggestions from code review

Co-authored-by: Sarah Rainsberger <sarah@rainsberger.ca>

* More markdoc

* Typo

* Apply suggestions from code review

Co-authored-by: Sarah Rainsberger <sarah@rainsberger.ca>

* Update

---------

Co-authored-by: Sarah Rainsberger <sarah@rainsberger.ca>

* Add changeset for content layer experimental release (#11644)

* Add changeset for content layer experimental release

* Update changeset

* Update .changeset/smooth-chicken-wash.md

Co-authored-by: Sarah Rainsberger <sarah@rainsberger.ca>

---------

Co-authored-by: Sarah Rainsberger <sarah@rainsberger.ca>

* feat: injectTypes (#11551)

* feat: make inline config 1st arg

* fix: run config done in sync

* feat: start working on injectTypes

* feat: write files

* feat: adapt core features

* feat: migrate db to injectTypes

* feat: special db handling

* feat: update settings instead of workarounds

* fix: create dotAstroDir

* feat: refactor sync tests

* fix: path

* fix: paths

* chore: add comments

* feat: overwrite content file if exists

* chore: remove unused db env related code

* feat: use dotAstroDir for settings

* chore: simplify astro env sync

* feat: use dotAstroDir for preferences

* feat: handle db in integration api

* chore: reorganize

* feat: format

* feat: add test

* Discard changes to examples/basics/astro.config.mjs

* Discard changes to examples/basics/package.json

* Discard changes to pnpm-lock.yaml

* chore: remove test files

* feat: update examples dts

* fix: dts

* chore: changesets

* fix: indentation

* Apply suggestions from code review

Co-authored-by: Chris Swithinbank <swithinbank@gmail.com>

* Apply suggestions from code review

Co-authored-by: Chris Swithinbank <swithinbank@gmail.com>

* chore: format

* Update packages/astro/src/integrations/hooks.ts

* Update .changeset/mean-horses-kiss.md

* feat: remove formatting

* feat: handle fs errors

* feat: remove astro:db special path handling

* feat: add fs error

* Update packages/astro/src/content/types-generator.ts

* Update .changeset/mean-horses-kiss.md

* Update errors-data.ts

* Update .changeset/mean-horses-kiss.md

Co-authored-by: Sarah Rainsberger <sarah@rainsberger.ca>

* Update .changeset/mean-horses-kiss.md

Co-authored-by: Sarah Rainsberger <sarah@rainsberger.ca>

---------

Co-authored-by: Chris Swithinbank <swithinbank@gmail.com>
Co-authored-by: Sarah Rainsberger <sarah@rainsberger.ca>
Co-authored-by: Erika <3019731+Princesseuh@users.noreply.github.com>

* Add file generation and flag for content intellisense (#11639)

* feat: add type to infer input type of collection

* refactor:

* feat: generate json schema for content too

* feat: generate a manifest of all the collections

* refactor: unnecessary type

* fix: only add content collections to manifest

* chore: changeset

* fix: generate file URLs

* fix: flag it properly

* fix: save in lower case

* docs: add jsdoc to experimental option

* nit: move function out

* fix: match vscode flag name

* Update packages/astro/src/@types/astro.ts

Co-authored-by: Sarah Rainsberger <sarah@rainsberger.ca>

* Update packages/astro/src/@types/astro.ts

Co-authored-by: Sarah Rainsberger <sarah@rainsberger.ca>

* Update serious-pumas-run.md

* test: add tests

* Add content layer support

* Apply suggestions from code review

* fix: test

* Update .changeset/serious-pumas-run.md

Co-authored-by: Sarah Rainsberger <sarah@rainsberger.ca>

* Apply suggestions from code review

* Remove check for json

---------

Co-authored-by: Matt Kane <m@mk.gg>
Co-authored-by: Sarah Rainsberger <sarah@rainsberger.ca>

* nit: use same filesystem error as injectTypes

* fix: code component was missing support for meta string (#11605)

* fix: code component was missing support for meta string

Fixed #11604

* Create odd-buttons-pay.md

* <Code>: add reference link for meta prop

* Apply suggestions from code review

Co-authored-by: Sarah Rainsberger <sarah@rainsberger.ca>

* Update .changeset/odd-buttons-pay.md

* Update .changeset/odd-buttons-pay.md

Co-authored-by: Sarah Rainsberger <sarah@rainsberger.ca>

---------

Co-authored-by: Erika <3019731+Princesseuh@users.noreply.github.com>
Co-authored-by: Sarah Rainsberger <sarah@rainsberger.ca>
Co-authored-by: Emanuele Stoppa <my.burning@gmail.com>

* Deprecates exporting prerender with dynamic values (#11657)

* wip

* done i think

* Add changeset

* Use hook instead

* Reorder hooks [skip ci]

* Update .changeset/eleven-pens-glow.md

Co-authored-by: Sarah Rainsberger <sarah@rainsberger.ca>

* Fix run

* Fix link

* Add link

Co-authored-by: Sarah Rainsberger <sarah11918@users.noreply.github.com>

* More accurate migration [skip ci]

---------

Co-authored-by: Sarah Rainsberger <sarah@rainsberger.ca>
Co-authored-by: Sarah Rainsberger <sarah11918@users.noreply.github.com>

* Use node parseArgs instead of yargs-parser and arg (#11645)

* wip

* done

* Add changeset

* Format

* Update

* Fix houston

* Fix test

* Fix test

* [ci] format

* resolve conflict

---------

Co-authored-by: Emanuele Stoppa <my.burning@gmail.com>
Co-authored-by: Sarah Rainsberger <sarah@rainsberger.ca>
Co-authored-by: Florian Lefebvre <contact@florian-lefebvre.dev>
Co-authored-by: Chris Swithinbank <swithinbank@gmail.com>
Co-authored-by: Erika <3019731+Princesseuh@users.noreply.github.com>
Co-authored-by: Julien Cayzac <jcayzac@users.noreply.github.com>
Co-authored-by: Bjorn Lu <bjornlu.dev@gmail.com>
Co-authored-by: Sarah Rainsberger <sarah11918@users.noreply.github.com>
Co-authored-by: Bjorn Lu <ematipico@users.noreply.github.com>
This commit is contained in:
Matt Kane 2024-08-14 11:49:19 +01:00 • committed by GitHub
parent 19a7259211
commit a79a8b0230
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
155 changed files with 5273 additions and 572 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

18
.changeset/fresh-fans-study.md Normal file
View file

@ -0,0 +1,18 @@
---
'@astrojs/db': minor
---
Changes how type generation works
The generated `.d.ts` file is now at a new location:
```diff
- .astro/db-types.d.ts
+ .astro/integrations/astro_db/db.d.ts
```
The following line can now be removed from `src/env.d.ts`:
```diff
- /// <reference path="../.astro/db-types.d.ts" />
```

35
.changeset/mean-horses-kiss.md Normal file
View file

@ -0,0 +1,35 @@
---
'astro': minor
---
Adds a new [`injectTypes()` utility](https://docs.astro.build/en/reference/integrations-reference/#injecttypes-options) to the Integration API and refactors how type generation works
Use `injectTypes()` in the `astro:config:done` hook to inject types into your user's project by adding a new a `*.d.ts` file.
The `filename` property will be used to generate a file at `/.astro/integrations/<normalized_integration_name>/<normalized_filename>.d.ts` and must end with `".d.ts"`.
The `content` property will create the body of the file, and must be valid TypeScript.
Additionally, `injectTypes()` returns a URL to the normalized path so you can overwrite its content later on, or manipulate it in any way you want.
```js
// my-integration/index.js
export default {
name: 'my-integration',
'astro:config:done': ({ injectTypes }) => {
injectTypes({
filename: "types.d.ts",
content: "declare module 'virtual:my-integration' {}"
})
}
};
```
Codegen has been refactored. Although `src/env.d.ts` will continue to work as is, we recommend you update it:
```diff
- /// <reference types="astro/client" />
+ /// <reference path="../.astro/types.d.ts" />
- /// <reference path="../.astro/env.d.ts" />
- /// <reference path="../.astro/actions.d.ts" />
```

21
.changeset/serious-pumas-run.md Normal file
View file

@ -0,0 +1,21 @@
---
'astro': minor
---
Adds support for Intellisense features (e.g. code completion, quick hints) for your content collection entries in compatible editors under the `experimental.contentIntellisense` flag.
```js
import { defineConfig } from 'astro';
export default defineConfig({
experimental: {
contentIntellisense: true
}
})
```
When enabled, this feature will generate and add JSON schemas to the `.astro` directory in your project. These files can be used by the Astro language server to provide Intellisense inside content files (`.md`, `.mdx`, `.mdoc`).
Note that at this time, this also require enabling the `astro.content-intellisense` option in your editor, or passing the `contentIntellisense: true` initialization parameter to the Astro language server for editors using it directly.
See the [experimental content Intellisense docs](https://docs.astro.build/en/reference/configuration-reference/#experimentalcontentintellisense) for more information updates as this feature develops.

107
.changeset/smooth-chicken-wash.md Normal file
View file

@ -0,0 +1,107 @@
---
'astro': minor
---
Adds experimental support for the Content Layer API.
The new Content Layer API builds upon content collections, taking them beyond local files in `src/content/` and allowing you to fetch content from anywhere, including remote APIs. These new collections work alongside your existing content collections, and you can migrate them to the new API at your own pace. There are significant improvements to performance with large collections of local files.
### Getting started
To try out the new Content Layer API, enable it in your Astro config:
```js
import { defineConfig } from 'astro';
export default defineConfig({
experimental: {
contentLayer: true
}
})
```
You can then create collections in your `src/content/config.ts` using the Content Layer API.
### Loading your content
The core of the new Content Layer API is the loader, a function that fetches content from a source and caches it in a local data store. Astro 4.14 ships with built-in `glob()` and `file()` loaders to handle your local Markdown, MDX, Markdoc, and JSON files:
```ts {3,7}
// src/content/config.ts
import { defineCollection, z } from 'astro:content';
import { glob } from 'astro/loaders';
const blog = defineCollection({
// The ID is a slug generated from the path of the file relative to `base`
loader: glob({ pattern: "**/*.md", base: "./src/data/blog" }),
schema: z.object({
title: z.string(),
description: z.string(),
publishDate: z.coerce.date(),
})
});
export const collections = { blog };
```
You can then query using the existing content collections functions, and enjoy a simplified `render()` function to display your content:
```astro
---
import { getEntry, render } from 'astro:content';
const post = await getEntry('blog', Astro.params.slug);
const { Content } = await render(entry);
---
<Content />
```
### Creating a loader
You're not restricted to the built-in loaders – we hope you'll try building your own. You can fetch content from anywhere and return an array of entries:
```ts
// src/content/config.ts
const countries = defineCollection({
loader: async () => {
const response = await fetch("https://restcountries.com/v3.1/all");
const data = await response.json();
// Must return an array of entries with an id property,
// or an object with IDs as keys and entries as values
return data.map((country) => ({
id: country.cca3,
...country,
}));
},
// optionally add a schema to validate the data and make it type-safe for users
// schema: z.object...
});
export const collections = { countries };
```
For more advanced loading logic, you can define an object loader. This allows incremental updates and conditional loading, and gives full access to the data store. It also allows a loader to define its own schema, including generating it dynamically based on the source API. See the [the Content Layer API RFC](https://github.com/withastro/roadmap/blob/content-layer/proposals/0047-content-layer.md#loaders) for more details.
### Sharing your loaders
Loaders are better when they're shared. You can create a package that exports a loader and publish it to npm, and then anyone can use it on their site. We're excited to see what the community comes up with! To get started, [take a look at some examples](https://github.com/ascorbic/astro-loaders/). Here's how to load content using an RSS/Atom feed loader:
```ts
// src/content/config.ts
import { defineCollection } from "astro:content";
import { feedLoader } from "@ascorbic/feed-loader";
const podcasts = defineCollection({
loader: feedLoader({
url: "https://feeds.99percentinvisible.org/99percentinvisible",
}),
});
export const collections = { podcasts };
```
### Learn more
To find out more about using the Content Layer API, check out [the Content Layer RFC](https://github.com/withastro/roadmap/blob/content-layer/proposals/0047-content-layer.md) and [share your feedback](https://github.com/withastro/roadmap/pull/982).

2
benchmark/bench/memory.js
View file

@ -18,7 +18,7 @@ export async function run(projectDir, outputFile) {
const outputFilePath = fileURLToPath(outputFile);
console.log('Building and benchmarking...');
await execaCommand(`node --expose-gc --max_old_space_size=256 ${astroBin} build`, {
await execaCommand(`node --expose-gc --max_old_space_size=10000 ${astroBin} build --silent`, {
cwd: root,
stdio: 'inherit',
env: {

BIN
benchmark/make-project/image.jpg Normal file
View file

Binary file not shown.
Side by side

After

Width:  |  Height:  |  Size: 170 KiB

63
benchmark/make-project/markdown-cc1.js Normal file
View file

@ -0,0 +1,63 @@
import fs from 'node:fs/promises';
import { loremIpsumMd } from './_util.js';
/**
* @param {URL} projectDir
*/
export async function run(projectDir) {
await fs.rm(projectDir, { recursive: true, force: true });
await fs.mkdir(new URL('./src/pages/blog', projectDir), { recursive: true });
await fs.mkdir(new URL('./src/content/blog', projectDir), { recursive: true });
await fs.copyFile(new URL('./image.jpg', import.meta.url), new URL('./src/image.jpg', projectDir));
const promises = [];
for (let i = 0; i < 10000; i++) {
const content = `\
# Article ${i}
${loremIpsumMd}
![image ${i}](../../image.jpg)
`;
promises.push(
fs.writeFile(new URL(`./src/content/blog/article-${i}.md`, projectDir), content, 'utf-8')
);
}
await fs.writeFile(
new URL(`./src/pages/blog/[...slug].astro`, projectDir),
`\
---
import { getCollection } from 'astro:content';
export async function getStaticPaths() {
const blogEntries = await getCollection('blog');
return blogEntries.map(entry => ({
params: { slug: entry.slug }, props: { entry },
}));
}
const { entry } = Astro.props;
const { Content } = await entry.render();
---
<h1>{entry.data.title}</h1>
<Content />
`,
'utf-8'
);
await Promise.all(promises);
await fs.writeFile(
new URL('./astro.config.js', projectDir),
`\
import { defineConfig } from 'astro/config';
export default defineConfig({
});`,
'utf-8'
);
}

80
benchmark/make-project/markdown-cc2.js Normal file
View file

@ -0,0 +1,80 @@
import fs from 'node:fs/promises';
import { loremIpsumMd } from './_util.js';
/**
* @param {URL} projectDir
*/
export async function run(projectDir) {
await fs.rm(projectDir, { recursive: true, force: true });
await fs.mkdir(new URL('./src/pages/blog', projectDir), { recursive: true });
await fs.mkdir(new URL('./data/blog', projectDir), { recursive: true });
await fs.mkdir(new URL('./src/content', projectDir), { recursive: true });
await fs.copyFile(new URL('./image.jpg', import.meta.url), new URL('./image.jpg', projectDir));
const promises = [];
for (let i = 0; i < 10000; i++) {
const content = `\
# Article ${i}
${loremIpsumMd}
![image ${i}](../../image.jpg)
`;
promises.push(
fs.writeFile(new URL(`./data/blog/article-${i}.md`, projectDir), content, 'utf-8')
);
}
await fs.writeFile(
new URL(`./src/content/config.ts`, projectDir),
/*ts */ `
import { defineCollection, z } from 'astro:content';
import { glob } from 'astro/loaders';
const blog = defineCollection({
loader: glob({ pattern: '*', base: './data/blog' }),
});
export const collections = { blog }
`
);
await fs.writeFile(
new URL(`./src/pages/blog/[...slug].astro`, projectDir),
`\
---
import { getCollection, render } from 'astro:content';
export async function getStaticPaths() {
const blogEntries = await getCollection('blog');
return blogEntries.map(entry => ({
params: { slug: entry.id }, props: { entry },
}));
}
const { entry } = Astro.props;
const { Content } = await render(entry);
---
<h1>{entry.data.title}</h1>
<Content />
`,
'utf-8'
);
await Promise.all(promises);
await fs.writeFile(
new URL('./astro.config.js', projectDir),
`\
import { defineConfig } from 'astro/config';
export default defineConfig({
experimental: {
contentLayer: true
}
});`,
'utf-8'
);
}

66
benchmark/make-project/mdx-cc1.js Normal file
View file

@ -0,0 +1,66 @@
import fs from 'node:fs/promises';
import { loremIpsumMd } from './_util.js';
/**
* @param {URL} projectDir
*/
export async function run(projectDir) {
await fs.rm(projectDir, { recursive: true, force: true });
await fs.mkdir(new URL('./src/pages/blog', projectDir), { recursive: true });
await fs.mkdir(new URL('./src/content/blog', projectDir), { recursive: true });
await fs.copyFile(new URL('./image.jpg', import.meta.url), new URL('./src/image.jpg', projectDir));
const promises = [];
for (let i = 0; i < 10000; i++) {
const content = `\
# Article ${i}
${loremIpsumMd}
![image ${i}](../../image.jpg)
`;
promises.push(
fs.writeFile(new URL(`./src/content/blog/article-${i}.mdx`, projectDir), content, 'utf-8')
);
}
await fs.writeFile(
new URL(`./src/pages/blog/[...slug].astro`, projectDir),
`\
---
import { getCollection } from 'astro:content';
export async function getStaticPaths() {
const blogEntries = await getCollection('blog');
return blogEntries.map(entry => ({
params: { slug: entry.slug }, props: { entry },
}));
}
const { entry } = Astro.props;
const { Content } = await entry.render();
---
<h1>{entry.data.title}</h1>
<Content />
`,
'utf-8'
);
await Promise.all(promises);
await fs.writeFile(
new URL('./astro.config.js', projectDir),
`\
import { defineConfig } from 'astro/config';
import mdx from '@astrojs/mdx';
export default defineConfig({
integrations: [mdx()],
});`,
'utf-8'
);
}

83
benchmark/make-project/mdx-cc2.js Normal file
View file

@ -0,0 +1,83 @@
import fs from 'node:fs/promises';
import { loremIpsumMd } from './_util.js';
/**
* @param {URL} projectDir
*/
export async function run(projectDir) {
await fs.rm(projectDir, { recursive: true, force: true });
await fs.mkdir(new URL('./src/pages/blog', projectDir), { recursive: true });
await fs.mkdir(new URL('./data/blog', projectDir), { recursive: true });
await fs.mkdir(new URL('./src/content', projectDir), { recursive: true });
await fs.copyFile(new URL('./image.jpg', import.meta.url), new URL('./image.jpg', projectDir));
const promises = [];
for (let i = 0; i < 10000; i++) {
const content = `\
# Article ${i}
${loremIpsumMd}
![image ${i}](../../image.jpg)
`;
promises.push(
fs.writeFile(new URL(`./data/blog/article-${i}.mdx`, projectDir), content, 'utf-8')
);
}
await fs.writeFile(
new URL(`./src/content/config.ts`, projectDir),
/*ts */ `
import { defineCollection, z } from 'astro:content';
import { glob } from 'astro/loaders';
const blog = defineCollection({
loader: glob({ pattern: '*', base: './data/blog' }),
});
export const collections = { blog }
`
);
await fs.writeFile(
new URL(`./src/pages/blog/[...slug].astro`, projectDir),
`\
---
import { getCollection, render } from 'astro:content';
export async function getStaticPaths() {
const blogEntries = await getCollection('blog');
return blogEntries.map(entry => ({
params: { slug: entry.id }, props: { entry },
}));
}
const { entry } = Astro.props;
const { Content } = await render(entry);
---
<h1>{entry.data.title}</h1>
<Content />
`,
'utf-8'
);
await Promise.all(promises);
await fs.writeFile(
new URL('./astro.config.js', projectDir),
`\
import { defineConfig } from 'astro/config';
import mdx from '@astrojs/mdx';
export default defineConfig({
integrations: [mdx()],
experimental: {
contentLayer: true
}
});`,
'utf-8'
);
}

3
benchmark/package.json
View file

@ -16,6 +16,7 @@
"markdown-table": "^3.0.3",
"mri": "^1.2.0",
"port-authority": "^2.0.1",
"pretty-bytes": "^6.1.1"
"pretty-bytes": "^6.1.1",
"sharp": "^0.33.3"
}
}

2
examples/basics/src/env.d.ts vendored
View file

@ -1 +1 @@
/// <reference types="astro/client" />
/// <reference path="../.astro/types.d.ts" />

1
examples/blog/src/env.d.ts vendored
View file

@ -1,2 +1 @@
/// <reference path="../.astro/types.d.ts" />
/// <reference types="astro/client" />

4
examples/container-with-vitest/package.json
View file

@ -19,7 +19,7 @@
"vitest": "^2.0.5"
},
"devDependencies": {
"@types/react-dom": "^18.3.0",
"@types/react": "^18.3.3"
"@types/react": "^18.3.3",
"@types/react-dom": "^18.3.0"
}
}

2
examples/framework-alpine/src/env.d.ts vendored
View file

@ -1 +1 @@
/// <reference types="astro/client" />
/// <reference path="../.astro/types.d.ts" />

2
examples/framework-lit/src/env.d.ts vendored
View file

@ -1 +1 @@
/// <reference types="astro/client" />
/// <reference path="../.astro/types.d.ts" />

2
examples/framework-multiple/src/env.d.ts vendored
View file

@ -1 +1 @@
/// <reference types="astro/client" />
/// <reference path="../.astro/types.d.ts" />

2
examples/framework-preact/src/env.d.ts vendored
View file

@ -1 +1 @@
/// <reference types="astro/client" />
/// <reference path="../.astro/types.d.ts" />

2
examples/framework-react/src/env.d.ts vendored
View file

@ -1 +1 @@
/// <reference types="astro/client" />
/// <reference path="../.astro/types.d.ts" />

2
examples/framework-solid/src/env.d.ts vendored
View file

@ -1 +1 @@
/// <reference types="astro/client" />
/// <reference path="../.astro/types.d.ts" />

2
examples/framework-svelte/src/env.d.ts vendored
View file

@ -1 +1 @@
/// <reference types="astro/client" />
/// <reference path="../.astro/types.d.ts" />

2
examples/framework-vue/src/env.d.ts vendored
View file

@ -1 +1 @@
/// <reference types="astro/client" />
/// <reference path="../.astro/types.d.ts" />

2
examples/hackernews/src/env.d.ts vendored
View file

@ -1 +1 @@
/// <reference types="astro/client" />
/// <reference path="../.astro/types.d.ts" />

2
examples/middleware/src/env.d.ts vendored
View file

@ -1,4 +1,4 @@
/// <reference types="astro/client" />
/// <reference path="../.astro/types.d.ts" />
declare namespace App {
interface Locals {
user: {

2
examples/minimal/src/env.d.ts vendored
View file

@ -1 +1 @@
/// <reference types="astro/client" />
/// <reference path="../.astro/types.d.ts" />

2
examples/non-html-pages/src/env.d.ts vendored
View file

@ -1 +1 @@
/// <reference types="astro/client" />
/// <reference path="../.astro/types.d.ts" />

1
examples/portfolio/src/env.d.ts vendored
View file

@ -1,2 +1 @@
/// <reference path="../.astro/types.d.ts" />
/// <reference types="astro/client" />

2
examples/ssr/src/env.d.ts vendored
View file

@ -1 +1 @@
/// <reference types="astro/client" />
/// <reference path="../.astro/types.d.ts" />

1
examples/with-markdoc/src/env.d.ts vendored
View file

@ -1,2 +1 @@
/// <reference path="../.astro/types.d.ts" />
/// <reference types="astro/client" />

2
examples/with-markdown-plugins/src/env.d.ts vendored
View file

@ -1 +1 @@
/// <reference types="astro/client" />
/// <reference path="../.astro/types.d.ts" />

2
examples/with-markdown-shiki/src/env.d.ts vendored
View file

@ -1 +1 @@
/// <reference types="astro/client" />
/// <reference path="../.astro/types.d.ts" />

2
examples/with-mdx/src/env.d.ts vendored
View file

@ -1 +1 @@
/// <reference types="astro/client" />
/// <reference path="../.astro/types.d.ts" />

2
examples/with-nanostores/src/env.d.ts vendored
View file

@ -1 +1 @@
/// <reference types="astro/client" />
/// <reference path="../.astro/types.d.ts" />

2
examples/with-tailwindcss/src/env.d.ts vendored
View file

@ -1 +1 @@
/// <reference types="astro/client" />
/// <reference path="../.astro/types.d.ts" />

9
packages/astro/package.json
View file

@ -68,6 +68,7 @@
"./assets/services/sharp": "./dist/assets/services/sharp.js",
"./assets/services/squoosh": "./dist/assets/services/squoosh.js",
"./assets/services/noop": "./dist/assets/services/noop.js",
"./loaders": "./dist/content/loaders/index.js",
"./content/runtime": "./dist/content/runtime.js",
"./content/runtime-assets": "./dist/content/runtime-assets.js",
"./debug": "./components/Debug.astro",
@ -132,6 +133,7 @@
"@babel/plugin-transform-react-jsx": "^7.25.2",
"@babel/traverse": "^7.25.3",
"@babel/types": "^7.25.2",
"@rollup/pluginutils": "^5.1.0",
"@oslojs/encoding": "^0.4.1",
"@types/babel__core": "^7.20.5",
"@types/cookie": "^0.6.0",
@ -163,7 +165,9 @@
"js-yaml": "^4.1.0",
"kleur": "^4.1.5",
"magic-string": "^0.30.11",
"micromatch": "^4.0.7",
"mrmime": "^2.0.0",
"neotraverse": "^0.6.9",
"ora": "^8.0.1",
"p-limit": "^6.1.0",
"p-queue": "^8.0.1",
@ -181,8 +185,10 @@
"vite": "^5.4.0",
"vitefu": "^0.2.5",
"which-pm": "^3.0.0",
"xxhash-wasm": "^1.0.2",
"zod": "^3.23.8",
"zod-to-json-schema": "^3.23.2"
"zod-to-json-schema": "^3.23.2",
"zod-to-ts": "^1.2.0"
},
"optionalDependencies": {
"sharp": "^0.33.3"
@ -203,6 +209,7 @@
"@types/html-escaper": "^3.0.2",
"@types/http-cache-semantics": "^4.0.4",
"@types/js-yaml": "^4.0.9",
"@types/micromatch": "^4.0.9",
"@types/prompts": "^2.4.9",
"@types/semver": "^7.5.8",
"astro-scripts": "workspace:*",

242
packages/astro/src/@types/astro.ts
View file

@ -1,5 +1,3 @@
import type { OutgoingHttpHeaders } from 'node:http';
import type { AddressInfo } from 'node:net';
import type {
MarkdownHeading,
MarkdownVFile,
@ -9,6 +7,8 @@ import type {
ShikiConfig,
} from '@astrojs/markdown-remark';
import type * as babel from '@babel/core';
import type { OutgoingHttpHeaders } from 'node:http';
import type { AddressInfo } from 'node:net';
import type * as rollup from 'rollup';
import type * as vite from 'vite';
import type {
@ -18,6 +18,7 @@ import type {
ActionReturnType,
} from '../actions/runtime/virtual/server.js';
import type { RemotePattern } from '../assets/utils/remotePattern.js';
import type { DataEntry, RenderedContent } from '../content/data-store.js';
import type { AssetsPrefix, SSRManifest, SerializedSSRManifest } from '../core/app/types.js';
import type { PageBuildData } from '../core/build/types.js';
import type { AstroConfigType } from '../core/config/index.js';
@ -78,7 +79,7 @@ export type {
UnresolvedImageTransform,
} from '../assets/types.js';
export type { RemotePattern } from '../assets/utils/remotePattern.js';
export type { SSRManifest, AssetsPrefix } from '../core/app/types.js';
export type { AssetsPrefix, SSRManifest } from '../core/app/types.js';
export type {
AstroCookieGetOptions,
AstroCookieSetOptions,
@ -123,6 +124,7 @@ export type TransitionAnimationValue =
| TransitionDirectionalAnimations;
// Allow users to extend this for astro-jsx.d.ts
// eslint-disable-next-line @typescript-eslint/no-empty-object-type
export interface AstroClientDirectives {}
@ -2183,6 +2185,225 @@ export interface AstroUserConfig {
* For a complete overview, and to give feedback on this experimental API, see the [Server Islands RFC](https://github.com/withastro/roadmap/pull/963).
*/
serverIslands?: boolean;
/**
* @docs
* @name experimental.contentIntellisense
* @type {boolean}
* @default `false`
* @version 4.14.0
* @description
*
* Enables Intellisense features (e.g. code completion, quick hints) for your content collection entries in compatible editors.
*
* When enabled, this feature will generate and add JSON schemas to the `.astro` directory in your project. These files can be used by the Astro language server to provide Intellisense inside content files (`.md`, `.mdx`, `.mdoc`).
*
* ```js
* {
* experimental: {
* contentIntellisense: true,
* },
* }
* ```
*
* To use this feature with the Astro VS Code extension, you must also enable the `astro.content-intellisense` option in your VS Code settings. For editors using the Astro language server directly, pass the `contentIntellisense: true` initialization parameter to enable this feature.
*/
contentIntellisense?: boolean;
/**
* @docs
* @name experimental.contentLayer
* @type {boolean}
* @default `false`
* @version 4.14.0
* @description
*
* The Content Layer API is a new way to handle content and data in Astro. It is similar to and builds upon [content collections](/en/guides/content-collections/), taking them beyond local files in `src/content/` and allowing you to fetch content from anywhere, including remote APIs, by adding a `loader` to your collection.
*
* Your existing content collections can be [migrated to the Content Layer API](#migrating-a-content-collection-to-content-layer) with a few small changes. However, it is not necessary to update all your collections at once to add a new collection powered by the Content Layer API. You may have collections using both the existing and new APIs defined in `src/content/config.ts` at the same time.
*
* The Content Layer API is designed to be more powerful and more performant, helping sites scale to thousands of pages. Data is cached between builds and updated incrementally. Markdown parsing is also 5-10 times faster, with similar scale reductions in memory, and MDX is 2-3 times faster.
*
* To enable, add the `contentLayer` flag to the `experimental` object in your Astro config:
*
* ```js
* // astro.config.mjs
* {
* experimental: {
* contentLayer: true,
* }
* }
* ```
*
* #### Fetching data with a `loader`
*
* The Content Layer API allows you to fetch your content from outside of the `src/content/` folder (whether stored locally in your project or remotely), and uses a `loader` property to retrieve your data.
*
* The `loader` is defined in the collection's schema and returns an array of entries. Astro provides two built-in loader functions (`glob()` and `file()`) for fetching your local content, as well as access to the API to [construct your own loader and fetch remote data](#creating-a-loader).
*
* The `glob()` loader creates entries from directories of Markdown, MDX, Markdoc, or JSON files from anywhere on the filesystem. It accepts a `pattern` of entry files to match, and a `base` file path of where your files are located. Use this when you have one file per entry.
*
* The `file()` loader creates multiple entries from a single local file. Use this when all your entries are stored in an array of objects.
*
* ```ts {3,8,19}
* // src/content/config.ts
* import { defineCollection, z } from 'astro:content';
* import { glob, file } from 'astro/loaders';
*
* const blog = defineCollection({
* // By default the ID is a slug generated from
* // the path of the file relative to `base`
* loader: glob({ pattern: "**\/*.md", base: "./src/data/blog" }),
* schema: z.object({
* title: z.string(),
* description: z.string(),
* pubDate: z.coerce.date(),
* updatedDate: z.coerce.date().optional(),
* })
* });
*
* const dogs = defineCollection({
* // The path is relative to the project root, or an absolute path.
* loader: file("src/data/dogs.json"),
* schema: z.object({
* id: z.string(),
* breed: z.string(),
* temperament: z.array(z.string()),
* }),
* });
*
* export const collections = { blog, dogs };
* ```
*
* #### Querying and rendering with the Content Layer API
*
* The collection can be [queried in the same way as content collections](/en/guides/content-collections/#querying-collections):
*
* ```ts
* // src/pages/index.astro
* import { getCollection, getEntry } from 'astro:content';
*
* // Get all entries from a collection.
* // Requires the name of the collection as an argument.
* const allBlogPosts = await getCollection('blog');
*
* // Get a single entry from a collection.
* // Requires the name of the collection and ID
* const labradorData = await getEntry('dogs', 'labrador-retriever');
* ```
*
* Entries generated from Markdown, MDX or Markdoc can be rendered directly to a page using the `render()` function.
*
* :::note
* The syntax for rendering collection entries is different from current content collections syntax.
* :::
*
* ```astro title="src/pages/[slug].astro"
* ---
* import { getEntry, render } from 'astro:content';
*
* const post = await getEntry('blog', Astro.params.slug);
*
* const { Content, headings } = await render(entry);
* ---
*
* <Content />
* ```
*
* #### Creating a loader
*
* With the Content Layer API, you can build loaders to load or generate content from anywhere.
*
* For example, you can create a loader that fetches collection entries from a remote API.
*
* ```ts
* // src/content/config.ts
* const countries = defineCollection({
* loader: async () => {
* const response = await fetch("https://restcountries.com/v3.1/all");
* const data = await response.json();
* // Must return an array of entries with an id property,
* // or an object with IDs as keys and entries as values
* return data.map((country) => ({
* id: country.cca3,
* ...country,
* }));
* },
* // optionally add a schema
* // schema: z.object...
* });
*
* export const collections = { countries };
* ```
*
* For more advanced loading logic, you can define an object loader. This allows incremental updates and conditional loading, and gives full access to the data store. See the API in [the Content Layer API RFC](https://github.com/withastro/roadmap/blob/content-layer/proposals/0047-content-layer.md#loaders).
*
* #### Migrating an existing content collection to use the Content Layer API
*
* You can convert an existing content collection with Markdown, MDX, Markdoc, or JSON entries to use the Content Layer API.
*
* 1. **Move the collection folder out of `src/content/`** (e.g. to `src/data/`). All collections located in the `src/content/` folder will use the existing Content Collections API.
*
* **Do not move the existing `src/content/config.ts` file**. This file will define all collections, using either API.
*
* 2. **Edit the collection definition**. Your updated collection requires a `loader`, and the option to select a collection `type` is no longer available.
*
* ```diff
* // src/content/config.ts
* import { defineCollection, z } from 'astro:content';
* + import { glob } from 'astro/loaders';
*
* const blog = defineCollection({
* // For content layer you no longer define a `type`
* - type: 'content',
* + loader: glob({ pattern: "**\/*.md", base: "./src/data/blog" }),
* schema: z.object({
* title: z.string(),
* description: z.string(),
* pubDate: z.coerce.date(),
* updatedDate: z.coerce.date().optional(),
* }),
* });
* ```
*
* 3. **Change references from `slug` to `id`**. Content layer collections do not have a `slug` field. Instead, all updated collections will have an `id`.
*
* ```diff
* // src/pages/index.astro
* ---
* export async function getStaticPaths() {
* const posts = await getCollection('blog');
* return posts.map((post) => ({
* - params: { slug: post.slug },
* + params: { slug: post.id },
* props: post,
* }));
* }
* ---
* ```
*
* 4. **Switch to the new `render()` function**. Entries no longer have a `render()` method, as they are now serializable plain objects. Instead, import the `render()` function from `astro:content`.
*
* ```diff
* // src/pages/index.astro
* ---
* - import { getEntry } from 'astro:content';
* + import { getEntry, render } from 'astro:content';
*
* const post = await getEntry('blog', params.slug);
*
* - const { Content, headings } = await post.render();
* + const { Content, headings } = await render(post);
* ---
*
* <Content />
* ```
*
* #### Learn more
*
* For a complete overview and the full API reference, see [the Content Layer API RFC](https://github.com/withastro/roadmap/blob/content-layer/proposals/0047-content-layer.md) and [share your feedback](https://github.com/withastro/roadmap/pull/982).
*/
contentLayer?: boolean;
};
}
@ -2279,6 +2500,10 @@ export interface AstroInlineOnlyConfig {
* @default "info"
*/
logLevel?: LoggerLevel;
/**
* Clear the content layer cache, forcing a rebuild of all content entries.
*/
force?: boolean;
/**
* @internal for testing only, use `logLevel` instead.
*/
@ -2307,6 +2532,8 @@ export type DataEntryModule = {
};
};
export type ContentEntryRenderFuction = (entry: DataEntry) => Promise<RenderedContent>;
export interface ContentEntryType {
extensions: string[];
getEntryInfo(params: {
@ -2322,6 +2549,8 @@ export interface ContentEntryType {
},
): rollup.LoadResult | Promise<rollup.LoadResult>;
contentModuleTypes?: string;
getRenderFunction?(settings: AstroSettings): Promise<ContentEntryRenderFuction>;
/**
* Handle asset propagation for rendered content to avoid bleed.
* Ex. MDX content can import styles and scripts, so `handlePropagation` should be true.
@ -2363,6 +2592,11 @@ export interface AstroAdapterFeatures {
functionPerRoute: boolean;
}
export interface InjectedType {
filename: string;
content: string;
}
export interface AstroSettings {
config: AstroConfig;
adapter: AstroAdapter | undefined;
@ -2398,6 +2632,7 @@ export interface AstroSettings {
latestAstroVersion: string | undefined;
serverIslandMap: NonNullable<SSRManifest['serverIslandMap']>;
serverIslandNameMap: NonNullable<SSRManifest['serverIslandNameMap']>;
injectedTypes: Array<InjectedType>;
}
export type AsyncRendererComponentFn<U> = (
@ -3099,6 +3334,7 @@ declare global {
'astro:config:done': (options: {
config: AstroConfig;
setAdapter: (adapter: AstroAdapter) => void;
injectTypes: (injectedType: InjectedType) => URL;
logger: AstroIntegrationLogger;
}) => void | Promise<void>;
'astro:server:setup': (options: {

2
packages/astro/src/actions/consts.ts
View file

@ -1,6 +1,6 @@
export const VIRTUAL_MODULE_ID = 'astro:actions';
export const RESOLVED_VIRTUAL_MODULE_ID = '\0' + VIRTUAL_MODULE_ID;
export const ACTIONS_TYPES_FILE = 'actions.d.ts';
export const ACTIONS_TYPES_FILE = 'astro/actions.d.ts';
export const VIRTUAL_INTERNAL_MODULE_ID = 'astro:internal-actions';
export const RESOLVED_VIRTUAL_INTERNAL_MODULE_ID = '\0astro:internal-actions';
export const NOOP_ACTIONS = '\0noop-actions';

39
packages/astro/src/actions/index.ts
View file

@ -30,9 +30,6 @@ export default function astroActions({
throw error;
}
const stringifiedActionsImport = JSON.stringify(
viteID(new URL('./actions', params.config.srcDir)),
);
params.updateConfig({
vite: {
plugins: [vitePluginUserActions({ settings }), vitePluginActions(fs)],
@ -49,11 +46,18 @@ export default function astroActions({
entrypoint: 'astro/actions/runtime/middleware.js',
order: 'post',
});
},
'astro:config:done': (params) => {
const stringifiedActionsImport = JSON.stringify(
viteID(new URL('./actions', params.config.srcDir)),
);
settings.injectedTypes.push({
filename: ACTIONS_TYPES_FILE,
content: `declare module "astro:actions" {
type Actions = typeof import(${stringifiedActionsImport})["server"];
await typegen({
stringifiedActionsImport,
root: params.config.root,
fs,
export const actions: Actions;
}`,
});
},
},
@ -119,24 +123,3 @@ const vitePluginActions = (fs: typeof fsMod): VitePlugin => ({
return code;
},
});
async function typegen({
stringifiedActionsImport,
root,
fs,
}: {
stringifiedActionsImport: string;
root: URL;
fs: typeof fsMod;
}) {
const content = `declare module "astro:actions" {
type Actions = typeof import(${stringifiedActionsImport})["server"];
export const actions: Actions;
}`;
const dotAstroDir = new URL('.astro/', root);
await fs.promises.mkdir(dotAstroDir, { recursive: true });
await fs.promises.writeFile(new URL(ACTIONS_TYPES_FILE, dotAstroDir), content);
}

40
packages/astro/src/assets/utils/resolveImports.ts Normal file
View file

@ -0,0 +1,40 @@
import { isRemotePath, removeBase } from '@astrojs/internal-helpers/path';
import { CONTENT_IMAGE_FLAG, IMAGE_IMPORT_PREFIX } from '../../content/consts.js';
import { shorthash } from '../../runtime/server/shorthash.js';
import { VALID_INPUT_FORMATS } from '../consts.js';
/**
* Resolves an image src from a content file (such as markdown) to a module ID or import that can be resolved by Vite.
*
* @param imageSrc The src attribute of an image tag
* @param filePath The path to the file that contains the imagem relative to the site root
* @returns A module id of the image that can be rsolved by Vite, or undefined if it is not a local image
*/
export function imageSrcToImportId(imageSrc: string, filePath: string): string | undefined {
// If the import is coming from the data store it will have a special prefix to identify it
// as an image import. We remove this prefix so that we can resolve the image correctly.
imageSrc = removeBase(imageSrc, IMAGE_IMPORT_PREFIX);
// We only care about local imports
if (isRemotePath(imageSrc) || imageSrc.startsWith('/')) {
return;
}
// We only care about images
const ext = imageSrc.split('.').at(-1) as (typeof VALID_INPUT_FORMATS)[number] | undefined;
if (!ext || !VALID_INPUT_FORMATS.includes(ext)) {
return;
}
// The import paths are relative to the content (md) file, but when it's actually resolved it will
// be in a single assets file, so relative paths will no longer work. To deal with this we use
// a query parameter to store the original path to the file and append a query param flag.
// This allows our Vite plugin to intercept the import and resolve the path relative to the
// importer and get the correct full path for the imported image.
const params = new URLSearchParams(CONTENT_IMAGE_FLAG);
params.set('importer', filePath);
return `${imageSrc}?${params.toString()}`;
}
export const importIdToSymbolName = (importId: string) =>
`__ASTRO_IMAGE_IMPORT_${shorthash(importId)}`;

6
packages/astro/src/cli/build/index.ts
View file

@ -14,6 +14,10 @@ export async function build({ flags }: BuildOptions) {
tables: {
Flags: [
['--outDir <directory>', `Specify the output directory for the build.`],
[
'--force',
'Clear the content layer and content collection cache, forcing a full rebuild.',
],
['--help (-h)', 'See all available flags.'],
],
},
@ -24,5 +28,5 @@ export async function build({ flags }: BuildOptions) {
const inlineConfig = flagsToAstroInlineConfig(flags);
await _build(inlineConfig, { force: !!flags.force });
await _build(inlineConfig);
}

2
packages/astro/src/cli/check/index.ts
View file

@ -32,7 +32,7 @@ export async function check(flags: Flags) {
// For now, we run this once as usually `astro check --watch` is ran alongside `astro dev` which also calls `astro sync`.
const { default: sync } = await import('../../core/sync/index.js');
try {
await sync({ inlineConfig: flagsToAstroInlineConfig(flags) });
await sync(flagsToAstroInlineConfig(flags));
} catch (_) {
return process.exit(1);
}

1
packages/astro/src/cli/dev/index.ts
View file

@ -18,6 +18,7 @@ export async function dev({ flags }: DevOptions) {
['--host', `Listen on all addresses, including LAN and public addresses.`],
['--host <custom-address>', `Expose on a network IP address at <custom-address>`],
['--open', 'Automatically open the app in the browser on server start'],
['--force', 'Clear the content layer cache, forcing a full rebuild.'],
['--help (-h)', 'See all available flags.'],
],
},

1
packages/astro/src/cli/flags.ts
View file

@ -12,6 +12,7 @@ export function flagsToAstroInlineConfig(flags: Flags): AstroInlineConfig {
configFile: typeof flags.config === 'string' ? flags.config : undefined,
mode: typeof flags.mode === 'string' ? (flags.mode as AstroInlineConfig['mode']) : undefined,
logLevel: flags.verbose ? 'debug' : flags.silent ? 'silent' : undefined,
force: flags.force ? true : undefined,
// Astro user configs
root: typeof flags.root === 'string' ? flags.root : undefined,

7
packages/astro/src/cli/sync/index.ts
View file

@ -12,7 +12,10 @@ export async function sync({ flags }: SyncOptions) {
commandName: 'astro sync',
usage: '[...flags]',
tables: {
Flags: [['--help (-h)', 'See all available flags.']],
Flags: [
['--force', 'Clear the content layer cache, forcing a full rebuild.'],
['--help (-h)', 'See all available flags.'],
],
},
description: `Generates TypeScript types for all Astro modules.`,
});
@ -20,7 +23,7 @@ export async function sync({ flags }: SyncOptions) {
}
try {
await _sync({ inlineConfig: flagsToAstroInlineConfig(flags), telemetry: true });
await _sync(flagsToAstroInlineConfig(flags), { telemetry: true });
return 0;
} catch (_) {
return 1;

26
packages/astro/src/content/consts.ts
View file

@ -2,18 +2,42 @@ export const PROPAGATED_ASSET_FLAG = 'astroPropagatedAssets';
export const CONTENT_RENDER_FLAG = 'astroRenderContent';
export const CONTENT_FLAG = 'astroContentCollectionEntry';
export const DATA_FLAG = 'astroDataCollectionEntry';
export const CONTENT_IMAGE_FLAG = 'astroContentImageFlag';
export const CONTENT_MODULE_FLAG = 'astroContentModuleFlag';
export const VIRTUAL_MODULE_ID = 'astro:content';
export const RESOLVED_VIRTUAL_MODULE_ID = '\0' + VIRTUAL_MODULE_ID;
export const DATA_STORE_VIRTUAL_ID = 'astro:data-layer-content';
export const RESOLVED_DATA_STORE_VIRTUAL_ID = '\0' + DATA_STORE_VIRTUAL_ID;
// Used by the content layer to create a virtual module that loads the `modules.mjs`, a file created by the content layer
// to map modules that are renderer at runtime
export const MODULES_MJS_ID = 'astro:content-module-imports';
export const MODULES_MJS_VIRTUAL_ID = '\0' + MODULES_MJS_ID;
export const DEFERRED_MODULE = 'astro:content-layer-deferred-module';
// Used by the content layer to create a virtual module that loads the `assets.mjs`
export const ASSET_IMPORTS_VIRTUAL_ID = 'astro:asset-imports';
export const ASSET_IMPORTS_RESOLVED_STUB_ID = '\0' + ASSET_IMPORTS_VIRTUAL_ID;
export const LINKS_PLACEHOLDER = '@@ASTRO-LINKS@@';
export const STYLES_PLACEHOLDER = '@@ASTRO-STYLES@@';
export const SCRIPTS_PLACEHOLDER = '@@ASTRO-SCRIPTS@@';
export const IMAGE_IMPORT_PREFIX = '__ASTRO_IMAGE_';
export const CONTENT_FLAGS = [
CONTENT_FLAG,
CONTENT_RENDER_FLAG,
DATA_FLAG,
PROPAGATED_ASSET_FLAG,
CONTENT_IMAGE_FLAG,
CONTENT_MODULE_FLAG,
] as const;
export const CONTENT_TYPES_FILE = 'types.d.ts';
export const CONTENT_TYPES_FILE = 'astro/content.d.ts';
export const DATA_STORE_FILE = 'data-store.json';
export const ASSET_IMPORTS_FILE = 'assets.mjs';
export const MODULES_IMPORTS_FILE = 'modules.mjs';
export const CONTENT_LAYER_TYPE = 'content_layer';

306
packages/astro/src/content/content-layer.ts Normal file
View file

@ -0,0 +1,306 @@
import { promises as fs, existsSync } from 'node:fs';
import { isAbsolute } from 'node:path';
import { fileURLToPath } from 'node:url';
import type { FSWatcher } from 'vite';
import xxhash from 'xxhash-wasm';
import type { AstroSettings } from '../@types/astro.js';
import { AstroUserError } from '../core/errors/errors.js';
import type { Logger } from '../core/logger/core.js';
import {
ASSET_IMPORTS_FILE,
CONTENT_LAYER_TYPE,
DATA_STORE_FILE,
MODULES_IMPORTS_FILE,
} from './consts.js';
import type { DataStore } from './data-store.js';
import type { LoaderContext } from './loaders/types.js';
import { getEntryDataAndImages, globalContentConfigObserver, posixRelative } from './utils.js';
export interface ContentLayerOptions {
store: DataStore;
settings: AstroSettings;
logger: Logger;
watcher?: FSWatcher;
}
export class ContentLayer {
#logger: Logger;
#store: DataStore;
#settings: AstroSettings;
#watcher?: FSWatcher;
#lastConfigDigest?: string;
#unsubscribe?: () => void;
#generateDigest?: (data: Record<string, unknown> | string) => string;
#loading = false;
constructor({ settings, logger, store, watcher }: ContentLayerOptions) {
// The default max listeners is 10, which can be exceeded when using a lot of loaders
watcher?.setMaxListeners(50);
this.#logger = logger;
this.#store = store;
this.#settings = settings;
this.#watcher = watcher;
}
/**
* Whether the content layer is currently loading content
*/
get loading() {
return this.#loading;
}
/**
* Watch for changes to the content config and trigger a sync when it changes.
*/
watchContentConfig() {
this.#unsubscribe?.();
this.#unsubscribe = globalContentConfigObserver.subscribe(async (ctx) => {
if (
!this.#loading &&
ctx.status === 'loaded' &&
ctx.config.digest !== this.#lastConfigDigest
) {
this.sync();
}
});
}
unwatchContentConfig() {
this.#unsubscribe?.();
}
/**
* Run the `load()` method of each collection's loader, which will load the data and save it in the data store.
* The loader itself is responsible for deciding whether this will clear and reload the full collection, or
* perform an incremental update. After the data is loaded, the data store is written to disk.
*/
async sync() {
if (this.#loading) {
return;
}
this.#loading = true;
try {
await this.#doSync();
} finally {
this.#loading = false;
}
}
async #getGenerateDigest() {
if (this.#generateDigest) {
return this.#generateDigest;
}
// xxhash is a very fast non-cryptographic hash function that is used to generate a content digest
// It uses wasm, so we need to load it asynchronously.
const { h64ToString } = await xxhash();
this.#generateDigest = (data: Record<string, unknown> | string) => {
const dataString = typeof data === 'string' ? data : JSON.stringify(data);
return h64ToString(dataString);
};
return this.#generateDigest;
}
async #getLoaderContext({
collectionName,
loaderName = 'content',
parseData,
}: {
collectionName: string;
loaderName: string;
parseData: LoaderContext['parseData'];
}): Promise<LoaderContext> {
return {
collection: collectionName,
store: this.#store.scopedStore(collectionName),
meta: this.#store.metaStore(collectionName),
logger: this.#logger.forkIntegrationLogger(loaderName),
settings: this.#settings,
parseData,
generateDigest: await this.#getGenerateDigest(),
watcher: this.#watcher,
};
}
async #doSync() {
const contentConfig = globalContentConfigObserver.get();
const logger = this.#logger.forkIntegrationLogger('content');
if (contentConfig?.status !== 'loaded') {
logger.debug('Content config not loaded, skipping sync');
return;
}
if (!this.#settings.config.experimental.contentLayer) {
const contentLayerCollections = Object.entries(contentConfig.config.collections).filter(
([_, collection]) => collection.type === CONTENT_LAYER_TYPE,
);
if (contentLayerCollections.length > 0) {
throw new AstroUserError(
`The following collections have a loader defined, but the content layer is not enabled: ${contentLayerCollections.map(([title]) => title).join(', ')}.`,
'To enable the Content Layer API, set `experimental: { contentLayer: true }` in your Astro config file.',
);
}
return;
}
logger.info('Syncing content');
const { digest: currentConfigDigest } = contentConfig.config;
this.#lastConfigDigest = currentConfigDigest;
const previousConfigDigest = await this.#store.metaStore().get('config-digest');
if (currentConfigDigest && previousConfigDigest !== currentConfigDigest) {
logger.info('Content config changed, clearing cache');
this.#store.clearAll();
await this.#store.metaStore().set('config-digest', currentConfigDigest);
}
await Promise.all(
Object.entries(contentConfig.config.collections).map(async ([name, collection]) => {
if (collection.type !== CONTENT_LAYER_TYPE) {
return;
}
let { schema } = collection;
if (!schema && typeof collection.loader === 'object') {
schema = collection.loader.schema;
if (typeof schema === 'function') {
schema = await schema();
}
}
const collectionWithResolvedSchema = { ...collection, schema };
const parseData: LoaderContext['parseData'] = async ({ id, data, filePath = '' }) => {
const { imageImports, data: parsedData } = await getEntryDataAndImages(
{
id,
collection: name,
unvalidatedData: data,
_internal: {
rawData: undefined,
filePath,
},
},
collectionWithResolvedSchema,
false,
);
if (imageImports?.length) {
this.#store.addAssetImports(
imageImports,
// This path may already be relative, if we're re-parsing an existing entry
isAbsolute(filePath)
? posixRelative(fileURLToPath(this.#settings.config.root), filePath)
: filePath,
);
}
return parsedData;
};
const context = await this.#getLoaderContext({
collectionName: name,
parseData,
loaderName: collection.loader.name,
});
if (typeof collection.loader === 'function') {
return simpleLoader(collection.loader, context);
}
if (!collection.loader.load) {
throw new Error(`Collection loader for ${name} does not have a load method`);
}
return collection.loader.load(context);
}),
);
if (!existsSync(this.#settings.config.cacheDir)) {
await fs.mkdir(this.#settings.config.cacheDir, { recursive: true });
}
const cacheFile = new URL(DATA_STORE_FILE, this.#settings.config.cacheDir);
await this.#store.writeToDisk(cacheFile);
if (!existsSync(this.#settings.dotAstroDir)) {
await fs.mkdir(this.#settings.dotAstroDir, { recursive: true });
}
const assetImportsFile = new URL(ASSET_IMPORTS_FILE, this.#settings.dotAstroDir);
await this.#store.writeAssetImports(assetImportsFile);
const modulesImportsFile = new URL(MODULES_IMPORTS_FILE, this.#settings.dotAstroDir);
await this.#store.writeModuleImports(modulesImportsFile);
logger.info('Synced content');
if (this.#settings.config.experimental.contentIntellisense) {
await this.regenerateCollectionFileManifest();
}
}
async regenerateCollectionFileManifest() {
const collectionsManifest = new URL('collections/collections.json', this.#settings.dotAstroDir);
this.#logger.debug('content', 'Regenerating collection file manifest');
if (existsSync(collectionsManifest)) {
try {
const collections = await fs.readFile(collectionsManifest, 'utf-8');
const collectionsJson = JSON.parse(collections);
collectionsJson.entries ??= {};
for (const { hasSchema, name } of collectionsJson.collections) {
if (!hasSchema) {
continue;
}
const entries = this.#store.values(name);
if (!entries?.[0]?.filePath) {
continue;
}
for (const { filePath } of entries) {
if (!filePath) {
continue;
}
const key = new URL(filePath, this.#settings.config.root).href.toLowerCase();
collectionsJson.entries[key] = name;
}
}
await fs.writeFile(collectionsManifest, JSON.stringify(collectionsJson, null, 2));
} catch {
this.#logger.error('content', 'Failed to regenerate collection file manifest');
}
}
this.#logger.debug('content', 'Regenerated collection file manifest');
}
}
export async function simpleLoader<TData extends { id: string }>(
handler: () => Array<TData> | Promise<Array<TData>>,
context: LoaderContext,
) {
const data = await handler();
context.store.clear();
for (const raw of data) {
const item = await context.parseData({ id: raw.id, data: raw });
context.store.set({ id: raw.id, data: item });
}
}
function contentLayerSingleton() {
let instance: ContentLayer | null = null;
return {
initialized: () => Boolean(instance),
init: (options: ContentLayerOptions) => {
instance?.unwatchContentConfig();
instance = new ContentLayer(options);
return instance;
},
get: () => {
if (!instance) {
throw new Error('Content layer not initialized');
}
return instance;
},
dispose: () => {
instance?.unwatchContentConfig();
instance = null;
},
};
}
export const globalContentLayer = contentLayerSingleton();

467
packages/astro/src/content/data-store.ts Normal file
View file

@ -0,0 +1,467 @@
import type { MarkdownHeading } from '@astrojs/markdown-remark';
import * as devalue from 'devalue';
import { existsSync, promises as fs, type PathLike } from 'fs';
import { imageSrcToImportId, importIdToSymbolName } from '../assets/utils/resolveImports.js';
import { AstroError, AstroErrorData } from '../core/errors/index.js';
import { CONTENT_MODULE_FLAG, DEFERRED_MODULE } from './consts.js';
const SAVE_DEBOUNCE_MS = 500;
export interface RenderedContent {
/** Rendered HTML string. If present then `render(entry)` will return a component that renders this HTML. */
html: string;
metadata?: {
/** Any images that are present in this entry. Relative to the {@link DataEntry} filePath. */
imagePaths?: Array<string>;
/** Any headings that are present in this file. */
headings?: MarkdownHeading[];
/** Raw frontmatter, parsed parsed from the file. This may include data from remark plugins. */
frontmatter?: Record<string, any>;
/** Any other metadata that is present in this file. */
[key: string]: unknown;
};
}
export interface DataEntry<TData extends Record<string, unknown> = Record<string, unknown>> {
/** The ID of the entry. Unique per collection. */
id: string;
/** The parsed entry data */
data: TData;
/** The file path of the content, if applicable. Relative to the site root. */
filePath?: string;
/** The raw body of the content, if applicable. */
body?: string;
/** An optional content digest, to check if the content has changed. */
digest?: number | string;
/** The rendered content of the entry, if applicable. */
rendered?: RenderedContent;
/**
* If an entry is a deferred, its rendering phase is delegated to a virtual module during the runtime phase when calling `renderEntry`.
*/
deferredRender?: boolean;
}
export class DataStore {
#collections = new Map<string, Map<string, any>>();
#file?: PathLike;
#assetsFile?: PathLike;
#modulesFile?: PathLike;
#saveTimeout: NodeJS.Timeout | undefined;
#assetsSaveTimeout: NodeJS.Timeout | undefined;
#modulesSaveTimeout: NodeJS.Timeout | undefined;
#dirty = false;
#assetsDirty = false;
#modulesDirty = false;
#assetImports = new Set<string>();
#moduleImports = new Map<string, string>();
constructor() {
this.#collections = new Map();
}
get<T = DataEntry>(collectionName: string, key: string): T | undefined {
return this.#collections.get(collectionName)?.get(String(key));
}
entries<T = DataEntry>(collectionName: string): Array<[id: string, T]> {
const collection = this.#collections.get(collectionName) ?? new Map();
return [...collection.entries()];
}
values<T = DataEntry>(collectionName: string): Array<T> {
const collection = this.#collections.get(collectionName) ?? new Map();
return [...collection.values()];
}
keys(collectionName: string): Array<string> {
const collection = this.#collections.get(collectionName) ?? new Map();
return [...collection.keys()];
}
set(collectionName: string, key: string, value: unknown) {
const collection = this.#collections.get(collectionName) ?? new Map();
collection.set(String(key), value);
this.#collections.set(collectionName, collection);
this.#saveToDiskDebounced();
}
delete(collectionName: string, key: string) {
const collection = this.#collections.get(collectionName);
if (collection) {
collection.delete(String(key));
this.#saveToDiskDebounced();
}
}
clear(collectionName: string) {
this.#collections.delete(collectionName);
this.#saveToDiskDebounced();
}
clearAll() {
this.#collections.clear();
this.#saveToDiskDebounced();
}
has(collectionName: string, key: string) {
const collection = this.#collections.get(collectionName);
if (collection) {
return collection.has(String(key));
}
return false;
}
hasCollection(collectionName: string) {
return this.#collections.has(collectionName);
}
collections() {
return this.#collections;
}
addAssetImport(assetImport: string, filePath: string) {
const id = imageSrcToImportId(assetImport, filePath);
if (id) {
this.#assetImports.add(id);
// We debounce the writes to disk because addAssetImport is called for every image in every file,
// and can be called many times in quick succession by a filesystem watcher. We only want to write
// the file once, after all the imports have been added.
this.#writeAssetsImportsDebounced();
}
}
addAssetImports(assets: Array<string>, filePath: string) {
assets.forEach((asset) => this.addAssetImport(asset, filePath));
}
addModuleImport(fileName: string) {
const id = contentModuleToId(fileName);
if (id) {
this.#moduleImports.set(fileName, id);
// We debounce the writes to disk because addAssetImport is called for every image in every file,
// and can be called many times in quick succession by a filesystem watcher. We only want to write
// the file once, after all the imports have been added.
this.#writeModulesImportsDebounced();
}
}
async writeAssetImports(filePath: PathLike) {
this.#assetsFile = filePath;
if (this.#assetImports.size === 0) {
try {
await fs.writeFile(filePath, 'export default new Map();');
} catch (err) {
throw new AstroError(AstroErrorData.UnknownFilesystemError, { cause: err });
}
}
if (!this.#assetsDirty && existsSync(filePath)) {
return;
}
// Import the assets, with a symbol name that is unique to the import id. The import
// for each asset is an object with path, format and dimensions.
// We then export them all, mapped by the import id, so we can find them again in the build.
const imports: Array<string> = [];
const exports: Array<string> = [];
this.#assetImports.forEach((id) => {
const symbol = importIdToSymbolName(id);
imports.push(`import ${symbol} from '${id}';`);
exports.push(`[${JSON.stringify(id)}, ${symbol}]`);
});
const code = /* js */ `
${imports.join('\n')}
export default new Map([${exports.join(', ')}]);
`;
try {
await fs.writeFile(filePath, code);
} catch (err) {
throw new AstroError(AstroErrorData.UnknownFilesystemError, { cause: err });
}
this.#assetsDirty = false;
}
async writeModuleImports(filePath: PathLike) {
this.#modulesFile = filePath;
if (this.#moduleImports.size === 0) {
try {
await fs.writeFile(filePath, 'export default new Map();');
} catch (err) {
throw new AstroError(AstroErrorData.UnknownFilesystemError, { cause: err });
}
}
if (!this.#modulesDirty && existsSync(filePath)) {
return;
}
// Import the assets, with a symbol name that is unique to the import id. The import
// for each asset is an object with path, format and dimensions.
// We then export them all, mapped by the import id, so we can find them again in the build.
const lines: Array<string> = [];
for (const [fileName, specifier] of this.#moduleImports) {
lines.push(`['${fileName}', () => import('${specifier}')]`);
}
const code = `
export default new Map([\n${lines.join(',\n')}]);
`;
try {
await fs.writeFile(filePath, code);
} catch (err) {
throw new AstroError(AstroErrorData.UnknownFilesystemError, { cause: err });
}
this.#modulesDirty = false;
}
#writeAssetsImportsDebounced() {
this.#assetsDirty = true;
if (this.#assetsFile) {
if (this.#assetsSaveTimeout) {
clearTimeout(this.#assetsSaveTimeout);
}
this.#assetsSaveTimeout = setTimeout(() => {
this.#assetsSaveTimeout = undefined;
this.writeAssetImports(this.#assetsFile!);
}, SAVE_DEBOUNCE_MS);
}
}
#writeModulesImportsDebounced() {
this.#modulesDirty = true;
if (this.#modulesFile) {
if (this.#modulesSaveTimeout) {
clearTimeout(this.#modulesSaveTimeout);
}
this.#modulesSaveTimeout = setTimeout(() => {
this.#modulesSaveTimeout = undefined;
this.writeModuleImports(this.#modulesFile!);
}, SAVE_DEBOUNCE_MS);
}
}
#saveToDiskDebounced() {
this.#dirty = true;
// Only save to disk if it has already been saved once
if (this.#file) {
if (this.#saveTimeout) {
clearTimeout(this.#saveTimeout);
}
this.#saveTimeout = setTimeout(() => {
this.#saveTimeout = undefined;
this.writeToDisk(this.#file!);
}, SAVE_DEBOUNCE_MS);
}
}
scopedStore(collectionName: string): ScopedDataStore {
return {
get: <TData extends Record<string, unknown> = Record<string, unknown>>(key: string) =>
this.get<DataEntry<TData>>(collectionName, key),
entries: () => this.entries(collectionName),
values: () => this.values(collectionName),
keys: () => this.keys(collectionName),
set: ({ id: key, data, body, filePath, deferredRender, digest, rendered }) => {
if (!key) {
throw new Error(`ID must be a non-empty string`);
}
const id = String(key);
if (digest) {
const existing = this.get<DataEntry>(collectionName, id);
if (existing && existing.digest === digest) {
return false;
}
}
const entry: DataEntry = {
id,
data,
};
// We do it like this so we don't waste space stringifying
// the fields if they are not set
if (body) {
entry.body = body;
}
if (filePath) {
if (filePath.startsWith('/')) {
throw new Error(`File path must be relative to the site root. Got: ${filePath}`);
}
entry.filePath = filePath;
}
if (digest) {
entry.digest = digest;
}
if (rendered) {
entry.rendered = rendered;
}
if (deferredRender) {
entry.deferredRender = deferredRender;
if (filePath) {
this.addModuleImport(filePath);
}
}
this.set(collectionName, id, entry);
return true;
},
delete: (key: string) => this.delete(collectionName, key),
clear: () => this.clear(collectionName),
has: (key: string) => this.has(collectionName, key),
addAssetImport: (assetImport: string, fileName: string) =>
this.addAssetImport(assetImport, fileName),
addAssetImports: (assets: Array<string>, fileName: string) =>
this.addAssetImports(assets, fileName),
addModuleImport: (fileName: string) => this.addModuleImport(fileName),
};
}
/**
* Returns a MetaStore for a given collection, or if no collection is provided, the default meta collection.
*/
metaStore(collectionName = ':meta'): MetaStore {
const collectionKey = `meta:${collectionName}`;
return {
get: (key: string) => this.get(collectionKey, key),
set: (key: string, data: string) => this.set(collectionKey, key, data),
delete: (key: string) => this.delete(collectionKey, key),
has: (key: string) => this.has(collectionKey, key),
};
}
toString() {
return devalue.stringify(this.#collections);
}
async writeToDisk(filePath: PathLike) {
if (!this.#dirty) {
return;
}
try {
await fs.writeFile(filePath, this.toString());
this.#file = filePath;
this.#dirty = false;
} catch (err) {
throw new AstroError(AstroErrorData.UnknownFilesystemError, { cause: err });
}
}
/**
* Attempts to load a DataStore from the virtual module.
* This only works in Vite.
*/
static async fromModule() {
try {
// @ts-expect-error - this is a virtual module
const data = await import('astro:data-layer-content');
const map = devalue.unflatten(data.default);
return DataStore.fromMap(map);
} catch {}
return new DataStore();
}
static async fromMap(data: Map<string, Map<string, any>>) {
const store = new DataStore();
store.#collections = data;
return store;
}
static async fromString(data: string) {
const map = devalue.parse(data);
return DataStore.fromMap(map);
}
static async fromFile(filePath: string | URL) {
try {
if (existsSync(filePath)) {
const data = await fs.readFile(filePath, 'utf-8');
return DataStore.fromString(data);
}
} catch {}
return new DataStore();
}
}
export interface ScopedDataStore {
get: <TData extends Record<string, unknown> = Record<string, unknown>>(
key: string,
) => DataEntry<TData> | undefined;
entries: () => Array<[id: string, DataEntry]>;
set: <TData extends Record<string, unknown>>(opts: {
/** The ID of the entry. Must be unique per collection. */
id: string;
/** The data to store. */
data: TData;
/** The raw body of the content, if applicable. */
body?: string;
/** The file path of the content, if applicable. Relative to the site root. */
filePath?: string;
/** A content digest, to check if the content has changed. */
digest?: number | string;
/** The rendered content, if applicable. */
rendered?: RenderedContent;
/**
* If an entry is a deferred, its rendering phase is delegated to a virtual module during the runtime phase.
*/
deferredRender?: boolean;
}) => boolean;
values: () => Array<DataEntry>;
keys: () => Array<string>;
delete: (key: string) => void;
clear: () => void;
has: (key: string) => boolean;
/**
* @internal Adds asset imports to the store. This is used to track image imports for the build. This API is subject to change.
*/
addAssetImports: (assets: Array<string>, fileName: string) => void;
/**
* @internal Adds an asset import to the store. This is used to track image imports for the build. This API is subject to change.
*/
addAssetImport: (assetImport: string, fileName: string) => void;
/**
* Adds a single asset to the store. This asset will be transformed
* by Vite, and the URL will be available in the final build.
* @param fileName
* @param specifier
* @returns
*/
addModuleImport: (fileName: string) => void;
}
/**
* A key-value store for metadata strings. Useful for storing things like sync tokens.
*/
export interface MetaStore {
get: (key: string) => string | undefined;
set: (key: string, value: string) => void;
has: (key: string) => boolean;
delete: (key: string) => void;
}
function dataStoreSingleton() {
let instance: Promise<DataStore> | DataStore | undefined = undefined;
return {
get: async () => {
if (!instance) {
instance = DataStore.fromModule();
}
return instance;
},
set: (store: DataStore) => {
instance = store;
},
};
}
// TODO: find a better place to put this image
export function contentModuleToId(fileName: string) {
const params = new URLSearchParams(DEFERRED_MODULE);
params.set('fileName', fileName);
params.set(CONTENT_MODULE_FLAG, 'true');
return `${DEFERRED_MODULE}?${params.toString()}`;
}
/** @internal */
export const globalDataStore = dataStoreSingleton();

83
packages/astro/src/content/loaders/file.ts Normal file
View file

@ -0,0 +1,83 @@
import { promises as fs, existsSync } from 'node:fs';
import { fileURLToPath } from 'node:url';
import { posixRelative } from '../utils.js';
import type { Loader, LoaderContext } from './types.js';
/**
* Loads entries from a JSON file. The file must contain an array of objects that contain unique `id` fields, or an object with string keys.
* @todo Add support for other file types, such as YAML, CSV etc.
* @param fileName The path to the JSON file to load, relative to the content directory.
*/
export function file(fileName: string): Loader {
if (fileName.includes('*')) {
// TODO: AstroError
throw new Error('Glob patterns are not supported in `file` loader. Use `glob` loader instead.');
}
async function syncData(filePath: string, { logger, parseData, store, settings }: LoaderContext) {
let json: Array<Record<string, unknown>>;
try {
const data = await fs.readFile(filePath, 'utf-8');
json = JSON.parse(data);
} catch (error: any) {
logger.error(`Error reading data from ${fileName}`);
logger.debug(error.message);
return;
}
if (Array.isArray(json)) {
if (json.length === 0) {
logger.warn(`No items found in ${fileName}`);
}
logger.debug(`Found ${json.length} item array in ${fileName}`);
store.clear();
for (const rawItem of json) {
const id = (rawItem.id ?? rawItem.slug)?.toString();
if (!id) {
logger.error(`Item in ${fileName} is missing an id or slug field.`);
continue;
}
const data = await parseData({ id, data: rawItem, filePath });
store.set({
id,
data,
filePath: posixRelative(fileURLToPath(settings.config.root), filePath),
});
}
} else if (typeof json === 'object') {
const entries = Object.entries<Record<string, unknown>>(json);
logger.debug(`Found object with ${entries.length} entries in ${fileName}`);
store.clear();
for (const [id, rawItem] of entries) {
const data = await parseData({ id, data: rawItem, filePath });
store.set({ id, data });
}
} else {
logger.error(`Invalid data in ${fileName}. Must be an array or object.`);
}
}
return {
name: 'file-loader',
load: async (options) => {
const { settings, logger, watcher } = options;
logger.debug(`Loading data from ${fileName}`);
const url = new URL(fileName, settings.config.root);
if (!existsSync(url)) {
logger.error(`File not found: ${fileName}`);
return;
}
const filePath = fileURLToPath(url);
await syncData(filePath, options);
watcher?.on('change', async (changedPath) => {
if (changedPath === filePath) {
logger.info(`Reloading data from ${fileName}`);
await syncData(filePath, options);
}
});
},
};
}

296
packages/astro/src/content/loaders/glob.ts Normal file
View file

@ -0,0 +1,296 @@
import { promises as fs } from 'node:fs';
import { fileURLToPath, pathToFileURL } from 'node:url';
import fastGlob from 'fast-glob';
import { bold, green } from 'kleur/colors';
import micromatch from 'micromatch';
import pLimit from 'p-limit';
import type { ContentEntryRenderFuction, ContentEntryType } from '../../@types/astro.js';
import type { RenderedContent } from '../data-store.js';
import { getContentEntryIdAndSlug, getEntryConfigByExtMap, posixRelative } from '../utils.js';
import type { Loader } from './types.js';
export interface GenerateIdOptions {
/** The path to the entry file, relative to the base directory. */
entry: string;
/** The base directory URL. */
base: URL;
/** The parsed, unvalidated data of the entry. */
data: Record<string, unknown>;
}
export interface GlobOptions {
/** The glob pattern to match files, relative to the base directory */
pattern: string;
/** The base directory to resolve the glob pattern from. Relative to the root directory, or an absolute file URL. Defaults to `.` */
base?: string | URL;
/**
* Function that generates an ID for an entry. Default implementation generates a slug from the entry path.
* @returns The ID of the entry. Must be unique per collection.
**/
generateId?: (options: GenerateIdOptions) => string;
}
function generateIdDefault({ entry, base, data }: GenerateIdOptions): string {
if (data.slug) {
return data.slug as string;
}
const entryURL = new URL(entry, base);
const { slug } = getContentEntryIdAndSlug({
entry: entryURL,
contentDir: base,
collection: '',
});
return slug;
}
/**
* Loads multiple entries, using a glob pattern to match files.
* @param pattern A glob pattern to match files, relative to the content directory.
*/
export function glob(globOptions: GlobOptions): Loader {
if (globOptions.pattern.startsWith('../')) {
throw new Error(
'Glob patterns cannot start with `../`. Set the `base` option to a parent directory instead.',
);
}
if (globOptions.pattern.startsWith('/')) {
throw new Error(
'Glob patterns cannot start with `/`. Set the `base` option to a parent directory or use a relative path instead.',
);
}
const generateId = globOptions?.generateId ?? generateIdDefault;
const fileToIdMap = new Map<string, string>();
return {
name: 'glob-loader',
load: async ({ settings, logger, watcher, parseData, store, generateDigest }) => {
const renderFunctionByContentType = new WeakMap<
ContentEntryType,
ContentEntryRenderFuction
>();
const untouchedEntries = new Set(store.keys());
async function syncData(entry: string, base: URL, entryType?: ContentEntryType) {
if (!entryType) {
logger.warn(`No entry type found for ${entry}`);
return;
}
const fileUrl = new URL(entry, base);
const contents = await fs.readFile(fileUrl, 'utf-8').catch((err) => {
logger.error(`Error reading ${entry}: ${err.message}`);
return;
});
if (!contents) {
logger.warn(`No contents found for ${entry}`);
return;
}
const { body, data } = await entryType.getEntryInfo({
contents,
fileUrl,
});
const id = generateId({ entry, base, data });
untouchedEntries.delete(id);
const existingEntry = store.get(id);
const digest = generateDigest(contents);
if (existingEntry && existingEntry.digest === digest && existingEntry.filePath) {
if (existingEntry.deferredRender) {
store.addModuleImport(existingEntry.filePath);
}
if (existingEntry.rendered?.metadata?.imagePaths?.length) {
// Add asset imports for existing entries
store.addAssetImports(
existingEntry.rendered.metadata.imagePaths,
existingEntry.filePath,
);
}
// Re-parsing to resolve images and other effects
await parseData(existingEntry);
return;
}
const filePath = fileURLToPath(fileUrl);
const relativePath = posixRelative(fileURLToPath(settings.config.root), filePath);
const parsedData = await parseData({
id,
data,
filePath,
});
if (entryType.getRenderFunction) {
let render = renderFunctionByContentType.get(entryType);
if (!render) {
render = await entryType.getRenderFunction(settings);
// Cache the render function for this content type, so it can re-use parsers and other expensive setup
renderFunctionByContentType.set(entryType, render);
}
let rendered: RenderedContent | undefined = undefined;
try {
rendered = await render?.({
id,
data: parsedData,
body,
filePath,
digest,
});
} catch (error: any) {
logger.error(`Error rendering ${entry}: ${error.message}`);
}
store.set({
id,
data: parsedData,
body,
filePath: relativePath,
digest,
rendered,
});
if (rendered?.metadata?.imagePaths?.length) {
store.addAssetImports(rendered.metadata.imagePaths, relativePath);
}
// todo: add an explicit way to opt in to deferred rendering
} else if ('contentModuleTypes' in entryType) {
store.set({
id,
data: parsedData,
body,
filePath: relativePath,
digest,
deferredRender: true,
});
} else {
store.set({ id, data: parsedData, body, filePath: relativePath, digest });
}
fileToIdMap.set(filePath, id);
}
const entryConfigByExt = getEntryConfigByExtMap([
...settings.contentEntryTypes,
...settings.dataEntryTypes,
] as Array<ContentEntryType>);
const baseDir = globOptions.base
? new URL(globOptions.base, settings.config.root)
: settings.config.root;
if (!baseDir.pathname.endsWith('/')) {
baseDir.pathname = `${baseDir.pathname}/`;
}
const files = await fastGlob(globOptions.pattern, {
cwd: fileURLToPath(baseDir),
});
function configForFile(file: string) {
const ext = file.split('.').at(-1);
if (!ext) {
logger.warn(`No extension found for ${file}`);
return;
}
return entryConfigByExt.get(`.${ext}`);
}
const limit = pLimit(10);
const skippedFiles: Array<string> = [];
const contentDir = new URL('content/', settings.config.srcDir);
function isInContentDir(file: string) {
const fileUrl = new URL(file, baseDir);
return fileUrl.href.startsWith(contentDir.href);
}
const configFiles = new Set(
['config.js', 'config.ts', 'config.mjs'].map((file) => new URL(file, contentDir).href),
);
function isConfigFile(file: string) {
const fileUrl = new URL(file, baseDir);
return configFiles.has(fileUrl.href);
}
await Promise.all(
files.map((entry) => {
if (isConfigFile(entry)) {
return;
}
if (isInContentDir(entry)) {
skippedFiles.push(entry);
return;
}
return limit(async () => {
const entryType = configForFile(entry);
await syncData(entry, baseDir, entryType);
});
}),
);
const skipCount = skippedFiles.length;
if (skipCount > 0) {
logger.warn(`The glob() loader cannot be used for files in ${bold('src/content')}.`);
if (skipCount > 10) {
logger.warn(
`Skipped ${green(skippedFiles.length)} files that matched ${green(globOptions.pattern)}.`,
);
} else {
logger.warn(`Skipped the following files that matched ${green(globOptions.pattern)}:`);
skippedFiles.forEach((file) => logger.warn(`• ${green(file)}`));
}
}
// Remove entries that were not found this time
untouchedEntries.forEach((id) => store.delete(id));
if (!watcher) {
return;
}
const matcher: RegExp = micromatch.makeRe(globOptions.pattern);
const matchesGlob = (entry: string) => !entry.startsWith('../') && matcher.test(entry);
const basePath = fileURLToPath(baseDir);
async function onChange(changedPath: string) {
const entry = posixRelative(basePath, changedPath);
if (!matchesGlob(entry)) {
return;
}
const entryType = configForFile(changedPath);
const baseUrl = pathToFileURL(basePath);
await syncData(entry, baseUrl, entryType);
logger.info(`Reloaded data from ${green(entry)}`);
}
watcher.on('change', onChange);
watcher.on('add', onChange);
watcher.on('unlink', async (deletedPath) => {
const entry = posixRelative(basePath, deletedPath);
if (!matchesGlob(entry)) {
return;
}
const id = fileToIdMap.get(deletedPath);
if (id) {
store.delete(id);
fileToIdMap.delete(deletedPath);
}
});
},
};
}

3
packages/astro/src/content/loaders/index.ts Normal file
View file

@ -0,0 +1,3 @@
export { file } from './file.js';
export { glob } from './glob.js';
export * from './types.js';

43
packages/astro/src/content/loaders/types.ts Normal file
View file

@ -0,0 +1,43 @@
import type { FSWatcher } from 'vite';
import type { ZodSchema } from 'zod';
import type { AstroIntegrationLogger, AstroSettings } from '../../@types/astro.js';
import type { MetaStore, ScopedDataStore } from '../data-store.js';
export interface ParseDataOptions<TData extends Record<string, unknown>> {
/** The ID of the entry. Unique per collection */
id: string;
/** The raw, unvalidated data of the entry */
data: TData;
/** An optional file path, where the entry represents a local file. */
filePath?: string;
}
export interface LoaderContext {
/** The unique name of the collection */
collection: string;
/** A database abstraction to store the actual data */
store: ScopedDataStore;
/** A simple KV store, designed for things like sync tokens */
meta: MetaStore;
logger: AstroIntegrationLogger;
settings: AstroSettings;
/** Validates and parses the data according to the collection schema */
parseData<TData extends Record<string, unknown>>(props: ParseDataOptions<TData>): Promise<TData>;
/** Generates a non-cryptographic content digest. This can be used to check if the data has changed */
generateDigest(data: Record<string, unknown> | string): string;
/** When running in dev, this is a filesystem watcher that can be used to trigger updates */
watcher?: FSWatcher;
}
export interface Loader {
/** Unique name of the loader, e.g. the npm package name */
name: string;
/** Do the actual loading of the data */
load: (context: LoaderContext) => Promise<void>;
/** Optionally, define the schema of the data. Will be overridden by user-defined schema */
schema?: ZodSchema | Promise<ZodSchema> | (() => ZodSchema | Promise<ZodSchema>);
}

333
packages/astro/src/content/runtime.ts
View file

@ -1,7 +1,10 @@
import type { MarkdownHeading } from '@astrojs/markdown-remark';
import { Traverse } from 'neotraverse/modern';
import pLimit from 'p-limit';
import { ZodIssueCode, string as zodString } from 'zod';
import { AstroError, AstroErrorData } from '../core/errors/index.js';
import { ZodIssueCode, z } from 'zod';
import type { GetImageResult, ImageMetadata } from '../@types/astro.js';
import { imageSrcToImportId } from '../assets/utils/resolveImports.js';
import { AstroError, AstroErrorData, AstroUserError } from '../core/errors/index.js';
import { prependForwardSlash } from '../core/path.js';
import {
type AstroComponentFactory,
@ -11,8 +14,11 @@ import {
renderScriptElement,
renderTemplate,
renderUniqueStylesheet,
render as serverRender,
unescapeHTML,
} from '../runtime/server/index.js';
import { CONTENT_LAYER_TYPE, IMAGE_IMPORT_PREFIX } from './consts.js';
import { type DataEntry, globalDataStore } from './data-store.js';
import type { ContentLookupMap } from './utils.js';
type LazyImport = () => Promise<any>;
@ -21,6 +27,15 @@ type CollectionToEntryMap = Record<string, GlobResult>;
type GetEntryImport = (collection: string, lookupId: string) => Promise<LazyImport>;
export function defineCollection(config: any) {
if ('loader' in config) {
if (config.type && config.type !== CONTENT_LAYER_TYPE) {
throw new AstroUserError(
'Collections that use the Content Layer API must have a `loader` defined and no `type` set.',
"Check your collection definitions in `src/content/config.*`.'",
);
}
config.type = CONTENT_LAYER_TYPE;
}
if (!config.type) config.type = 'content';
return config;
}
@ -56,11 +71,34 @@ export function createGetCollection({
cacheEntriesByCollection: Map<string, any[]>;
}) {
return async function getCollection(collection: string, filter?: (entry: any) => unknown) {
const hasFilter = typeof filter === 'function';
const store = await globalDataStore.get();
let type: 'content' | 'data';
if (collection in contentCollectionToEntryMap) {
type = 'content';
} else if (collection in dataCollectionToEntryMap) {
type = 'data';
} else if (store.hasCollection(collection)) {
// @ts-expect-error virtual module
const { default: imageAssetMap } = await import('astro:asset-imports');
const result = [];
for (const rawEntry of store.values<DataEntry>(collection)) {
const data = rawEntry.filePath
? updateImageReferencesInData(rawEntry.data, rawEntry.filePath, imageAssetMap)
: rawEntry.data;
const entry = {
...rawEntry,
data,
collection,
};
if (hasFilter && !filter(entry)) {
continue;
}
result.push(entry);
}
return result;
} else {
// eslint-disable-next-line no-console
console.warn(
@ -70,6 +108,7 @@ export function createGetCollection({
);
return [];
}
const lazyImports = Object.values(
type === 'content'
? contentCollectionToEntryMap[collection]
@ -111,7 +150,7 @@ export function createGetCollection({
);
cacheEntriesByCollection.set(collection, entries);
}
if (typeof filter === 'function') {
if (hasFilter) {
return entries.filter(filter);
} else {
// Clone the array so users can safely mutate it.
@ -124,11 +163,27 @@ export function createGetCollection({
export function createGetEntryBySlug({
getEntryImport,
getRenderEntryImport,
collectionNames,
}: {
getEntryImport: GetEntryImport;
getRenderEntryImport: GetEntryImport;
collectionNames: Set<string>;
}) {
return async function getEntryBySlug(collection: string, slug: string) {
const store = await globalDataStore.get();
if (!collectionNames.has(collection)) {
if (store.hasCollection(collection)) {
throw new AstroError({
...AstroErrorData.GetEntryDeprecationError,
message: AstroErrorData.GetEntryDeprecationError.message(collection, 'getEntryBySlug'),
});
}
// eslint-disable-next-line no-console
console.warn(`The collection ${JSON.stringify(collection)} does not exist.`);
return undefined;
}
const entryImport = await getEntryImport(collection, slug);
if (typeof entryImport !== 'function') return undefined;
@ -151,8 +206,28 @@ export function createGetEntryBySlug({
};
}
export function createGetDataEntryById({ getEntryImport }: { getEntryImport: GetEntryImport }) {
export function createGetDataEntryById({
getEntryImport,
collectionNames,
}: {
getEntryImport: GetEntryImport;
collectionNames: Set<string>;
}) {
return async function getDataEntryById(collection: string, id: string) {
const store = await globalDataStore.get();
if (!collectionNames.has(collection)) {
if (store.hasCollection(collection)) {
throw new AstroError({
...AstroErrorData.GetEntryDeprecationError,
message: AstroErrorData.GetEntryDeprecationError.message(collection, 'getDataEntryById'),
});
}
// eslint-disable-next-line no-console
console.warn(`The collection ${JSON.stringify(collection)} does not exist.`);
return undefined;
}
const lazyImport = await getEntryImport(collection, id);
// TODO: AstroError
@ -187,9 +262,11 @@ type EntryLookupObject = { collection: string; id: string } | { collection: stri
export function createGetEntry({
getEntryImport,
getRenderEntryImport,
collectionNames,
}: {
getEntryImport: GetEntryImport;
getRenderEntryImport: GetEntryImport;
collectionNames: Set<string>;
}) {
return async function getEntry(
// Can either pass collection and identifier as 2 positional args,
@ -216,6 +293,33 @@ export function createGetEntry({
: collectionOrLookupObject.slug;
}
const store = await globalDataStore.get();
if (store.hasCollection(collection)) {
const entry = store.get<DataEntry>(collection, lookupId);
if (!entry) {
// eslint-disable-next-line no-console
console.warn(`Entry ${collection} → ${lookupId} was not found.`);
return;
}
if (entry.filePath) {
// @ts-expect-error virtual module
const { default: imageAssetMap } = await import('astro:asset-imports');
entry.data = updateImageReferencesInData(entry.data, entry.filePath, imageAssetMap);
}
return {
...entry,
collection,
} as DataEntryResult | ContentEntryResult;
}
if (!collectionNames.has(collection)) {
// eslint-disable-next-line no-console
console.warn(`The collection ${JSON.stringify(collection)} does not exist.`);
return undefined;
}
const entryImport = await getEntryImport(collection, lookupId);
if (typeof entryImport !== 'function') return undefined;
@ -261,6 +365,115 @@ type RenderResult = {
remarkPluginFrontmatter: Record<string, any>;
};
const CONTENT_LAYER_IMAGE_REGEX = /__ASTRO_IMAGE_="([^"]+)"/g;
async function updateImageReferencesInBody(html: string, fileName: string) {
// @ts-expect-error Virtual module
const { default: imageAssetMap } = await import('astro:asset-imports');
const imageObjects = new Map<string, GetImageResult>();
// @ts-expect-error Virtual module resolved at runtime
const { getImage } = await import('astro:assets');
// First load all the images. This is done outside of the replaceAll
// function because getImage is async.
for (const [_full, imagePath] of html.matchAll(CONTENT_LAYER_IMAGE_REGEX)) {
try {
const decodedImagePath = JSON.parse(imagePath.replaceAll('&#x22;', '"'));
const id = imageSrcToImportId(decodedImagePath.src, fileName);
const imported = imageAssetMap.get(id);
if (!id || imageObjects.has(id) || !imported) {
continue;
}
const image: GetImageResult = await getImage({ ...decodedImagePath, src: imported });
imageObjects.set(imagePath, image);
} catch {
throw new Error(`Failed to parse image reference: ${imagePath}`);
}
}
return html.replaceAll(CONTENT_LAYER_IMAGE_REGEX, (full, imagePath) => {
const image = imageObjects.get(imagePath);
if (!image) {
return full;
}
const { index, ...attributes } = image.attributes;
return Object.entries({
...attributes,
src: image.src,
srcset: image.srcSet.attribute,
})
.map(([key, value]) => (value ? `${key}=${JSON.stringify(String(value))}` : ''))
.join(' ');
});
}
function updateImageReferencesInData<T extends Record<string, unknown>>(
data: T,
fileName: string,
imageAssetMap: Map<string, ImageMetadata>,
): T {
return new Traverse(data).map(function (ctx, val) {
if (typeof val === 'string' && val.startsWith(IMAGE_IMPORT_PREFIX)) {
const src = val.replace(IMAGE_IMPORT_PREFIX, '');
const id = imageSrcToImportId(src, fileName);
if (!id) {
ctx.update(src);
return;
}
const imported = imageAssetMap.get(id);
if (imported) {
ctx.update(imported);
} else {
ctx.update(src);
}
}
});
}
export async function renderEntry(
entry: DataEntry | { render: () => Promise<{ Content: AstroComponentFactory }> },
) {
if (entry && 'render' in entry) {
// This is an old content collection entry, so we use its render method
return entry.render();
}
if (entry.deferredRender) {
try {
// @ts-expect-error virtual module
const { default: contentModules } = await import('astro:content-module-imports');
const module = contentModules.get(entry.filePath);
const deferredMod = await module();
return {
Content: deferredMod.Content,
headings: deferredMod.getHeadings?.() ?? [],
remarkPluginFrontmatter: deferredMod.frontmatter ?? {},
};
} catch (e) {
// eslint-disable-next-line
console.error(e);
}
}
const html =
entry?.rendered?.metadata?.imagePaths?.length && entry.filePath
? await updateImageReferencesInBody(entry.rendered.html, entry.filePath)
: entry?.rendered?.html;
const Content = createComponent(() => serverRender`${unescapeHTML(html)}`);
return {
Content,
headings: entry?.rendered?.metadata?.headings ?? [],
remarkPluginFrontmatter: entry?.rendered?.metadata?.frontmatter ?? {},
};
}
async function render({
collection,
id,
@ -357,36 +570,92 @@ async function render({
export function createReference({ lookupMap }: { lookupMap: ContentLookupMap }) {
return function reference(collection: string) {
return zodString().transform((lookupId: string, ctx) => {
const flattenedErrorPath = ctx.path.join('.');
if (!lookupMap[collection]) {
ctx.addIssue({
code: ZodIssueCode.custom,
message: `**${flattenedErrorPath}:** Reference to ${collection} invalid. Collection does not exist or is empty.`,
});
return;
}
return z
.union([
z.string(),
z.object({
id: z.string(),
collection: z.string(),
}),
z.object({
slug: z.string(),
collection: z.string(),
}),
])
.transform(
async (
lookup:
| string
| { id: string; collection: string }
| { slug: string; collection: string },
ctx,
) => {
const flattenedErrorPath = ctx.path.join('.');
const store = await globalDataStore.get();
const collectionIsInStore = store.hasCollection(collection);
const { type, entries } = lookupMap[collection];
const entry = entries[lookupId];
if (typeof lookup === 'object') {
// If these don't match then something is wrong with the reference
if (lookup.collection !== collection) {
ctx.addIssue({
code: ZodIssueCode.custom,
message: `**${flattenedErrorPath}**: Reference to ${collection} invalid. Expected ${collection}. Received ${lookup.collection}.`,
});
return;
}
if (!entry) {
ctx.addIssue({
code: ZodIssueCode.custom,
message: `**${flattenedErrorPath}**: Reference to ${collection} invalid. Expected ${Object.keys(
entries,
)
.map((c) => JSON.stringify(c))
.join(' | ')}. Received ${JSON.stringify(lookupId)}.`,
});
return;
}
// Content is still identified by slugs, so map to a `slug` key for consistency.
if (type === 'content') {
return { slug: lookupId, collection };
}
return { id: lookupId, collection };
});
// A reference object might refer to an invalid collection, because when we convert it we don't have access to the store.
// If it is an object then we're validating later in the pipeline, so we can check the collection at that point.
if (!lookupMap[collection] && !collectionIsInStore) {
ctx.addIssue({
code: ZodIssueCode.custom,
message: `**${flattenedErrorPath}:** Reference to ${collection} invalid. Collection does not exist or is empty.`,
});
return;
}
return lookup;
}
if (collectionIsInStore) {
const entry = store.get(collection, lookup);
if (!entry) {
ctx.addIssue({
code: ZodIssueCode.custom,
message: `**${flattenedErrorPath}**: Reference to ${collection} invalid. Entry ${lookup} does not exist.`,
});
return;
}
return { id: lookup, collection };
}
if (!lookupMap[collection] && store.collections().size === 0) {
// If the collection is not in the lookup map or store, it may be a content layer collection and the store may not yet be populated.
// For now, we can't validate this reference, so we'll optimistically convert it to a reference object which we'll validate
// later in the pipeline when we do have access to the store.
return { id: lookup, collection };
}
const { type, entries } = lookupMap[collection];
const entry = entries[lookup];
if (!entry) {
ctx.addIssue({
code: ZodIssueCode.custom,
message: `**${flattenedErrorPath}**: Reference to ${collection} invalid. Expected ${Object.keys(
entries,
)
.map((c) => JSON.stringify(c))
.join(' | ')}. Received ${JSON.stringify(lookup)}.`,
});
return;
}
// Content is still identified by slugs, so map to a `slug` key for consistency.
if (type === 'content') {
return { slug: lookup, collection };
}
return { id: lookup, collection };
},
);
};
}

217
packages/astro/src/content/types-generator.ts
View file

@ -1,18 +1,20 @@
import glob from 'fast-glob';
import { bold, cyan } from 'kleur/colors';
import type fsMod from 'node:fs';
import * as path from 'node:path';
import { fileURLToPath, pathToFileURL } from 'node:url';
import glob from 'fast-glob';
import { bold, cyan } from 'kleur/colors';
import { type ViteDevServer, normalizePath } from 'vite';
import { z } from 'zod';
import { z, type ZodSchema } from 'zod';
import { zodToJsonSchema } from 'zod-to-json-schema';
import { printNode, zodToTs } from 'zod-to-ts';
import type { AstroSettings, ContentEntryType } from '../@types/astro.js';
import { AstroError } from '../core/errors/errors.js';
import { AstroErrorData } from '../core/errors/index.js';
import type { Logger } from '../core/logger/core.js';
import { isRelativePath } from '../core/path.js';
import { CONTENT_TYPES_FILE, VIRTUAL_MODULE_ID } from './consts.js';
import { CONTENT_LAYER_TYPE, CONTENT_TYPES_FILE, VIRTUAL_MODULE_ID } from './consts.js';
import {
type CollectionConfig,
type ContentConfig,
type ContentObservable,
type ContentPaths,
@ -44,7 +46,7 @@ type CollectionEntryMap = {
entries: Record<string, ContentEntryMetadata>;
}
| {
type: 'data';
type: 'data' | typeof CONTENT_LAYER_TYPE;
entries: Record<string, DataEntryMetadata>;
};
};
@ -245,7 +247,7 @@ export async function createContentTypesGenerator({
collectionEntryMap[collectionKey] = {
type: 'content',
entries: {
...collectionInfo.entries,
...(collectionInfo.entries as Record<string, ContentEntryMetadata>),
[entryKey]: { slug: addedSlug },
},
};
@ -356,6 +358,51 @@ function normalizeConfigPath(from: string, to: string) {
return `"${isRelativePath(configPath) ? '' : './'}${normalizedPath}"` as const;
}
const schemaCache = new Map<string, ZodSchema>();
async function getContentLayerSchema<T extends keyof ContentConfig['collections']>(
collection: ContentConfig['collections'][T],
collectionKey: T,
): Promise<ZodSchema | undefined> {
const cached = schemaCache.get(collectionKey);
if (cached) {
return cached;
}
if (
collection?.type === CONTENT_LAYER_TYPE &&
typeof collection.loader === 'object' &&
collection.loader.schema
) {
let schema = collection.loader.schema;
if (typeof schema === 'function') {
schema = await schema();
}
if (schema) {
schemaCache.set(collectionKey, await schema);
return schema;
}
}
}
async function typeForCollection<T extends keyof ContentConfig['collections']>(
collection: ContentConfig['collections'][T] | undefined,
collectionKey: T,
): Promise<string> {
if (collection?.schema) {
return `InferEntrySchema<${collectionKey}>`;
}
if (collection?.type === CONTENT_LAYER_TYPE) {
const schema = await getContentLayerSchema(collection, collectionKey);
if (schema) {
const ast = zodToTs(schema);
return printNode(ast.node);
}
}
return 'any';
}
async function writeContentFiles({
fs,
contentPaths,
@ -391,12 +438,15 @@ async function writeContentFiles({
entries: {},
};
}
let contentCollectionsMap: CollectionEntryMap = {};
for (const collectionKey of Object.keys(collectionEntryMap).sort()) {
const collectionConfig = contentConfig?.collections[JSON.parse(collectionKey)];
const collection = collectionEntryMap[collectionKey];
if (
collectionConfig?.type &&
collection.type !== 'unknown' &&
collectionConfig.type !== CONTENT_LAYER_TYPE &&
collection.type !== collectionConfig.type
) {
viteServer.hot.send({
@ -419,15 +469,15 @@ async function writeContentFiles({
});
return;
}
const resolvedType: 'content' | 'data' =
const resolvedType =
collection.type === 'unknown'
? // Add empty / unknown collections to the data type map by default
// This ensures `getCollection('empty-collection')` doesn't raise a type error
collectionConfig?.type ?? 'data'
(collectionConfig?.type ?? 'data')
: collection.type;
const collectionEntryKeys = Object.keys(collection.entries).sort();
const dataType = collectionConfig?.schema ? `InferEntrySchema<${collectionKey}>` : 'any';
const dataType = await typeForCollection(collectionConfig, collectionKey);
switch (resolvedType) {
case 'content':
if (collectionEntryKeys.length === 0) {
@ -446,6 +496,9 @@ async function writeContentFiles({
}
contentTypesStr += `};\n`;
break;
case CONTENT_LAYER_TYPE:
dataTypesStr += `${collectionKey}: Record<string, {\n id: string;\n collection: ${collectionKey};\n data: ${dataType};\n rendered?: RenderedContent \n}>;\n`;
break;
case 'data':
if (collectionEntryKeys.length === 0) {
dataTypesStr += `${collectionKey}: Record<string, {\n id: string;\n collection: ${collectionKey};\n data: ${dataType};\n}>;\n`;
@ -458,40 +511,60 @@ async function writeContentFiles({
}
if (collectionConfig?.schema) {
let zodSchemaForJson =
typeof collectionConfig.schema === 'function'
? collectionConfig.schema({ image: () => z.string() })
: collectionConfig.schema;
if (zodSchemaForJson instanceof z.ZodObject) {
zodSchemaForJson = zodSchemaForJson.extend({
$schema: z.string().optional(),
});
}
try {
await fs.promises.writeFile(
new URL(`./${collectionKey.replace(/"/g, '')}.schema.json`, collectionSchemasDir),
JSON.stringify(
zodToJsonSchema(zodSchemaForJson, {
name: collectionKey.replace(/"/g, ''),
markdownDescription: true,
errorMessages: true,
// Fix for https://github.com/StefanTerdell/zod-to-json-schema/issues/110
dateStrategy: ['format:date-time', 'format:date', 'integer'],
}),
null,
2,
),
);
} catch (err) {
// This should error gracefully and not crash the dev server
logger.warn(
'content',
`An error was encountered while creating the JSON schema for the ${collectionKey} collection. Proceeding without it. Error: ${err}`,
);
}
await generateJSONSchema(
fs,
collectionConfig,
collectionKey,
collectionSchemasDir,
logger,
);
}
break;
}
if (
settings.config.experimental.contentIntellisense &&
collectionConfig &&
(collectionConfig.schema || (await getContentLayerSchema(collectionConfig, collectionKey)))
) {
await generateJSONSchema(fs, collectionConfig, collectionKey, collectionSchemasDir, logger);
contentCollectionsMap[collectionKey] = collection;
}
}
if (settings.config.experimental.contentIntellisense) {
let contentCollectionManifest: {
collections: { hasSchema: boolean; name: string }[];
entries: Record<string, string>;
} = {
collections: [],
entries: {},
};
Object.entries(contentCollectionsMap).forEach(([collectionKey, collection]) => {
const collectionConfig = contentConfig?.collections[JSON.parse(collectionKey)];
const key = JSON.parse(collectionKey);
contentCollectionManifest.collections.push({
hasSchema: Boolean(collectionConfig?.schema || schemaCache.has(collectionKey)),
name: key,
});
Object.keys(collection.entries).forEach((entryKey) => {
const entryPath = new URL(
JSON.parse(entryKey),
contentPaths.contentDir + `${key}/`,
).toString();
// Save entry path in lower case to avoid case sensitivity issues between Windows and Unix
contentCollectionManifest.entries[entryPath.toLowerCase()] = key;
});
});
await fs.promises.writeFile(
new URL('./collections.json', collectionSchemasDir),
JSON.stringify(contentCollectionManifest, null, 2),
);
}
if (!fs.existsSync(settings.dotAstroDir)) {
@ -499,7 +572,7 @@ async function writeContentFiles({
}
const configPathRelativeToCacheDir = normalizeConfigPath(
settings.dotAstroDir.pathname,
new URL('astro', settings.dotAstroDir).pathname,
contentPaths.config.url.pathname,
);
@ -515,8 +588,62 @@ async function writeContentFiles({
contentConfig ? `typeof import(${configPathRelativeToCacheDir})` : 'never',
);
await fs.promises.writeFile(
new URL(CONTENT_TYPES_FILE, settings.dotAstroDir),
typeTemplateContent,
);
// If it's the first time, we inject types the usual way. sync() will handle creating files and references. If it's not the first time, we just override the dts content
if (settings.injectedTypes.some((t) => t.filename === CONTENT_TYPES_FILE)) {
fs.promises.writeFile(
new URL(CONTENT_TYPES_FILE, settings.dotAstroDir),
typeTemplateContent,
'utf-8',
);
} else {
settings.injectedTypes.push({
filename: CONTENT_TYPES_FILE,
content: typeTemplateContent,
});
}
}
async function generateJSONSchema(
fsMod: typeof import('node:fs'),
collectionConfig: CollectionConfig,
collectionKey: string,
collectionSchemasDir: URL,
logger: Logger,
) {
let zodSchemaForJson =
typeof collectionConfig.schema === 'function'
? collectionConfig.schema({ image: () => z.string() })
: collectionConfig.schema;
if (!zodSchemaForJson && collectionConfig.type === CONTENT_LAYER_TYPE) {
zodSchemaForJson = await getContentLayerSchema(collectionConfig, collectionKey);
}
if (zodSchemaForJson instanceof z.ZodObject) {
zodSchemaForJson = zodSchemaForJson.extend({
$schema: z.string().optional(),
});
}
try {
await fsMod.promises.writeFile(
new URL(`./${collectionKey.replace(/"/g, '')}.schema.json`, collectionSchemasDir),
JSON.stringify(
zodToJsonSchema(zodSchemaForJson, {
name: collectionKey.replace(/"/g, ''),
markdownDescription: true,
errorMessages: true,
// Fix for https://github.com/StefanTerdell/zod-to-json-schema/issues/110
dateStrategy: ['format:date-time', 'format:date', 'integer'],
}),
null,
2,
),
);
} catch (err) {
// This should error gracefully and not crash the dev server
logger.warn(
'content',
`An error was encountered while creating the JSON schema for the ${collectionKey} collection. Proceeding without it. Error: ${err}`,
);
}
}

144
packages/astro/src/content/utils.ts
View file

@ -5,6 +5,7 @@ import { slug as githubSlug } from 'github-slugger';
import matter from 'gray-matter';
import type { PluginContext } from 'rollup';
import { type ViteDevServer, normalizePath } from 'vite';
import xxhash from 'xxhash-wasm';
import { z } from 'zod';
import type {
AstroConfig,
@ -15,7 +16,13 @@ import type {
import { AstroError, AstroErrorData, MarkdownError, errorMap } from '../core/errors/index.js';
import { isYAMLException } from '../core/errors/utils.js';
import type { Logger } from '../core/logger/core.js';
import { CONTENT_FLAGS, PROPAGATED_ASSET_FLAG } from './consts.js';
import {
CONTENT_FLAGS,
CONTENT_LAYER_TYPE,
CONTENT_MODULE_FLAG,
IMAGE_IMPORT_PREFIX,
PROPAGATED_ASSET_FLAG,
} from './consts.js';
import { createImage } from './runtime-assets.js';
/**
* Amap from a collection + slug to the local file path.
@ -35,6 +42,54 @@ const collectionConfigParser = z.union([
type: z.literal('data'),
schema: z.any().optional(),
}),
z.object({
type: z.literal(CONTENT_LAYER_TYPE),
schema: z.any().optional(),
loader: z.union([
z.function().returns(
z.union([
z.array(
z
.object({
id: z.string(),
})
.catchall(z.unknown()),
),
z.promise(
z.array(
z
.object({
id: z.string(),
})
.catchall(z.unknown()),
),
),
]),
),
z.object({
name: z.string(),
load: z.function(
z.tuple(
[
z.object({
collection: z.string(),
store: z.any(),
meta: z.any(),
logger: z.any(),
settings: z.any(),
parseData: z.any(),
generateDigest: z.function(z.tuple([z.any()], z.string())),
watcher: z.any().optional(),
}),
],
z.unknown(),
),
),
schema: z.any().optional(),
render: z.function(z.tuple([z.any()], z.unknown())).optional(),
}),
]),
}),
]);
const contentConfigParser = z.object({
@ -42,7 +97,7 @@ const contentConfigParser = z.object({
});
export type CollectionConfig = z.infer<typeof collectionConfigParser>;
export type ContentConfig = z.infer<typeof contentConfigParser>;
export type ContentConfig = z.infer<typeof contentConfigParser> & { digest?: string };
type EntryInternal = { rawData: string | undefined; filePath: string };
@ -67,30 +122,46 @@ export function parseEntrySlug({
}
}
export async function getEntryData(
export async function getEntryDataAndImages<
TInputData extends Record<string, unknown> = Record<string, unknown>,
TOutputData extends TInputData = TInputData,
>(
entry: {
id: string;
collection: string;
unvalidatedData: Record<string, unknown>;
unvalidatedData: TInputData;
_internal: EntryInternal;
},
collectionConfig: CollectionConfig,
shouldEmitFile: boolean,
pluginContext: PluginContext,
) {
let data;
if (collectionConfig.type === 'data') {
data = entry.unvalidatedData;
pluginContext?: PluginContext,
): Promise<{ data: TOutputData; imageImports: Array<string> }> {
let data: TOutputData;
if (collectionConfig.type === 'data' || collectionConfig.type === CONTENT_LAYER_TYPE) {
data = entry.unvalidatedData as TOutputData;
} else {
const { slug, ...unvalidatedData } = entry.unvalidatedData;
data = unvalidatedData;
data = unvalidatedData as TOutputData;
}
let schema = collectionConfig.schema;
const imageImports = new Set<string>();
if (typeof schema === 'function') {
schema = schema({
image: createImage(pluginContext, shouldEmitFile, entry._internal.filePath),
});
if (pluginContext) {
schema = schema({
image: createImage(pluginContext, shouldEmitFile, entry._internal.filePath),
});
} else if (collectionConfig.type === CONTENT_LAYER_TYPE) {
schema = schema({
image: () =>
z.string().transform((val) => {
imageImports.add(val);
return `${IMAGE_IMPORT_PREFIX}${val}`;
}),
});
}
}
if (schema) {
@ -119,7 +190,7 @@ export async function getEntryData(
},
});
if (parsed.success) {
data = parsed.data as Record<string, unknown>;
data = parsed.data as TOutputData;
} else {
if (!formattedError) {
formattedError = new AstroError({
@ -139,6 +210,27 @@ export async function getEntryData(
throw formattedError;
}
}
return { data, imageImports: Array.from(imageImports) };
}
export async function getEntryData(
entry: {
id: string;
collection: string;
unvalidatedData: Record<string, unknown>;
_internal: EntryInternal;
},
collectionConfig: CollectionConfig,
shouldEmitFile: boolean,
pluginContext?: PluginContext,
) {
const { data } = await getEntryDataAndImages(
entry,
collectionConfig,
shouldEmitFile,
pluginContext,
);
return data;
}
@ -383,6 +475,11 @@ export function hasContentFlag(viteId: string, flag: (typeof CONTENT_FLAGS)[numb
return flags.has(flag);
}
export function isDeferredModule(viteId: string): boolean {
const flags = new URLSearchParams(viteId.split('?')[1] ?? '');
return flags.has(CONTENT_MODULE_FLAG);
}
async function loadContentConfig({
fs,
settings,
@ -402,7 +499,10 @@ async function loadContentConfig({
const config = contentConfigParser.safeParse(unparsedConfig);
if (config.success) {
return config.data;
// Generate a digest of the config file so we can invalidate the cache if it changes
const hasher = await xxhash();
const digest = await hasher.h64ToString(await fs.promises.readFile(configPathname, 'utf-8'));
return { ...config.data, digest };
} else {
return undefined;
}
@ -556,3 +656,17 @@ export function hasAssetPropagationFlag(id: string): boolean {
return false;
}
}
/**
* Convert a platform path to a posix path.
*/
export function posixifyPath(filePath: string) {
return filePath.split(path.sep).join('/');
}
/**
* Unlike `path.posix.relative`, this function will accept a platform path and return a posix path.
*/
export function posixRelative(from: string, to: string) {
return posixifyPath(path.relative(from, to));
}

14
packages/astro/src/content/vite-plugin-content-assets.ts
View file

@ -1,5 +1,5 @@
import { extname } from 'node:path';
import { pathToFileURL } from 'node:url';
import { fileURLToPath, pathToFileURL } from 'node:url';
import type { Plugin } from 'vite';
import type { AstroSettings, SSRElement } from '../@types/astro.js';
import { getAssetsPrefix } from '../assets/utils/getAssetsPrefix.js';
@ -12,6 +12,7 @@ import { joinPaths, prependForwardSlash } from '../core/path.js';
import { getStylesForURL } from '../vite-plugin-astro-server/css.js';
import { getScriptsForURL } from '../vite-plugin-astro-server/scripts.js';
import {
CONTENT_IMAGE_FLAG,
CONTENT_RENDER_FLAG,
LINKS_PLACEHOLDER,
PROPAGATED_ASSET_FLAG,
@ -32,6 +33,17 @@ export function astroContentAssetPropagationPlugin({
name: 'astro:content-asset-propagation',
enforce: 'pre',
async resolveId(id, importer, opts) {
if (hasContentFlag(id, CONTENT_IMAGE_FLAG)) {
const [base, query] = id.split('?');
const params = new URLSearchParams(query);
const importerParam = params.get('importer');
const importerPath = importerParam
? fileURLToPath(new URL(importerParam, settings.config.root))
: importer;
return this.resolve(base, importerPath, { skipSelf: true, ...opts });
}
if (hasContentFlag(id, CONTENT_RENDER_FLAG)) {
const base = id.split('?')[0];

1
packages/astro/src/content/vite-plugin-content-imports.ts
View file

@ -158,6 +158,7 @@ export const _internal = {
// The content config could depend on collection entries via `reference()`.
// Reload the config in case of changes.
// Changes to the config file itself are handled in types-generator.ts, so we skip them here
if (entryType === 'content' || entryType === 'data') {
await reloadContentConfigObserver({ fs, settings, viteServer });
}

106
packages/astro/src/content/vite-plugin-content-virtual-mod.ts
View file

@ -1,6 +1,7 @@
import nodeFs from 'node:fs';
import { extname } from 'node:path';
import { fileURLToPath, pathToFileURL } from 'node:url';
import { dataToEsm } from '@rollup/pluginutils';
import glob from 'fast-glob';
import pLimit from 'p-limit';
import type { Plugin } from 'vite';
@ -13,9 +14,18 @@ import { rootRelativePath } from '../core/viteUtils.js';
import type { AstroPluginMetadata } from '../vite-plugin-astro/index.js';
import { createDefaultAstroMetadata } from '../vite-plugin-astro/metadata.js';
import {
ASSET_IMPORTS_FILE,
ASSET_IMPORTS_RESOLVED_STUB_ID,
ASSET_IMPORTS_VIRTUAL_ID,
CONTENT_FLAG,
CONTENT_RENDER_FLAG,
DATA_FLAG,
DATA_STORE_FILE,
DATA_STORE_VIRTUAL_ID,
MODULES_IMPORTS_FILE,
MODULES_MJS_ID,
MODULES_MJS_VIRTUAL_ID,
RESOLVED_DATA_STORE_VIRTUAL_ID,
RESOLVED_VIRTUAL_MODULE_ID,
VIRTUAL_MODULE_ID,
} from './consts.js';
@ -30,6 +40,7 @@ import {
getEntrySlug,
getEntryType,
getExtGlob,
isDeferredModule,
} from './utils.js';
interface AstroContentVirtualModPluginParams {
@ -43,13 +54,14 @@ export function astroContentVirtualModPlugin({
}: AstroContentVirtualModPluginParams): Plugin {
let IS_DEV = false;
const IS_SERVER = isServerLikeOutput(settings.config);
const dataStoreFile = new URL(DATA_STORE_FILE, settings.config.cacheDir);
return {
name: 'astro-content-virtual-mod-plugin',
enforce: 'pre',
configResolved(config) {
IS_DEV = config.mode === 'development';
},
resolveId(id) {
async resolveId(id) {
if (id === VIRTUAL_MODULE_ID) {
if (!settings.config.experimental.contentCollectionCache) {
return RESOLVED_VIRTUAL_MODULE_ID;
@ -61,6 +73,38 @@ export function astroContentVirtualModPlugin({
return { id: RESOLVED_VIRTUAL_MODULE_ID, external: true };
}
}
if (id === DATA_STORE_VIRTUAL_ID) {
return RESOLVED_DATA_STORE_VIRTUAL_ID;
}
if (isDeferredModule(id)) {
const [, query] = id.split('?');
const params = new URLSearchParams(query);
const fileName = params.get('fileName');
let importerPath = undefined;
if (fileName && URL.canParse(fileName, settings.config.root.toString())) {
importerPath = fileURLToPath(new URL(fileName, settings.config.root));
}
if (importerPath) {
return await this.resolve(importerPath);
}
}
if (id === MODULES_MJS_ID) {
const modules = new URL(MODULES_IMPORTS_FILE, settings.dotAstroDir);
if (fs.existsSync(modules)) {
return fileURLToPath(modules);
}
return MODULES_MJS_VIRTUAL_ID;
}
if (id === ASSET_IMPORTS_VIRTUAL_ID) {
const assetImportsFile = new URL(ASSET_IMPORTS_FILE, settings.dotAstroDir);
if (fs.existsSync(assetImportsFile)) {
return fileURLToPath(assetImportsFile);
}
return ASSET_IMPORTS_RESOLVED_STUB_ID;
}
},
async load(id, args) {
if (id === RESOLVED_VIRTUAL_MODULE_ID) {
@ -87,6 +131,41 @@ export function astroContentVirtualModPlugin({
} satisfies AstroPluginMetadata,
};
}
if (id === RESOLVED_DATA_STORE_VIRTUAL_ID) {
if (!fs.existsSync(dataStoreFile)) {
return 'export default new Map()';
}
const jsonData = await fs.promises.readFile(dataStoreFile, 'utf-8');
try {
const parsed = JSON.parse(jsonData);
return {
code: dataToEsm(parsed, {
compact: true,
}),
map: { mappings: '' },
};
} catch (err) {
const message = 'Could not parse JSON file';
this.error({ message, id, cause: err });
}
}
if (id === ASSET_IMPORTS_RESOLVED_STUB_ID) {
const assetImportsFile = new URL(ASSET_IMPORTS_FILE, settings.dotAstroDir);
if (!fs.existsSync(assetImportsFile)) {
return 'export default new Map()';
}
return fs.readFileSync(assetImportsFile, 'utf-8');
}
if (id === MODULES_MJS_VIRTUAL_ID) {
const modules = new URL(MODULES_IMPORTS_FILE, settings.dotAstroDir);
if (!fs.existsSync(modules)) {
return 'export default new Map()';
}
return fs.readFileSync(modules, 'utf-8');
}
},
renderChunk(code, chunk) {
if (!settings.config.experimental.contentCollectionCache) {
@ -98,6 +177,31 @@ export function astroContentVirtualModPlugin({
return code.replaceAll(RESOLVED_VIRTUAL_MODULE_ID, `${prefix}content/entry.mjs`);
}
},
configureServer(server) {
const dataStorePath = fileURLToPath(dataStoreFile);
// Watch for changes to the data store file
if (Array.isArray(server.watcher.options.ignored)) {
// The data store file is in node_modules, so is ignored by default,
// so we need to un-ignore it.
server.watcher.options.ignored.push(`!${dataStorePath}`);
}
server.watcher.add(dataStorePath);
server.watcher.on('change', (changedPath) => {
// If the datastore file changes, invalidate the virtual module
if (changedPath === dataStorePath) {
const module = server.moduleGraph.getModuleById(RESOLVED_DATA_STORE_VIRTUAL_ID);
if (module) {
server.moduleGraph.invalidateModule(module);
}
server.ws.send({
type: 'full-reload',
path: '*',
});
}
});
},
};
}

34
packages/astro/src/core/build/index.ts
View file

@ -29,12 +29,12 @@ import { levels, timerMessage } from '../logger/core.js';
import { apply as applyPolyfill } from '../polyfill.js';
import { createRouteManifest } from '../routing/index.js';
import { getServerIslandRouteData } from '../server-islands/endpoint.js';
import { clearContentLayerCache } from '../sync/index.js';
import { ensureProcessNodeEnv, isServerLikeOutput } from '../util.js';
import { collectPagesData } from './page-data.js';
import { staticBuild, viteBuild } from './static-build.js';
import type { StaticBuildOptions } from './types.js';
import { getTimeStat } from './util.js';
export interface BuildOptions {
/**
* Teardown the compiler WASM instance after build. This can improve performance when
@ -44,14 +44,6 @@ export interface BuildOptions {
* @default true
*/
teardownCompiler?: boolean;
/**
* If `experimental.contentCollectionCache` is enabled, this flag will clear the cache before building
*
* @internal not part of our public api
* @default false
*/
force?: boolean;
}
/**
@ -69,13 +61,16 @@ export default async function build(
const logger = createNodeLogger(inlineConfig);
const { userConfig, astroConfig } = await resolveConfig(inlineConfig, 'build');
telemetry.record(eventCliSession('build', userConfig));
if (astroConfig.experimental.contentCollectionCache && options.force) {
const contentCacheDir = new URL('./content/', astroConfig.cacheDir);
if (fs.existsSync(contentCacheDir)) {
logger.debug('content', 'clearing content cache');
await fs.promises.rm(contentCacheDir, { force: true, recursive: true });
logger.warn('content', 'content cache cleared (force)');
if (inlineConfig.force) {
if (astroConfig.experimental.contentCollectionCache) {
const contentCacheDir = new URL('./content/', astroConfig.cacheDir);
if (fs.existsSync(contentCacheDir)) {
logger.debug('content', 'clearing content cache');
await fs.promises.rm(contentCacheDir, { force: true, recursive: true });
logger.warn('content', 'content cache cleared (force)');
}
}
await clearContentLayerCache({ astroConfig, logger, fs });
}
const settings = await createSettings(astroConfig, fileURLToPath(astroConfig.root));
@ -243,18 +238,21 @@ class AstroBuilder {
buildMode: this.settings.config.output,
});
}
// Benchmark results
this.settings.timer.writeStats();
}
/** Build the given Astro project. */
async run() {
this.settings.timer.start('Total build');
const setupData = await this.setup();
try {
await this.build(setupData);
} catch (_err) {
throw _err;
} finally {
this.settings.timer.end('Total build');
// Benchmark results
this.settings.timer.writeStats();
}
}

7
packages/astro/src/core/config/schema.ts
View file

@ -89,9 +89,11 @@ export const ASTRO_CONFIG_DEFAULTS = {
clientPrerender: false,
globalRoutePriority: false,
serverIslands: false,
contentIntellisense: false,
env: {
validateSecrets: false,
},
contentLayer: false,
},
} satisfies AstroUserConfig & { server: { open: boolean } };
@ -538,6 +540,11 @@ export const AstroConfigSchema = z.object({
.boolean()
.optional()
.default(ASTRO_CONFIG_DEFAULTS.experimental.serverIslands),
contentIntellisense: z
.boolean()
.optional()
.default(ASTRO_CONFIG_DEFAULTS.experimental.contentIntellisense),
contentLayer: z.boolean().optional().default(ASTRO_CONFIG_DEFAULTS.experimental.contentLayer),
})
.strict(
`Invalid or outdated experimental feature.\nCheck for incorrect spelling or outdated Astro version.\nSee https://docs.astro.build/en/reference/configuration-reference/#experimental-flags for a list of all current experiments.`,

6
packages/astro/src/core/config/settings.ts
View file

@ -14,7 +14,8 @@ import { loadTSConfig } from './tsconfig.js';
export function createBaseSettings(config: AstroConfig): AstroSettings {
const { contentDir } = getContentPaths(config);
const preferences = createPreferences(config);
const dotAstroDir = new URL('.astro/', config.root);
const preferences = createPreferences(config, dotAstroDir);
return {
config,
preferences,
@ -106,8 +107,9 @@ export function createBaseSettings(config: AstroConfig): AstroSettings {
watchFiles: [],
devToolbarApps: [],
timer: new AstroTimer(),
dotAstroDir: new URL('.astro/', config.root),
dotAstroDir,
latestAstroVersion: undefined, // Will be set later if applicable when the dev server starts
injectedTypes: [],
};
}

1
packages/astro/src/core/dev/container.ts
View file

@ -97,6 +97,7 @@ export async function createContainer({
skip: {
content: true,
},
force: inlineConfig?.force,
});
const viteServer = await vite.createServer(viteConfig);

36
packages/astro/src/core/dev/dev.ts
View file

@ -1,4 +1,4 @@
import fs from 'node:fs';
import fs, { existsSync } from 'node:fs';
import type http from 'node:http';
import type { AddressInfo } from 'node:net';
import { green } from 'kleur/colors';
@ -6,7 +6,11 @@ import { performance } from 'perf_hooks';
import { gt, major, minor, patch } from 'semver';
import type * as vite from 'vite';
import type { AstroInlineConfig } from '../../@types/astro.js';
import { DATA_STORE_FILE } from '../../content/consts.js';
import { globalContentLayer } from '../../content/content-layer.js';
import { DataStore, globalDataStore } from '../../content/data-store.js';
import { attachContentServerListeners } from '../../content/index.js';
import { globalContentConfigObserver } from '../../content/utils.js';
import { telemetry } from '../../events/index.js';
import * as msg from '../messages.js';
import { ensureProcessNodeEnv } from '../util.js';
@ -102,6 +106,36 @@ export default async function dev(inlineConfig: AstroInlineConfig): Promise<DevS
await attachContentServerListeners(restart.container);
let store: DataStore | undefined;
try {
const dataStoreFile = new URL(DATA_STORE_FILE, restart.container.settings.config.cacheDir);
if (existsSync(dataStoreFile)) {
store = await DataStore.fromFile(dataStoreFile);
globalDataStore.set(store);
}
} catch (err: any) {
logger.error('content', err.message);
}
if (!store) {
store = new DataStore();
globalDataStore.set(store);
}
const config = globalContentConfigObserver.get();
if (config.status === 'error') {
logger.error('content', config.error.message);
}
if (config.status === 'loaded') {
const contentLayer = globalContentLayer.init({
settings: restart.container.settings,
logger,
watcher: restart.container.viteServer.watcher,
store,
});
contentLayer.watchContentConfig();
await contentLayer.sync();
}
logger.info(null, green('watching for file changes...'));
return {

40
packages/astro/src/core/dev/restart.ts
View file

@ -2,6 +2,7 @@ import type nodeFs from 'node:fs';
import { fileURLToPath } from 'node:url';
import * as vite from 'vite';
import type { AstroInlineConfig, AstroSettings } from '../../@types/astro.js';
import { globalContentLayer } from '../../content/content-layer.js';
import { eventCliSession, telemetry } from '../../events/index.js';
import { createNodeLogger, createSettings, resolveConfig } from '../config/index.js';
import { collectErrorMetadata } from '../errors/dev/utils.js';
@ -30,7 +31,6 @@ async function createRestartedContainer(
}
const configRE = /.*astro.config.(?:mjs|cjs|js|ts)$/;
const preferencesRE = /.*\.astro\/settings.json$/;
function shouldRestartContainer(
{ settings, inlineConfig, restartInFlight }: Container,
@ -39,17 +39,19 @@ function shouldRestartContainer(
if (restartInFlight) return false;
let shouldRestart = false;
const normalizedChangedFile = vite.normalizePath(changedFile);
// If the config file changed, reload the config and restart the server.
if (inlineConfig.configFile) {
shouldRestart = vite.normalizePath(inlineConfig.configFile) === vite.normalizePath(changedFile);
shouldRestart = vite.normalizePath(inlineConfig.configFile) === normalizedChangedFile;
}
// Otherwise, watch for any astro.config.* file changes in project root
else {
const normalizedChangedFile = vite.normalizePath(changedFile);
shouldRestart = configRE.test(normalizedChangedFile);
if (preferencesRE.test(normalizedChangedFile)) {
const settingsPath = vite.normalizePath(
fileURLToPath(new URL('settings.json', settings.dotAstroDir)),
);
if (settingsPath.match(normalizedChangedFile)) {
shouldRestart = settings.preferences.ignoreNextPreferenceReload ? false : true;
settings.preferences.ignoreNextPreferenceReload = false;
@ -169,14 +171,28 @@ export async function createContainerWithAutomaticRestart({
// Ignore the `forceOptimize` parameter for now.
restart.container.viteServer.restart = () => handleServerRestart();
// Set up shortcuts, overriding Vite's default shortcuts so it works for Astro
// Set up shortcuts
const customShortcuts: Array<vite.CLIShortcut> = [
// Disable default Vite shortcuts that don't work well with Astro
{ key: 'r', description: '' },
{ key: 'u', description: '' },
{ key: 'c', description: '' },
];
if (restart.container.settings.config.experimental.contentLayer) {
customShortcuts.push({
key: 's',
description: 'sync content layer',
action: () => {
if (globalContentLayer.initialized()) {
globalContentLayer.get().sync();
}
},
});
}
restart.container.viteServer.bindCLIShortcuts({
customShortcuts: [
// Disable Vite's builtin "r" (restart server), "u" (print server urls) and "c" (clear console) shortcuts
{ key: 'r', description: '' },
{ key: 'u', description: '' },
{ key: 'c', description: '' },
],
customShortcuts,
});
}
setupContainer();

25
packages/astro/src/core/errors/errors-data.ts
View file

@ -1291,6 +1291,17 @@ export const RewriteWithBodyUsed = {
'Astro.rewrite() cannot be used if the request body has already been read. If you need to read the body, first clone the request.',
} satisfies ErrorData;
/**
* @docs
* @description
* An unknown error occured while reading or writing files to disk. It can be caused by many things, eg. missing permissions or a file not existing we attempt to read.
*/
export const UnknownFilesystemError = {
name: 'UnknownFilesystemError',
title: 'An unknown error occured while reading or writing files to disk.',
hint: 'It can be caused by many things, eg. missing permissions or a file not existing we attempt to read. Check the error cause for more details.',
} satisfies ErrorData;
/**
* @docs
* @kind heading
@ -1469,6 +1480,20 @@ export const UnknownContentCollectionError = {
name: 'UnknownContentCollectionError',
title: 'Unknown Content Collection Error.',
} satisfies ErrorData;
/**
* @docs
* @description
* The `getDataEntryById` and `getEntryBySlug` functions are deprecated and cannot be used with content layer collections. Use the `getEntry` function instead.
*/
export const GetEntryDeprecationError = {
name: 'GetEntryDeprecationError',
title: 'Invalid use of `getDataEntryById` or `getEntryBySlug` function.',
message: (collection: string, method: string) =>
`The \`${method}\` function is deprecated and cannot be used to query the "${collection}" collection. Use \`getEntry\` instead.`,
hint: 'Use the `getEntry` or `getCollection` functions to query content layer collections.',
} satisfies ErrorData;
/**
* @docs
* @message

2
packages/astro/src/core/index.ts
View file

@ -23,4 +23,4 @@ export const build = (inlineConfig: AstroInlineConfig) => _build(inlineConfig);
* @experimental The JavaScript API is experimental
*/
// Wrap `_sync` to prevent exposing internal options
export const sync = (inlineConfig: AstroInlineConfig) => _sync({ inlineConfig });
export const sync = (inlineConfig: AstroInlineConfig) => _sync(inlineConfig);

2
packages/astro/src/core/sync/constants.ts Normal file
View file

@ -0,0 +1,2 @@
// TODO: use types.d.ts for backward compatibility. Use astro.d.ts in Astro 5.0
export const REFERENCE_FILE = './types.d.ts';

83
packages/astro/src/core/sync/index.ts
View file

@ -1,16 +1,17 @@
import fsMod from 'node:fs';
import fsMod, { existsSync } from 'node:fs';
import { performance } from 'node:perf_hooks';
import { fileURLToPath } from 'node:url';
import { dim } from 'kleur/colors';
import { type HMRPayload, createServer } from 'vite';
import type { AstroConfig, AstroInlineConfig, AstroSettings } from '../../@types/astro.js';
import { getPackage } from '../../cli/install-package.js';
import { DATA_STORE_FILE } from '../../content/consts.js';
import { globalContentLayer } from '../../content/content-layer.js';
import { DataStore, globalDataStore } from '../../content/data-store.js';
import { createContentTypesGenerator } from '../../content/index.js';
import { globalContentConfigObserver } from '../../content/utils.js';
import { syncAstroEnv } from '../../env/sync.js';
import { telemetry } from '../../events/index.js';
import { eventCliSession } from '../../events/session.js';
import { runHookConfigSetup } from '../../integrations/hooks.js';
import { runHookConfigDone, runHookConfigSetup } from '../../integrations/hooks.js';
import { getTimeStat } from '../build/util.js';
import { resolveConfig } from '../config/config.js';
import { createNodeLogger } from '../config/logging.js';
@ -27,7 +28,7 @@ import {
import type { Logger } from '../logger/core.js';
import { formatErrorMessage } from '../messages.js';
import { ensureProcessNodeEnv } from '../util.js';
import { setUpEnvTs } from './setup-env-ts.js';
import { writeFiles } from './write-files.js';
export type SyncOptions = {
/**
@ -36,21 +37,17 @@ export type SyncOptions = {
fs?: typeof fsMod;
logger: Logger;
settings: AstroSettings;
force?: boolean;
skip?: {
// Must be skipped in dev
content?: boolean;
};
};
type DBPackage = {
typegen?: (args: Pick<AstroConfig, 'root' | 'integrations'>) => Promise<void>;
};
export default async function sync({
inlineConfig,
fs,
telemetry: _telemetry = false,
}: { inlineConfig: AstroInlineConfig; fs?: typeof fsMod; telemetry?: boolean }) {
export default async function sync(
inlineConfig: AstroInlineConfig,
{ fs, telemetry: _telemetry = false }: { fs?: typeof fsMod; telemetry?: boolean } = {},
) {
ensureProcessNodeEnv('production');
const logger = createNodeLogger(inlineConfig);
const { astroConfig, userConfig } = await resolveConfig(inlineConfig ?? {}, 'sync');
@ -63,7 +60,24 @@ export default async function sync({
settings,
logger,
});
return await syncInternal({ settings, logger, fs });
await runHookConfigDone({ settings, logger });
return await syncInternal({ settings, logger, fs, force: inlineConfig.force });
}
/**
* Clears the content layer and content collection cache, forcing a full rebuild.
*/
export async function clearContentLayerCache({
astroConfig,
logger,
fs = fsMod,
}: { astroConfig: AstroConfig; logger: Logger; fs?: typeof fsMod }) {
const dataStore = new URL(DATA_STORE_FILE, astroConfig.cacheDir);
if (fs.existsSync(dataStore)) {
logger.debug('content', 'clearing data store');
await fs.promises.rm(dataStore, { force: true });
logger.warn('content', 'data store cleared (force)');
}
}
/**
@ -77,28 +91,43 @@ export async function syncInternal({
fs = fsMod,
settings,
skip,
force,
}: SyncOptions): Promise<void> {
const cwd = fileURLToPath(settings.config.root);
if (force) {
await clearContentLayerCache({ astroConfig: settings.config, logger, fs });
}
const timerStart = performance.now();
const dbPackage = await getPackage<DBPackage>(
'@astrojs/db',
logger,
{
optional: true,
cwd,
},
[],
);
try {
await dbPackage?.typegen?.(settings.config);
if (!skip?.content) {
await syncContentCollections(settings, { fs, logger });
settings.timer.start('Sync content layer');
let store: DataStore | undefined;
try {
const dataStoreFile = new URL(DATA_STORE_FILE, settings.config.cacheDir);
if (existsSync(dataStoreFile)) {
store = await DataStore.fromFile(dataStoreFile);
globalDataStore.set(store);
}
} catch (err: any) {
logger.error('content', err.message);
}
if (!store) {
store = new DataStore();
globalDataStore.set(store);
}
const contentLayer = globalContentLayer.init({
settings,
logger,
store,
});
await contentLayer.sync();
settings.timer.end('Sync content layer');
}
syncAstroEnv(settings, fs);
await setUpEnvTs({ settings, logger, fs });
await writeFiles(settings, fs, logger);
logger.info('types', `Generated ${dim(getTimeStat(timerStart, performance.now()))}`);
} catch (err) {
const error = createSafeError(err);

94
packages/astro/src/core/sync/setup-env-ts.ts
View file

@ -1,94 +0,0 @@
import type fsMod from 'node:fs';
import path from 'node:path';
import { fileURLToPath } from 'node:url';
import { bold } from 'kleur/colors';
import { normalizePath } from 'vite';
import type { AstroSettings } from '../../@types/astro.js';
import { ACTIONS_TYPES_FILE } from '../../actions/consts.js';
import { CONTENT_TYPES_FILE } from '../../content/consts.js';
import { ENV_TYPES_FILE } from '../../env/constants.js';
import { type Logger } from '../logger/core.js';
function getDotAstroTypeReference({
settings,
filename,
}: { settings: AstroSettings; filename: string }) {
const relativePath = normalizePath(
path.relative(
fileURLToPath(settings.config.srcDir),
fileURLToPath(new URL(filename, settings.dotAstroDir)),
),
);
return `/// <reference path=${JSON.stringify(relativePath)} />`;
}
type InjectedType = { filename: string; meetsCondition?: () => boolean | Promise<boolean> };
export async function setUpEnvTs({
settings,
logger,
fs,
}: {
settings: AstroSettings;
logger: Logger;
fs: typeof fsMod;
}) {
const envTsPath = new URL('env.d.ts', settings.config.srcDir);
const envTsPathRelativetoRoot = normalizePath(
path.relative(fileURLToPath(settings.config.root), fileURLToPath(envTsPath)),
);
const injectedTypes: Array<InjectedType> = [
{
filename: CONTENT_TYPES_FILE,
meetsCondition: () => fs.existsSync(new URL(CONTENT_TYPES_FILE, settings.dotAstroDir)),
},
{
filename: ACTIONS_TYPES_FILE,
meetsCondition: () => fs.existsSync(new URL(ACTIONS_TYPES_FILE, settings.dotAstroDir)),
},
];
if (settings.config.experimental.env) {
injectedTypes.push({
filename: ENV_TYPES_FILE,
});
}
if (fs.existsSync(envTsPath)) {
const initialEnvContents = await fs.promises.readFile(envTsPath, 'utf-8');
let typesEnvContents = initialEnvContents;
for (const injectedType of injectedTypes) {
if (!injectedType.meetsCondition || (await injectedType.meetsCondition?.())) {
const expectedTypeReference = getDotAstroTypeReference({
settings,
filename: injectedType.filename,
});
if (!typesEnvContents.includes(expectedTypeReference)) {
typesEnvContents = `${expectedTypeReference}\n${typesEnvContents}`;
}
}
}
if (initialEnvContents !== typesEnvContents) {
logger.info('types', `Updated ${bold(envTsPathRelativetoRoot)} type declarations.`);
await fs.promises.writeFile(envTsPath, typesEnvContents, 'utf-8');
}
} else {
// Otherwise, inject the `env.d.ts` file
let referenceDefs: string[] = [];
referenceDefs.push('/// <reference types="astro/client" />');
for (const injectedType of injectedTypes) {
if (!injectedType.meetsCondition || (await injectedType.meetsCondition?.())) {
referenceDefs.push(getDotAstroTypeReference({ settings, filename: injectedType.filename }));
}
}
await fs.promises.mkdir(settings.config.srcDir, { recursive: true });
await fs.promises.writeFile(envTsPath, referenceDefs.join('\n'), 'utf-8');
logger.info('types', `Added ${bold(envTsPathRelativetoRoot)} type declarations`);
}
}

78
packages/astro/src/core/sync/write-files.ts Normal file
View file

@ -0,0 +1,78 @@
import type fsMod from 'node:fs';
import { dirname, relative } from 'node:path';
import { fileURLToPath } from 'node:url';
import { bold } from 'kleur/colors';
import { normalizePath } from 'vite';
import type { AstroSettings } from '../../@types/astro.js';
import type { Logger } from '../logger/core.js';
import { REFERENCE_FILE } from './constants.js';
import { AstroError, AstroErrorData } from '../errors/index.js';
export async function writeFiles(settings: AstroSettings, fs: typeof fsMod, logger: Logger) {
try {
writeInjectedTypes(settings, fs);
await setUpEnvTs(settings, fs, logger);
} catch (e) {
throw new AstroError(AstroErrorData.UnknownFilesystemError, { cause: e });
}
}
function getTsReference(type: 'path' | 'types', value: string) {
return `/// <reference ${type}=${JSON.stringify(value)} />`;
}
const CLIENT_TYPES_REFERENCE = getTsReference('types', 'astro/client');
function writeInjectedTypes(settings: AstroSettings, fs: typeof fsMod) {
const references: Array<string> = [];
for (const { filename, content } of settings.injectedTypes) {
const filepath = normalizePath(fileURLToPath(new URL(filename, settings.dotAstroDir)));
fs.mkdirSync(dirname(filepath), { recursive: true });
fs.writeFileSync(filepath, content, 'utf-8');
references.push(normalizePath(relative(fileURLToPath(settings.dotAstroDir), filepath)));
}
const astroDtsContent = `${CLIENT_TYPES_REFERENCE}\n${references.map((reference) => getTsReference('path', reference)).join('\n')}`;
if (references.length === 0) {
fs.mkdirSync(settings.dotAstroDir, { recursive: true });
}
fs.writeFileSync(
normalizePath(fileURLToPath(new URL(REFERENCE_FILE, settings.dotAstroDir))),
astroDtsContent,
'utf-8',
);
}
async function setUpEnvTs(settings: AstroSettings, fs: typeof fsMod, logger: Logger) {
const envTsPath = normalizePath(fileURLToPath(new URL('env.d.ts', settings.config.srcDir)));
const envTsPathRelativetoRoot = normalizePath(
relative(fileURLToPath(settings.config.root), envTsPath),
);
const relativePath = normalizePath(
relative(
fileURLToPath(settings.config.srcDir),
fileURLToPath(new URL(REFERENCE_FILE, settings.dotAstroDir)),
),
);
const expectedTypeReference = getTsReference('path', relativePath);
if (fs.existsSync(envTsPath)) {
const initialEnvContents = await fs.promises.readFile(envTsPath, 'utf-8');
let typesEnvContents = initialEnvContents;
if (!typesEnvContents.includes(expectedTypeReference)) {
typesEnvContents = `${expectedTypeReference}\n${typesEnvContents}`;
}
if (initialEnvContents !== typesEnvContents) {
logger.info('types', `Updated ${bold(envTsPathRelativetoRoot)} type declarations.`);
await fs.promises.writeFile(envTsPath, typesEnvContents, 'utf-8');
}
} else {
// Otherwise, inject the `env.d.ts` file
await fs.promises.mkdir(settings.config.srcDir, { recursive: true });
await fs.promises.writeFile(envTsPath, expectedTypeReference, 'utf-8');
logger.info('types', `Added ${bold(envTsPathRelativetoRoot)} type declarations`);
}
}

12
packages/astro/src/env/sync.ts vendored
View file

@ -1,9 +1,9 @@
import fsMod from 'node:fs';
import type { AstroSettings } from '../@types/astro.js';
import { ENV_TYPES_FILE, TYPES_TEMPLATE_URL } from './constants.js';
import { TYPES_TEMPLATE_URL } from './constants.js';
import { getEnvFieldType } from './validators.js';
export function syncAstroEnv(settings: AstroSettings, fs = fsMod) {
export function syncAstroEnv(settings: AstroSettings, fs = fsMod): void {
if (!settings.config.experimental.env) {
return;
}
@ -23,8 +23,10 @@ export function syncAstroEnv(settings: AstroSettings, fs = fsMod) {
}
const template = fs.readFileSync(TYPES_TEMPLATE_URL, 'utf-8');
const dts = template.replace('// @@CLIENT@@', client).replace('// @@SERVER@@', server);
const content = template.replace('// @@CLIENT@@', client).replace('// @@SERVER@@', server);
fs.mkdirSync(settings.dotAstroDir, { recursive: true });
fs.writeFileSync(new URL(ENV_TYPES_FILE, settings.dotAstroDir), dts, 'utf-8');
settings.injectedTypes.push({
filename: 'astro/env.d.ts',
content,
});
}

25
packages/astro/src/integrations/hooks.ts
View file

@ -101,6 +101,18 @@ export function getToolbarServerCommunicationHelpers(server: ViteDevServer) {
};
}
// Will match any invalid characters (will be converted to _). We only allow a-zA-Z0-9.-_
const SAFE_CHARS_RE = /[^\w.-]/g;
export function normalizeInjectedTypeFilename(filename: string, integrationName: string): string {
if (!filename.endsWith('.d.ts')) {
throw new Error(
`Integration ${bold(integrationName)} is injecting a type that does not end with "${bold('.d.ts')}"`,
);
}
return `./integrations/${integrationName.replace(SAFE_CHARS_RE, '_')}/${filename.replace(SAFE_CHARS_RE, '_')}`;
}
export async function runHookConfigSetup({
settings,
command,
@ -328,6 +340,19 @@ export async function runHookConfigDone({
}
settings.adapter = adapter;
},
injectTypes(injectedType) {
const normalizedFilename = normalizeInjectedTypeFilename(
injectedType.filename,
integration.name,
);
settings.injectedTypes.push({
filename: normalizedFilename,
content: injectedType.content,
});
return new URL(normalizedFilename, settings.config.root);
},
logger: getLogger(integration, logger),
}),
logger,

4
packages/astro/src/preferences/index.ts
View file

@ -82,9 +82,9 @@ export function coerce(key: string, value: unknown) {
return value as any;
}
export default function createPreferences(config: AstroConfig): AstroPreferences {
export default function createPreferences(config: AstroConfig, dotAstroDir: URL): AstroPreferences {
const global = new PreferenceStore(getGlobalPreferenceDir());
const project = new PreferenceStore(fileURLToPath(new URL('./.astro/', config.root)));
const project = new PreferenceStore(fileURLToPath(dotAstroDir));
const stores: Record<PreferenceLocation, PreferenceStore> = { global, project };
return {

26
packages/astro/src/vite-plugin-markdown/content-entry-type.ts
View file

@ -1,4 +1,5 @@
import { fileURLToPath } from 'node:url';
import { fileURLToPath, pathToFileURL } from 'node:url';
import { createMarkdownProcessor } from '@astrojs/markdown-remark';
import type { ContentEntryType } from '../@types/astro.js';
import { safeParseFrontmatter } from '../content/utils.js';
@ -15,4 +16,27 @@ export const markdownContentEntryType: ContentEntryType = {
},
// We need to handle propagation for Markdown because they support layouts which will bring in styles.
handlePropagation: true,
async getRenderFunction(settings) {
const processor = await createMarkdownProcessor(settings.config.markdown);
return async function renderToString(entry) {
if (!entry.body) {
return {
html: '',
};
}
const result = await processor.render(entry.body, {
frontmatter: entry.data,
// @ts-expect-error Internal API
fileURL: entry.filePath ? pathToFileURL(entry.filePath) : undefined,
});
return {
html: result.code,
metadata: {
...result.metadata,
imagePaths: Array.from(result.metadata.imagePaths),
},
};
};
},
};

7
packages/astro/templates/content/module.mjs
View file

@ -9,7 +9,7 @@ import {
createReference,
} from 'astro/content/runtime';
export { defineCollection } from 'astro/content/runtime';
export { defineCollection, renderEntry as render } from 'astro/content/runtime';
export { z } from 'astro/zod';
const contentDir = '@@CONTENT_DIR@@';
@ -33,6 +33,8 @@ const collectionToEntryMap = createCollectionToGlobResultMap({
let lookupMap = {};
/* @@LOOKUP_MAP_ASSIGNMENT@@ */
const collectionNames = new Set(Object.keys(lookupMap));
function createGlobLookup(glob) {
return async (collection, lookupId) => {
const filePath = lookupMap[collection]?.entries[lookupId];
@ -59,15 +61,18 @@ export const getCollection = createGetCollection({
export const getEntryBySlug = createGetEntryBySlug({
getEntryImport: createGlobLookup(contentCollectionToEntryMap),
getRenderEntryImport: createGlobLookup(collectionToRenderEntryMap),
collectionNames,
});
export const getDataEntryById = createGetDataEntryById({
getEntryImport: createGlobLookup(dataCollectionToEntryMap),
collectionNames,
});
export const getEntry = createGetEntry({
getEntryImport: createGlobLookup(collectionToEntryMap),
getRenderEntryImport: createGlobLookup(collectionToRenderEntryMap),
collectionNames,
});
export const getEntries = createGetEntries(getEntry);

25
packages/astro/templates/content/types.d.ts vendored
View file

@ -1,10 +1,21 @@
declare module 'astro:content' {
interface RenderResult {
Content: import('astro/runtime/server/index.js').AstroComponentFactory;
headings: import('astro').MarkdownHeading[];
remarkPluginFrontmatter: Record<string, any>;
}
interface Render {
'.md': Promise<{
Content: import('astro').MarkdownInstance<{}>['Content'];
headings: import('astro').MarkdownHeading[];
remarkPluginFrontmatter: Record<string, any>;
}>;
'.md': Promise<RenderResult>;
}
export interface RenderedContent {
html: string;
metadata?: {
imagePaths: Array<string>;
[key: string]: unknown;
};
}
}
@ -100,6 +111,10 @@ declare module 'astro:content' {
}[],
): Promise<CollectionEntry<C>[]>;
export function render<C extends keyof AnyEntryMap>(
entry: AnyEntryMap[C][string],
): Promise<RenderResult>;
export function reference<C extends keyof AnyEntryMap>(
collection: C,
): import('astro/zod').ZodEffects<

216
packages/astro/test/astro-sync.test.js
View file

@ -1,8 +1,10 @@
// @ts-check
import assert from 'node:assert/strict';
import * as fs from 'node:fs';
import { before, describe, it } from 'node:test';
import { beforeEach, describe, it } from 'node:test';
import { fileURLToPath } from 'node:url';
import ts from 'typescript';
import { normalizePath } from 'vite';
import { loadFixture } from './test-utils.js';
const createFixture = () => {
@ -11,66 +13,79 @@ const createFixture = () => {
/** @type {Record<string, string>} */
const writtenFiles = {};
/**
* @param {string} path
*/
const getExpectedPath = (path) =>
normalizePath(fileURLToPath(new URL(path, astroFixture.config.root)));
return {
/** @param {string} root */
async whenSyncing(root) {
async load(root) {
astroFixture = await loadFixture({ root });
const envPath = new URL('env.d.ts', astroFixture.config.srcDir).href;
const typesDtsPath = new URL('.astro/types.d.ts', astroFixture.config.root).href;
return astroFixture.config;
},
clean() {
const envPath = new URL('env.d.ts', astroFixture.config.srcDir);
if (fs.existsSync(envPath)) {
fs.unlinkSync(new URL('env.d.ts', astroFixture.config.srcDir));
}
fs.rmSync(new URL('./.astro/', astroFixture.config.root), { force: true, recursive: true });
},
async whenSyncing() {
const fsMock = {
...fs,
existsSync(path, ...args) {
if (path.toString() === envPath) {
return false;
}
if (path.toString() === typesDtsPath) {
return true;
}
return fs.existsSync(path, ...args);
},
/**
* @param {fs.PathLike} path
* @param {string} contents
*/
writeFileSync(path, contents) {
writtenFiles[path.toString()] = contents;
return fs.writeFileSync(path, contents);
},
promises: {
...fs.promises,
async readFile(path, ...args) {
if (path.toString() === envPath) {
return `/// <reference path="astro/client" />`;
} else {
return fs.promises.readFile(path, ...args);
}
},
async writeFile(path, contents) {
/**
* @param {fs.PathLike} path
* @param {string} contents
*/
writeFile(path, contents) {
writtenFiles[path.toString()] = contents;
return fs.promises.writeFile(path, contents);
},
},
};
await astroFixture.sync({
inlineConfig: { root: fileURLToPath(new URL(root, import.meta.url)) },
fs: fsMock,
});
await astroFixture.sync(
{ root: fileURLToPath(astroFixture.config.root) },
{
// @ts-ignore
fs: fsMock,
},
);
},
/** @param {string} path */
thenFileShouldExist(path) {
const expectedPath = new URL(path, astroFixture.config.root).href;
assert.equal(writtenFiles.hasOwnProperty(expectedPath), true, `${path} does not exist`);
assert.equal(
writtenFiles.hasOwnProperty(getExpectedPath(path)),
true,
`${path} does not exist`,
);
},
/**
* @param {string} path
* @param {string} content
* @param {string | undefined} error
*/
thenFileContentShouldInclude(path, content, error) {
const expectedPath = new URL(path, astroFixture.config.root).href;
assert.equal(writtenFiles[expectedPath].includes(content), true, error);
thenFileContentShouldInclude(path, content, error = undefined) {
assert.equal(writtenFiles[getExpectedPath(path)].includes(content), true, error);
},
/**
* @param {string} path
*/
thenFileShouldBeValidTypescript(path) {
const expectedPath = new URL(path, astroFixture.config.root).href;
try {
const content = writtenFiles[expectedPath];
const content = writtenFiles[getExpectedPath(path)];
const result = ts.transpileModule(content, {
compilerOptions: {
module: ts.ModuleKind.ESNext,
@ -91,33 +106,71 @@ const createFixture = () => {
describe('astro sync', () => {
/** @type {ReturnType<typeof createFixture>} */
let fixture;
before(async () => {
beforeEach(async () => {
fixture = createFixture();
});
it('Writes `src/env.d.ts` if none exists', async () => {
await fixture.whenSyncing('./fixtures/astro-basic/');
fixture.thenFileShouldExist('src/env.d.ts');
fixture.thenFileContentShouldInclude('src/env.d.ts', `/// <reference types="astro/client" />`);
describe('References', () => {
it('Writes `src/env.d.ts` if none exists', async () => {
await fixture.load('./fixtures/astro-basic/');
fixture.clean();
await fixture.whenSyncing();
fixture.thenFileShouldExist('src/env.d.ts');
fixture.thenFileContentShouldInclude(
'src/env.d.ts',
`/// <reference path="../.astro/types.d.ts" />`,
);
});
it('Updates `src/env.d.ts` if one exists', async () => {
const config = await fixture.load('./fixtures/astro-basic/');
fixture.clean();
fs.writeFileSync(new URL('./env.d.ts', config.srcDir), '// whatever', 'utf-8');
await fixture.whenSyncing();
fixture.thenFileShouldExist('src/env.d.ts');
fixture.thenFileContentShouldInclude(
'src/env.d.ts',
`/// <reference path="../.astro/types.d.ts" />`,
);
});
it('Writes `src/types.d.ts`', async () => {
await fixture.load('./fixtures/astro-basic/');
fixture.clean();
await fixture.whenSyncing();
fixture.thenFileShouldExist('.astro/types.d.ts');
fixture.thenFileContentShouldInclude(
'.astro/types.d.ts',
`/// <reference types="astro/client" />`,
);
});
});
describe('Content collections', () => {
it('Writes types to `.astro`', async () => {
await fixture.whenSyncing('./fixtures/content-collections/');
it('Adds reference to `.astro/types.d.ts`', async () => {
await fixture.load('./fixtures/content-collections/');
fixture.clean();
await fixture.whenSyncing();
fixture.thenFileShouldExist('.astro/types.d.ts');
fixture.thenFileContentShouldInclude(
'.astro/types.d.ts',
`/// <reference path="astro/content.d.ts" />`,
);
fixture.thenFileShouldExist('.astro/astro/content.d.ts');
fixture.thenFileContentShouldInclude(
'.astro/astro/content.d.ts',
`declare module 'astro:content' {`,
'Types file does not include `astro:content` module declaration',
);
fixture.thenFileShouldBeValidTypescript('.astro/types.d.ts');
fixture.thenFileShouldBeValidTypescript('.astro/astro/content.d.ts');
});
it('Writes types for empty collections', async () => {
await fixture.whenSyncing('./fixtures/content-collections-empty-dir/');
fixture.thenFileShouldExist('.astro/types.d.ts');
await fixture.load('./fixtures/content-collections-empty-dir/');
fixture.clean();
await fixture.whenSyncing();
fixture.thenFileContentShouldInclude(
'.astro/types.d.ts',
'.astro/astro/content.d.ts',
`"blog": Record<string, {
id: string;
slug: string;
@ -129,7 +182,7 @@ describe('astro sync', () => {
'Types file does not include empty collection type',
);
fixture.thenFileContentShouldInclude(
'.astro/types.d.ts',
'.astro/astro/content.d.ts',
`"blogMeta": Record<string, {
id: string;
collection: "blogMeta";
@ -138,60 +191,55 @@ describe('astro sync', () => {
'Types file does not include empty collection type',
);
});
it('Adds type reference to `src/env.d.ts`', async () => {
await fixture.whenSyncing('./fixtures/content-collections/');
fixture.thenFileShouldExist('src/env.d.ts');
fixture.thenFileContentShouldInclude(
'src/env.d.ts',
`/// <reference path="../.astro/types.d.ts" />`,
);
});
});
describe('Astro Env', () => {
it('Writes types to `.astro`', async () => {
await fixture.whenSyncing('./fixtures/astro-env/');
fixture.thenFileShouldExist('.astro/env.d.ts');
describe('astro:env', () => {
it('Adds reference to `.astro/types.d.ts`', async () => {
await fixture.load('./fixtures/astro-env/');
fixture.clean();
await fixture.whenSyncing();
fixture.thenFileShouldExist('.astro/types.d.ts');
fixture.thenFileContentShouldInclude(
'.astro/env.d.ts',
'.astro/types.d.ts',
`/// <reference path="astro/env.d.ts" />`,
);
fixture.thenFileShouldExist('.astro/astro/env.d.ts');
fixture.thenFileContentShouldInclude(
'.astro/astro/env.d.ts',
`declare module 'astro:env/client' {`,
'Types file does not include `astro:env` module declaration',
);
});
it('Adds type reference to `src/env.d.ts`', async () => {
await fixture.whenSyncing('./fixtures/astro-env/');
fixture.thenFileShouldExist('src/env.d.ts');
fixture.thenFileContentShouldInclude(
'src/env.d.ts',
`/// <reference path="../.astro/env.d.ts" />`,
);
});
it('Does not throw if a public variable is required', async () => {
let error = null;
try {
await fixture.whenSyncing('./fixtures/astro-env-required-public/');
} catch (e) {
error = e;
await fixture.load('./fixtures/astro-env-required-public/');
fixture.clean();
await fixture.whenSyncing();
assert.ok(true);
} catch {
assert.fail();
}
assert.equal(error, null, 'Syncing should not throw astro:env validation errors');
});
});
describe('Astro Actions', () => {
// We can't check for the file existence or content yet because
// it's an integration and does not use the fs mock
it('Adds type reference to `src/env.d.ts`', async () => {
await fixture.whenSyncing('./fixtures/actions/');
fixture.thenFileShouldExist('src/env.d.ts');
describe('astro:actions', () => {
it('Adds reference to `.astro/types.d.ts`', async () => {
await fixture.load('./fixtures/actions/');
fixture.clean();
await fixture.whenSyncing();
fixture.thenFileShouldExist('.astro/types.d.ts');
fixture.thenFileContentShouldInclude(
'src/env.d.ts',
`/// <reference path="../.astro/actions.d.ts" />`,
'.astro/types.d.ts',
`/// <reference path="astro/actions.d.ts" />`,
);
fixture.thenFileShouldExist('.astro/astro/actions.d.ts');
fixture.thenFileContentShouldInclude(
'.astro/astro/actions.d.ts',
`declare module "astro:actions" {`,
'Types file does not include `astro:actions` module declaration',
);
fixture.thenFileShouldBeValidTypescript('.astro/astro/actions.d.ts');
});
});
});

79
packages/astro/test/content-intellisense.test.js Normal file
View file

@ -0,0 +1,79 @@
import assert from 'node:assert/strict';
import { before, describe, it } from 'node:test';
import { loadFixture } from './test-utils.js';
describe('Content Intellisense', () => {
/** @type {import("./test-utils.js").Fixture} */
let fixture;
/** @type {string[]} */
let collectionsDir = [];
/** @type {{collections: {hasSchema: boolean, name: string}[], entries: Record<string, string>}} */
let manifest = undefined;
before(async () => {
fixture = await loadFixture({ root: './fixtures/content-intellisense/' });
await fixture.build();
collectionsDir = await fixture.readdir('../.astro/collections');
manifest = JSON.parse(await fixture.readFile('../.astro/collections/collections.json'));
});
it('generate JSON schemas for content collections', async () => {
assert.deepEqual(collectionsDir.includes('blog-cc.schema.json'), true);
});
it('generate JSON schemas for content layer', async () => {
assert.deepEqual(collectionsDir.includes('blog-cl.schema.json'), true);
});
it('manifest exists', async () => {
assert.notEqual(manifest, undefined);
});
it('manifest has content collections', async () => {
const manifestCollections = manifest.collections.map((collection) => collection.name);
assert.equal(
manifestCollections.includes('blog-cc'),
true,
"Expected 'blog-cc' collection in manifest",
);
});
it('manifest has content layer', async () => {
const manifestCollections = manifest.collections.map((collection) => collection.name);
assert.equal(
manifestCollections.includes('blog-cl'),
true,
"Expected 'blog-cl' collection in manifest",
);
});
it('has entries for content collections', async () => {
const collectionEntries = Object.entries(manifest.entries).filter((entry) =>
entry[0].includes(
'/astro/packages/astro/test/fixtures/content-intellisense/src/content/blog-cc/',
),
);
assert.equal(collectionEntries.length, 3, "Expected 3 entries for 'blog-cc' collection");
assert.equal(
collectionEntries.every((entry) => entry[1] === 'blog-cc'),
true,
"Expected 3 entries for 'blog-cc' collection to have 'blog-cc' as collection",
);
});
it('has entries for content layer', async () => {
const collectionEntries = Object.entries(manifest.entries).filter((entry) =>
entry[0].includes('/astro/packages/astro/test/fixtures/content-intellisense/src/blog-cl/'),
);
assert.equal(collectionEntries.length, 3, "Expected 3 entries for 'blog-cl' collection");
assert.equal(
collectionEntries.every((entry) => entry[1] === 'blog-cl'),
true,
"Expected 3 entries for 'blog-cl' collection to have 'blog-cl' as collection name",
);
});
});

88
packages/astro/test/content-layer-markdoc.test.js Normal file
View file

@ -0,0 +1,88 @@
import assert from 'node:assert/strict';
import { after, before, describe, it } from 'node:test';
import * as cheerio from 'cheerio';
import { loadFixture } from './test-utils.js';
describe('Content layer markdoc', () => {
let fixture;
before(async () => {
fixture = await loadFixture({
root: './fixtures/content-layer-markdoc/',
});
});
describe('dev', () => {
let devServer;
before(async () => {
devServer = await fixture.startDevServer();
});
after(async () => {
await devServer.stop();
});
it('renders content - with components', async () => {
const res = await fixture.fetch('/');
const html = await res.text();
renderComponentsChecks(html);
});
it('renders content - with components inside partials', async () => {
const res = await fixture.fetch('/');
const html = await res.text();
renderComponentsInsidePartialsChecks(html);
});
});
describe('build', () => {
before(async () => {
await fixture.build();
});
it('renders content - with components', async () => {
const html = await fixture.readFile('/index.html');
renderComponentsChecks(html);
});
it('renders content - with components inside partials', async () => {
const html = await fixture.readFile('/index.html');
renderComponentsInsidePartialsChecks(html);
});
});
});
/** @param {string} html */
function renderComponentsChecks(html) {
const $ = cheerio.load(html);
const h2 = $('h2');
assert.equal(h2.text(), 'Post with components');
// Renders custom shortcode component
const marquee = $('marquee');
assert.notEqual(marquee, null);
assert.equal(marquee.attr('data-custom-marquee'), '');
// Renders Astro Code component
const pre = $('pre');
assert.notEqual(pre, null);
assert.ok(pre.hasClass('github-dark'));
assert.ok(pre.hasClass('astro-code'));
}
/** @param {string} html */
function renderComponentsInsidePartialsChecks(html) {
const $ = cheerio.load(html);
// renders Counter.tsx
const button = $('#counter');
assert.equal(button.text(), '1');
// renders DeeplyNested.astro
const deeplyNested = $('#deeply-nested');
assert.equal(deeplyNested.text(), 'Deeply nested partial');
}

45
packages/astro/test/content-layer-render.test.js Normal file
View file

@ -0,0 +1,45 @@
import assert from 'node:assert/strict';
import { after, before, describe, it } from 'node:test';
import { loadFixture } from './test-utils.js';
describe('Content Layer MDX rendering dev', () => {
/** @type {import("./test-utils.js").Fixture} */
let fixture;
let devServer;
before(async () => {
fixture = await loadFixture({
root: './fixtures/content-layer-rendering/',
});
devServer = await fixture.startDevServer();
});
after(async () => {
devServer?.stop();
});
it('Render an MDX file', async () => {
const html = await fixture.fetch('/reptiles/iguana').then((r) => r.text());
assert.match(html, /Iguana/);
assert.match(html, /This is a rendered entry/);
});
});
describe('Content Layer MDX rendering build', () => {
/** @type {import("./test-utils.js").Fixture} */
let fixture;
before(async () => {
fixture = await loadFixture({
root: './fixtures/content-layer-rendering/',
});
await fixture.build();
});
it('Render an MDX file', async () => {
const html = await fixture.readFile('/reptiles/iguana/index.html');
assert.match(html, /Iguana/);
assert.match(html, /This is a rendered entry/);
});
});

279
packages/astro/test/content-layer.test.js Normal file
View file

@ -0,0 +1,279 @@
import assert from 'node:assert/strict';
import { promises as fs } from 'node:fs';
import { sep } from 'node:path';
import { sep as posixSep } from 'node:path/posix';
import { after, before, describe, it } from 'node:test';
import * as devalue from 'devalue';
import { loadFixture } from './test-utils.js';
describe('Content Layer', () => {
/** @type {import("./test-utils.js").Fixture} */
let fixture;
before(async () => {
fixture = await loadFixture({ root: './fixtures/content-layer/' });
});
describe('Build', () => {
let json;
before(async () => {
fixture = await loadFixture({ root: './fixtures/content-layer/' });
await fs
.unlink(new URL('./node_modules/.astro/data-store.json', fixture.config.root))
.catch(() => {});
await fixture.build();
const rawJson = await fixture.readFile('/collections.json');
json = devalue.parse(rawJson);
});
it('Returns custom loader collection', async () => {
assert.ok(json.hasOwnProperty('customLoader'));
assert.ok(Array.isArray(json.customLoader));
const item = json.customLoader[0];
assert.deepEqual(item, {
id: '1',
collection: 'blog',
data: {
userId: 1,
id: 1,
title: 'sunt aut facere repellat provident occaecati excepturi optio reprehenderit',
body: 'quia et suscipit\nsuscipit recusandae consequuntur expedita et cum\nreprehenderit molestiae ut ut quas totam\nnostrum rerum est autem sunt rem eveniet architecto',
},
});
});
it('filters collection items', async () => {
assert.ok(json.hasOwnProperty('customLoader'));
assert.ok(Array.isArray(json.customLoader));
assert.equal(json.customLoader.length, 5);
});
it('Returns `file()` loader collection', async () => {
assert.ok(json.hasOwnProperty('fileLoader'));
assert.ok(Array.isArray(json.fileLoader));
const ids = json.fileLoader.map((item) => item.data.id);
assert.deepEqual(ids, [
'labrador-retriever',
'german-shepherd',
'golden-retriever',
'french-bulldog',
'bulldog',
'beagle',
'poodle',
'rottweiler',
'german-shorthaired-pointer',
'yorkshire-terrier',
'boxer',
'dachshund',
'siberian-husky',
'great-dane',
'doberman-pinscher',
'australian-shepherd',
'miniature-schnauzer',
'cavalier-king-charles-spaniel',
'shih-tzu',
'boston-terrier',
'bernese-mountain-dog',
'pomeranian',
'havanese',
'english-springer-spaniel',
'shetland-sheepdog',
]);
});
it('Returns data entry by id', async () => {
assert.ok(json.hasOwnProperty('dataEntry'));
assert.equal(json.dataEntry.filePath?.split(sep).join(posixSep), 'src/data/dogs.json');
delete json.dataEntry.filePath;
assert.deepEqual(json.dataEntry, {
id: 'beagle',
collection: 'dogs',
data: {
breed: 'Beagle',
id: 'beagle',
size: 'Small to Medium',
origin: 'England',
lifespan: '12-15 years',
temperament: ['Friendly', 'Curious', 'Merry'],
},
});
});
it('returns collection from a simple loader', async () => {
assert.ok(json.hasOwnProperty('simpleLoader'));
assert.ok(Array.isArray(json.simpleLoader));
const item = json.simpleLoader[0];
assert.deepEqual(item, {
id: 'siamese',
collection: 'cats',
data: {
breed: 'Siamese',
id: 'siamese',
size: 'Medium',
origin: 'Thailand',
lifespan: '15 years',
temperament: ['Active', 'Affectionate', 'Social', 'Playful'],
},
});
});
it('transforms a reference id to a reference object', async () => {
assert.ok(json.hasOwnProperty('entryWithReference'));
assert.deepEqual(json.entryWithReference.data.cat, { collection: 'cats', id: 'tabby' });
});
it('can store Date objects', async () => {
assert.ok(json.entryWithReference.data.publishedDate instanceof Date);
});
it('returns a referenced entry', async () => {
assert.ok(json.hasOwnProperty('referencedEntry'));
assert.deepEqual(json.referencedEntry, {
collection: 'cats',
data: {
breed: 'Tabby',
id: 'tabby',
size: 'Medium',
origin: 'Egypt',
lifespan: '15 years',
temperament: ['Curious', 'Playful', 'Independent'],
},
id: 'tabby',
});
});
it('updates the store on new builds', async () => {
assert.equal(json.increment.data.lastValue, 1);
await fixture.build();
const newJson = devalue.parse(await fixture.readFile('/collections.json'));
assert.equal(newJson.increment.data.lastValue, 2);
});
it('clears the store on new build with force flag', async () => {
let newJson = devalue.parse(await fixture.readFile('/collections.json'));
assert.equal(newJson.increment.data.lastValue, 2);
await fixture.build({ force: true }, {});
newJson = devalue.parse(await fixture.readFile('/collections.json'));
assert.equal(newJson.increment.data.lastValue, 1);
});
it('clears the store on new build if the config has changed', async () => {
let newJson = devalue.parse(await fixture.readFile('/collections.json'));
assert.equal(newJson.increment.data.lastValue, 1);
await fixture.editFile('src/content/config.ts', (prev) => {
return `${prev}\nexport const foo = 'bar';`;
});
await fixture.build();
newJson = devalue.parse(await fixture.readFile('/collections.json'));
assert.equal(newJson.increment.data.lastValue, 1);
await fixture.resetAllFiles();
});
});
describe('Dev', () => {
let devServer;
let json;
before(async () => {
devServer = await fixture.startDevServer();
const rawJsonResponse = await fixture.fetch('/collections.json');
const rawJson = await rawJsonResponse.text();
json = devalue.parse(rawJson);
});
after(async () => {
devServer?.stop();
});
it('Returns custom loader collection', async () => {
assert.ok(json.hasOwnProperty('customLoader'));
assert.ok(Array.isArray(json.customLoader));
const item = json.customLoader[0];
assert.deepEqual(item, {
id: '1',
collection: 'blog',
data: {
userId: 1,
id: 1,
title: 'sunt aut facere repellat provident occaecati excepturi optio reprehenderit',
body: 'quia et suscipit\nsuscipit recusandae consequuntur expedita et cum\nreprehenderit molestiae ut ut quas totam\nnostrum rerum est autem sunt rem eveniet architecto',
},
});
});
it('Returns `file()` loader collection', async () => {
assert.ok(json.hasOwnProperty('fileLoader'));
assert.ok(Array.isArray(json.fileLoader));
const ids = json.fileLoader.map((item) => item.data.id);
assert.deepEqual(ids, [
'labrador-retriever',
'german-shepherd',
'golden-retriever',
'french-bulldog',
'bulldog',
'beagle',
'poodle',
'rottweiler',
'german-shorthaired-pointer',
'yorkshire-terrier',
'boxer',
'dachshund',
'siberian-husky',
'great-dane',
'doberman-pinscher',
'australian-shepherd',
'miniature-schnauzer',
'cavalier-king-charles-spaniel',
'shih-tzu',
'boston-terrier',
'bernese-mountain-dog',
'pomeranian',
'havanese',
'english-springer-spaniel',
'shetland-sheepdog',
]);
});
it('Returns data entry by id', async () => {
assert.ok(json.hasOwnProperty('dataEntry'));
assert.equal(json.dataEntry.filePath?.split(sep).join(posixSep), 'src/data/dogs.json');
delete json.dataEntry.filePath;
assert.deepEqual(json.dataEntry, {
id: 'beagle',
collection: 'dogs',
data: {
breed: 'Beagle',
id: 'beagle',
size: 'Small to Medium',
origin: 'England',
lifespan: '12-15 years',
temperament: ['Friendly', 'Curious', 'Merry'],
},
});
});
it('updates collection when data file is changed', async () => {
const rawJsonResponse = await fixture.fetch('/collections.json');
const initialJson = devalue.parse(await rawJsonResponse.text());
assert.equal(initialJson.fileLoader[0].data.temperament.includes('Bouncy'), false);
await fixture.editFile('/src/data/dogs.json', (prev) => {
const data = JSON.parse(prev);
data[0].temperament.push('Bouncy');
return JSON.stringify(data, null, 2);
});
// Writes are debounced to 500ms
await new Promise((r) => setTimeout(r, 700));
const updatedJsonResponse = await fixture.fetch('/collections.json');
const updated = devalue.parse(await updatedJsonResponse.text());
assert.ok(updated.fileLoader[0].data.temperament.includes('Bouncy'));
await fixture.resetAllFiles();
});
});
});

3
packages/astro/test/fixtures/content-collections/astro.config.mjs vendored
View file

@ -4,4 +4,7 @@ import { defineConfig } from 'astro/config';
// https://astro.build/config
export default defineConfig({
integrations: [mdx()],
experimental: {
contentIntellisense: true
}
});

12
packages/astro/test/fixtures/content-intellisense/astro.config.mjs vendored Normal file
View file

@ -0,0 +1,12 @@
import markdoc from "@astrojs/markdoc";
import mdx from '@astrojs/mdx';
import { defineConfig } from 'astro/config';
// https://astro.build/config
export default defineConfig({
integrations: [mdx(), markdoc()],
experimental: {
contentLayer: true,
contentIntellisense: true
}
});

10
packages/astro/test/fixtures/content-intellisense/package.json vendored Normal file
View file

@ -0,0 +1,10 @@
{
"name": "@test/content-intellisense",
"version": "0.0.0",
"private": true,
"dependencies": {
"astro": "workspace:*",
"@astrojs/mdx": "workspace:*",
"@astrojs/markdoc": "workspace:*"
}
}

3
packages/astro/test/fixtures/content-intellisense/src/blog-cl/entry.md vendored Normal file
View file

@ -0,0 +1,3 @@
---
title: "Markdown"
---

3
packages/astro/test/fixtures/content-intellisense/src/blog-cl/entry2.mdx vendored Normal file
View file

@ -0,0 +1,3 @@
---
title: "MDX"
---

3
packages/astro/test/fixtures/content-intellisense/src/blog-cl/entry3.mdoc vendored Normal file
View file

@ -0,0 +1,3 @@
---
title: "Markdoc"
---

3
packages/astro/test/fixtures/content-intellisense/src/content/blog-cc/entry.md vendored Normal file
View file

@ -0,0 +1,3 @@
---
title: "Markdown"
---

3
packages/astro/test/fixtures/content-intellisense/src/content/blog-cc/entry2.mdx vendored Normal file
View file

@ -0,0 +1,3 @@
---
title: "MDX"
---

3
packages/astro/test/fixtures/content-intellisense/src/content/blog-cc/entry3.mdoc vendored Normal file
View file

@ -0,0 +1,3 @@
---
title: "Markdoc"
---

24
packages/astro/test/fixtures/content-intellisense/src/content/config.ts vendored Normal file
View file

@ -0,0 +1,24 @@
import { glob } from 'astro/loaders';
import { defineCollection, z } from 'astro:content';
const blogCC = defineCollection({
type: 'content',
schema: z.object({
title: z.string(),
description: z.string().optional(),
}),
});
const blogCL = defineCollection({
// By default the ID is a slug, generated from the path of the file relative to `base`
loader: glob({ pattern: "**/*", base: "./src/blog-cl" }),
schema: z.object({
title: z.string(),
description: z.string().optional(),
}),
});
export const collections = {
"blog-cc": blogCC,
"blog-cl": blogCL,
};

8
packages/astro/test/fixtures/content-intellisense/src/utils.js vendored Normal file
View file

@ -0,0 +1,8 @@
export function stripRenderFn(entryWithRender) {
const { render, ...entry } = entryWithRender;
return entry;
}
export function stripAllRenderFn(collection = []) {
return collection.map(stripRenderFn);
}

9
packages/astro/test/fixtures/content-layer-markdoc/astro.config.mjs vendored Normal file
View file

@ -0,0 +1,9 @@
import markdoc from '@astrojs/markdoc';
import preact from '@astrojs/preact';
import { defineConfig } from 'astro/config';
// https://astro.build/config
export default defineConfig({
integrations: [markdoc(), preact()],
experimental: { contentLayer: true }
});

3
packages/astro/test/fixtures/content-layer-markdoc/content/_nested.mdoc vendored Normal file
View file

@ -0,0 +1,3 @@
Render components from a deeply nested partial:
{% deeply-nested /%}

7
packages/astro/test/fixtures/content-layer-markdoc/content/blog/_counter.mdoc vendored Normal file
View file

@ -0,0 +1,7 @@
# Hello from a partial!
Render a component from a partial:
{% counter /%}
{% partial file="../_nested.mdoc" /%}

19
packages/astro/test/fixtures/content-layer-markdoc/content/blog/with-components.mdoc vendored Normal file
View file

@ -0,0 +1,19 @@
---
title: Post with components
---
## Post with components
This uses a custom marquee component with a shortcode:
{% marquee-element direction="right" %}
I'm a marquee too!
{% /marquee-element %}
{% partial file="_counter.mdoc" /%}
And a code component for code blocks:
```js
const isRenderedWithShiki = true;
```

32
packages/astro/test/fixtures/content-layer-markdoc/markdoc.config.ts vendored Normal file
View file

@ -0,0 +1,32 @@
import { Markdoc, component, defineMarkdocConfig } from '@astrojs/markdoc/config';
export default defineMarkdocConfig({
nodes: {
fence: {
render: component('./src/components/Code.astro'),
attributes: {
language: { type: String },
content: { type: String },
},
},
},
tags: {
'marquee-element': {
render: component('./src/components/CustomMarquee.astro'),
attributes: {
direction: {
type: String,
default: 'left',
matches: ['left', 'right', 'up', 'down'],
errorLevel: 'critical',
},
},
},
counter: {
render: component('./src/components/CounterWrapper.astro'),
},
'deeply-nested': {
render: component('./src/components/DeeplyNested.astro'),
},
},
});

11
packages/astro/test/fixtures/content-layer-markdoc/package.json vendored Normal file
View file

@ -0,0 +1,11 @@
{
"name": "@test/content-layer-markdoc",
"version": "0.0.0",
"private": true,
"dependencies": {
"@astrojs/markdoc": "workspace:*",
"@astrojs/preact": "workspace:*",
"astro": "workspace:*",
"preact": "^10.23.1"
}
}

12
packages/astro/test/fixtures/content-layer-markdoc/src/components/Code.astro vendored Normal file
View file

@ -0,0 +1,12 @@
---
import { Code } from 'astro/components';
type Props = {
content: string;
language: string;
}
const { content, language } = Astro.props as Props;
---
<Code lang={language} code={content} />

10
packages/astro/test/fixtures/content-layer-markdoc/src/components/Counter.tsx vendored Normal file
View file

@ -0,0 +1,10 @@
import { useState } from 'preact/hooks';
export default function Counter() {
const [count, setCount] = useState(1);
return (
<button id="counter" onClick={() => setCount(count + 1)}>
{count}
</button>
);
}

5
packages/astro/test/fixtures/content-layer-markdoc/src/components/CounterWrapper.astro vendored Normal file
View file

@ -0,0 +1,5 @@
---
import Counter from './Counter';
---
<Counter client:load />

Some files were not shown because too many files have changed in this diff Show more