Add Astro.fetchContent API (#91)

README.md

@@ -33,7 +33,7 @@ npm run dev
 
 ### ⚙️ Configuration
 
-To configure Astro, add a `astro.config.mjs` file in the root of your project. All of the options can be omitted. Here are the defaults:
+To configure Astro, add a `astro.config.mjs` file in the root of your project. All settings are optional. Here are the defaults:
 
 ```js
 export default {
@@ -50,6 +50,8 @@ export default {
     /** Set this to "preact" or "react" to determine what *.jsx files should load */
     '.jsx': 'react',
   },
+  /** Your public domain, e.g.: https://my-site.dev/ */
+  site: '',
 };
 ```
 
@@ -76,7 +78,7 @@ Our goal is to support all popular state management libraries, as long as there
 - **Vue:**
   - [ ] **Vuex: Partial Support** (Note: You can access a vuex store directly, but full `vuex` support requires the ability to set a custom `vue.use(store)` call to every component island. Planned.)
 
-*Are we missing your favorite state management library? Add it to the list above in a PR (or create an issue)!*
+_Are we missing your favorite state management library? Add it to the list above in a PR (or create an issue)!_
 
 ### 💅 Styling
 
@@ -113,7 +115,6 @@ Supports:
 - `lang="scss"`: load as the `.scss` extension
 - `lang="sass"`: load as the `.sass` extension (no brackets; indent-style)
 
-
 #### 🦊 Autoprefixer
 
 We also automatically add browser prefixes using [Autoprefixer][autoprefixer]. By default, Astro loads the default values, but you may also specify your own by placing a [Browserslist][browserslist] file in your project root.
@@ -146,9 +147,32 @@ Then write Tailwind in your project just like you’re used to:
 </style>
 ```
 
-#### 🍱 Collections (beta)
+## 🚀 Build & Deployment
 
-Astro’s Collections API can be used for paginating content whether local `*.md` files or data from a headless CMS.
+Add a `build` npm script to your `/package.json` file:
+
+```json
+{
+  "scripts": {
+    "dev": "astro dev .",
+    "build": "astro build ."
+  }
+}
+```
+
+Then run:
+
+```
+npm run build
+```
+
+Now upload the contents of `/_site_` to your favorite static site host.
+
+## 🥾 Guides
+
+### 🍱 Collections (beta)
+
+By default, any Astro component can fetch data from any API or local `*.md` files. But what if you had a blog you wanted to paginate? What if you wanted to generate dynamic URLs based on metadata (e.g. `/tag/[tag]/`)? Or do both together? Astro Collections are a way to do all of that. It’s perfect for generating blog-like content, or scaffolding out dynamic URLs from your data.
 
 First, decide on a URL schema. For our example, perhaps you want all your paginated posts at `/posts/1`, `/posts/2`, etc. But in addition, you also wanted `/tag/[tag]` and `/year/[year]` collections where posts are filtered by tag or year.
 
@@ -180,7 +204,7 @@ export async function createCollection() {
 
 These are important so your data is exposed to the page as a prop, and also Astro has everything it needs to gather your data and generate the proper routes. How it does this is more clear if we walk through a practical example.
 
-##### Example 1: Simple pagination
+#### Example 1: Simple pagination
 
 Assume we have Markdown files that have `title`, `tag`, and `date` in their frontmatter, like so:
 
@@ -215,7 +239,7 @@ import PostPreview from '../components/PostPreview.astro';
 export let collection: any;
 
 export async function createCollection() {
-  const allPosts = import.meta.fetchContent('./post/*.md');     // load data that already lives at `/post/[slug]`
+  const allPosts = Astro.fetchContent('./post/*.md');           // load data that already lives at `/post/[slug]`
   allPosts.sort((a, b) => new Date(b.date) - new Date(a.date)); // sort newest -> oldest (we got "date" from frontmatter!)
 
   // (load more data here, if needed)
@@ -264,7 +288,7 @@ Let’s walk through some of the key parts:
 
 It should be noted that the above example shows `<PostPreview />` and `<Pagination />` components. Pretend those are custom components that you made to display the post data, and the pagination navigation. There’s nothing special about them; only consider those examples of how you’d use collection data to display everything the way you’d like.
 
-##### Example 2: Advanced filtering & pagination
+#### Example 2: Advanced filtering & pagination
 
 In our earlier example, we covered simple pagination for `/posts/1`, but we’d still like to make `/tag/[tag]/1` and `/year/[year]/1`. To do that, we’ll create 2 more collections: `/astro/pages/$tag.astro` and `astro/pages/$year.astro`. Assume that the markup is the same, but we’ve expanded the `createCollection()` function with more data.
 
@@ -277,7 +301,7 @@ In our earlier example, we covered simple pagination for `/posts/1`, but we’d
   export let collection: any;
 
   export async function createCollection() {
-    const allPosts = import.meta.fetchContent('./post/*.md');
+    const allPosts = Astro.fetchContent('./post/*.md');
     allPosts.sort((a, b) => new Date(b.date) - new Date(a.date));
 +   const allTags = [...new Set(allPosts.map((post) => post.tags).flat())];  // gather all unique tags (we got "tag" from frontmatter!)
 +   allTags.sort((a, b) => a.localeCompare(b));                              // sort tags A -> Z
@@ -309,36 +333,63 @@ These are still paginated, too! But since there are other conditions applied, th
 
 Lastly, what about `/year/*`? Well hopefully you can figure that out from here. It follows the exact same pattern, except using `post.date` frontmatter. You’ll grab the year from that date string, and sort probably newest to oldest rather than alphabetical. You’ll also change `params.tag` to `params.year` (or whatever you name it), but otherwise most everything else should be the same.
 
-##### Tips
+#### Tips
 
 - Having to load different collections in different `$[collection].astro` files might seem like a pain at first, until you remember **you can create reusable components!** Treat `/pages/*.astro` files as your one-off routing & data fetching logic, and treat `/components/*.astro` as your reusable markup. If you find yourself duplicating things too much, you can probably use a component instead!
 - Stay true to `/pages/$[collection].astro` naming. If you have an `/all-posts/*` route, then use `/pages/$all-posts.astro` to manage that. Don’t try and trick `permalink` to generate too many URL trees; it’ll only result in pages being missed when it comes time to build.
-- Need to load local markdown? Try `import.meta.fetchContent('./data/*.md')`
-- Need to load remote data? Simply `fetch()` to make it happen!
+- Need to load local markdown? Try [`Astro.fetchContent()`][fetch-content]
+- Need to load remote data from an API? Simply `fetch()` to make it happen!
 
-## 🚀 Build & Deployment
+## 📚 API
 
-Add a `build` npm script to your `/package.json` file:
+### `Astro` global
 
-```json
+The `Astro` global is available in all contexts in `.astro` files. It has the following functions:
+
+#### `config`
+
+`Astro.config` returns an object with the following properties:
+
+| Name   | Type     | Description                                                                                                |
+| :----- | :------- | :--------------------------------------------------------------------------------------------------------- |
+| `site` | `string` | Your website’s public root domain. Set it with `site: "https://mysite.com"` in your [Astro config][config] |
+
+#### `fetchContent()`
+
+`Astro.fetchContent()` is a way to load local `*.md` files into your static site setup. You can either use this on its own, or within [Astro Collections][collections].
+
+```
+// ./astro/components/my-component.astro
+---
+const data = Astro.fetchContent('../pages/post/*.md'); // returns an array of posts that live at ./astro/pages/post/*.md
+---
+
+<div>
+{data.slice(0, 3).map((post) => (
+  <article>
+    <h1>{post.title}</h1>
+    <p>{post.description}</p>
+    <a href={post.url}>Read more</a>
+  </article>
+))}
+</div>
+```
+
+`.fetchContent()` only takes one parameter: a relative URL glob of which local files you’d like to import. Currently only `*.md` files are supported. It’s synchronous, and returns an array of items of type:
+
+```
 {
-  "scripts": {
-    "dev": "astro dev .",
-    "build": "astro build ."
-  }
-}
+  url: string;     // the URL of this item (if it’s in pages/)
+  content: string; // the HTML of this item
+  // frontmatter data expanded here
+}[];
 ```
 
-Then run:
-
-```
-npm run build
-```
-
-Now upload the contents of `/_site_` to your favorite static site host.
-
 [autoprefixer]: https://github.com/postcss/autoprefixer
 [browserslist]: https://github.com/browserslist/browserslist
+[collections]: #-collections-beta
+[config]: #%EF%B8%8F-configuration
+[fetch-content]: #fetchContent--
 [intersection-observer]: https://developer.mozilla.org/en-US/docs/Web/API/Intersection_Observer_API
 [request-idle-cb]: https://developer.mozilla.org/en-US/docs/Web/API/Window/requestIdleCallback
 [sass]: https://sass-lang.com/

examples/blog/.gitignore

@@ -0,0 +1 @@
+_site

examples/blog/astro/pages/$posts.astro

@@ -14,7 +14,7 @@ export let collection: any;
 export async function createCollection() {
   return {
     async data() {
-      let allPosts = await import.meta.fetchContent('./post/*.md');
+      let allPosts = Astro.fetchContent('./post/*.md');
       allPosts.sort((a, b) => new Date(b.date) - new Date(a.date));
       return allPosts;
     },

examples/blog/astro/pages/$tag.astro

@@ -12,7 +12,7 @@ let description = 'An example blog on Astro';
 import authorData from '../data/authors.json';
 export let collection: any;
 export async function createCollection() {
-  let allPosts = import.meta.fetchContent('./post/*.md');
+  let allPosts = Astro.fetchContent('./post/*.md');
   let allTags = new Set();
   let routes = [];
   for (const post of allPosts) {

examples/blog/astro/pages/index.astro

@@ -13,7 +13,7 @@ let description = 'An example blog on Astro';
 // so we show a preview of posts here, but actually paginate from $posts.astro
 import authorData from '../data/authors.json';
 
-let allPosts = import.meta.fetchContent('./post/*.md');
+let allPosts = Astro.fetchContent('./post/*.md');
 allPosts.sort((a, b) => new Date(b.date) - new Date(a.date));
 let firstThree = allPosts.slice(0, 3);
 ---

package.json

@@ -93,5 +93,8 @@
     "typescript": "^4.2.3",
     "uvu": "^0.5.1"
   },
-  "engines": { "node": "~14.0.0", "npm" : ">=6.14.0 <7.0.0" }
+  "engines": {
+    "node": "~14.0.0",
+    "npm": ">=6.14.0 <7.0.0"
+  }
 }

src/compiler/codegen/content.ts

@@ -52,7 +52,7 @@ function globSearch(spec: string, { filename }: { filename: string }): string[]
   }
 }
 
-/** import.meta.fetchContent() */
+/** Astro.fetchContent() */
 export function fetchContent(spec: string, { namespace, filename }: GlobOptions): GlobResult {
   let code = '';
   const imports = new Set<string>();

src/compiler/codegen/index.ts

@@ -13,7 +13,7 @@ import * as babelTraverse from '@babel/traverse';
 import { ImportDeclaration, ExportNamedDeclaration, VariableDeclarator, Identifier } from '@babel/types';
 import { warn } from '../../logger.js';
 import { fetchContent } from './content.js';
-import { isImportMetaDeclaration } from './utils.js';
+import { isFetchContent, isImportMetaDeclaration } from './utils.js';
 import { yellow } from 'kleur/colors';
 
 const traverse: typeof babelTraverse.default = (babelTraverse.default as any).default;
@@ -305,7 +305,7 @@ function compileModule(module: Script, state: CodegenState, compileOptions: Comp
 
   let script = '';
   let propsStatement = '';
-  let contentCode = ''; // code for handling import.meta.fetchContent(), if any;
+  let contentCode = ''; // code for handling Astro.fetchContent(), if any;
   let createCollection = ''; // function for executing collection
   const componentPlugins = new Set<ValidExtensionPlugins>();
 
@@ -354,8 +354,8 @@ function compileModule(module: Script, state: CodegenState, compileOptions: Comp
         }
         case 'VariableDeclaration': {
           for (const declaration of node.declarations) {
-            // only select import.meta.fetchContent() calls here. this utility filters those out for us.
-            if (!isImportMetaDeclaration(declaration, 'fetchContent')) continue;
+            // only select Astro.fetchContent() calls here. this utility filters those out for us.
+            if (!isFetchContent(declaration)) continue;
 
             // remove node
             body.splice(i, 1);
@@ -366,7 +366,7 @@ function compileModule(module: Script, state: CodegenState, compileOptions: Comp
             if (init.type === 'AwaitExpression') {
               init = init.argument;
               const shortname = path.relative(compileOptions.astroConfig.projectRoot.pathname, state.filename);
-              warn(compileOptions.logging, shortname, yellow('awaiting import.meta.fetchContent() not necessary'));
+              warn(compileOptions.logging, shortname, yellow('awaiting Astro.fetchContent() not necessary'));
             }
             if (init.type !== 'CallExpression') continue;
 
@@ -374,7 +374,7 @@ function compileModule(module: Script, state: CodegenState, compileOptions: Comp
             const namespace = id.name;
 
             if ((init as any).arguments[0].type !== 'StringLiteral') {
-              throw new Error(`[import.meta.fetchContent] Only string literals allowed, ex: \`import.meta.fetchContent('./post/*.md')\`\n  ${state.filename}`);
+              throw new Error(`[Astro.fetchContent] Only string literals allowed, ex: \`Astro.fetchContent('./post/*.md')\`\n  ${state.filename}`);
             }
             const spec = (init as any).arguments[0].value;
             if (typeof spec === 'string') contentImports.set(namespace, { spec, declarator: node.kind });
@@ -432,8 +432,8 @@ function compileModule(module: Script, state: CodegenState, compileOptions: Comp
           switch (node.type) {
             case 'VariableDeclaration': {
               for (const declaration of node.declarations) {
-                // only select import.meta.collection() calls here. this utility filters those out for us.
-                if (!isImportMetaDeclaration(declaration, 'fetchContent')) continue;
+                // only select Astro.fetchContent() calls here. this utility filters those out for us.
+                if (!isFetchContent(declaration)) continue;
 
                 // a bit of munging
                 let { id, init } = declaration;
@@ -441,7 +441,7 @@ function compileModule(module: Script, state: CodegenState, compileOptions: Comp
                 if (init.type === 'AwaitExpression') {
                   init = init.argument;
                   const shortname = path.relative(compileOptions.astroConfig.projectRoot.pathname, state.filename);
-                  warn(compileOptions.logging, shortname, yellow('awaiting import.meta.fetchContent() not necessary'));
+                  warn(compileOptions.logging, shortname, yellow('awaiting Astro.fetchContent() not necessary'));
                 }
                 if (init.type !== 'CallExpression') continue;
 
@@ -449,7 +449,7 @@ function compileModule(module: Script, state: CodegenState, compileOptions: Comp
                 const namespace = id.name;
 
                 if ((init as any).arguments[0].type !== 'StringLiteral') {
-                  throw new Error(`[import.meta.fetchContent] Only string literals allowed, ex: \`import.meta.fetchContent('./post/*.md')\`\n  ${state.filename}`);
+                  throw new Error(`[Astro.fetchContent] Only string literals allowed, ex: \`Astro.fetchContent('./post/*.md')\`\n  ${state.filename}`);
                 }
                 const spec = (init as any).arguments[0].value;
                 if (typeof spec !== 'string') break;
@@ -462,7 +462,7 @@ function compileModule(module: Script, state: CodegenState, compileOptions: Comp
                 }
 
                 createCollection =
-                  imports + '\n\nexport ' + createCollection.substring(0, declaration.start || 0) + globResult.code + createCollection.substring(declaration.end || 0);
+                  imports + '\nexport ' + createCollection.substring(0, declaration.start || 0) + globResult.code + createCollection.substring(declaration.end || 0);
               }
               break;
             }
@@ -471,8 +471,8 @@ function compileModule(module: Script, state: CodegenState, compileOptions: Comp
       });
     }
 
-    // import.meta.fetchContent()
-    for (const [namespace, { declarator, spec }] of contentImports.entries()) {
+    // Astro.fetchContent()
+    for (const [namespace, { spec }] of contentImports.entries()) {
       const globResult = fetchContent(spec, { namespace, filename: state.filename });
       for (const importStatement of globResult.imports) {
         state.importExportStatements.add(importStatement);

src/compiler/codegen/utils.ts

@@ -18,3 +18,22 @@ export function isImportMetaDeclaration(declaration: VariableDeclarator, metaNam
   if (metaName && (init.callee.property.type !== 'Identifier' || init.callee.property.name !== metaName)) return false;
   return true;
 }
+
+/** Is this an Astro.fetchContent() call? */
+export function isFetchContent(declaration: VariableDeclarator): boolean {
+  let { init } = declaration;
+  if (!init) return false; // definitely not import.meta
+  // this could be `await import.meta`; if so, evaluate that:
+  if (init.type === 'AwaitExpression') {
+    init = init.argument;
+  }
+  // continue evaluating
+  if (
+    init.type !== 'CallExpression' ||
+    init.callee.type !== 'MemberExpression' ||
+    (init.callee.object as any).name !== 'Astro' ||
+    (init.callee.property as any).name !== 'fetchContent'
+  )
+    return false;
+  return true;
+}

src/runtime.ts

@@ -126,10 +126,12 @@ async function load(config: RuntimeConfig, rawPathname: string | undefined): Pro
         collection.page.last = Math.ceil(data.length / pageSize);
         // TODO: fix the .replace() hack
         if (end < data.length) {
-          collection.url.next = collection.url.current.replace(/\d+$/, `${searchResult.currentPage + 1}`);
+          collection.url.next = collection.url.current.replace(/(\/\d+)?$/, `/${searchResult.currentPage + 1}`);
         }
         if (searchResult.currentPage > 1) {
-          collection.url.prev = collection.url.current.replace(/\d+$/, `${searchResult.currentPage - 1 || 1}`);
+          collection.url.prev = collection.url.current
+            .replace(/\d+$/, `${searchResult.currentPage - 1 || 1}`) // update page #
+            .replace(/\/1$/, ''); // if end is `/1`, then just omit
         }
 
         data = data.slice(start, end);

src/search.ts

@@ -85,7 +85,7 @@ export function searchForPage(url: URL, astroRoot: URL): SearchResult {
         statusCode: 200,
         location: collection.location,
         pathname: reqPath,
-        currentPage: collection.currentPage,
+        currentPage: collection.currentPage || 1,
       };
     }
   }

test/astro-basic.test.js

@@ -3,7 +3,7 @@ import * as assert from 'uvu/assert';
 import { doc } from './test-utils.js';
 import { setup } from './helpers.js';
 
-const Basics = suite('Search paths');
+const Basics = suite('Basic test');
 
 setup(Basics, './fixtures/astro-basic');
 

test/astro-collection.test.js

@@ -0,0 +1,30 @@
+import { suite } from 'uvu';
+import * as assert from 'uvu/assert';
+import { doc } from './test-utils.js';
+import { setup } from './helpers.js';
+
+const Collections = suite('Collections');
+
+setup(Collections, './fixtures/astro-collection');
+
+Collections('generates list & sorts successfully', async ({ runtime }) => {
+  const result = await runtime.load('/posts');
+  const $ = doc(result.contents);
+  const urls = [
+    ...$('#posts a').map(function () {
+      return $(this).attr('href');
+    }),
+  ];
+  assert.equal(urls, ['/post/three', '/post/two']);
+});
+
+Collections('generates pagination successfully', async ({ runtime }) => {
+  const result = await runtime.load('/posts');
+  const $ = doc(result.contents);
+  const prev = $('#prev-page');
+  const next = $('#next-page');
+  assert.equal(prev.length, 0); // this is first page; should be missing
+  assert.equal(next.length, 1); // this should be on-page
+});
+
+Collections.run();

test/astro-styles-ssr.test.js

@@ -1,13 +1,10 @@
 import { suite } from 'uvu';
 import * as assert from 'uvu/assert';
-import { createRuntime } from '../lib/runtime.js';
-import { loadConfig } from '../lib/config.js';
 import { doc } from './test-utils.js';
+import { setup } from './helpers.js';
 
 const StylesSSR = suite('Styles SSR');
 
-let runtime;
-
 /** Basic CSS minification; removes some flakiness in testing CSS */
 function cssMinify(css) {
   return css
@@ -18,22 +15,9 @@ function cssMinify(css) {
     .replace(/;}/g, '}'); // collapse block
 }
 
-StylesSSR.before(async () => {
-  const astroConfig = await loadConfig(new URL('./fixtures/astro-styles-ssr', import.meta.url).pathname);
+setup(StylesSSR, './fixtures/astro-styles-ssr');
 
-  const logging = {
-    level: 'error',
-    dest: process.stderr,
-  };
-
-  runtime = await createRuntime(astroConfig, { logging });
-});
-
-StylesSSR.after(async () => {
-  (await runtime) && runtime.shutdown();
-});
-
-StylesSSR('Has <link> tags', async () => {
+StylesSSR('Has <link> tags', async ({ runtime }) => {
   const MUST_HAVE_LINK_TAGS = [
     '/_astro/components/ReactCSS.css',
     '/_astro/components/SvelteScoped.svelte.css',
@@ -51,7 +35,7 @@ StylesSSR('Has <link> tags', async () => {
   }
 });
 
-StylesSSR('Has correct CSS classes', async () => {
+StylesSSR('Has correct CSS classes', async ({ runtime }) => {
   const result = await runtime.load('/');
   const $ = doc(result.contents);
 
@@ -81,7 +65,7 @@ StylesSSR('Has correct CSS classes', async () => {
   }
 });
 
-StylesSSR('CSS Module support in .astro', async () => {
+StylesSSR('CSS Module support in .astro', async ({ runtime }) => {
   const result = await runtime.load('/');
   const $ = doc(result.contents);
 

test/fixtures/astro-collection/astro/pages/$posts.astro

@@ -0,0 +1,28 @@
+---
+export let collection: any;
+
+export async function createCollection() {
+  return {
+    async data() {
+      let data = Astro.fetchContent('./post/*.md');
+      data.sort((a, b) => new Date(b.date) - new Date(a.date));
+      return data;
+    },
+    pageSize: 2
+  };
+}
+---
+
+<div id="posts">
+{collection.data.map((post) => (
+  <article>
+    <h1>{post.title}</h1>
+    <a href={post.url}>Read more</a>
+  </article>
+))}
+</div>
+
+<nav>
+  {collection.url.prev && <a id="prev-page" href={collection.url.prev}>Previous page</a>}
+  {collection.url.next && <a id="next-page" href={collection.url.next}>Next page</a>}
+</nav>

test/fixtures/astro-collection/astro/pages/post/one.md

@@ -0,0 +1,8 @@
+---
+title: Post One
+date: 2021-04-13 00:00:00
+---
+
+# Post One
+
+I’m the first blog post

test/fixtures/astro-collection/astro/pages/post/three.md

@@ -0,0 +1,8 @@
+---
+title: Post Three
+date: 2021-04-15 00:00:00
+---
+
+# Post Three
+
+I’m the third blog post

test/fixtures/astro-collection/astro/pages/post/two.md

@@ -0,0 +1,8 @@
+---
+title: Post Two
+date: 2021-04-14 00:00:00
+---
+
+# Post Two
+
+I’m the second blog post