Add Astro.fetchContent API (#91)

README.md

@@ -8,7 +8,7 @@ A next-generation static-site generator with partial hydration. Use your favorit
 # currently hidden during private beta, please don't share :)
 npm install astro@shhhhh
 
-# NOTE: There is currently a bug in Snowpack that prevents you 
+# NOTE: There is currently a bug in Snowpack that prevents you
 # from using astro outside of the monorepo setup that we have here.
 # For now, do all development inside the `examples/` directory for this repo.
 ```
@@ -33,7 +33,7 @@ npm run dev
 
 ### ⚙️ Configuration
 
-To configure Astro, add a `astro.config.mjs` file in the root of your project. All of the options can be omitted. Here are the defaults:
+To configure Astro, add a `astro.config.mjs` file in the root of your project. All settings are optional. Here are the defaults:
 
 ```js
 export default {
@@ -50,6 +50,8 @@ export default {
     /** Set this to "preact" or "react" to determine what *.jsx files should load */
     '.jsx': 'react',
   },
+  /** Your public domain, e.g.: https://my-site.dev/ */
+  site: '',
 };
 ```
 
@@ -64,7 +66,7 @@ By default, Astro outputs zero client-side JS. If you'd like to include an inter
 
 ### ⚛️ State Management
 
-Frontend state management depends on your framework of choice. Below is a list of popular frontend state management libraries, and their current support with Astro. 
+Frontend state management depends on your framework of choice. Below is a list of popular frontend state management libraries, and their current support with Astro.
 
 Our goal is to support all popular state management libraries, as long as there is no technical reason that we cannot.
 
@@ -72,11 +74,11 @@ Our goal is to support all popular state management libraries, as long as there
   - [ ] **Redux: Partial Support** (Note: You can access a Redux store directly, but full `react-redux` support requires the ability to set a custom `<Provider>` wrapper to every component island. Planned.)
   - [x] **Recoil: Full Support**
 - **Svelte**
-  - [x] **Svelte Stores: Full Support** 
+  - [x] **Svelte Stores: Full Support**
 - **Vue:**
   - [ ] **Vuex: Partial Support** (Note: You can access a vuex store directly, but full `vuex` support requires the ability to set a custom `vue.use(store)` call to every component island. Planned.)
 
-*Are we missing your favorite state management library? Add it to the list above in a PR (or create an issue)!*
+_Are we missing your favorite state management library? Add it to the list above in a PR (or create an issue)!_
 
 ### 💅 Styling
 
@@ -113,7 +115,6 @@ Supports:
 - `lang="scss"`: load as the `.scss` extension
 - `lang="sass"`: load as the `.sass` extension (no brackets; indent-style)
 
-
 #### 🦊 Autoprefixer
 
 We also automatically add browser prefixes using [Autoprefixer][autoprefixer]. By default, Astro loads the default values, but you may also specify your own by placing a [Browserslist][browserslist] file in your project root.
@@ -146,9 +147,32 @@ Then write Tailwind in your project just like you’re used to:
 </style>
 ```
 
-#### 🍱 Collections (beta)
+## 🚀 Build & Deployment
 
-Astro’s Collections API can be used for paginating content whether local `*.md` files or data from a headless CMS.
+Add a `build` npm script to your `/package.json` file:
+
+```json
+{
+  "scripts": {
+    "dev": "astro dev .",
+    "build": "astro build ."
+  }
+}
+```
+
+Then run:
+
+```
+npm run build
+```
+
+Now upload the contents of `/_site_` to your favorite static site host.
+
+## 🥾 Guides
+
+### 🍱 Collections (beta)
+
+By default, any Astro component can fetch data from any API or local `*.md` files. But what if you had a blog you wanted to paginate? What if you wanted to generate dynamic URLs based on metadata (e.g. `/tag/[tag]/`)? Or do both together? Astro Collections are a way to do all of that. It’s perfect for generating blog-like content, or scaffolding out dynamic URLs from your data.
 
 First, decide on a URL schema. For our example, perhaps you want all your paginated posts at `/posts/1`, `/posts/2`, etc. But in addition, you also wanted `/tag/[tag]` and `/year/[year]` collections where posts are filtered by tag or year.
 
@@ -180,7 +204,7 @@ export async function createCollection() {
 
 These are important so your data is exposed to the page as a prop, and also Astro has everything it needs to gather your data and generate the proper routes. How it does this is more clear if we walk through a practical example.
 
-##### Example 1: Simple pagination
+#### Example 1: Simple pagination
 
 Assume we have Markdown files that have `title`, `tag`, and `date` in their frontmatter, like so:
 
@@ -215,7 +239,7 @@ import PostPreview from '../components/PostPreview.astro';
 export let collection: any;
 
 export async function createCollection() {
-  const allPosts = import.meta.fetchContent('./post/*.md');     // load data that already lives at `/post/[slug]`
+  const allPosts = Astro.fetchContent('./post/*.md');           // load data that already lives at `/post/[slug]`
   allPosts.sort((a, b) => new Date(b.date) - new Date(a.date)); // sort newest -> oldest (we got "date" from frontmatter!)
 
   // (load more data here, if needed)
@@ -264,7 +288,7 @@ Let’s walk through some of the key parts:
 
 It should be noted that the above example shows `<PostPreview />` and `<Pagination />` components. Pretend those are custom components that you made to display the post data, and the pagination navigation. There’s nothing special about them; only consider those examples of how you’d use collection data to display everything the way you’d like.
 
-##### Example 2: Advanced filtering & pagination
+#### Example 2: Advanced filtering & pagination
 
 In our earlier example, we covered simple pagination for `/posts/1`, but we’d still like to make `/tag/[tag]/1` and `/year/[year]/1`. To do that, we’ll create 2 more collections: `/astro/pages/$tag.astro` and `astro/pages/$year.astro`. Assume that the markup is the same, but we’ve expanded the `createCollection()` function with more data.
 
@@ -277,7 +301,7 @@ In our earlier example, we covered simple pagination for `/posts/1`, but we’d
   export let collection: any;
 
   export async function createCollection() {
-    const allPosts = import.meta.fetchContent('./post/*.md');
+    const allPosts = Astro.fetchContent('./post/*.md');
     allPosts.sort((a, b) => new Date(b.date) - new Date(a.date));
 +   const allTags = [...new Set(allPosts.map((post) => post.tags).flat())];  // gather all unique tags (we got "tag" from frontmatter!)
 +   allTags.sort((a, b) => a.localeCompare(b));                              // sort tags A -> Z
@@ -309,36 +333,63 @@ These are still paginated, too! But since there are other conditions applied, th
 
 Lastly, what about `/year/*`? Well hopefully you can figure that out from here. It follows the exact same pattern, except using `post.date` frontmatter. You’ll grab the year from that date string, and sort probably newest to oldest rather than alphabetical. You’ll also change `params.tag` to `params.year` (or whatever you name it), but otherwise most everything else should be the same.
 
-##### Tips
+#### Tips
 
 - Having to load different collections in different `$[collection].astro` files might seem like a pain at first, until you remember **you can create reusable components!** Treat `/pages/*.astro` files as your one-off routing & data fetching logic, and treat `/components/*.astro` as your reusable markup. If you find yourself duplicating things too much, you can probably use a component instead!
 - Stay true to `/pages/$[collection].astro` naming. If you have an `/all-posts/*` route, then use `/pages/$all-posts.astro` to manage that. Don’t try and trick `permalink` to generate too many URL trees; it’ll only result in pages being missed when it comes time to build.
-- Need to load local markdown? Try `import.meta.fetchContent('./data/*.md')`
+- Need to load local markdown? Try [`Astro.fetchContent()`][fetch-content]
-- Need to load remote data? Simply `fetch()` to make it happen!
+- Need to load remote data from an API? Simply `fetch()` to make it happen!
 
-## 🚀 Build & Deployment
+## 📚 API
 
-Add a `build` npm script to your `/package.json` file:
+### `Astro` global
 
-```json
+The `Astro` global is available in all contexts in `.astro` files. It has the following functions:
+
+#### `config`
+
+`Astro.config` returns an object with the following properties:
+
+| Name   | Type     | Description                                                                                                |
+| :----- | :------- | :--------------------------------------------------------------------------------------------------------- |
+| `site` | `string` | Your website’s public root domain. Set it with `site: "https://mysite.com"` in your [Astro config][config] |
+
+#### `fetchContent()`
+
+`Astro.fetchContent()` is a way to load local `*.md` files into your static site setup. You can either use this on its own, or within [Astro Collections][collections].
+
+```
+// ./astro/components/my-component.astro
+---
+const data = Astro.fetchContent('../pages/post/*.md'); // returns an array of posts that live at ./astro/pages/post/*.md
+---
+
+<div>
+{data.slice(0, 3).map((post) => (
+  <article>
+    <h1>{post.title}</h1>
+    <p>{post.description}</p>
+    <a href={post.url}>Read more</a>
+  </article>
+))}
+</div>
+```
+
+`.fetchContent()` only takes one parameter: a relative URL glob of which local files you’d like to import. Currently only `*.md` files are supported. It’s synchronous, and returns an array of items of type:
+
+```
 {
-  "scripts": {
+  url: string;     // the URL of this item (if it’s in pages/)
-    "dev": "astro dev .",
+  content: string; // the HTML of this item
-    "build": "astro build ."
+  // frontmatter data expanded here
-  }
+}[];
-}
 ```
 
-Then run:
-
-```
-npm run build
-```
-
-Now upload the contents of `/_site_` to your favorite static site host.
-
 [autoprefixer]: https://github.com/postcss/autoprefixer
 [browserslist]: https://github.com/browserslist/browserslist
+[collections]: #-collections-beta
+[config]: #%EF%B8%8F-configuration
+[fetch-content]: #fetchContent--
 [intersection-observer]: https://developer.mozilla.org/en-US/docs/Web/API/Intersection_Observer_API
 [request-idle-cb]: https://developer.mozilla.org/en-US/docs/Web/API/Window/requestIdleCallback
 [sass]: https://sass-lang.com/

examples/blog/.gitignore

@@ -0,0 +1 @@
+_site

examples/blog/astro/pages/$posts.astro

@@ -14,7 +14,7 @@ export let collection: any;
 export async function createCollection() {
   return {
     async data() {
-      let allPosts = await import.meta.fetchContent('./post/*.md');
+      let allPosts = Astro.fetchContent('./post/*.md');
       allPosts.sort((a, b) => new Date(b.date) - new Date(a.date));
       return allPosts;
     },

examples/blog/astro/pages/$tag.astro

@@ -12,7 +12,7 @@ let description = 'An example blog on Astro';
 import authorData from '../data/authors.json';
 export let collection: any;
 export async function createCollection() {
-  let allPosts = import.meta.fetchContent('./post/*.md');
+  let allPosts = Astro.fetchContent('./post/*.md');
   let allTags = new Set();
   let routes = [];
   for (const post of allPosts) {

examples/blog/astro/pages/index.astro

@@ -13,7 +13,7 @@ let description = 'An example blog on Astro';
 // so we show a preview of posts here, but actually paginate from $posts.astro
 import authorData from '../data/authors.json';
 
-let allPosts = import.meta.fetchContent('./post/*.md');
+let allPosts = Astro.fetchContent('./post/*.md');
 allPosts.sort((a, b) => new Date(b.date) - new Date(a.date));
 let firstThree = allPosts.slice(0, 3);
 ---

package.json

@@ -93,5 +93,8 @@
     "typescript": "^4.2.3",
     "uvu": "^0.5.1"
   },
-  "engines": { "node": "~14.0.0", "npm" : ">=6.14.0 <7.0.0" }
+  "engines": {
+    "node": "~14.0.0",
+    "npm": ">=6.14.0 <7.0.0"
+  }
 }

src/compiler/codegen/content.ts

@@ -52,7 +52,7 @@ function globSearch(spec: string, { filename }: { filename: string }): string[]
   }
 }
 
-/** import.meta.fetchContent() */
+/** Astro.fetchContent() */
 export function fetchContent(spec: string, { namespace, filename }: GlobOptions): GlobResult {
   let code = '';
   const imports = new Set<string>();

src/compiler/codegen/index.ts

@@ -13,7 +13,7 @@ import * as babelTraverse from '@babel/traverse';
 import { ImportDeclaration, ExportNamedDeclaration, VariableDeclarator, Identifier } from '@babel/types';
 import { warn } from '../../logger.js';
 import { fetchContent } from './content.js';
-import { isImportMetaDeclaration } from './utils.js';
+import { isFetchContent, isImportMetaDeclaration } from './utils.js';
 import { yellow } from 'kleur/colors';
 
 const traverse: typeof babelTraverse.default = (babelTraverse.default as any).default;
@@ -305,7 +305,7 @@ function compileModule(module: Script, state: CodegenState, compileOptions: Comp
 
   let script = '';
   let propsStatement = '';
-  let contentCode = ''; // code for handling import.meta.fetchContent(), if any;
+  let contentCode = ''; // code for handling Astro.fetchContent(), if any;
   let createCollection = ''; // function for executing collection
   const componentPlugins = new Set<ValidExtensionPlugins>();
 
@@ -354,8 +354,8 @@ function compileModule(module: Script, state: CodegenState, compileOptions: Comp
         }
         case 'VariableDeclaration': {
           for (const declaration of node.declarations) {
-            // only select import.meta.fetchContent() calls here. this utility filters those out for us.
+            // only select Astro.fetchContent() calls here. this utility filters those out for us.
-            if (!isImportMetaDeclaration(declaration, 'fetchContent')) continue;
+            if (!isFetchContent(declaration)) continue;
 
             // remove node
             body.splice(i, 1);
@@ -366,7 +366,7 @@ function compileModule(module: Script, state: CodegenState, compileOptions: Comp
             if (init.type === 'AwaitExpression') {
               init = init.argument;
               const shortname = path.relative(compileOptions.astroConfig.projectRoot.pathname, state.filename);
-              warn(compileOptions.logging, shortname, yellow('awaiting import.meta.fetchContent() not necessary'));
+              warn(compileOptions.logging, shortname, yellow('awaiting Astro.fetchContent() not necessary'));
             }
             if (init.type !== 'CallExpression') continue;
 
@@ -374,7 +374,7 @@ function compileModule(module: Script, state: CodegenState, compileOptions: Comp
             const namespace = id.name;
 
             if ((init as any).arguments[0].type !== 'StringLiteral') {
-              throw new Error(`[import.meta.fetchContent] Only string literals allowed, ex: \`import.meta.fetchContent('./post/*.md')\`\n  ${state.filename}`);
+              throw new Error(`[Astro.fetchContent] Only string literals allowed, ex: \`Astro.fetchContent('./post/*.md')\`\n  ${state.filename}`);
             }
             const spec = (init as any).arguments[0].value;
             if (typeof spec === 'string') contentImports.set(namespace, { spec, declarator: node.kind });
@@ -432,8 +432,8 @@ function compileModule(module: Script, state: CodegenState, compileOptions: Comp
           switch (node.type) {
             case 'VariableDeclaration': {
               for (const declaration of node.declarations) {
-                // only select import.meta.collection() calls here. this utility filters those out for us.
+                // only select Astro.fetchContent() calls here. this utility filters those out for us.
-                if (!isImportMetaDeclaration(declaration, 'fetchContent')) continue;
+                if (!isFetchContent(declaration)) continue;
 
                 // a bit of munging
                 let { id, init } = declaration;
@@ -441,7 +441,7 @@ function compileModule(module: Script, state: CodegenState, compileOptions: Comp
                 if (init.type === 'AwaitExpression') {
                   init = init.argument;
                   const shortname = path.relative(compileOptions.astroConfig.projectRoot.pathname, state.filename);
-                  warn(compileOptions.logging, shortname, yellow('awaiting import.meta.fetchContent() not necessary'));
+                  warn(compileOptions.logging, shortname, yellow('awaiting Astro.fetchContent() not necessary'));
                 }
                 if (init.type !== 'CallExpression') continue;
 
@@ -449,7 +449,7 @@ function compileModule(module: Script, state: CodegenState, compileOptions: Comp
                 const namespace = id.name;
 
                 if ((init as any).arguments[0].type !== 'StringLiteral') {
-                  throw new Error(`[import.meta.fetchContent] Only string literals allowed, ex: \`import.meta.fetchContent('./post/*.md')\`\n  ${state.filename}`);
+                  throw new Error(`[Astro.fetchContent] Only string literals allowed, ex: \`Astro.fetchContent('./post/*.md')\`\n  ${state.filename}`);
                 }
                 const spec = (init as any).arguments[0].value;
                 if (typeof spec !== 'string') break;
@@ -462,7 +462,7 @@ function compileModule(module: Script, state: CodegenState, compileOptions: Comp
                 }
 
                 createCollection =
-                  imports + '\n\nexport ' + createCollection.substring(0, declaration.start || 0) + globResult.code + createCollection.substring(declaration.end || 0);
+                  imports + '\nexport ' + createCollection.substring(0, declaration.start || 0) + globResult.code + createCollection.substring(declaration.end || 0);
               }
               break;
             }
@@ -471,8 +471,8 @@ function compileModule(module: Script, state: CodegenState, compileOptions: Comp
       });
     }
 
-    // import.meta.fetchContent()
+    // Astro.fetchContent()
-    for (const [namespace, { declarator, spec }] of contentImports.entries()) {
+    for (const [namespace, { spec }] of contentImports.entries()) {
       const globResult = fetchContent(spec, { namespace, filename: state.filename });
       for (const importStatement of globResult.imports) {
         state.importExportStatements.add(importStatement);

src/compiler/codegen/utils.ts

@@ -18,3 +18,22 @@ export function isImportMetaDeclaration(declaration: VariableDeclarator, metaNam
   if (metaName && (init.callee.property.type !== 'Identifier' || init.callee.property.name !== metaName)) return false;
   return true;
 }
+
+/** Is this an Astro.fetchContent() call? */
+export function isFetchContent(declaration: VariableDeclarator): boolean {
+  let { init } = declaration;
+  if (!init) return false; // definitely not import.meta
+  // this could be `await import.meta`; if so, evaluate that:
+  if (init.type === 'AwaitExpression') {
+    init = init.argument;
+  }
+  // continue evaluating
+  if (
+    init.type !== 'CallExpression' ||
+    init.callee.type !== 'MemberExpression' ||
+    (init.callee.object as any).name !== 'Astro' ||
+    (init.callee.property as any).name !== 'fetchContent'
+  )
+    return false;
+  return true;
+}

src/runtime.ts

@@ -126,10 +126,12 @@ async function load(config: RuntimeConfig, rawPathname: string | undefined): Pro
         collection.page.last = Math.ceil(data.length / pageSize);
         // TODO: fix the .replace() hack
         if (end < data.length) {
-          collection.url.next = collection.url.current.replace(/\d+$/, `${searchResult.currentPage + 1}`);
+          collection.url.next = collection.url.current.replace(/(\/\d+)?$/, `/${searchResult.currentPage + 1}`);
         }
         if (searchResult.currentPage > 1) {
-          collection.url.prev = collection.url.current.replace(/\d+$/, `${searchResult.currentPage - 1 || 1}`);
+          collection.url.prev = collection.url.current
+            .replace(/\d+$/, `${searchResult.currentPage - 1 || 1}`) // update page #
+            .replace(/\/1$/, ''); // if end is `/1`, then just omit
         }
 
         data = data.slice(start, end);

src/search.ts

@@ -85,7 +85,7 @@ export function searchForPage(url: URL, astroRoot: URL): SearchResult {
         statusCode: 200,
         location: collection.location,
         pathname: reqPath,
-        currentPage: collection.currentPage,
+        currentPage: collection.currentPage || 1,
       };
     }
   }

test/astro-basic.test.js

@@ -3,7 +3,7 @@ import * as assert from 'uvu/assert';
 import { doc } from './test-utils.js';
 import { setup } from './helpers.js';
 
-const Basics = suite('Search paths');
+const Basics = suite('Basic test');
 
 setup(Basics, './fixtures/astro-basic');
 

test/astro-collection.test.js

@@ -0,0 +1,30 @@
+import { suite } from 'uvu';
+import * as assert from 'uvu/assert';
+import { doc } from './test-utils.js';
+import { setup } from './helpers.js';
+
+const Collections = suite('Collections');
+
+setup(Collections, './fixtures/astro-collection');
+
+Collections('generates list & sorts successfully', async ({ runtime }) => {
+  const result = await runtime.load('/posts');
+  const $ = doc(result.contents);
+  const urls = [
+    ...$('#posts a').map(function () {
+      return $(this).attr('href');
+    }),
+  ];
+  assert.equal(urls, ['/post/three', '/post/two']);
+});
+
+Collections('generates pagination successfully', async ({ runtime }) => {
+  const result = await runtime.load('/posts');
+  const $ = doc(result.contents);
+  const prev = $('#prev-page');
+  const next = $('#next-page');
+  assert.equal(prev.length, 0); // this is first page; should be missing
+  assert.equal(next.length, 1); // this should be on-page
+});
+
+Collections.run();

test/astro-styles-ssr.test.js

@@ -1,13 +1,10 @@
 import { suite } from 'uvu';
 import * as assert from 'uvu/assert';
-import { createRuntime } from '../lib/runtime.js';
-import { loadConfig } from '../lib/config.js';
 import { doc } from './test-utils.js';
+import { setup } from './helpers.js';
 
 const StylesSSR = suite('Styles SSR');
 
-let runtime;
-
 /** Basic CSS minification; removes some flakiness in testing CSS */
 function cssMinify(css) {
   return css
@@ -18,22 +15,9 @@ function cssMinify(css) {
     .replace(/;}/g, '}'); // collapse block
 }
 
-StylesSSR.before(async () => {
+setup(StylesSSR, './fixtures/astro-styles-ssr');
-  const astroConfig = await loadConfig(new URL('./fixtures/astro-styles-ssr', import.meta.url).pathname);
 
-  const logging = {
+StylesSSR('Has <link> tags', async ({ runtime }) => {
-    level: 'error',
-    dest: process.stderr,
-  };
-
-  runtime = await createRuntime(astroConfig, { logging });
-});
-
-StylesSSR.after(async () => {
-  (await runtime) && runtime.shutdown();
-});
-
-StylesSSR('Has <link> tags', async () => {
   const MUST_HAVE_LINK_TAGS = [
     '/_astro/components/ReactCSS.css',
     '/_astro/components/SvelteScoped.svelte.css',
@@ -51,7 +35,7 @@ StylesSSR('Has <link> tags', async () => {
   }
 });
 
-StylesSSR('Has correct CSS classes', async () => {
+StylesSSR('Has correct CSS classes', async ({ runtime }) => {
   const result = await runtime.load('/');
   const $ = doc(result.contents);
 
@@ -81,7 +65,7 @@ StylesSSR('Has correct CSS classes', async () => {
   }
 });
 
-StylesSSR('CSS Module support in .astro', async () => {
+StylesSSR('CSS Module support in .astro', async ({ runtime }) => {
   const result = await runtime.load('/');
   const $ = doc(result.contents);
 

test/fixtures/astro-collection/astro/pages/$posts.astro

@@ -0,0 +1,28 @@
+---
+export let collection: any;
+
+export async function createCollection() {
+  return {
+    async data() {
+      let data = Astro.fetchContent('./post/*.md');
+      data.sort((a, b) => new Date(b.date) - new Date(a.date));
+      return data;
+    },
+    pageSize: 2
+  };
+}
+---
+
+<div id="posts">
+{collection.data.map((post) => (
+  <article>
+    <h1>{post.title}</h1>
+    <a href={post.url}>Read more</a>
+  </article>
+))}
+</div>
+
+<nav>
+  {collection.url.prev && <a id="prev-page" href={collection.url.prev}>Previous page</a>}
+  {collection.url.next && <a id="next-page" href={collection.url.next}>Next page</a>}
+</nav>

test/fixtures/astro-collection/astro/pages/post/one.md

@@ -0,0 +1,8 @@
+---
+title: Post One
+date: 2021-04-13 00:00:00
+---
+
+# Post One
+
+I’m the first blog post

test/fixtures/astro-collection/astro/pages/post/three.md

@@ -0,0 +1,8 @@
+---
+title: Post Three
+date: 2021-04-15 00:00:00
+---
+
+# Post Three
+
+I’m the third blog post

test/fixtures/astro-collection/astro/pages/post/two.md

@@ -0,0 +1,8 @@
+---
+title: Post Two
+date: 2021-04-14 00:00:00
+---
+
+# Post Two
+
+I’m the second blog post