- Stable
3.0.0
Toggle Menu
1.93s
29.05s
Fetch
Contents
Fetch network resources and cache them so you don’t bombard your API (or other resources). Do this at configurable intervals—not with every build! Once per minute, or once per hour, once per day, or however often you like!
With the added benefit that if one successful request completes, you can now work offline!
This plugin can save any kind of asset—JSON, HTML, images, videos, etc.
- Fetch a remote URL and saves it to a local cache.
- If the remote server goes down or linkrots away—we keep and continue to use the local asset (save remote images!)
- If cache expires and the network connection fails, will continue to use the cached request and make a new request when the network connectivity is restored.
- Control concurrency so we don’t make too many network requests at the same time.
- Requires Node 12+
@11ty/eleventy-cache-assets
. Installation
npm install @11ty/eleventy-fetch
Formerly known as @11ty/eleventy-cache-assets
.
This plugin caches complete network responses. Unless you’re willing to perform a full review of everything this plugin caches to disk for privacy and security exposure, it is strongly recommended that you add the .cache
folder to your .gitignore
file so that network responses aren’t checked in to your git repository.
Are you 100% sure that private e-mail addresses aren’t being returned from a cached API? I’m guessing no—add .cache
to your .gitignore
file. Right now. Do it.
Usage
Cache a JSON file from an API
Consider the following example, perhaps in an Eleventy Global Data File.
import EleventyFetch from "@11ty/eleventy-fetch";
export default async function () {
let url = "https://api.github.com/repos/11ty/eleventy";
/* This returns a promise */
return EleventyFetch(url, {
duration: "1d", // save for 1 day
type: "json", // we’ll parse JSON for you
});
};
const EleventyFetch = require("@11ty/eleventy-fetch");
module.exports = async function () {
let url = "https://api.github.com/repos/11ty/eleventy";
/* This returns a promise */
return EleventyFetch(url, {
duration: "1d", // save for 1 day
type: "json", // we’ll parse JSON for you
});
};
Options
Verbose Output
Added in Fetch 3.0 Option to log requested remote URLs to the console.
verbose: true
(The default isfalse
starting in Fetch 3.0)
Change the Cache Duration
After this amount of time has passed, we’ll make a new network request to the URL to fetch fresh data.
The duration
option supports the following shorthand values:
s
is seconds (e.g.duration: "43s"
)m
is minutes (e.g.duration: "2m"
)h
is hours (e.g.duration: "99h"
)d
is days (The default isduration: "1d"
)w
is weeks, or shorthand for 7 days (e.g.duration: 2w
is 14 days)y
is years, or shorthand for 365 days (not exactly one year) (e.g.duration: 2y
is 730 days)
Here are a few more values you can use:
duration: "*"
will never fetch new data (after the first success).duration: "0s"
will always fetch new data.
Type
type: "json"
type: "text"
type: "buffer"
(default: use this for non-text things)
Cache Directory
The directory
option let’s you change where the cache is stored. It is strongly recommended that you add this folder to your .gitignore
file.
import EleventyFetch from "@11ty/eleventy-fetch";
await EleventyFetch("https://…", {
directory: ".cache",
});
If you want to use this utility inside of a Netlify Function (or AWS Lambda), use a writeable location (/tmp/
) like directory: "/tmp/.cache/"
. You can also use dryRun: true
to skip writing to the file system.
Remove URL query params from Cache Identifier
(Version 2.0.3 and newer) If your fetched URL contains some query parameters that aren’t relevant to the identifier used in the cache, remove them using the removeUrlQueryParams
option. This is useful if an API adds extra junk to your request URLs.
removeUrlQueryParams: true
(false
is default)
import EleventyFetch from "@11ty/eleventy-fetch";
await EleventyFetch(
"https://www.zachleat.com/img/avatar-2017-big.png?Get=rid&of=these",
{
removeUrlQueryParams: true,
}
);
Note that query params are removed before—and are relevant to how—the hash key is calculated.
What happens when a request fails?
- If this is the first ever request to this URL (no entry exists in your cache folder), it will fail. Use a
try
/catch
if you’d like to handle this gracefully. - If a failure happens and a cache entry already exists (even if it’s expired), it will use the cached entry.
- If you prefer the build to fail when your API requests fail, leave out the
try
catch
and let the error throw without handling it!
import EleventyFetch from "@11ty/eleventy-fetch";
export default async function () {
try {
let url = "https://api.github.com/repos/11ty/eleventy";
/* This returns a promise */
return EleventyFetch(url, {
duration: "1d",
type: "json",
});
} catch (e) {
return {
// my failure fallback data
};
}
};
const EleventyFetch = require("@11ty/eleventy-fetch");
module.exports = async function () {
try {
let url = "https://api.github.com/repos/11ty/eleventy";
/* This returns a promise */
return EleventyFetch(url, {
duration: "1d",
type: "json",
});
} catch (e) {
return {
// my failure fallback data
};
}
};
Running this on your Build Server
This documentation has moved to the Deployment page.
More Examples
Cache a Remote Image
This is what eleventy-img
uses internally.
import EleventyFetch from "@11ty/eleventy-fetch";
export default async function () {
let url = "https://www.zachleat.com/img/avatar-2017-big.png";
let imageBuffer = await EleventyFetch(url, {
duration: "1d",
type: "buffer",
});
// Use imageBuffer as an input to the `sharp` plugin, for example
// (Example truncated)
};
const EleventyFetch = require("@11ty/eleventy-fetch");
module.exports = async function () {
let url = "https://www.zachleat.com/img/avatar-2017-big.png";
let imageBuffer = await EleventyFetch(url, {
duration: "1d",
type: "buffer",
});
// Use imageBuffer as an input to the `sharp` plugin, for example
// (Example truncated)
};
Fetch Google Fonts CSS
Also a good example of using fetchOptions
to pass in a custom user agent. Full option list is available on the node-fetch
documentation.
import EleventyFetch from "@11ty/eleventy-fetch";
export default async function() {
let url = "https://fonts.googleapis.com/css?family=Roboto+Mono:400&display=swap";
let fontCss = await EleventyFetch(url, {
duration: "1d",
type: "text",
fetchOptions: {
headers: {
// lol
"user-agent":
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.169 Safari/537.36",
},
},
});
};
const EleventyFetch = require("@11ty/eleventy-fetch");
module.exports = async function() {
let url = "https://fonts.googleapis.com/css?family=Roboto+Mono:400&display=swap";
let fontCss = await EleventyFetch(url, {
duration: "1d",
type: "text",
fetchOptions: {
headers: {
// lol
"user-agent":
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.169 Safari/537.36",
},
},
});
};
Fetching GitHub Stars for a repo
- This specific example has been previously described in our quick tips section: head over to read Quick Tip #009—Cache Data Requests.
Advanced Usage
Manually store your own data in the cache
You probably won’t need to do this. If you’d like to store data of your own choosing in the cache (some expensive thing, but perhaps not related to a network request), you may do so! Consider the following Global Data File:
import { AssetCache } from "@11ty/eleventy-fetch";
export default async function () {
// Pass in your unique custom cache key
// (normally this would be tied to your API URL)
let asset = new AssetCache("zachleat_twitter_followers");
// check if the cache is fresh within the last day
if (asset.isCacheValid("1d")) {
// return cached data.
return asset.getCachedValue(); // a promise
}
// do some expensive operation here, this is simplified for brevity
let fakeTwitterApiContents = { followerCount: 1000 };
await asset.save(fakeTwitterApiContents, "json");
return fakeTwitterApiContents;
};
const { AssetCache } = require("@11ty/eleventy-fetch");
module.exports = async function () {
// Pass in your unique custom cache key
// (normally this would be tied to your API URL)
let asset = new AssetCache("zachleat_twitter_followers");
// check if the cache is fresh within the last day
if (asset.isCacheValid("1d")) {
// return cached data.
return asset.getCachedValue(); // a promise
}
// do some expensive operation here, this is simplified for brevity
let fakeTwitterApiContents = { followerCount: 1000 };
await asset.save(fakeTwitterApiContents, "json");
return fakeTwitterApiContents;
};
Change Global Concurrency
import EleventyFetch from "@11ty/eleventy-fetch";
EleventyFetch.concurrency = 4; // default is 10
DEBUG mode
DEBUG=EleventyCacheAssets* node your-node-script.js
DEBUG=EleventyCacheAssets* npx @11ty/eleventy