Reducing API calls for Next.js SSG

I have been using Next.js to create a statically generated site based upon data from an external API. I’ve been having a lot of problems with repeated calls to the same api route (which returns the same data) each time.

The reason this is a bit of a headache for me is that:

• I am paying for the api and I only have a fixed number of calls per day
• If I hit the api too rapidly the api won’t return the data to me due to rate limiting.
• It’s a waste of time and bandwidth hitting the same route again and again when I should already have that data.

I tried a few things to start with.

In the _app.js component I tried to use getInitialProps as this would then share the props to all components in the app which for my use case would have been fine. It did do that however it seemed that whenever a new static page was generated a new _app.js was being fired up and therefore was doing the same amount of api calls just from a different place in the code.

I also tried using react context and wrapping the App component in a Context.Provider but I think the same thing as above was happening where the _app.js was fired up for everypage so there was no reduction in api calls.

The solution I came to was to cache the responses locally myself as .json files.

import fs from 'fs'
export const fileExists = (path: string) => {
  return fs.existsSync(path)
}
export const writeFile = async (path: string, data: any) => {
  const dataToWrite = JSON.stringify(data)
  await fs.writeFile(path, dataToWrite, (err) => { err ?     console.error(err) : console.log('files saved to json: ', path) } )
}
export const readFile = async (path: string) => {
  return new Promise((resolve, reject) => {
    fs.readFile(path, 'utf8', (error, data) => {
      if (error) {
        reject(error);
      }
      resolve(JSON.parse(data));
    });
  });
}
export const routeStringToPathName = (route: string) => {
  const baseUrlRemoved = route.slice(34)
  const fileName = baseUrlRemoved.replace('/', '_')
  const pathName = './json/' + fileName + '.json'
  return pathName
}

The above was created in a separate .js file.

There are four functions one to check if the file already exists (ie have I tried to hit this api endpoint before), another that writes to a .json file, another that reads from a .json file and another that converts a url so a file name and path.

I then import and use these functions as below (this can be done in a separate .js file and then imported and used in getStaticProps) :

export const fetchApiData = async (param1, param2) => {
  const myHeaders = new Headers();
  myHeaders.append('api-key', '123456');
  const requestString = `https://data-api.io/data?  param1=${param1}&param2=${param2}`;
  const myRequest = new Request(requestString, {
    method: 'GET',
    headers: myHeaders,
    mode: 'cors',
    cache: 'default',
  })
  const pathName = routeStringToPathName(requestString)
  let timeout = await new Promise(resolve => setTimeout(resolve, 1000))
  if(fileExists(pathName)){
    const fileReturned = await readFile(pathName)
    return await fileReturned
  } else {
    let res = await fetch(myRequest)
    let result = await res.json()
    await writeFile(pathName, result)
    return result
  }
}

I’ve included params and an api key example but this would need editing for the particular api and end point you are using.

The api fetch is set up and then the requestString is passed into the routeStringToPathName function where it is returned in the format ./json/filename.json (ie saved to the json folder).

I have included a timeout as I think that Next.js can fire out the static pages so quickly it can be hitting the api route again before it has returned the first time and the result written to json.

Next it looks at the pathName and checks if the file exists (ie the endpoint has been hit, returned data and a file created.) If it does exist it will return the contents of the file and if it doesn’t it will carry out the fetch and then right the result to a file.

While this is far from perfect and there are undoubtedly many improvements to be made it works and reduced the number of api calls for me by a lot.

I was given lots of suggestions of things that might work by people but most seemed not to (or would in SSR but not SSG). This also feels like it must be a common problem and will probably be solved in a future version of Next.

One last caveat is to be careful if the data you are fetching from the api will change over time and you are caching the end result as it will become out of date. I solved this by editing my package.json and adding a deploy script that will remove the json folder and create a new empty one before building and exporting the SSG site.

"deploy": "rm -rf json && mkdir json && npm run build && npm run export"