async await geocode api call within map with bottleneck

Question

i'm using geocode api which have 10 request per second with in gatsby, i'm creating pages during develop, inside map function i'm calling geocode api to get the lat,lon from address to pass it to context for each page

but i'm hitting api limit 10 request per second, tried with enter link description here

but i think i'm doing it wrong, because page creation failed "gatsby-node.js" threw an error while running the createPages lifecycle:

const getAllData = async () =>
  Promise.all(
   //data from graphql 
    data.map(async (node) => {

      //googleapi function send fetch request to  api  
      const geo = await limiter.schedule(() => googleapi({ address }));

      results = await anotherapi_base_on_res(res.latitude, res.longitude);

      return {
        path: `/${slug}`,
        component: require.resolve(`./src/templates/abc.js`),
        context: {
          slug: node.url,
        },
      };
    })
  );
const dataResult = await getAllData();

pages are not being created, is it the right way to use Bottleneck

Ferran Buireu · Answer 1 · 2022-08-02T20:06:24.650

Pages are created when you run the gatsby develop or gatsby build commands, on runtime, once. I'm afraid this approach will never work as is.

More details about createPage API:

Create pages dynamically. This extension point is called only after the initial sourcing and transformation of nodes plus creation of the GraphQL schema are complete so you can query your data in order to create pages

i want to pass longitude and latitude to context

There are a lot of implementations detail lacking, but I think I get what you are trying to do. I think your code should look like:

const path = require("path");

exports.createPages = async ({ graphql, actions, reporter }) => {
  const { createPage } = actions;

  // somewhere you are getting the data in a GraphQL query

  data.forEach(node=> {
    const geo = await limiter.schedule(() => googleapi({ address }));

    const { latitude, longitude } = someFancyFunction(res.latitude, res.longitude)

    createPage({
      path: `/${slug}`,
      component: require.resolve(`./src/templates/abc.js`),
      context: {
        slug: node.url,
        latitude, 
        longitude
      },
    });
  });
};


const someFancyFunction = async () => {
   return await anotherapi_base_on_res(res.latitude, res.longitude);
}

The idea, once you get your data (I assume is a GraphQL query) is to loop through it but using a forEach, not a map since you don't want to return an array, you just want to createPage for each node, as Gatsby docs points out.

However, using a forEach you won't be able to use an async modifier, so you will need to change your workaround/approach to call an async function externally with the needed data (someFancyFunction), assuming it returns a latitude and longitude variables (tweak it accordingly). After that, can pass your data via context into your template.

Another workaround would be doing the loop to get the coordinates before the forEach (using an async map if needed or a for..of, etc.) to set an array of coordinates, then you can provide that data to the template for the corresponding position.

Thanks @Ferran ... this code is from gatsby-node.js, i want to pass `longitude` and `latitude` to `context` .. but hitting api call limiit — Yasir, Aug 02 '22 at 19:46
I updated the answer. There are *a lot* implementation details missing, but the main reason your code is failing is that `createPage` should be called in a `forEach` not a `map` loop. Moreover, I'm not sure about the `async` in your approach and because that implementation details missing it's difficult to provide a complete snippet. Option 1) Inside the `forEach` call an external `async` function to get the coordinates. Option 2) Do the loop before to get the full coordinates and assign them to a node, then `createPage` for each node and provide that data — Ferran Buireu, Aug 02 '22 at 20:09

async await geocode api call within map with bottleneck

1 Answers1