0

I am writing a node.js app with the http module and am trying to combine data from multiple GET requests to produce a result. I checked similar problems but they all involve async.map to init requests in parallel. In my case, I need the result from the first GET request to get the URL for the next one, etc. Here is a sample of what I am dealing with:

// First GET request
https.get('URL', (resp) => {
    let data = '';
    resp.on('data', (chunk) => {
        data += chunk;
    });

    resp.on('end', () => {
        let parsedData = JSON.parse(data);
        let names = parsedData.name;
        let before_id = parsedData.messages[99].id;
    });
});

I then want to pass names and before_id to another GET request like this:

// Second GET request
https.get('URL&before_id=' + before_id, (resp) => {
    let data = '';
    resp.on('data', (chunk) => {
        data += chunk;
    });

    resp.on('end', () => {
        let parsedData = JSON.parse(data);
        names = //names from previous request + this parsedData.name
        let before_id = parsedData.messages[99].id;
    });
});

For context: I am using the GroupMe API, which only allows me to retrieve 100 messages at a time. I would like to retrieve 1000 messages, which is done by done by finding the message id of the last message in a batch of 100 and calling another GET request, passing that message id as a parameter in the URL. (This will request the 100 messages immediately before the specified id). Not only then do I need to pass the id from each request to the next one, but I also need to store the data I need from each batch of requests, combine them, then serve them together. How do I do this?

Daccache
  • 125
  • 2
  • 8

2 Answers2

1

there will be better ways to structure code than what I'm going to present in answer. But it's the bare minimum example of what you want to do.

To use the result of one asynchronous operation to initiate some other task, you have to embed that task in the callback of the asynchronous operation.

// First GET request
https.get('URL', (resp) => {
    let data = '';
    resp.on('data', (chunk) => {
        data += chunk;
    });

    resp.on('end', () => {
        let parsedData = JSON.parse(data);
        let names = parsedData.name;
        let before_id = parsedData.messages[99].id;
        // Second GET request
        https.get('URL&before_id=' + before_id, (resp) => {
            let data = '';
            resp.on('data', (chunk) => {
                data += chunk;
            });

            resp.on('end', () => {
                let parsedData = JSON.parse(data);
                names = //names from previous request + this parsedData.name
                let before_id = parsedData.messages[99].id;
            });
        });
    });
});
tbking
  • 8,796
  • 2
  • 20
  • 33
  • I'm sure there are better/more concise ways to do this, but this method suits my needs perfectly. Thanks! – Daccache Dec 27 '20 at 22:03
0

You could try to declare a global variable and update it in the first get request by adding on the variable and later using that data in the next get request.

let before_id = ''
// First GET request
https.get('URL', (resp) => {
    resp.on('data', (chunk) => {
        data += chunk;
    });
    before_id += parsedData.messages[99].id;
//Do other stuff
});
// Second GET request
https.get( 'URL&before_id=' + before_id, (resp)()=>{
//Do stuff
})
Manaweb
  • 56
  • 4