0

I'm trying to extract a piece of information as soon as it is available. It could be available first on 4 different websites, so I have created regex expressions for each URL I can find the information and stored these values in an array:

var sources = [{
    url: URL1
    regex: REGEX1
},
{
    url: URL2
    regex: REGEX2
}, etc.];

I then loop over these with a for loop and want to test them one at a time for the information. This for loop is set on a 30 second interval, so I want all of the websites to be tested every 30 seconds.

for (var i = 0; i < data.sources.length; i++)
{
    var source = data.sources[i];
    var url = source.url;
    var regex = source.regex;

    // this is npm request
    request(url, function(error, response, html)
    {
        //EDIT INCLUDED HERE: HIGH-LEVEL OVERVIEW
        ->check to see if data has been updated
        ->if yes, use the capturing regex groups to enter data into MySQL table
    }
 }

The issue is, because request is a callback function (or 'has' a callback function?), regex and url as overwritten before anything can be done with them. So the request will fire and I will get valid HTML, but then regex is set to a value that isn't valid for that url. I need help coming up with one of two solutions:

  1. (Preferable): pass the regex to the request function so that it is local to that function and doesn't get over written. This would be great because it would allow all of the requests to in a non-blocking manner.
  2. Have the request functions wait until the previous one is finished before continuing through the for loop.
mscdex
  • 104,356
  • 15
  • 192
  • 153
Kramhsiri
  • 119
  • 9
  • You will have to show more context here for what you're trying to accomplish with the loop. Asynchronous functions inside a loop require special handling, but we don't know what the objective of the loop is. You CAN'T make anything "wait" in Javascript. You have to handle the async result in the callback. – jfriend00 Aug 21 '16 at 02:47
  • Ok I tried to add a little context. Basically I'm trying to search through 4 webpages to see if data has been updated. If it is updated on one of the pages, then I want to stop searching on all of the pages (the updated data I'm looking for will be the same, I just want to get it as soon as it is available from one of the 4 sites.) – Kramhsiri Aug 21 '16 at 02:51
  • @Kramhsiri If you don't feel like reading the full answer including all updates in the duplicate, I'll summarize it for you. Use `let` instead of `var`. EG: `let regex = source.regex;`. That would acheive the first solution you proposed (each request would use its own regex). – Paul Aug 21 '16 at 02:52
  • FYI, if you had described your real problem as the subject of your question, it would probably have gotten an answer rather then getting marked a dup. but, you asked about a detail of your implementation which is described in the dup even though there's a bit more to your overall problem than is described in that dup. Next time, please describe the real problem, not just what's wrong with your attempted solution. – jfriend00 Aug 21 '16 at 03:02
  • Maybe that I don't fully understand my problem is part of the problem? – Kramhsiri Aug 21 '16 at 03:08
  • @Kramhsiri Did using `let` instead of `var` solve your problem? – Paul Aug 21 '16 at 18:51

0 Answers0