My For loop is too fast and I miss getting critical data

Question

The API i'm using to get JSON data only lets you put in a single date, and they don't have any methods for getting multiple dates. Therefor, I decided to create a for loop to loop through some dates, to pick up all the json data I need. The problem is, the loop does not wait for the json file to return the data, so even though I get some of the data, I don't get all of the data. What is worse, is some dates are slow to come back, so they don't even get pushed into the array in the right order. It just pushs them in, in a somewhat random fasion. How can I tell the loop to "wait" until I get the data, until looping again?

jsonDataArray = [];
    for (var k = 10; k < 20; k++){
        var num = k;
        var n = num.toString();
        var mydate = "2017-02-"+n; 
        d3.json("http://api.fixer.io/"+mydate, function (jsonData) {
            jsonDataArray.push(jsonData);   
        }
    )}

Since you have to wait for the result to come back for the next one, you basically want a recursive function. Pass in a start, pass in an end. If start < end, call self again with an incremented start. — tymeJV, Nov 01 '17 at 19:46
Possible duplicate of [How to make JQuery-AJAX request synchronous](https://stackoverflow.com/questions/13971769/how-to-make-jquery-ajax-request-synchronous) — Camilo, Nov 01 '17 at 19:48
@Shawn - It's the same principle tho - making async requests in order. — tymeJV, Nov 01 '17 at 19:50
@Shawn, yes it is, d3.json() makes an ajax request using the url that you pass — Patrick Evans, Nov 01 '17 at 19:50
@tymeJV No, you don't need to wait for the next one.. each call is not dependent on the previous call. Why slow your program down when you can make multiple requests at once? — Brian Glaz, Nov 01 '17 at 19:50
@tymeJV You just can't use push, and should assign directly to an index. `jsonDataArray[k] = jsonData;` for example. Promise approach is better though, so you actually know when all requests are complete. — Brian Glaz, Nov 01 '17 at 19:53
@BrianGlaz - Yep, OP would have to use `let` tho in the loop, else `k` is always going to be the final loop counter. I like the promise solution below — tymeJV, Nov 01 '17 at 19:54

score 3 · Answer 1 · answered Nov 01 '17 at 19:50

3

I/O in JavaScript is non-blocking, so data will be returned in your callback handler after your loop is done.

You could return an array of Promises from your loop and then use Promise.all. Here's the idea:

results = [];
for (var k = 10; k < 20; k++){
        var num = k;
        var n = num.toString();
        var mydate = "2017-02-"+n;
        results.push(new Promise(function(resolve, reject) {
          d3.json("http://api.fixer.io/"+mydate, function (jsonData) {
              resolve(jsonData);
          }}));
    )}  

Promise.all(results).then(function (jsonArray) {
    ... work on the jsonArray
}

answered Nov 01 '17 at 19:50

Jochen Bedersdorfer

4,093
24
26

Your callback is wrong. You're resolving to the error messages, not the data. – Thomas Nov 01 '17 at 20:36
you mean the d3 callback? – Jochen Bedersdorfer Nov 01 '17 at 20:49
Oh my. Didn't know that one. I'd never put something like that into a lib. You can not rely on *Function#length* in JS. – Thomas Nov 01 '17 at 21:01
to be fair, I didn't either. I assumed the OPs code was correct for that part – Jochen Bedersdorfer Nov 01 '17 at 21:02

score 1 · Accepted Answer · answered Nov 01 '17 at 20:28

1

check out d3.promise (3rd party module) and d3.range()

Promise.all(
  d3.range(10, 20)
    .map(day => d3.promise.json("http://api.fixer.io/2017-02-" + day))
).then(data => {
  console.log(data);
})

.as-console-wrapper{top:0;max-height:100%!important}

<script src="https://cdnjs.cloudflare.com/ajax/libs/d3/3.4.11/d3.min.js"></script>
<script src="https://cdn.jsdelivr.net/npm/d3.promise@2.0.0/dist/d3.promise.js"></script>

answered Nov 01 '17 at 20:28

Thomas

11,958
1
14
23

I'm still running into issues of getting the complete data, for example: a month, but this little code does what its intended to do, which is pull every single data I requested without missing any or putting the dates in random order in the array. Thank you. – Shawn Nov 02 '17 at 12:31

My For loop is too fast and I miss getting critical data

2 Answers2