Node.js async loop issue

Question

I'm trying to make a simple price checking scraper. I'm struggling with the part of the program that gets updates for the items that are in the database which are over an hour old.

Everything works fine, apart from when I put the URL and the updated prices back into the database as new entries. I get the last URL from the for loop against all the entries, so not much use for future checks!

From what I've read I'm pretty sure its an async issue, but I just can't fathom what I need to do resolve it. Should I be looking at callbacks, promises or something else completely?

connection.query("SELECT * FROM productprice WHERE DATE_ADD(date, INTERVAL 1 HOUR) <= CURRENT_DATE()", function(err, rows, fields){
if (err) throw err;

    for (var i in rows) {

        request(rows[i].urlsource, function(error, response, html){
            if(!error){
                var $ = cheerio.load(html);

                var priceup, urls;

                $('#priceblock_ourprice').filter(function(){
                    var data = $(this);
                    priceup = data.text();
                    urls = rows[i].urlsource;

                    console.log(rows[i].urlsource, priceup);
                })

            }

            var sql = "INSERT INTO productprice (urlsource, price, date) VALUES (?, ?, ?)";
            connection.query(sql, [urls, priceup, now], function (err, result) {
                if (err) throw err;
                console.log("something got inserted");
                });

        }) ;//REQUEST

    }//FOR LOOP 

})//CONNECTION.QUERY

I think it's an async issue and the for loop continuing to loop through the entries while cheerio is still looking at the DOM and getting the price, so that by the time it gets put back into the database it just grabs the last URL value.

Possible duplicate of [JavaScript closure inside loops – simple practical example](https://stackoverflow.com/questions/750486/javascript-closure-inside-loops-simple-practical-example) — CertainPerformance, Mar 30 '19 at 08:34
As a learner, consider learning modern code instead of ancient one. What you are doing is handled with promises nowadays, not callbacks — baao, Mar 30 '19 at 08:56
@bambam thanks. I had a scraper project in mind and all the tutorials used node so that’s where I started. Is there anything you’d suggest I use instead? — Chris, Mar 30 '19 at 12:58
No, node is good. Promise is a node object that is used instead of callbacks though — baao, Mar 30 '19 at 14:56
Ok looked into promises last night and although I understand the concept, my noob brain is struggling to understand how I should apply it/adapt my above code. Any suggestions? I’m thinking I need to push it all into a new array and do the html loading and traversing as the promise and then push that into the array when it’s done — Chris, Mar 31 '19 at 09:10

Node.js async loop issue

0 Answers0