0

I tried to make crawling module but I can't get return value.

.then() is no use because it works in then itself. I have to get result out of .then().

I tried to use await but it is no use.

const request = require("request");
const cheerio = require("cheerio");


let data = new Array();

const getHTML = async(url) => {
    
    request(url, (error, response, body) => {
        if (error) throw error;
        let $ = cheerio.load(body);
        $('body .web-results-list .web-result').each(function (index, elem){  
            let title = $(elem).find('.web-result-title').text();
            let linkadd = $(elem).find('.web-result-url').text();
            let content = $(elem).find('.web-result-description').text();
            data.push([title, linkadd, content]);
            //console.log(title);
            //console.log(linkadd);
            //console.log(content);
        })      
    }
    );
    return data
}

const func = async() => {
    data = await getHTML("https://www.search.com/web?q=apple");
}

func();
console.log(data);

console result = []

How can I fix it?

Eugenie
  • 1
  • 1
  • By not using the deprecated `request` package (for some time now, see https://github.com/request/request/issues/3142) and using [the fetch package](https://www.npmjs.com/package/fetch) instead, which you can `await` just fine, instead of having to use a callback function. – Mike 'Pomax' Kamermans Jul 24 '21 at 14:20
  • 1
    BTW, using `async` without `await` in the same function, is useless. Also, expecting `console.log(data)` to print the data after a call to `func`, is like ordering a pizza online by pressing the "Confirm Order" button and being surprised that your plate is still empty. – trincot Jul 24 '21 at 14:24
  • @trincot no? Using `async` makes your function return a promise that can be used with `await` or `.then()` downstream. which we see happening at the end of the code. (I'm assumping this is heavily reduced code for the purpose of asking the question). You need `async` if you want to `await`, but you don't need `await` inside an `async`. (even if the vast majority of the time, the whole reason you want `async` is for those `await`s). – Mike 'Pomax' Kamermans Jul 24 '21 at 14:27
  • Sure, you can *use* the promise returned by `async` function as a promise, but my point is: if you have a function that doesn't use `await`, then you don't really *need* `async`, as there is nothing asynchronous in the first place then. You might as well execute it synchronously. And if it happens to be a function that returns a promise, you don't need `async` either, as then ... well then you already have the promise returned. Either way, `async` gives no benefit to the program logic. I think we actually agree on this. – trincot Jul 24 '21 at 14:30

0 Answers0