0

I'm creating an API using the Cheerio library to do a webscrapping on the imdb site and get a movie ranking. I catch all the information on the site and send it to the items array as an object and I would like to return this in json to the user making a get request. The web scrapping is working, because in the console.log(items) inside var cheerioData returns me all the items I would like. But putting this console.log outside of axios, it returns an empty array as I make the request and I get a status 200 with the array empty.

import * as cheerio from 'cheerio';
import { Request, Response } from "express";
import axios from 'axios';


class ScrappingRaking { 
    async index(req: Request, res: Response) {
        const items: object [] = [];

        try {
            axios("https://www.imdb.com/chart/tvmeter/?ref_=nv_tvv_mptv").then(res => {
                const data = res.data;
                const $ = cheerio.load(data);
            
                var cheerioData = [...$('.lister-list>tr')].map((e, i) => {
                    if (i < 15) {
                        const ranking = i + 1;
                        const title = $(e).find('.titleColumn a').text().trim();
                        const year = $(e).find('.titleColumn span').text().trim();
                        items.push({ranking, title, year});

                        console.log(items);
                    }
                })
            })        
            return res.status(200).json(items);
        } catch (err) {
            return res.status(500).json(err);
        }
    }
}
 
 
export default new ScrappingRaking();

How could I make the items returned inside my array be displayed to my user by a req get in status 200

Routfin
  • 397
  • 1
  • 11
  • You're trying to send your response before you have it. Move the `res.status(200).json(items)` **into** the `then` callback, after you've populated `items`. (There's no need to declare it outside, either. Just do it all once the promise fulfills.) Or better yet, since `index` is an `async` function, use `await` rather than `.then`. There's no point in having an `async` function that doesn't have `await`, the purpose of `async` functions is to let you use syntax rather than callbacks for consuming promises. – T.J. Crowder Feb 07 '23 at 18:04
  • (Side note: "[scrapping](https://www.collinsdictionary.com/dictionary/english/scrap)" is throwing something away. "[Scraping](https://www.collinsdictionary.com/dictionary/english/scrape)" [one `p`] is what you're doing when reading content from web pages. :-) ) (Yes, English is **that** weird. :-) ) – T.J. Crowder Feb 07 '23 at 18:05
  • FWIW, here's a bit of simplification: https://pastebin.com/TgndZBmX or https://pastebin.com/A8DJgycj In particular, I suggest not using `map` just for side-effects (like pushing to an array). More on my blog [here](https://thenewtoys.dev/blog/2021/04/17/misusing-map/). – T.J. Crowder Feb 07 '23 at 18:17

0 Answers0