how to scrape data from a website using node

Question

I am new in Node JS. from morning i am trying to scrape data from website https://www.pmkisan.gov.in/StateDist_Beneficiery.aspx

[![enter image description here][1]][1]

I want to store that date also in db table. Currently i am putting hard coded date. But how can i scrape that date?

my code

(async () => {
    
    

    await performDbActions(webData);

})();


async function performDbActions(data) {
    let dataToBeInsert = {};
    // console.log(data);
    for (const d of data) {
        if (Object.keys(d).length) {
            // console.log(d);
            let district = await db.sequelize.query("select * from abc where name=:districtName or other_names like :districtLike", {
                replacements: {districtName: d['name'], districtLike: '%' + d['name'] + '%'},
                raw: true,
                type: db.sequelize.QueryTypes.SELECT
            });
            
                    delete d['sno'];
                    delete d['name'];
           
                    d['as_on'] = '2020-02-06';
                
                
            }
        }
    }
   
}

eol · Accepted Answer · 2020-08-28T15:03:56.197

1

According to the page's source code the date you're looking for is inside a <span> that has the id ContentPlaceHolder1_lbldate. So you can just use cheerio to get its text-content and pass the result to performDbActions as an additional parameter:

//...
const date = $('#ContentPlaceHolder1_lbldate').text();
//...
await performDbActions(webData, date);
// ...

async function performDbActions(data, date) {
  // ...
  // it would be safer to use an external date-library like moment.js but here's a way to convert the date in plain js 
  const dateParts =date.split('/');
  const dateObj = new Date(dateParts[2], dateParts[1] - 1, dateParts[0]);
  d['created_at'] = dateObj;
}

Note that the date is in format dd/mm/yyyy, so you may have to convert it to your desired format.

edited Aug 28 '20 at 15:03

answered Aug 28 '20 at 14:38

eol

23,236
5
46
64

it is in text format and in my table that column data type is date – aashu1807 Aug 28 '20 at 14:44
As mentioned in the answer, you need to make sure to convert it to a valid format/date. Check this for example: https://stackoverflow.com/questions/5619202/converting-a-string-to-a-date-in-javascript – eol Aug 28 '20 at 14:48
I saw this link already. But not able to solve my problem – aashu1807 Aug 28 '20 at 14:53
const date_temp = $('#ContentPlaceHolder1_lbldate').text(); console.log(date_temp); const date=new Date(date_temp); console.log(date); – aashu1807 Aug 28 '20 at 14:53
I have written this. but getting "Invalid Date" – aashu1807 Aug 28 '20 at 14:54
Yeah you did not carefully read the answers obviously :) I'll edit the answer and give you a solution ... – eol Aug 28 '20 at 14:54
Updated, please check. – eol Aug 28 '20 at 14:55
Let us [continue this discussion in chat](https://chat.stackoverflow.com/rooms/220626/discussion-between-aashu1807-and-eol). – aashu1807 Aug 28 '20 at 15:01

how to scrape data from a website using node

1 Answers1