Firing onclick event which triggers a POST request in jsdom

Question

I'm trying to scrape a poorly designed governmental website which uses POST requests triggered from JavaScript for navigation (I'm trying to navigate the calendar).

I'm trying to do this the elegant way, with jsdom and jQuery in node (and possibly jsdom-simulant), but I'm not sure I understand how I'm supposed to fire the events within the simulator, given that the events themselves are supposed to go back to jsdom and trigger a new HTTP POST request.

I don't expect you guys to write the code for me, I only need a couple of pointers as to the structure, philosophy or an existing code base which does something similar.

Answer 1

Regarding the scraping part, this is a POST request sending form url encoded data. There are 2 fields which seem necessary in the payload :

• __EVENTTARGET=ctl00$B_Center$VoturiPlen1$calVOT
• __EVENTARGUMENT=XXXX (with XXXX some value)

The __EVENTARGUMENT value is incrementing each days. For instance on 04/04/2018 it's 6668, on 05/04/2018 it would be 6669. Looking at the oldest date which is 01/01/1998, the index is -730, so this index can be calculated using the difference in days between the target date and 01/01/1998 minus 730

Using curl & bash and dateutils :

target_date="2018-04-04"
index=$(($(dateutils.ddiff 1998-01-01 "$target_date") - 730))

curl 'https://www.senat.ro/Voturiplen.aspx' \
     -H 'User-Agent: Mozilla' \
     -H 'Content-Type: application/x-www-form-urlencoded; charset=UTF-8' \
     --data "__EVENTTARGET=ctl00%24B_Center%24VoturiPlen1%24calVOT&__EVENTARGUMENT=$index"

And using pup html parser :

curl 'https://www.senat.ro/Voturiplen.aspx' \
     -H 'User-Agent: Mozilla' \
     -H 'Content-Type: application/x-www-form-urlencoded; charset=UTF-8' \
     --data "__EVENTTARGET=ctl00%24B_Center%24VoturiPlen1%24calVOT&__EVENTARGUMENT=$index" | \
     pup 'table#ctl00_B_Center_VoturiPlen1_GridVoturi'

Using nodejs you can use node-request, moment :

const request = require('request');
const moment = require('moment');
const jsdom = require("jsdom");
const {JSDOM} = jsdom;

var a = moment('21/12/2017', 'DD/MM/YYYY');
var b = moment('01/01/1998', 'DD/MM/YYYY');
var index = a.diff(b, 'days') - 730;

request.post({
    url: 'https://www.senat.ro/Voturiplen.aspx',
    form: {
        "__EVENTTARGET": "ctl00$B_Center$VoturiPlen1$calVOT",
        "__EVENTARGUMENT": index
    },
    headers: {
        'User-Agent': 'Mozilla'
    }
},
function(err, httpResponse, body) {
    const dom = new JSDOM(body);
    var table = dom.window.document.querySelector("#ctl00_B_Center_VoturiPlen1_GridVoturi");
    console.log(table.textContent);
});

check this post for date diff with moment