Fetch a POST request and convert result to DOM on NodeJs when crawling a website

Question

I'm using node-fetch and dom-parser to fetch a POST request and convert its result to a DOM Document. I'm using this:

var data = new FormData();
    data.append("dataInicial", "01/01/2020");
    data.append("dataFinal", "01/04/2020");
    data.append("valorCorrecao", "100,00");
    data.append("percentualCorrecao", "120,00");
    data.append("percentualCorrecao", "120,00");

const response = await fetch('https://www3.bcb.gov.br/CALCIDADAO/publico/corrigirPeloCDI.do?method=corrigirPeloCDI', {
        method: "post",
        body: data,
})
const messageData = await response.text();

Printing with console.log(messageData),, I get the html code as a string:

https://gist.github.com/hofstede-matheus/4c9c10f0edc229dfece61f869f621301

But using when DOMParser to convert it to a DOM Document with:

const parser = new DOMParser();
const htmlDocument = parser.parseFromString(messageData, "text/html");

It returns a rawHTML whose I cant use a querySelector to crawl a string inside a table:

https://gist.github.com/hofstede-matheus/f173e3a3f66a277a1e43ed9711af53c7

I have also tried regex get the table with no success. There are other ways to archive what I'm trying to get

Have you tried https://stackoverflow.com/a/10585079/8993539? — Saddy, Apr 21 '20 at 23:58
I'm using nodejs, so I cannot access document, it throws ReferenceError: document is not defined — Matheus Hofstede, Apr 22 '20 at 00:17
Sorry about that, I think you want this then: https://stackoverflow.com/questions/11398419/trying-to-use-the-domparser-with-node-js — Saddy, Apr 22 '20 at 00:19
You may want to try one of the alternatives to DOMParser (like JSDOM) — Saddy, Apr 22 '20 at 00:19
I've made a regex to extract the text that I needed in messageData, thanks! — Matheus Hofstede, Apr 30 '20 at 14:05

marpenter · Answer 1 · 2022-11-03T20:40:29.700

0

You can use something like cheerio: https://www.npmjs.com/package/cheerio

const cheerio = require('cheerio');
const $ = cheerio.load(messsageData);

Then use jQuery selectors to access/manipulate any element you need:

$('h1').addClass('myClass');

Example from Cheerio's page:

const cheerio = require('cheerio');
const $ = cheerio.load('<h2 class="title">Hello world</h2>');

$('h2.title').text('Hello there!');
$('h2').addClass('welcome');

$.html();
//=> <html><head></head><body><h2 class="title welcome">Hello there!</h2></body></html>

edited Nov 03 '22 at 20:40

answered Oct 25 '22 at 18:25

marpenter

11
3

Your answer could be improved by providing an example of the solution and how it helps the OP. – Tyler2P Oct 28 '22 at 15:28
While this link may answer the question, it is better to include the essential parts of the answer here and provide the link for reference. Link-only answers can become invalid if the linked page changes. - [From Review](/review/late-answers/33028717) – Meet Bhalodiya Nov 01 '22 at 06:58

Fetch a POST request and convert result to DOM on NodeJs when crawling a website

1 Answers1