0

Im trying to fetch the HTML from this url which works by using JSDOM with:

JSDOM.fromURL("https://rarible.com/collection/" + collection, { runScripts: "dangerously" },{ pretendToBeVisual: true },{resources: "usable"}).then(dom => {
      console.log(dom.serialize()); //turn the page back into HTML
      var rendered= dom.serialize();
      res.status(200).json([{ rendered}]);
});

The thing is that the response i get seems to be unrendered content, meaning there's practically no html, mostly script(see the reponse in the pastebin)

How can i get elements of this page after all the scripts are rendered?

Quentin
  • 914,110
  • 126
  • 1,211
  • 1,335
Ted's Projects
  • 99
  • 1
  • 7
  • 1
    You can't scrape such content with just one single HTP request - you need a _headless browser_, that will actually fetch all those scripts & execute them. – CBroe Nov 01 '22 at 13:57
  • @CBroe — They aren't making a single HTTP request (well, they wouldn't be if they hadn't made the mistake described in my answer). JSDOM is **designed** to fetch dependant resources (hence `resources: "usable"`) in a similar way to a headless browser. – Quentin Nov 01 '22 at 14:06

1 Answers1

0

JSDOM takes a single options argument. You're putting each option in a separate object and passing each as a separate argument.

{resources: "usable"} is ignored because it is in the wrong place so it doesn't trigger the download of external scripts.

You need to combine them, like so:

JSDOM.fromURL("https://rarible.com/collection/" + collection, {
     runScripts: "dangerously",
     pretendToBeVisual: true,
     resources: "usable"
}).then(...
Quentin
  • 914,110
  • 126
  • 1,211
  • 1,335
  • Thanks for the answer but still getting the same result, it doesnt seem to be rendering thhe page... – Ted's Projects Nov 01 '22 at 14:37
  • @Ted'sProjects — When I run it it throws an error "Uncaught [TypeError: window.matchMedia is not a function]" because it uses a feature not implemented by JSDOM that you'd need to polyfill. – Quentin Nov 01 '22 at 14:40
  • Oh yeah i see that on my side too... You got any fix for that brother? – Ted's Projects Nov 01 '22 at 14:44
  • Not off the top of my head. You'd need to write a polyfill. (Or find an existing one). https://duckduckgo.com/?va=j&t=hc&q=jsdom+polyfill+matchmedia&ia=web – Quentin Nov 01 '22 at 14:46
  • Here's what i ended up finishing with: `const {window} = new JSDOM(JSON.stringify(dom)); matchMediaPolyfill(window); window.matchMedia('(min-width: 920px)'); window.matchMedia('(min-width: 920px)') // Create MediaQueryList instance ` But im still getting an error: `error - unhandledRejection: TypeError: Converting circular structure to JSON` – Ted's Projects Nov 01 '22 at 15:36
  • https://stackoverflow.com/questions/4816099/chrome-sendrequest-error-typeerror-converting-circular-structure-to-json – Quentin Nov 01 '22 at 15:40