How to extract a javascript object as json from a HTML page using python or nodejs?

Question

https://yeastmine.yeastgenome.org/yeastmine/customQuery.do

The above webpage has something like this. As far as I understand, JSON does not support single quote, only double quote is allowed. So the things in {} is not a valid JSON object. What is the best way to extract this object from the resulted HTML page and convert it to JSON? Thanks.

var helpMap = {'NcRNAGene': ...

This one mentions JSON.stringify. But I am not sure how to first get helpMap as JS object in the first place in python or nodejs.

Convert JS object to JSON string

What data do you want from that website exactly? It does look like they have an API you can interact with.. See [here](https://yeastmine.yeastgenome.org/yeastmine/api.do?subtab=python) and [here](https://github.com/intermine/intermine-ws-python-docs/) for more. It would appear they're using Intermine to serve their API. — Matt Oestreich, Sep 27 '19 at 22:24
But the things that I want extract here is about the metadata, not the data saved in intermine. — user1424739, Sep 28 '19 at 04:18
Maybe I'm not seeing the same thing as you... Can you post a screenshot of what you want to gather? — Matt Oestreich, Sep 28 '19 at 04:44

score 1 · Answer 1 · answered Sep 27 '19 at 21:02

1

In the console of that website you can write javascript. In this case you are right that JSON.Stringify is what you want here, you use it by passing the javascript object helpMap into it as a parameter, the result is the JSON-encoded string:

jsonString = JSON.stringify(helpMap)
console.log(jsonString)

You should be able to copy that json string out of your console (in chrome there will be a "Copy" button at the end of it).

answered Sep 27 '19 at 21:02

Patrick Fay

552
3
15

No. I need it to be at the command line instead of in the browser, as I mentioned python or nodejs. – user1424739 Sep 27 '19 at 21:08
Sorry I misunderstood the question, I thought you just wanted to get the JSON once, but it sounds like you want to reload the page to get changes. Probably you want to download/scrape the webpage https://blog.bitsrc.io/https-blog-bitsrc-io-how-to-perform-web-scraping-using-node-js-5a96203cb7cb then find the bit of javascript you're concerned with, either by searching for the line it's on looking for `var helpMap = ` or something like that, then parse that line as javascript https://www.npmjs.com/package/esprima, then convert that into JSON. – Patrick Fay Sep 27 '19 at 21:33
Once I called `esprima.parseScript("var helpMap = ...")`, how eval it to get the object of helpMap? – user1424739 Sep 28 '19 at 00:34

score 1 · Answer 2 · answered Sep 28 '19 at 05:09

Suppose the webpage is downloaded to x.html, run the following.

grep '^ \+var helpMap' < x.html | ./main.js

main.js has the following code.

fs=require('fs');
data = fs.readFileSync(process.stdin.fd);
eval(data.toString());
console.log(helpMap);

Then use JSON.stringify() on helpMap if necesssary.

How to extract a javascript object as json from a HTML page using python or nodejs?

2 Answers2