Python 3.4, how to execute Javascript on a website and catch result? (like it appears in the browser)

Question

So I'm making this scraper program but I would need to catch information that doesn't show up in the source code.

Here is what it looks like in the source :

<td><script> <type="text/javascript">SCRIPT("SUPERLONGVARIABLE")</script></td>

And I'd like to be able to catch the result (which is just a short string of about 5 - 20 letters).

Any idea how I could do that? Not sure if the requests module can help me on this one.

Thanks in advance guys! :)

you want to execute javascript or scrape javascript content? — taesu, Sep 14 '15 at 16:29
I wanna execute it and the catch the result it would show in a regular browser — taze totero, Sep 14 '15 at 16:37
use `selenium` look at [this](https://selenium-python.readthedocs.org/) good luck — taesu, Sep 14 '15 at 16:39
Okay thanks! If I understand well, driver.execute_script("") would be what I'm looking for right? — taze totero, Sep 14 '15 at 16:45
Awesome! But just one thing, I'd need to "catch" the result of the javascript (it's just a rather short string), can I do it with this or can I just execute it ? (because executing it on it self won't be enough) — taze totero, Sep 14 '15 at 16:50

score 1 · Answer 1 · edited May 23 '17 at 11:58

1

Basically, you'd want to learn how to use selenium for python : link.
By utilizing driver.execute_script('foo'), you can actually get the result of that execution.

For simliar answer, look at this question on so

edited May 23 '17 at 11:58

Community

1
1

answered Sep 14 '15 at 16:53

taesu

4,482
4
23
41

My bad I thought I had answered, I tried with the command I wrote on the last comment but the thing is it litterally goes through firefox (as in the firefox page pops up) and I don't end up getting the result I want in python (meaning the decoded string) – taze totero Sep 17 '15 at 18:36

Python 3.4, how to execute Javascript on a website and catch result? (like it appears in the browser)

1 Answers1