8

Is there any way to get the executed javascript contents from a webpage? I have tried requests + BeautifulSoup, mechanize, these yield me with "source code" of the webpage and not the executed javascript. For example, this website :- http://listen.tidal.com/login

As you can see, in the source code, there is un-execute JS, but, when you inspect the element, you'll see the executed code.

Now, is there any way I could get that EXECUTED code in python? Hints please, because I have tried emulating a browser using mechanize and it does the same like reuqests. Thank You

Xonshiz
  • 1,307
  • 2
  • 20
  • 48
  • 1
    See [Python browser emulator with JS support](http://stackoverflow.com/questions/21777306/python-browser-emulator-with-js-support). – approxiblue Oct 05 '15 at 20:36
  • 1
    also have a look: http://stackoverflow.com/questions/8049520/web-scraping-javascript-page-with-python – mico Oct 06 '15 at 06:39
  • http://stackoverflow.com/questions/5793414/mechanize-and-javascript , too – mico Oct 06 '15 at 06:41
  • Ah! your quote "but, when you inspect the element, you'll see the executed code." how inspecting element executes js- could you explain.Why don't you mention what do you want to do using JS? – Learner Oct 13 '15 at 11:38

3 Answers3

2

In fact, JavaScript engine is needed for execution of javascript. Python is a language with its own interpreter(compiler!) to execute python code. These are two different technology. So if you want to execute javascript from python, python must have api or sort of bindings that interacts with the engine executes javascript. Fortunately python has interactivity with several JS Engine for implementing web related works(testing etc.). This interoperable JS can be divided into two groups as below-

  1. Browser without Graphical User Interface(GUI) aka Headless browser: e.g. PhantomJS a Webkit rendering engine based headless browser, SlimerJS a Gecko rendering engine based headless browser for more see here. You can interoperate with PhantomJS with selenium ( a glue between python and PhantomJS) or you can use PyQt and use python to run JS like here.
  2. Browser with Graphical User Interface(GUI): e.g. Firefox, Chromium, Safari etc. In this case also you can execute JS through selenium python.

Simple execution example of JS in selenium python as below-

from selenium import webdriver
#define driver- firefox, chrome  or phantomjs etc.
driver = webdriver.Firefox()
#Open the url
driver.get('https://www.google.com')
#see how javascript simple alert is being executed
driver.execute_script("alert('hello world');")
#close the driver  i.e. closing opened Firefox instance!
driver.close()
Community
  • 1
  • 1
Learner
  • 5,192
  • 1
  • 24
  • 36
1

Just to highlight - Python doesn't execute your Js code, but runtime does. Here is an example of the python module that picks available runtime and evaluates code for you.

Look at PyExecJS, here you can find some examples, but take into account that it might not contain any browser APIs like DOM, Html5 Api, etc. It's mostly based on js engine capabilities.

Another big question, what is the reason to evaluate code in the python?

Artemis
  • 4,821
  • 3
  • 21
  • 24
0

Yes, you have to pick a tool that supports Javascript content, other to mechanics. Mechanics is only for static content like you already had observed. There are several, found by search words "python alternative to mechanics". I would test PhantomJS, if I had to pick one.

Also several others are found in the other answers linked in comments, just I wanted to leave these as comments due to avoid "SO is not a pick your favorite tool recommending site" problem and thus only the universal solution is mentioned. So, please do have a little search ;)

mico
  • 12,730
  • 12
  • 59
  • 99