Load web page in python AFTER JavaScripts executes

Question

I am trting to get the definition of words in spanish (like a dictionary) based on what the user inputs. The idea would be:

>>> hola
'1. interj. U. como salutación familiar.'

I first tried with urllib2, but since the definition appeared after the execution of JS (makes sense duh) it didn't work. I also tried selenium, but from what I understood it has to open a navigator window, right? I need it to be like urllib2, invisible.

If you want to try, the page where I search the definition is http://lema.rae.es/drae/?val=word where word is the word the user inputs.

Any thoughts, anyone?

score 1 · Answer 1 · answered Mar 19 '15 at 13:09

1

You can automate a headless PhantomJS browser through selenium:

>>> from selenium import webdriver
>>>
>>> driver = webdriver.PhantomJS()
>>> driver.get('http://lema.rae.es/drae/?val=word')
>>>
>>> description = driver.find_element_by_css_selector('div.field-content p.azul')
>>> print description.text
El Diccionario de la lengua española (DRAE) es la obra de referencia de la Academia. La última edición es la 23.ª, publicada en octubre de 2014. Mientras se trabaja en la edición digital, que estará disponible próximamente, esta versión electrónica permite acceder al contenido de la 22.ª edición y las enmiendas incorporadas hasta 2012.

answered Mar 19 '15 at 13:09

alecxe

462,703
120
1,088
1,195

When I try to execute driver = webdriver.PhantomJS() I get an error: selenium.common.exceptions.WebDriverException: Message: Unable to start phantomjs with ghostdriver. Screenshot: available via screen – francisco sollima Mar 19 '15 at 13:20
@franciscosollima well, see http://superuser.com/questions/674322/python-selenium-phantomjs-unable-to-start-phantomjs-with-ghostdriver and http://stackoverflow.com/questions/16389938/using-selenium-in-the-background. First of all, upgrade `selenium` to the latest version: `pip install --upgrade selenium`. Also, download `phantojs` executeable and start it via `driver = webdriver.PhantomJS('path_here')`. Hope that helps. – alecxe Mar 19 '15 at 13:32
Ok, I'll have to try it in a few hours, sorry for the delay. – francisco sollima Mar 19 '15 at 13:40

score 1 · Accepted Answer · edited May 23 '17 at 12:28

I might do it like alecxe suggested, but I'd use the URL that loads the definition itself. For instance, searching for azul:

from selenium import webdriver

driver = webdriver.PhantomJS()
driver.get('http://lema.rae.es/drae/srv/search?val=azul')
print driver.find_element_by_css_selector("body>div").text

The URL that appears in the question loads a page that then loads the definition's URL in an iframe element. Loading the definition directly with the URL I show above saves some work and some complexity: the entire definition is contained in the first div child of body. Unfortunately, it does not remove the need for JavaScript.

Running the code above produces:

azul.
(Quizá alterac. del ár. hisp. lazawárd, este del ár. lāzaward, este del persa laǧvard o lažvard, y este del sánscr. rājāvarta, rizo del rey).
1. adj. Del color del cielo sin nubes. Es el quinto color del espectro solar. U. t. c. s.
2. m. El cielo, el espacio. U. m. en leng. poét.
3. m. Méx. Miembro del cuerpo de Policía.
~ de cobalto.
[... etc ...]

Note that I've not detected the need to use any wait mechanism to detect that the content of the page is ready. Looking at the page in a debugger a) I did not see any Ajax request and b) looking at the JavaScript and the page itself, it looks like what is served is an obfuscated page that the JavaScript then deobfuscates synchronously. So by the time driver.get returns, the content should be ready to be used.

Load web page in python AFTER JavaScripts executes

2 Answers2

Linked