0

I'm using requests_html to extract the element <div id="TranslationsHead">...</div> in this url in which <span id="LangBar"> ... </span> is rendered by javascript.

enter image description here

from requests_html import HTMLSession
session = HTMLSession()
from bs4 import BeautifulSoup

url = 'https://www.thefreedictionary.com/love'
headers = {'User-Agent': 'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:78.0) Gecko/20100101 Firefox/78.0'}

r = session.get(url, headers = headers)
soup = BeautifulSoup(r.content, 'html.parser')


soup.select_one('#TranslationsHead')

and its result is <div id="TranslationsHead"><span id="TranslationsTitle">Translations</span></div>. Sadly, it still does not capture <span id="LangBar"> ... </span>.

Could you please elaborate on how to capture such content?

Thank you so much for your help!

Akira
  • 2,594
  • 3
  • 20
  • 45

3 Answers3

2

You need to call r.html.render() to render the page with JavaScript:

from requests_html import HTMLSession

url = 'https://www.thefreedictionary.com/love'
session = HTMLSession()
r = session.get(url)
r.html.render()
lang_bar = r.html.find('#LangBar', first=True)
print(lang_bar.html)

If you want to prettify the output import BeautifulSoup and use:

soup = BeautifulSoup(lang_bar.html, 'html.parser')
print(soup.prettify())

If you want the languages:

for lcd in lang_bar.find('div.lcd'):
    print(lcd.text)

Outputs:

Afrikaans / Afrikaans
Arabic / العربية
Bulgarian / Български
Chinese Simplified / 中文简体
Chinese Traditional / 中文繁體
Croatian / Hrvatski
Czech / Česky
Danish / Dansk
Dutch / Nederlands
Esperanto / Esperanto
Estonian / eesti keel
Farsi / فارسی
Finnish / Suomi
etc

If you want to get all the translations note es is the default:

from requests_html import HTMLSession

url = 'https://www.thefreedictionary.com/love'
session = HTMLSession()
r = session.get(url)
r.html.render()
for span in r.html.find('span.trans'):
    print(span, span.text)

Outputs:

<Element 'span' class=('trans',) lang='af' style='display: none;'> liefde
<Element 'span' class=('trans',) lang='ar' style='display: none;'> حُب
<Element 'span' class=('trans',) lang='bg' style='display: none;'> любов
<Element 'span' class=('trans',) lang='br' style='display: none;'> amor
<Element 'span' class=('trans',) lang='cs' style='display: none;'> láska
<Element 'span' class=('trans',) lang='de' style='display: none;'> die Liebe
<Element 'span' class=('trans',) lang='da' style='display: none;'> kærlighed
<Element 'span' class=('trans',) lang='el' style='display: none;'> αγάπη
<Element 'span' class=('trans',) lang='es' style='display: inline;'> amor

If you want to simulate a click on one language and display the results:

from requests_html import HTMLSession

url = 'https://www.thefreedictionary.com/love'
session = HTMLSession()
r = session.get(url)
script = """
 () => {
              if ( document.readyState === "complete" ) {
                   document.getElementsByClassName("fl_ko")[0].click();
              }
        }
         """
r.html.render(script=script, timeout=10, sleep=2)
for span in r.html.find('span.trans[style="display: inline;"]'):
    print(span, span.text)

Outputs:

<Element 'span' class=('trans',) lang='ko' style='display: inline;'> 애정
<Element 'span' class=('trans',) lang='ko' style='display: inline;'> 연애
<Element 'span' class=('trans',) lang='ko' style='display: inline;'> 사랑하는 사람
<Element 'span' class=('trans',) lang='ko' style='display: inline;'> (테니스) 영점
<Element 'span' class=('trans',) lang='ko' style='display: inline;'> 사랑하다

UPDATED IN RESPONSE TO COMMENT

Jupyter, Spyder etc.use an event loop under the hood and request-html calls loop.run_until_complete which rise that exception when the loop is already running. Have you tried using AsyncHTMLSession?

from requests_html import AsyncHTMLSession

url = 'https://www.thefreedictionary.com/love'

asession = AsyncHTMLSession()

async def get_results():
    r = await asession.get(url)
    await r.html.arender()
    return r

r = asession.run(get_results)
lang_bar = r[0].html.find('#LangBar', first=True)
print(lang_bar.html)

Or:

from requests_html import AsyncHTMLSession

url = 'https://www.thefreedictionary.com/love'

asession = AsyncHTMLSession()
script = """
 () => {
              if ( document.readyState === "complete" ) {
                   document.getElementsByClassName("fl_ko")[0].click();
              }
        }
         """


async def get_results():
    r = await asession.get(url)
    await r.html.arender(script=script, timeout=10, sleep=2)
    return r

r = asession.run(get_results)
for span in r[0].html.find('span.trans[style="display: inline;"]'):
    print(span, span.text)
marc_s
  • 732,580
  • 175
  • 1,330
  • 1,459
Dan-Dev
  • 8,957
  • 3
  • 38
  • 55
  • It's weird that the error `Cannot use HTMLSession within an existing event loop. Use AsyncHTMLSession instead.` appears. – Akira Aug 23 '20 at 08:41
  • I have posted a question about this error [here](https://stackoverflow.com/questions/63545082/why-is-the-error-cannot-use-htmlsession-within-an-existing-event-loop-use-asyn). – Akira Aug 23 '20 at 08:49
  • I have updated the answer with a version of the code using AsyncHTMLSession. Can you let me know if it works? – Dan-Dev Aug 23 '20 at 17:38
  • 1
    Sadly, both of your versions do not work in my jupyter notebook. – Akira Aug 23 '20 at 18:43
0
from bs4 import BeautifulSoup
from requests_html import HTMLSession

url = 'https://www.thefreedictionary.com/love'
headers = {'User-Agent': 'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:78.0) Gecko/20100101 Firefox/78.0'}

session = HTMLSession()

resp = session.get(url, headers = headers)
resp.html.render()

soup = bs(resp.html.html, "lxml")
soup.find("div", {"id": "TranslationsHead"})
bot8080
  • 46
  • 3
0

remove r.html.render() if you are using juypter and it should work for you.

CASEDEARTISTA
  • 1
  • 2