1

I am using an example code I found here on Stackoverflow Specifically, I use part of the proposed code from the second most voted answer. Code follows:

import urllib
from bs4 import BeautifulSoup

url = "https://www.dailymotion.com/search/1234"
html = urllib.urlopen(url).read()
soup = BeautifulSoup(html)
print(soup.find('body'))

As you may have noticed I call dailymotion.com where I want to analyze the search results for some random text "1234".

This is how the browser looks like when I open it: Picture

When I press F12 and analyse the source code, especially the body part I notice that there is a difference. Picture2

Wherease my output from the python code looks like this: 

<body>\n<div id="root"></div>\n<!-- Google Analytics -->\n\n<!-- End Google Analytics -->\n\n\n\n<!-- pv5template --><!-- Mon Apr 29 2019 14:40:17 GMT+0200 (CEST) --><!-- b465253ce32a2c1e664af00ed756d25b830c890a --><!-- v-0.0.1770-rc2 --><link as="script" href="https://ajax.googleapis.com/ajax/libs/jquery/1.12.4/jquery.min.js" rel="preload"/><link as="script" href="https://static1.dmcdn.net/playerv5/dmp.manifest.7fbdfdd01f6244e8feb4.es5.js" rel="preload"/><link as="script" href="https://static1.dmcdn.net/playerv5/dmp.vendor.6dc443eed0ba7568dbab.es5.js" rel="preload"/><link as="script" href="https://static1.dmcdn.net/playerv5/dmp.main.cbf70e2a7d9818475b7b.es5.js" rel="preload"/><link as="script" href="https://static1.dmcdn.net/playerv5/dmp.theme_neon.f7b007935e44806b2e97.es5.js" rel="preload"/><link href="https://dmxleo.dailymotion.com" rel="preconnect"/><link href="https://pebed.dm-event.net" rel="preconnect"/><link crossorigin="anonymous" href="https://static1.dmcdn.net" rel="preconnect"/><link href="https://graphql.api.dailymotion.com" rel="preconnect"/>\n\n \n\n<!-- Google Tag Manager (noscript) -->\n<noscript><iframe height="0" src="https://www.googletagmanager.com/ns.html?id=GTM-P8L2J6G&amp;gtm_auth=-_fJL9BsWrxKWr76EuFTEA&amp;gtm_preview=env-5&amp;gtm_cookies_win=x" style="display:none;visibility:hidden" width="0"></iframe></noscript>\n<!-- End Google Tag Manager (noscript) -->\n</body>

The important difference is right on the beginning. It looks like it is not giving me the contents of id="root". Since my goal is to analyze the content of the elements in "root", I have to ask is there a way to get that part of the code as well?

CroatiaHR
  • 615
  • 6
  • 24
  • Seems the page content is generated by a js call. You should take a look at the `selenium` package – Maaz Apr 30 '19 at 10:03

1 Answers1

2

As I said in comment, you can have the content of the div id=root using the selenium package.

Here is a snippet which get the div

from selenium import webdriver
from bs4 import BeautifulSoup

browser=webdriver.Firefox()
browser.get('https://www.dailymotion.com/search/1234')

soup=BeautifulSoup(browser.page_source)


div = soup.find('div', {'id':'root'})
print(div)

Then the different card are inside a section tag. You just have to apply a find_all method on your div:

for section in div.find_all('section'):
    print(section)
Maaz
  • 2,405
  • 1
  • 15
  • 21