Code not on BS4, but can be found in 'Inspect Element'

Question

I have tried making a website which uses Beautiful Soup 4 to search g2a for the prices of games (by class). The problem is that when I look in the HTML code, it clearly shows the price of the first result (£2.30), but when I search for the class in Beautiful Soup 4, there is nothing between the same class's tags:

#summoningg2a
r = requests.get('https://www.g2a.com/?search=x')
data = r.text
soup = BeautifulSoup(data, 'html.parser')

#finding prices
prices = soup.find_all("strong", class_="mp-pi-price-min")

print(soup.prettify())

wrt. screenshots -- one shows `mp-pi-price-min` while the other `mp-pi-price` those are different classes. — Dima Tisnek, Mar 09 '17 at 19:57

score 0 · Accepted Answer · answered Mar 08 '17 at 20:50

0

requests doesn't handle dynamic page content. You're best bet is using Selenium to drive a browser. From there you can parse page_source with BeautifulSoup to get the results you're looking for.

answered Mar 08 '17 at 20:50

crookedleaf

2,118
4
16
38

score 0 · Answer 2 · answered Mar 09 '17 at 01:11

0

In chrome development tools, you can check the ajax request(made by Javascript) URL. you can mimic that requests and get data back.

r = requests.get('the ajax requests url')
data = r.text

answered Mar 09 '17 at 01:11

宏杰李

11,820
2
28
35

Code not on BS4, but can be found in 'Inspect Element'

2 Answers2