Python Can not parse website Human problem but open in website

Question

i try to parse website but there is error You need to enable support for <a href="https://yandex.ru/support/common/browsers-settings/browsers-java-js-settings.html">js</a> in your browser to visit this site

I try this code

import requests
from bs4 import BeautifulSoup

URL = "https://siteurl"
headers={'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/102.0.0.0 Safari/537.36'}
page = requests.get(URL.strip(), headers=headers, timeout=100)

soup = BeautifulSoup(page.content, "html.parser")
print(soup.contents)

when i try to open in browser , it's work . Any solution?

That response is not an error. The website returned that content because it knows that you are using a method that does not understand javascript. — John Gordon, Nov 10 '22 at 19:12

score 0 · Answer 1 · answered Nov 10 '22 at 19:13

0

The website asks for the enabled JavaScript. BeatifulSoup does not mimick a full-fledged web-browser, so it lacks JavaScript functionality. You can try using Selenium + BeatifulSoup together since Selenium behaves as a full fledged browser.

answered Nov 10 '22 at 19:13

supsayan

21
3

score -2 · Answer 2 · answered Nov 10 '22 at 19:10

-2

import requests
from bs4 import BeautifulSoup

URL = "https://siteurl"
page = requests.get(URL)

soup = BeautifulSoup(page.text, "html.parser")
print(soup)

answered Nov 10 '22 at 19:10

Harez

30
5

not working , same problem – russiaisoccupant Nov 10 '22 at 19:11
3

This is nearly identical to the code that the questioner already tried. How is it supposed to solve the problem? – John Gordon Nov 10 '22 at 19:15

Python Can not parse website Human problem but open in website

2 Answers2