5

I would like to scrape some ads for personal use from mobile.de.

I am using python 3.6 with requests lib, but I am facing issue with some bot inspection. How could I pass this gateway from their website?

import requests
from bs4 import BeautifulSoup

r = requests.get("https://www.mobile.de/?lang=en")
bs = BeautifulSoup(r.content, 'lxml')
print(bs)

This part of code displays me following:

<p>To continue your browser has to accept cookies and has to have JavaScript enabled.</p>

Where can I find the logic that I need to solve in order to pass this?

Pavel Fedotov
  • 748
  • 1
  • 7
  • 29
ldragicevic
  • 621
  • 4
  • 11
  • 22

3 Answers3

4

The reason you got unexpected content is you do not have a valid header. Just like @afit said. But To continue your browser has to accept cookies and has to have JavaScript enabled. is making sense, because if you do not enable JavaScript you won't load full of content.

Note: I recommend you use selenium to do this. requests_html can't access website successfully dues to lack of suitable header while it is rendering. Btw, if you want to access the url inside JavaScript and grab content, it will be tough job.

from bs4 import BeautifulSoup
from selenium import webdriver

dr = webdriver.Chrome()
dr.get("https://www.mobile.de/?lang=en")
bs = BeautifulSoup(dr.page_source,"lxml")
KC.
  • 2,981
  • 2
  • 12
  • 22
  • It succeeded. Could you tell me if its possible to disable opening chrome window? – ldragicevic Dec 25 '18 at 21:08
  • I have tried everything from -> https://stackoverflow.com/questions/46920243/how-to-configure-chromedriver-to-initiate-chrome-browser-in-headless-mode-throug , but I cannot pass this with headless mode. – ldragicevic Dec 25 '18 at 21:42
  • See the last part of this answer https://stackoverflow.com/a/49582462/9380694. It works for me. – KC. Dec 26 '18 at 01:53
2

They could be doing this a number of different ways, ranging from trivial to tricky to bypass at scale. One approach would be to modify your User-Agent, as their simplest approach would be to deny requests based on that.

r = requests.get(
    'https://yoursite.com',
    headers = {
        'User-Agent': 'Popular browser\'s user-agent',
    }
)

It doesn't look like it from the example URL you show, but they could be expecting that URL to be hit after hitting another page on the site that drops a cookie. If that's the case, make the earlier request and provide the cookie in your requests call.

Aidan Fitzpatrick
  • 1,950
  • 1
  • 21
  • 26
-2

In order for requests can handle and keep cookies sent by the server, you must use request.Session()

so, instead of

r = requests.get("https://www.mobile.de/?lang=en")

use it like this:

req = requests.Session()
req.get("https://www.mobile.de/?lang=en")
Art
  • 2,836
  • 4
  • 17
  • 34
Adán Escobar
  • 1,729
  • 9
  • 15