Accepting cookies while scraping page with requests and beautifulsoup

Question

I did a script that tracks the price of a product on many different pages. The problem is that some websites uses cookies and you have to click accept cookies to see the price.

This will probably not help but this is the website it's in swedish so many of you won't understand.

How do I accept cookies while web scraping?

Harshith Reddy · Answer 1 · 2020-12-06T19:35:26.373

0

There are no cookies being involved in doing a request. I feel you shouldn't be facing any problem doing a get or a post request.

Edit: try this peice of code:

r = requests.get('https://www.google.com/')

with open('test.html', 'w') as f:
    f.write(r.text)
    f.close()

Run the test.html file in your web browser and try to see the difference. The test.html is what your code sees which is different from what a normal person sees in their web browser with the full GUI.

edited Dec 06 '20 at 19:35

answered Dec 06 '20 at 19:19

Harshith Reddy

36
1
3

I just guessed this was the problem but I may be wrong. This is my second python script so I don't have much knowledge – Dojje Dec 06 '20 at 19:21
Its Okay, no problem. – Harshith Reddy Dec 06 '20 at 19:29

score 0 · Answer 2 · answered Dec 06 '20 at 19:24

When you scrape a site you don't have to accept those cookies. But if you want to accept then you can simply click on the "accept-button" from the website. You can do this with this method:

Clicking with Selenium

Get the X-Path with right clicking on the website and inspect the cookie button.

Accepting cookies while scraping page with requests and beautifulsoup

2 Answers2