Does requests rely on selenium?

Question

I am running some crawls in order to test if the results I get deviate. For this effort I created two test suites, the first one I created with the requests and BeautifulSoup library, the other one is based on selenium. I would like to find out, if pages detect both bots in the same way.

But I am still unsure if I am right, by assuming that requests and BeautifulSoup are independent from Selenium.

I hope its not a dump question, but I haven't find any proper answer yet (maybe because of the wrong keywords). However, any help would be appreciated. Thanks in advance

I checked the requests documentation. I wrote a mail to the developer, without any answer. And of course I checked on google. I found something about scrapy vs selenium but well... are requests and BeautyfulSoup related to scrapy?

score 1 · Accepted Answer · answered Jun 23 '19 at 23:23

1

The python requests module does not use Selenium, neither does BeautifulSoup. Both will run independent of a web browser. Both are pure python implementations.

answered Jun 23 '19 at 23:23

vy32

28,461
37
122
246

score 1 · Answer 2 · answered Jun 23 '19 at 23:33

Selenium automates browsers, so you'll present to a web service with the user-agent string and other variables that the browser you choose to drive with Selenium would present.

You can specify user-agent string when you use requests, or not, but requests doesn't drive a browser inherently, so you'll be presenting as a different entity from the user-agent perspective, like python-requests/2.18.4.

BeautifulSoup is a parser, and so it presents to a web service through another library (like requests); it doesn't have its own native presentation.

Does requests rely on selenium?

2 Answers2