4

I want to make 10 requests to https://www.google.com/ but with random user agents using selenium and python. I've a loop and inside that loop I'm making 10 requests with random user agents (using fake-user agent). The main problem is for every request web driver is opening a new instance of google chrome and I want to do this in one single instance but with different user agents. How can I make this possible ? 1 google chrome instance and 10 requests with 10 random user agents. Here is my code:

chrome_options = Options()
chrome_options.add_argument('no-sandbox')
chrome_options.add_argument("--start-maximized")
ua = UserAgent()
for i in range(0, 10):
    userAgent = ua.random
    chrome_options.add_argument('--user-agent="' + str(userAgent) + '"')
    driver1 = webdriver.Chrome(chrome_options=chrome_options, 
    executable_path="C:/Python34/chromedriver")
    driver1.get('https://www.google.com/')
    time.sleep(5)
undetected Selenium
  • 183,867
  • 41
  • 278
  • 352
Arsalan
  • 51
  • 1
  • 3

3 Answers3

5

First the update 1

execute_cdp_cmd(): With the availability of execute_cdp_cmd(cmd, cmd_args) command now you can easily execute commands using Selenium. Using this feature you can modify the easily to prevent Selenium from getting detected.

  • Code Block:

    from selenium import webdriver

driver = webdriver.Chrome(executable_path=r'C:\WebDrivers\chromedriver.exe')
print(driver.execute_script("return navigator.userAgent;"))
# Setting user agent as Chrome/83.0.4103.97
driver.execute_cdp_cmd('Network.setUserAgentOverride', {"userAgent": 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.97 Safari/537.36'})
print(driver.execute_script("return navigator.userAgent;"))
# Setting user agent as Chrome/83.0.4103.53
driver.execute_cdp_cmd('Network.setUserAgentOverride', {"userAgent": 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.53 Safari/537.36'})
print(driver.execute_script("return navigator.userAgent;"))
driver.get('https://www.httpbin.org/headers')

  • Console Output:

    Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.106 Safari/537.36
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.97 Safari/537.36
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.53 Safari/537.36

  • Browser Snapshot:

useragent

Legend: 1 - Applicable only to Selenium clients.

Originally answered Nov 6 '18 at 8:00

No. When you configure an instance of a ChromeDriver with ChromeOptions to initiate a new Chrome Browser Session the configuration of the ChromeDriver remains unchanged throughout the lifetime of the ChromeDriver and remains uneditable. So you can't change the user agent when the WebDriver instance is executing the loop making 10 requests.

Even if you are able to extract the ChromeDriver and ChromeSession attributes e.g. UserAgent, Session ID, Cookies and other session attributes from the already initiated Browsing Session still you won't be able to change those attributes of the ChromeDriver.

A cleaner way would be to call driver.quit() within tearDown(){} method to close and destroy the ChromeDriver and Chrome Browser instances gracefully and then span a new set of ChromeDriver and Chrome Browser instance with the new set of configurations.

Here you can find a relevant discussion on How can I reconnect to the browser opened by webdriver with selenium?

Reference

You can find a couple of relevant detailed discussions in:

undetected Selenium
  • 183,867
  • 41
  • 278
  • 352
  • 1
    So I think I should use python requests library to do this task because with that I can change IP and user agent on each request and that's how we can make 10 requests using different user agents. Thank you @DebanjanB – Arsalan Nov 06 '18 at 09:34
2

Yes. It is possible now with cdp:

driver.execute_cdp_cmd("Network.enable", {})
driver.execute_cdp_cmd("Network.setExtraHTTPHeaders", {"headers": {"User-Agent": "browser1"}})
driver.execute_cdp_cmd("Network.setExtraHTTPHeaders", {"headers": {"User-Agent": "browser2"}})
driver.execute_cdp_cmd("Network.setExtraHTTPHeaders", {"headers": {"User-Agent": "browser3"}})

driver.get('https://www.httpbin.org/headers')
pguardiario
  • 53,827
  • 19
  • 119
  • 159
0

it open 10 chrome instance because you did notclose() it, try

...
...
driver1.get('https://www.whatsmyua.info/')
time.sleep(5)
driver1.close()
ewwink
  • 18,382
  • 2
  • 44
  • 54
  • Thank you for answering. The problem is I don't want to close the driver instance and make a new request using same instance but with different user agent. is it possible ? – Arsalan Nov 06 '18 at 03:18