How can I change remote controller of FireFox in Selenium Python?

Question

When I FireFox is loaded through Selenium, my browser is under remote controller and a bot image show in URL section in my browser. For deal with this problem I changed User-Agent by this code:

from selenium import webdriver
profile = webdriver.FirefoxProfile()
profile.set_preference("general.useragent.override", "whatever you want")
driver = webdriver.Firefox(profile)

User-Agent was changed successfully but, bot image in URL section of my browser was remained. Would you help me, please? I used this URL for changing User-Agent:

Change user agent for selenium driver My whole code is:

MainLink="https://blog.feedspot.com/iot_blogs/"
from selenium import webdriver
from selenium.webdriver.common.desired_capabilities import DesiredCapabilities

caps = DesiredCapabilities.PHANTOMJS

caps["phantomjs.page.settings.userAgent"] = "whatever you want"
driver = webdriver.Firefox()
from selenium import webdriver
profile = webdriver.FirefoxProfile()
profile.set_preference("general.useragent.override", "whatever you want")
driver = webdriver.Firefox(profile)
agent = driver.execute_script("return navigator.userAgent")
print(agent)
driver.get(MainLink)

So your problem is that you want to remote control your browser but does not want it to display it to the enduser? The fact it shows the icon and the message you are referring to would not compromise any tests or scraping you might be doing in terms of functionality. — Lafa, Sep 30 '18 at 20:14
Who do you want to be hiding the bot action from? Some end user or the page you're accessing? — Lafa, Sep 30 '18 at 20:38

Lafa · Accepted Answer · 2018-09-30T21:25:38.853

Your code is a little confusing. You don't need to use both phantomjs and firefox as the driver for selenium. Which one are you going to use?

As I understood, you would like to avoid being detected from the page you're interacting with. This is usually a greater concern if you're operating with a headless browser, which is the case when using phantomjs but not when using firefox without explicitly telling it to run in this mode, which apparently is your case.

If you are in fact having issues of this nature, there are many ways to try to mitigate this, starting from changing the user agent, as mentioned by you. Assuming you want to go with firefox, A code example would be:

from selenium import webdriver

profile = webdriver.FirefoxProfile()
user_agent = 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.47 Safari/537.36'


profile.set_preference("general.useragent.override", user_agent)
driver = webdriver.Firefox(profile)
MainLink="https://blog.feedspot.com/iot_blogs/"

driver.get(MainLink)

In addition, you could set a different user agent each time you make a request, combining with changing the ip address from where the requests are originated, of course, but this is besides the point...

Hope this helps...

Tanks for your guidance about headless browser. But your code doesn't work in headless way. But, I could run it in headless way that I share it in response section. — Hamed Baziyad, Oct 02 '18 at 19:04

score 1 · Answer 2 · answered Jun 16 '20 at 20:57

Depending on how "deep" you want to go, you could also add cookies copied from a trusted client:

from selenium import webdriver

driver = webdriver.Chrome()

driver.get("http://www.example.com")

# Adds the cookie into current browser context
driver.add_cookie({"name": "key", "value": "value"})

See also: https://www.selenium.dev/documentation/en/support_packages/working_with_cookies/

score 0 · Answer 3 · answered Oct 02 '18 at 19:06

I could run my code without loading page or in headless way (headless browser). from selenium import webdriver

from selenium.webdriver.firefox.options import Options
options = Options()
options.add_argument("--headless")
driver = webdriver.Firefox(firefox_options=options)
print("Firefox Headless Browser Invoked")
driver.get('https://blog.feedspot.com/iot_blogs/')

s=driver.find_element_by_xpath("""/html/body/div[1]/div[2]/div/div/div[1]/article/div[1]/h1""")
print(s.text)

How can I change remote controller of FireFox in Selenium Python?

3 Answers3