Setting custom User Agent in Selenium Webdriver for PhantomJS with Ruby

Question

I've been all over the net tonight, for about 3-4 hours now. I've tried every suggestion I've come across. I've even checked the "capabilities" object on my Selenium driver object to ensure that it is actually set there, and indeed, it is:

#<Selenium::WebDriver::Remote::Capabilities:0x00000007475cf0
 @capabilities=
  {:browser_name=>"phantomjs",
   :version=>"1.9.7",
   :platform=>:"linux-unknown-64bit",
   :javascript_enabled=>true,
   :css_selectors_enabled=>true,
   :takes_screenshot=>true,
   :native_events=>true,
   :rotatable=>false,
   :firefox_profile=>nil,
   :proxy=>#<Selenium::WebDriver::Proxy:0x00000007475908 @type=:direct>,
   "driverName"=>"ghostdriver",
   "driverVersion"=>"1.1.0",
   "handlesAlerts"=>false,
   "databaseEnabled"=>false,
   "locationContextEnabled"=>false,
   "applicationCacheEnabled"=>false,
   "browserConnectionEnabled"=>false,
   "webStorageEnabled"=>false,
   "acceptSslCerts"=>false,
   "proxy"=>{"proxyType"=>"direct"},
   "phantomjs.page.settings.userAgent"=>
    "Mozilla/5.0 (X11; Ubuntu; Linux i686; rv:27.0) Gecko/20100101 Firefox/27.0"}>

Besides "phantomjs.page.settings.userAgent", I've tried "userAgent", etc. Every thing I could find online in the last 3-4 hours, I've tried. Apparently, near the beginning of 2013, this was a quite common question, and the solutions I'm speaking of were apparently the common solutions. None of these are working, and in fact, I know this for sure from this bit of information (note that User-Agent is "Ruby"):

UNCAUGHT EXCEPTION: {"errorMessage"=>"Element is not currently visible and may not be manipulated",
 "request"=>
  {"headers"=>
    {"Accept"=>"application/json",
     "Accept-Encoding"=>"gzip;q=1.0,deflate;q=0.6,identity;q=0.3",
     "Connection"=>"close",
     "Content-Length"=>"2",
     "Content-Type"=>"application/x-www-form-urlencoded",
     "Host"=>"localhost:9876",
     "User-Agent"=>"Ruby"},
   "httpVersion"=>"1.1",
   "method"=>"POST",
   "post"=>"{}",
   "postRaw"=>"{}",
   "url"=>"/click",
   "urlParsed"=>
    {"anchor"=>"",
     "query"=>"",
     "file"=>"click",
     "directory"=>"/",
     "path"=>"/click",
     "relative"=>"/click",
     "port"=>"",
     "host"=>"",
     "password"=>"",
     "user"=>"",
     "userInfo"=>"",
     "authority"=>"",
     "protocol"=>"",
     "source"=>"/click",
     "queryKey"=>{},
     "chunks"=>["click"]},
   "urlOriginal"=>
    "/session/a03cc440-4f5c-11e4-8854-ed9c22bf60af/element/%3Awdc%3A1412822036214/click"}}

Unfortunately, there is a lot more information and discussion on these kinds of Selenium issues, and many others, if you're using Java. At this point, I wish in every way I had gone with Java for this project, but now I have 30,000 lines of code that I wrote entirely myself over the past 2 months. Losing this work, now at least, would not only be devastating for me personally, but would be disastrous for my job.

What gives? Am I really going to have to dig in and customize the source to get what I want, or is this feature really implemented now? Again, I saw all the answers from early 2013, but they don't work for me, and I have no idea why, or how to easily fix it. I'm on a deadline so this is beginning to be very stressful.

Does anyone have any ideas for me? Keep in mind I'm using Ruby, not Java.

Selenium-webdriver is 2.43. PhantomJS is 1.9.7. GhostDriver is 1.1.0.

This seems incredulous, to me, that there is no way to modify your User-Agent.

Please let me know if I can supply any other information that might be of assistance.

I appreciate in advance if you'd be so kind to share a few ideas or some information to get me pointed in the right direction.

Answer 1

In Java I did the following: (PhantomJS 1.9.8, Selenium 2.39)

String userAgent = "Mozilla/5.0 (Linux; U; Android 2.3.3; en-us; LG-LU3000 Build/GRI40) AppleWebKit/533.1 (KHTML, like Gecko) Version/4.0 Mobile Safari/533.1";
DesiredCapabilities caps = new DesiredCapabilities();
caps.setCapability(PhantomJSDriverService.PHANTOMJS_PAGE_SETTINGS_PREFIX + "userAgent", userAgent);
PhantomJSDriver driver = new PhantomJSDriver(caps);

Maybe there are similar setting options for Ruby. Take a look at "page.settings.userAgent"

Answer 2

capabilities = Selenium::WebDriver::Remote::Capabilities.phantomjs('phantomjs.page.settings.userAgent' => 'some user Agent')
@instance = Selenium::WebDriver.for(:remote,:url=>'http://localhost:8910',:desired_capabilities=>capabilities)

Answer 3

This ended up not being "possible" out-of-the-box, as moonfly mentioned in his comments to my question. However, this turns out to be relatively easy to do. I should note, that this did not solve my particular problem. I assumed, perhaps naively, that I was getting strange results because of the User-Agent 'Ruby'. Turns out I spent hours figuring this out for nothing; I still get the same result.

But, for those of you that this will help, you'll have to make a quick monkey-patch to Selenium. Nasty, gross, ugly, hacky, I know. But, it did the trick.

Note, that this is a monkey-patch for Ruby's selenium-webdriver, version 2.43.0 -- if you have another version, there is no guarantee this will work.

module Selenium
    module WebDriver
        module Remote
            module Http

                class Default < Common
                    private

                    def request(verb, url, headers, payload, redirects = 0)
            # THIS IS WHERE OUR CUSTOM CHANGE BEGINS
            headers.merge!('User-Agent' => 'Whatever User Agent You May Want')
            # THIS IS WHERE OUR CUSTOM CHANGE ENDS

            request = new_request_for(verb, url, headers, payload)

            retries = 0
            begin
              response = response_for(request)
            rescue Errno::ECONNABORTED, Errno::ECONNRESET, Errno::EADDRINUSE
              # a retry is sometimes needed on Windows XP where we may quickly
              # run out of ephemeral ports
              #
              # A more robust solution is bumping the MaxUserPort setting
              # as described here:
              #
              # http://msdn.microsoft.com/en-us/library/aa560610%28v=bts.20%29.aspx
              raise if retries >= MAX_RETRIES

              request = new_request_for(verb, url, headers, payload)
              retries += 1

              retry
            rescue Errno::ECONNREFUSED => ex
              if use_proxy?
                raise ex.class, "using proxy: #{proxy.http}"
              else
                raise
              end
            end

            if response.kind_of? Net::HTTPRedirection
              raise Error::WebDriverError, "too many redirects" if redirects >= MAX_REDIRECTS
              request(:get, URI.parse(response['Location']), DEFAULT_HEADERS.dup, nil, redirects + 1)
            else
              create_response response.code, response.body, response.content_type
            end
          end

        end

      end
    end
  end
end