Blacklist URLs with headless Chrome

Question

I'm trying to block URLs in my specs, achieving something like I had when using capybara_webkit:

Capybara::Webkit.configure do |config|
  config.block_url("*google*")
  config.allow_url('*my_website.com')
end

After reading this article, I tried to do something like:

require 'webmock/rspec'

module WebmockConfig
  def self.default_disabled_urls
    [
      '*google*'
    ]
  end
end

WebMock.disable_net_connect!(allow_localhost: true)
WebMock.disable_net_connect!(allow: WebmockConfig.default_disabled_urls)

but I'm getting

Real HTTP connections are disabled. Unregistered request: POST http://127.0.0.1/session

even if that should be solved by WebMock.disable_net_connect!(allow_localhost: true).

When running the specs without WebMock.disable_net_connect!(allow: WebmockConfig.default_disabled_urls), everything is working fine.

Thomas Walpole · Accepted Answer · 2018-07-27T15:09:56.740

The capybara-webkit white/blacklisting affects the requests made by the browser, whereas WebMock can only affect requests made by your app. This means WebMock is useless for what you want since it wouldn't actually stop your browser from loading anything from google, etc. To do that while using the selenium driver you need to use a programmable proxy like puffing-billy which will allow you to customize the responses for any matching requests the browser makes.

To configure a driver using headless chrome and puffing_billy you could do something like

Capybara.register_driver :headless_chrome do |app|
 browser_options = ::Selenium::WebDriver::Chrome::Options.new
 browser_options.headless!
 browser_options.add_argument("--proxy-server=#{Billy.proxy.host}:#{Billy.proxy.port}")
 Capybara::Selenium::Driver.new(app, browser: :chrome, options: browser_options)
end

Whether or not you need any other options is dependent on your system config, etc but you should be able to tell by looking at your current driver registration.

Hey Thomas. From what I can read here https://github.com/oesmith/puffing-billy#setup-for-capybara it's not possible to use my actual driver (`headless_chrome`) with `puffing-billy` fucntionalities? — fabdurso, Jul 27 '18 at 09:40
@fabersky It can be used with any Capybara driver that supports a proxy - you just need to register your own custom driver configuration - https://github.com/oesmith/puffing-billy#customising-the-javascript-driver and https://github.com/oesmith/puffing-billy/blob/master/lib/billy/browsers/capybara.rb . I've added a configuration that could be used with headless chrome to my answer — Thomas Walpole, Jul 27 '18 at 15:10

score 1 · Answer 2 · answered Jul 26 '18 at 12:16

1

The allow_localhost: true settings are overwritten by allow: WebmockConfig.default_disabled_urls you have to call WebMock.disable_net_connect! once with both settings or by adding 'localhost', '127.0.0.1' entries into self.default_disabled_urls

answered Jul 26 '18 at 12:16

narze

66
4

Hey @narze I know how to add multiple urls in `WebMock.disable_net_connect!(allow:` by adding an array of URLs, but how can I combine `allow_localhost` and `allow`? – fabdurso Jul 26 '18 at 12:34
@fabersky You can try `WebMock.disable_net_connect!(allow: [...], allow_localhost: true)` and see if it works or not. If not, manually add localhost ips into `allow` array – narze Jul 26 '18 at 15:29
anyway, even if i achieve that in this way, I don't think it would be the expected result. What I want to achieve is just blocking requests from blacklisted URLs. This way, my specs will probably fail once they will get a request from one of that URLs. – fabdurso Jul 26 '18 at 15:51

Blacklist URLs with headless Chrome

2 Answers2