2

I am reading through the documentation of the socksify gem on Rubyforge. I have installed the gem successfully, and I have run this documented code with success to test that my local implementation can replicate it:

require 'socksify/http'
uri = URI.parse('http://rubyforge.org/')
Net::HTTP.SOCKSProxy('127.0.0.1', 9050).start(uri.host, uri.port) do |http|
  http.get(uri.path)
end
# => #<Net::HTTPOK 200 OK readbody=true>

But how do I e.g. scrape 'http://google.com/', and get the html content? I wish to parse it with e.g. Nokogiri like this:

Nokogiri::HTML(open("http://google.com/))
JohnSmith1976
  • 536
  • 2
  • 12
  • 35

1 Answers1

3
 require 'socksify/http'
 http = Net::HTTP::SOCKSProxy(addr, port)
 html = http.get(URI('http://google.de'))
 html_doc = Nokogiri::HTML(html)
mattes
  • 8,936
  • 5
  • 48
  • 73
  • 1
    Thanks @mattes. But when I initiate with `http = Net::HTTP::SOCKSProxy('127.0.0.1', 9050)`, and then run `html = http.get(URI('http://google.de'))`I get the response `301 Moved`. – JohnSmith1976 Apr 05 '14 at 14:21
  • That ist correct. `http://google.de` redirects to `http://www.google.de`. For more information, see http://en.wikipedia.org/wiki/URL_redirection#HTTP_status_codes_3xx – mattes Apr 07 '14 at 11:47
  • Right you are. Thanks! – JohnSmith1976 May 02 '14 at 08:39
  • @mattes do you have any idea how to set user-agent using this method? – medBouzid Mar 20 '21 at 18:47