3

Is it possible to change your IP Address and User Agent when using NodeJS/X-Ray to make requests to an external site?

Randy
  • 71
  • 1
  • 8

1 Answers1

2

Yes you can.

But instead of passing the url(s) you want to scrape to x-ray, rather use the request module to get the response, and pass that to x-ray. This will allow you to pass options into the request module, which will allow you to change your User Agent as well as use a proxy (which is the best way to 'change' your IP).

var options = {
    headers: {'User-Agent': 'Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 7.0; InfoPath.3; .NET CLR 3.1.40767; Trident/6.0; en-IN)'},
    proxy: 'http://us-ny.proxymesh.com:31280',
    strictSSL: false
};

request(url, options, function (err, response) {

    xray(response.body, {
        //x-ray selectors            
    })
    (function (err, obj) {
    //parse results
    }
});
Mark Gibaud
  • 2,061
  • 23
  • 35
  • That's an interesting approach. It looks like you're saying I should be able to spoof the ip in an http custom header but I cant figure out how to do that. I'm able to spoof the user agent but what would be the setting for the ip address. I tried 'Forwarded:' : 'for=192.0.2.43, for=198.51.100.17' and 'X-Forwarded-For:' : 'xxx.xxx.xxx.xxx' but neither of those seem to work. – Randy Mar 25 '16 at 21:59
  • As far as I'm aware you can't actually change your IP by doing anything on your side. All you can do is go through a proxy, which means your end destination (the servers hosting the website you want to screenscrape) have a source IP that is not your own (but rather the IP of one of the proxy servers you're going through). – Mark Gibaud Mar 25 '16 at 22:10
  • how would you handle proxie authentication here? – Kelvin U Nov 09 '16 at 02:57
  • Try setting your proxy like this: proxy: "http://username:password@proxy.foo.com:8080" – Mark Gibaud Nov 09 '16 at 18:41