0

I am sending a request to a website ( I am using request module ) and it returns data in response. Everything is working fine until I send request behind a proxy ( even when the proxy is not banned on the site ).

my problem is the same How to stop NodeJS "Request" module changes request when using proxy

I tried every solution in the above post but nothing helped

Headers I am using in request

Host: www.somesite.com
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:69.0) Gecko/20100101 Firefox/69.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate, br
Connection: keep-alive
Upgrade-Insecure-Requests: 1
webmaker
  • 1
  • You should post some code. Also don't set Host. – pguardiario Oct 20 '19 at 00:20
  • @pguardiario host ? – webmaker Oct 20 '19 at 07:44
  • The first header you set is Host.Let the library set that. – pguardiario Oct 20 '19 at 10:17
  • @pguardiario Headers are really important here , If I remove any header or change the order of the headers The request will get redirected to the bot protection because the request should look like it was made from a browser – webmaker Oct 20 '19 at 16:23
  • The Headers are really important , even the order of the headers. If I remove any header or change the order the request will get redirected towards to bot protection. The request should totally look like it was made by a browser otherwise the bot protection catches it. you can check https://stackoverflow.com/questions/55243887/how-to-stop-nodejs-request-module-changes-request-when-using-proxy . I tried every solution answers in that post but nothing worked , even my proxies are not modifying headers. I don't know what to do @pguardiario – webmaker Oct 20 '19 at 16:25
  • The library sets the host header. When you use a proxy it gets set differently than a direct request. – pguardiario Oct 21 '19 at 01:15
  • No the library does not set host header automatically , I tested a request on my own webserver @pguardiario – webmaker Oct 21 '19 at 04:13
  • No you're wrong. Look at [the examples](https://github.com/request/request). Nowhere does anybody ever set the Host header because the library needs to set that based on the url and proxy. – pguardiario Oct 21 '19 at 04:48
  • If I don't set the host header the request gets redirected towards the bot protection @pguardiario – webmaker Oct 21 '19 at 05:40
  • What bot protection? If you decide you need to set those headers you need the more low level 'http' library. – pguardiario Oct 21 '19 at 06:52
  • The request works good and returns data but as soon as I use a proxy it gets redirected towards the bot protection. Should I use native http module ? @pguardiario – webmaker Oct 21 '19 at 08:04
  • You should post some code, otherwise we have no idea what you're doing. – pguardiario Oct 21 '19 at 08:27
  • Can I contact you somewhere ? Do not want to reveal the site as per their security reasons. @pguardiario – webmaker Oct 21 '19 at 09:34
  • @pguardiario can I ? please I need help – webmaker Oct 23 '19 at 12:52

0 Answers0