1

I had configure the spider to access the Tor with setup Privoxy but this only work when I use in localhost as the setting I configure is pointed to 127.0.0.1: port. But when i deploy to the Scapinghub, the server side do not setup tor and privoxy as i do. Is that any solution that i can use to let the spider go through my machine through my network and port ?

As i know, if on the same network, we can use the internal IP. Can I just replace the public IP to 127.0.0.1 but i wonder how the network to forward to which machine.

Below is the configuration to access tor:

middlewares.py

class ProxyMiddleware(object):
    @classmethod
    def process_request(self, request, spider):
        request.meta['proxy'] = "http://127.0.0.1:8118"

setting.py 

DOWNLOADER_MIDDLEWARES = {
    'tutorial.middlewares.ProxyMiddleware': 1
}
Terence Goh
  • 13
  • 1
  • 4

1 Answers1

0

You can deploy a custom docker image with tor set up on it. And then point to the 127.0.0.1. https://shub.readthedocs.io/en/stable/deploy-custom-image.html#deploy-custom-image

Grg_Lnt
  • 61
  • 10