Scrapyd with Polipo and Tor

Question

UPDATE: I am now running this command:

scrapyd-deploy <project_name>

And getting this error:

504 Connect to localhost:8123 failed: General SOCKS server failure

I am trying to deploy my scrapy spider through scrapyd-deploy, the following is the command I use:

scrapyd-deploy -L <project_name>

I get the following error message:

    Traceback (most recent call last):
  File "/usr/local/bin/scrapyd-deploy", line 269, in <module>
    main()
  File "/usr/local/bin/scrapyd-deploy", line 74, in main
    f = urllib2.urlopen(req)
  File "/usr/lib/python2.7/urllib2.py", line 127, in urlopen
    return _opener.open(url, data, timeout)
  File "/usr/lib/python2.7/urllib2.py", line 410, in open
    response = meth(req, response)
  File "/usr/lib/python2.7/urllib2.py", line 523, in http_response
    'http', request, response, code, msg, hdrs)
  File "/usr/lib/python2.7/urllib2.py", line 448, in error
    return self._call_chain(*args)
  File "/usr/lib/python2.7/urllib2.py", line 382, in _call_chain
    result = func(*args)
  File "/usr/lib/python2.7/urllib2.py", line 531, in http_error_default
    raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
urllib2.HTTPError: HTTP Error 404: Not found

The following is my scrapy.cfg file:

[settings]
default = <project_name>.settings

[deploy:<project_name>]
url = http://localhost:8123
project = <project_name>
eggs_dir    = eggs
logs_dir    = logs
items_dir   = items
jobs_to_keep = 5
dbs_dir     = dbs
max_proc    = 0
max_proc_per_cpu = 4
finished_to_keep = 100
poll_interval = 5
http_port = 8123
debug       = on
runner      = scrapyd.runner
application = scrapyd.app.application
launcher    = scrapyd.launcher.Launcher

[services]
schedule.json     = scrapyd.webservice.Schedule
cancel.json       = scrapyd.webservice.Cancel
addversion.json   = scrapyd.webservice.AddVersion
listprojects.json = scrapyd.webservice.ListProjects
listversions.json = scrapyd.webservice.ListVersions
listspiders.json  = scrapyd.webservice.ListSpiders
delproject.json   = scrapyd.webservice.DeleteProject
delversion.json   = scrapyd.webservice.DeleteVersion
listjobs.json     = scrapyd.webservice.ListJobs

I am running tor and polipo, with the polipo proxy on port 'http://localhost:8123'. I can perform a wget and download that page without any problems. The proxy is correctly working, I can connect to the internet and so on. Please ask if you need more clarification.

Thanks!

score 0 · Answer 1 · answered Jun 13 '14 at 17:54

0

urllib2.HTTPError: HTTP Error 404: Not found

The url is not reached.

answered Jun 13 '14 at 17:54

elmonkeylp

2,674
1
16
14

Yes, I am aware of that. That's the problem I am trying to solve. – Moataz Elmasry Jun 16 '14 at 09:46

score 0 · Answer 2 · answered Jun 14 '14 at 05:55

0

Anything interesting in /var/log/polipo/polipo.log? What comes from tail -100 /var/log/polipo/polipo.log?

answered Jun 14 '14 at 05:55

There

498
6
18

Nothing interesting, just the following messages: Couldn't parse ETag. Unsupported Cache-Control directive post-check -- ignored. – Moataz Elmasry Jun 16 '14 at 09:43

score 0 · Accepted Answer · answered Jun 17 '14 at 16:03

0

Apparently this is because I forgot to run the main command. It is easy to miss because it is mentioned in the Overview page of the documentation, and not the Deployment page. The following is the command:

scrapyd

answered Jun 17 '14 at 16:03

Moataz Elmasry

542
1
5
18

score 0 · Answer 4 · answered Oct 10 '14 at 12:47

504 Connect to localhost:8123 failed: General SOCKS server failure

You're asking Polipo to connect to localhost:8123; Polipo passes the request to tor, which returns a failure result which is dutifully returned by Polipo ("General SOCKS server failure").

url = http://localhost:8123

This is certainly not what you meant.

http_port = 8123

I'm also pretty sure you didn't want to run scrapyd on the same port as Polipo.

Scrapyd with Polipo and Tor

4 Answers4