38

I want to pass all Python's traffics through a http proxy server, I checked urlib2 and requests packages for instance, they can be configured to use proxies but how could I use something like a system-wide proxy for Python to proxy all the data out?

YanetP1988
  • 1,346
  • 3
  • 18
  • 43
Hassan Abedi
  • 790
  • 1
  • 7
  • 22
  • What do you mean by system-wide proxy and how do you configure it? If there is one, you should extract it and set it in urllib2. – helloV Jul 26 '15 at 17:19
  • I mean, I don't want to use a proxy for just a particular Python library(like requests), instead I want to pass all traffics originating from Python process running my script through a proxy. – Hassan Abedi Jul 26 '15 at 17:31
  • I think you are trying to approach this from a wrong direction. If you need something like "system wide proxy" you should try solve this issue on network level. I don't think there are system wide config files for urlib or requests libraries. – Boris Jul 26 '15 at 17:37
  • What kind of script? I am assuming you have a shell script that starts few python processes and you want them to use the proxy. – helloV Jul 26 '15 at 17:37

2 Answers2

73

Linux system first export the environment variables like this

$ export http_proxy="http://<user>:<pass>@<proxy>:<port>"
$ export HTTP_PROXY="http://<user>:<pass>@<proxy>:<port>"

$ export https_proxy="http://<user>:<pass>@<proxy>:<port>"
$ export HTTPS_PROXY="http://<user>:<pass>@<proxy>:<port>"

or in the script that you want to pass through the proxy

import os

proxy = 'http://<user>:<pass>@<proxy>:<port>'

os.environ['http_proxy'] = proxy 
os.environ['HTTP_PROXY'] = proxy
os.environ['https_proxy'] = proxy
os.environ['HTTPS_PROXY'] = proxy

#your code goes here.............

then run python script

$ python my_script.py

UPDATE

And also you can use redsocks With this tool you can redirect silently all your TCP connections to a PROXY with or without authentication. But you have to be carefull because it's for all connections not only for the python.

Windows systems you can use tools like freecap, proxifier, proxycap, and configure to run behind the python executable

Xavier Lopez
  • 67
  • 8
efirvida
  • 4,592
  • 3
  • 42
  • 68
  • 1
    @efirvida Its not working in my case: https://stackoverflow.com/questions/48676514/how-to-resolve-proxy-error-in-reading-writing-to-hdfs-using-python I came across your answer after I posted this question. I tried exporting the proxies like you say here, and then os.environ... But the error persists. – Kristada673 Feb 08 '18 at 03:24
  • faced with encoded = binascii.b2a_base64(s, newline=False) TypeError: a bytes-like object is required, not 'str' – Yuseferi Jun 30 '18 at 13:24
  • @zhilevan, i dont know about that error, but maybe try this solution: https://stackoverflow.com/questions/45482272/typeerror-a-bytes-like-object-is-required-not-str-python-2-to-3 – efirvida Jun 30 '18 at 20:00
  • Just tried and the process stuck after assigning this to the four environment variables. The log is not coming out as well. – Terry Windwalker Jul 17 '21 at 06:55
  • You're a gem. Solved an issue I was trying to solve since yesterday. Thank you!! – Arsh Arora Aug 28 '22 at 11:51
6

Admittedly, this isn't exactly what you are looking for, but if you know the programmatic source of all your network traffic in your python code, you can do this proxy wrapping in the code itself. A pure python solution is suggested using the PySocks module in the following link:

https://github.com/httplib2/httplib2/issues/22

import httplib2
# detect presense of proxy and use env varibles if they exist
pi = httplib2.proxy_info_from_environment()
if pi:
    import socks
    socks.setdefaultproxy(pi.proxy_type, pi.proxy_host, pi.proxy_port)
    socks.wrapmodule(httplib2)    

# now all calls through httplib2 should use the proxy settings
httplib2.Http()

Now every call made using httplib2 will use those proxy settings. You should be able to wrap any networking module that uses sockets to use this proxy.

https://github.com/Anorov/PySocks

They mention there that this isn't recommended, but it seems to be working fine for me.

chisaipete
  • 884
  • 9
  • 20