how to set proxy value and run linux command in python

Question

In linux to set a proxy value you do the following.

proxy=http://$user:$password@proxy.server.com:${NUM}
http_proxy="$proxy" https_proxy="$proxy" ${COMMAND}

For security reasons, if you run this in a subshell, you are not explicitly letting your password in the open, or in the logs. Problem with this approach is, I have to set user name and password everytime I want to run a command.

Therefore decided to write a python code for it. I have a working version in C. Just wanted to learn more in Python. I have found nice ways to encode and decode my password, and after most of the hooplahs, I pass it to this function to test proxy connection.

def test_connection(creds,proxy_url):
    import pycurl
    import cStringIO

    buf = cStringIO.StringIO()
    test_url="http://www.google.com"

    c = pycurl.Curl()
    c.setopt(c.URL, test_url)
    c.setopt(c.WRITEFUNCTION, buf.write)
    c.setopt(c.PROXY, proxy_url)
    c.setopt(c.PROXYPORT, 8080)
    c.setopt(c.PROXYTYPE, c.PROXYTYPE_HTTP) 
    c.setopt(c.PROXYAUTH, c.HTTPAUTH_NTLM) 
    c.setopt(c.PROXYUSERPWD, creds) 
    c.perform()
    buf.close()
    return c.getinfo(c.RESPONSE_CODE)

Where I'm having problems is using suprocess, I do understand that subprocess does not allow you to use export, since it is not really a command. Subprocess module errors with 'export' in python on linux?

this is my implementation

finalCommand = ["/bin/sh", "-c"]
finalCommand.append(http_proxy)
finalCommand.append(https_proxy)
for x in bashCommand:
    finalCommand.append(x)

print subprocess.call(finalCommand)
process = subprocess.Popen(finalCommand,stdout=subprocess.PIPE)


out, err = process.communicate()

print "Output ... \n %s" % (out)

if err == None: 
    print "No errors"
else: 
    print "Errors ... \n %s" %(err)

Unfortunately, after several tests, my program always return no output and no error. I have printed the output of the curl, so i know the decode, encode, or proxy isn't the issue. Any suggestions?

POST-ANSWER EDIT: Interaction between Python script and linux shell

env did solve my problem, but I also had to refer to the thread above. Some of the commands I ran were interactive one, and as it explains it well in the thread, PIPE doesn't work properly with interactive programs.

Are you just looking for the `env` argument to [`Popen`](https://docs.python.org/2/library/subprocess.html#subprocess.Popen)? Or are you actually trying to launch a shell and run scripts directly within it instead of in subshells, or something else funky? — abarnert, Jul 23 '14 at 20:15
Meanwhile, please give us a [MCVE](http://stackoverflow.com/help/mcve)—in particular, notice the "complete" part. Without knowing what's in `http_proxy`, `https_proxy`, and `bashCommand`, it's hard to debug what might be going wrong. (Also, is there's a reason you're first doing `call` on the command, and then creating a `Popen` with the exact same command and calling `communicate` on it?) — abarnert, Jul 23 '14 at 20:22
It's not so much that "PIPE doesn't work properly with interactive programs" (although that is true before Python 3.2, you're probably not running into that, and if you are, install the `subprocess32` backport to fix that), it's that some interactive programs require you to change the buffering, and others aren't talking to stdin at all (e.g., they're reading a password without echoing it and need a real tty, or they just ask libpam to do it for them, etc.) so piping to stdin doesn't do any good. And also that writing code to talk interactively over a pipe without blocking is hard. — abarnert, Jul 24 '14 at 17:40

abarnert · Accepted Answer · 2014-07-23T20:25:14.577

It's hard to be sure without knowing exactly what commands you're trying to run, but I'm pretty sure what you want to do here is just set up the environment for your subprocess, using the env argument to Popen:

env = dict(os.environ)
env['http_proxy'] = proxy
env['https_proxy'] = proxy
for command in commands:
    out = subprocess.check_output(command, env=env)

If you want to modify your own environment, rather than just the subprocesses' environments, just modify os.environ in place. (See the documentation for platform-specific issues, and how to deal with them.)

Meanwhile, the reason you're getting no errors is simple:

process = subprocess.Popen(finalCommand,stdout=subprocess.PIPE)
out, err = process.communicate()

If you don't pass stderr=subprocess.PIPE to the Popen constructor, it doesn't capture stderr, so err ends up as None.

As a side note, you almost never want to check == None. Often, just if not err: is sufficient. When it's not, if err is not None: is almost always required. The set of cases where == is necessary but not insufficient is vanishingly small. See the Programming Recommendations in PEP 8 for (slightly) more details.

And one more side note: You can just write finalCommand.extend(x). The list.extend method does the same thing as looping over an iterable and appending each element one by one, except that it's more readable, harder to get anything wrong, more concise, and faster.

thanks for the tips.. I'll have to dig this out myself, but setting an env variable, wouldn't that show up inside the env variables of the command line.. — kmassada, Jul 24 '14 at 12:13
@kmassada: I'm not sure what your question means. If a child process (like your Python script) modifies its environment, or launches a child of its own with a different environment, that won't affect the parent shell, much less any other shells you have running. But if you launch a shell with `subprocess.call(['/bin/sh'], env=env`) or something, then yes, the keys in `env` will all show up as the variables of the shell you launched. — abarnert, Jul 24 '14 at 17:34
If you're using nodejs `puppeteer`, don't forget to add `env[no_proxy]=somehost` if necessary. — caram, Feb 04 '20 at 17:49

how to set proxy value and run linux command in python

1 Answers1