Using curl in Popen in Python

Question

I run this curl command in the unix shell and it works (see below). I was able to redirect the returned data to a file but now I want to process the data in my code instead of wasting a bunch of space in a file.

curl -k -o outputfile.txt 'obfuscatedandVeryLongAddress'
#curl command above, python representation below
addr = "obfuscatedandVeryLongAddress"
theFile = subprocess.Popen(["curl", "-k", addr], stdout = subprocess.PIPE, stderr = subprocess.PIPE, shell = True)

theFile.stdout is empty after this. The data returned in the curl command should be something like 4,000 lines (verified when running the command in the shell). Is the size breaking theFile.stdout? Am I doing something else wrong? I tried using:

out, err = theFile.communicate()

and then printing the out variable but still nothing

edit: formatting and clarification

Why don't you use the `requests` library? Or system default `urllib` ? — UltraInstinct, Jul 07 '16 at 20:37
related: [Why does shell=True eat my subprocess.Popen stdout?](http://stackoverflow.com/q/10661457/4279) — jfs, Jul 07 '16 at 22:51

Eugene K · Accepted Answer · 2016-07-08T00:07:27.277

3

You need to remove shell=True.

theFile = subprocess.Popen(["curl", "-k", addr], stdout = subprocess.PIPE, stderr = subprocess.PIPE)

Should work.

If you do shell=True, you should pass a string. Otherwise, what you're actually doing is passing those arguments -k, and addr as arguments to the shell. So if your shell is sh, what you're doing is sh 'curl' -k addr.

edited Jul 08 '16 at 00:07

answered Jul 07 '16 at 20:38

Eugene K

3,381
2
23
36

score 0 · Answer 2 · answered Jul 07 '16 at 20:46

Eugene's is a direct answer to your question, but I thought I'd add one on using the requests library, since it will require less code and be easier to read for anyone that needs to look at your code (and has the benefit of being cross-platform).

import requests

response = requests.get('longaddress', verify=False)
print response.text

If the response is json, you can automatically convert it to a python object

print response.json()

score 0 · Answer 3 · answered Jul 07 '16 at 21:15

You can put the curl command in a string like:

theFile = subprocess.Popen('curl -k {}'.format(addr), stdout = subprocess.PIPE, stderr = subprocess.PIPE, shell = True)

Or you can remove the shell argument:

theFile = subprocess.Popen(["curl", "-k", addr], stdout = subprocess.PIPE, stderr = subprocess.PIPE)

Or you could use the pycurl module to use libcurl library directly and skip the whole additional process.

Using curl in Popen in Python

3 Answers3