How to use Python to execute a cURL command?

Question

I want to execute a curl command in Python.

Usually, I just need to enter the command in the terminal and press the return key. However, I don't know how it works in Python.

The command shows below:

curl -d @request.json --header "Content-Type: application/json" https://www.googleapis.com/qpxExpress/v1/trips/search?key=mykeyhere

There is a request.json file to be sent to get a response.

I searched a lot and got confused. I tried to write a piece of code, although I could not fully understand it and it didn't work.

import pycurl
import StringIO

response = StringIO.StringIO()
c = pycurl.Curl()
c.setopt(c.URL, 'https://www.googleapis.com/qpxExpress/v1/trips/search?key=mykeyhere')
c.setopt(c.WRITEFUNCTION, response.write)
c.setopt(c.HTTPHEADER, ['Content-Type: application/json','Accept-Charset: UTF-8'])
c.setopt(c.POSTFIELDS, '@request.json')
c.perform()
c.close()
print response.getvalue()
response.close()

The error message is Parse Error. How to get a response from the server correctly?

Does this help? http://stackoverflow.com/questions/2667509/curl-alternative-in-python — br3w5, Aug 25 '14 at 17:25
FWIW, have you considered using [pycurl](http://curl.haxx.se/libcurl/python/) the _"Python binding to cURL"_ ? Depending your needs, it might be more efficient/convenient than invoking the command line utility behind the scene. — Sylvain Leroux, Aug 25 '14 at 17:25
Do you need to use cURL? Have you considered [Requests](http://docs.python-requests.org/en/latest/)? Might be simpler, especially if you're new to python, which tends to be unforgiving. — vch, Aug 25 '14 at 17:26
I just wanted to also point you to this great answer on how to execute a shell command in Python: https://stackoverflow.com/a/92395/778533 — tommy.carstensen, Apr 02 '18 at 03:40

score 305 · Answer 1 · edited Sep 07 '22 at 07:51

305

For the sake of simplicity, you should consider using the Requests library.

An example with JSON response content would be something like:

import requests
r = requests.get('https://github.com/timeline.json')
r.json()

If you look for further information, in the Quickstart section, they have lots of working examples.

For your specific curl translation:

import requests

url = 'https://www.googleapis.com/qpxExpress/v1/trips/search?key=mykeyhere'
payload = open("request.json")
headers = {'content-type': 'application/json', 'Accept-Charset': 'UTF-8'}
r = requests.post(url, data=payload, headers=headers)

edited Sep 07 '22 at 07:51

Boris Verkhovskiy

14,854
11
100
103

answered Aug 25 '14 at 17:33

otorrillas

4,357
1
21
34

For some reason Requests is eating up the response data.. My response should be something like `One=1&Two=2&Three=3` but instead it is returning `One=1&&Three=3` I cURLed it and the response is coming through in the format I am expecting.. weird bug. – tricknology Apr 24 '15 at 21:09
1

Please @tricknology, try to search for the bug and, if you do not happen to find a proper solution, post a new question. – otorrillas Apr 24 '15 at 21:32
4

Should anyone else happen to see this, the reason why that was happening to me is that I was giving a string as my payload instead of a dictionary object. – tricknology Apr 24 '15 at 22:37
1

It seems there's a small typo in the headers, which should read `'Accept-Charset': 'UTF-8'` – Stephen Lead Feb 04 '16 at 02:25
thanks for reporting @StephenLead. I have just corrected it. – otorrillas Feb 04 '16 at 19:26
1

Opening the file and parsing JSON before sending it is needlessly inefficient. You parse the JSON then convert it back into a string with json.dumps(). This is a bad answer. – Nathan K Dec 15 '17 at 21:44
4

`Requests` is an extra dependency you need to install and manage. For a simple solution using just standard library, see https://stackoverflow.com/a/13921930/111995 – geekQ May 29 '18 at 15:40
1

This is all very nice, but doesn't actually answer the question the OP posed? – blamblambunny Jul 07 '19 at 20:49

score 195 · Answer 2 · edited Sep 07 '22 at 08:14

195

Use curlconverter.com. It'll convert almost any curl command into Python, Node.js, PHP, R, Go and more.

Example:

curl -X POST -H 'Content-type: application/json' --data '{"text":"Hello, World!"}' https://hooks.slack.com/services/asdfasdfasdf

becomes this in Python

import requests

json_data = {
    'text': 'Hello, World!',
}

response = requests.post('https://hooks.slack.com/services/asdfasdfasdf', json=json_data)

edited Sep 07 '22 at 08:14

Boris Verkhovskiy

14,854
11
100
103

answered Jun 05 '18 at 16:02

Kyle Bridenstine

6,055
11
62
100

Curl converter does not seem to work well with fields, such as `curl -F name=John -F shoesize=11 https://example.com/`. It treats them as files. – w. Patrick Gale Jun 22 '23 at 14:21
1

https://curlconverter.com/ was very useful! – Benas Jul 24 '23 at 14:17

score 36 · Answer 3 · edited Oct 01 '22 at 08:02

curl -d @request.json --header "Content-Type: application/json" https://www.googleapis.com/qpxExpress/v1/trips/search?key=mykeyhere

its Python implementation looks like this:

import requests

headers = {
    'Content-Type': 'application/json',
}

params = {
    'key': 'mykeyhere',
}

with open('request.json') as f:
    data = f.read().replace('\n', '')

response = requests.post('https://www.googleapis.com/qpxExpress/v1/trips/search', params=params, headers=headers, data=data)

Check this link, it will help convert cURL commands to Python, PHP and Node.js

Joran Beasley · Answer 4 · 2016-06-14T16:01:19.613

29

import requests
url = "https://www.googleapis.com/qpxExpress/v1/trips/search?key=mykeyhere"
data = requests.get(url).json

maybe?

if you are trying to send a file

files = {'request_file': open('request.json', 'rb')}
r = requests.post(url, files=files)
print r.text, print r.json

ahh thanks @LukasGraf now i better understand what his original code is doing

import requests,json
url = "https://www.googleapis.com/qpxExpress/v1/trips/search?key=mykeyhere"
my_json_data = json.load(open("request.json"))
req = requests.post(url,data=my_json_data)
print req.text
print 
print req.json # maybe?

edited Jun 14 '16 at 16:01

answered Aug 25 '14 at 17:28

Joran Beasley

110,522
12
160
179

That doesn't include data from the `requests.json` file though, and doesn't set the `Content-Type: application/json` header - also, this will send a `GET` request, not a `POST`. – Lukas Graf Aug 25 '14 at 17:29
1

`curl -d @` will read the *fields* to post from `` - that's not the same as file upload. – Lukas Graf Aug 25 '14 at 17:32
@LukasGraf thanks :) ... I dont use curl much (read: almost never) – Joran Beasley Aug 25 '14 at 17:40
2

One small note, `data = requests.get(url).json` should be `data = requests.get(url).json()` – dpg5000 Apr 28 '20 at 04:26
1

in 2014 it was a property now its a function :) good call though – Joran Beasley Apr 28 '20 at 23:33

score 15 · Answer 5 · edited Jul 10 '18 at 02:28

15

My answer is WRT python 2.6.2.

import commands

status, output = commands.getstatusoutput("curl -H \"Content-Type:application/json\" -k -u (few other parameters required) -X GET https://example.org -s")

print output

I apologize for not providing the required parameters 'coz it's confidential.

edited Jul 10 '18 at 02:28

nikoskip

1,860
1
21
38

answered Mar 15 '18 at 16:41

apoorva_bhat

169
1
5

If you need to use some specials options from curl like --resolve this is the way. Thank you. – nikoskip Jul 10 '18 at 15:08
how can i only get the returned json without the tabular stat – Grant Gubatan Oct 12 '18 at 04:03

score 13 · Answer 6 · edited Sep 07 '22 at 07:59

I had this exact question because I had to do something to retrieve content, but all I had available was an old version of Python with inadequate SSL support. If you're on an older MacBook, you know what I'm talking about. In any case, curl runs fine from a shell (I suspect it has modern SSL support linked in) so sometimes you want to do this without using requests or urllib.request.

You can use the subprocess module to execute curl and get at the retrieved content:

import subprocess

# 'response' contains a []byte with the retrieved content.
# use '-s' to keep curl quiet while it does its job, but
# it's useful to omit that while you're still writing code
# so you know if curl is working
response = subprocess.check_output(['curl', '-s', baseURL % page_num])

Python 3's subprocess module also contains .run() with a number of useful options.

I having the same issues with the company MacBook with proxy security analysis. I'll write my wapper around CURL to process some requests. — Ângelo Polotto, Apr 13 '23 at 17:39

Jacob Mead · Answer 7 · 2021-03-29T12:04:53.147

4

I use os library.

import os

os.system("sh script.sh")

script.sh literally only contains the curl.

edited Mar 29 '21 at 12:04

answered Mar 28 '21 at 06:37

Jacob Mead

57
1
2

chainstair · Answer 8 · 2021-05-06T16:46:38.987

PYTHON 3

Only works within UNIX (Linux / Mac) (!)

Executing a cURL with Python 3 and parsing its JSON data.

import shlex
import json
import subprocess

# Make sure that cURL has Silent mode (--silent) activated
# otherwise we receive progress data inside err message later
cURL = r"""curl -X --silent POST http://www.test.testtestest/ -d 'username=test'"""

lCmd = shlex.split(cURL) # Splits cURL into an array

p = subprocess.Popen(lCmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
out, err = p.communicate() # Get the output and the err message

json_data = json.loads(out.decode("utf-8"))

print(json_data) # Display now the data

Sometimes you also need to install these dependencies on UNIX if you experience strange errors:

# Dependencies  
sudo apt install libcurl4-openssl-dev libssl-dev
sudo apt install curl

score 1 · Answer 9 · answered Dec 17 '21 at 13:36

if you os supporting curl you can do something like this:

import os

os.system("curl -d @request.json --header "Content-Type: application/json" https://www.googleapis.com/qpxExpress/v1/trips/search?key=mykeyhere")

I'm using this way... And I think you can use this too!

by the way.. the module "os" is auto-installing when you install python. soo, you don't need to install packages ;)

BARIS KURT · Answer 10 · 2022-09-13T13:43:37.020

1

use requests lib.. this code is :

curl -LH "Accept: text/x-bibliography; style=apa" https://doi.org/10.5438/0000-0C2G

equal to this:

import requests

headers = {
    'Accept': 'text/x-bibliography; style=apa',
}

r = requests.get('https://doi.org/10.5438/0000-0C2G', headers=headers)


print(r.text)

edited Sep 13 '22 at 13:43

answered Mar 31 '22 at 21:57

BARIS KURT

477
4
15

score -2 · Answer 11 · edited Aug 30 '22 at 06:50

-2

This is one approach:

Import os
import requests 
Data = os.execute(curl URL)
R= Data.json()

edited Aug 30 '22 at 06:50

Elisha Habinsky

131
10

answered May 09 '18 at 20:49

AliRazaRizwe

35
1

os.system instead of os.execute, and requests seems unnecessary in this case – SeanFromIT Jun 10 '18 at 02:02

How to use Python to execute a cURL command?

11 Answers11

Linked

Related