Curl is generating status 400 in subprocess

Question

I run a curl command on windows command line prompt. It produces a json output. The command looks like this:

curl --data "action=details&user=user&project=project1&problemid=2021" https://website:9020/

I issue the same command in python as following:

import subprocess
output = subprocess.run(
                [
                    "curl",
                    "--data",
                    "\"action=details&user=user&project=project1&problemid=2021\""
                    "https://website:9020/",
                ],
                stdout=subprocess.PIPE,
                stderr=subprocess.STDOUT,
                shell=True,
            )
print(output.stdout.decode("utf-8"))

The output is the following:

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   199  100    64  100   123     64    123  0:00:01 --:--:--  0:00:01  1254
{"status":400,"message":"Action parameter is missing"}

But the command line produces a json output. Yet the same command issued through subprocess.run produces this error. I also tried it with subprocess.Popen and subprocess.check_output. Same issue persists. What am I doing wrong here that is causing this error?

schilli · Answer 1 · 2021-07-07T20:13:38.310

1

Did you try running your code without the quotation marks around the action parameter? Because the API is complaining about missing action parameter. I think it's just not recognising it due to the escaped quotation marks.

import subprocess
output = subprocess.run(
    [
        "curl",
        "--data",
        "action=details&user=user&project=project1&problemid=2021"
        "https://website:9020/",
    ],
    stdout=subprocess.PIPE,
    stderr=subprocess.STDOUT,
    shell=True,
)
print(output.stdout.decode("utf-8"))

EDIT: I'm not sure if this is the case, but it could be that subprocess implicitly surrounds all its parameters with quotation marks in order to avoid code injection and globbing.

I was just googling again and found this answer, which states it quite well: why would you use curl, if you could use requests?

No guarantee, but I think something like this should work:

import requests
url = "https://website:9020/"
payload = {
    "action": "details",
    "user": "user",
    "project": "project1",
    "problemid": "2021"
}
res = requests.get(url, params=payload)

edited Jul 07 '21 at 20:13

answered Jul 07 '21 at 19:48

schilli

1,700
1
9
17

No, it's the opposite. When you run the command in the shell, you need to quote the data to protect it from modification by the shell; but here you have no shell, and so the quotes are not only unnecessary, but wrong. – tripleee Jul 07 '21 at 19:59
Yeah, it does not work without the quotation marks. If I run without them, it says "action" is not recognized as internal command. Curl does not automatically wrap the parameters with quotation marks. – Vance Pyton Jul 07 '21 at 19:59
That's explained in more detail in [When to wrap quotes around a shell variable?](https://stackoverflow.com/questions/10067266/when-to-wrap-quotes-around-a-shell-variable) but again, this answer is correct, just not the speculation in the edit. I guess "implicitly" makes it kind of right, but not really. – tripleee Jul 07 '21 at 20:01
Wait, the `shell=True` is horribly wrong here. – tripleee Jul 07 '21 at 20:06
@tripleee should it be false? or that option should not exist at all? I tried with both but error persists. – Vance Pyton Jul 07 '21 at 20:07
Omitting it defaults to `False` so both are fine, but usually just leave it out. – tripleee Jul 07 '21 at 20:08
I printed `output` it gave me this: `CompletedProcess(args=['curl', '--data', '"action=details&user=user&project=project1&problemid=2021"', 'https://website:9020/'], returncode=0, stdout=b'` – Vance Pyton Jul 07 '21 at 20:11
@VancePyton Did you check my latest edit of the answer? Try using the requests package instead of calling curl with subprocess. That is the more pythonic and clean solution. – schilli Jul 07 '21 at 21:39

score 1 · Accepted Answer · answered Jul 07 '21 at 20:30

1

In some cases (e.g. if they require running command line as administrator), subprocess might not be the right choice to execute the command line commands.

You can use os.system() to see the output or os.popen() to read and store the output.

import os
import json

# to see the output 
print(os.system("curl --data \"action=details&user=user&project=project1&problemid=2021\" https://website:9020/")

output = os.popen("curl --data \"action=details&user=user&project=project1&problemid=2021\" https://website:9020/").read()

outputjson = json.loads(output)

then you can access the json information.

answered Jul 07 '21 at 20:30

Gravity Mass

605
7
13

You'll notice that the `os.system` documentation recommends that you use `subprocess` instead, and that `os.popen` is just a thin and basically redundant wrapper around `subprocess`. – tripleee Jul 08 '21 at 04:08
But it did not work for me. `os.popen` did. – Vance Pyton Jul 08 '21 at 16:24

Curl is generating status 400 in subprocess

2 Answers2