Passing Python JSON argument with whitespace

Question

I am trying to pass JSON as an argument to python script in command prompt. It works if the element in JSON does not have white spaces in value but it does not work if there are white spaces.

Here is script

import json, sys, traceback
    if(len(sys.argv)>1):
    print(sys.argv[1])
    jsonInput = json.loads(sys.argv[1]);
    print(jsonInput['name'])
    print(jsonInput['kingdom'])
    print(jsonInput['slogan'])

Passing below JSON as an argument in the power Shell. I have spaces in values e.g. Jon Snow

python C:\Users\user1\Desktop\myTest.py '{\"name\":\"Jon Snow\",\"kingdom\":\"Winterfell\",\"slogan\":\"King in the North\"}'

Output:

python : Traceback (most recent call last):
At line:1 char:1
+ python C:\Users\kiran.patil\Desktop\myTest.py '{\"name\":\"Jon Snow\" ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (Traceback (most recent call last)::String) [], RemoteException
    + FullyQualifiedErrorId : NativeCommandError
 
  File "C:\Users\User\Desktop\myTest.py", line 5, in <module>
    jsonInput = json.loads(sys.argv[1]);
  File "C:\Program Files\Python38\lib\json\__init__.py", line 357, in loads
    return _default_decoder.decode(s)
  File "C:\Program Files\Python38\lib\json\decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "C:\Program Files\Python38\lib\json\decoder.py", line 353, in raw_decode
    obj, end = self.scan_once(s, idx)
json.decoder.JSONDecodeError: Unterminated string starting at: line 1 column 9 (char 8)

Please any advise to fix it.

The bottom line: your command works as-is in PowerShell [Core] v6+. In Windows PowerShell, you can make it work by inserting at least one _space character_ in the string; alternatively, use `""` instead of `\"`, as in the accepted answer. Note, however, that _neither_ form of escaping should be necessary at all, and is only required due to a longstanding PowerShell bug in argument-passing to external programs. — mklement0, Oct 20 '20 at 19:43
Re-reading the question, I see that you make the _opposite_ claim: that it works _without_ whitespace - but that contradicts your actual command (and my analysis). Is this a remnant from an earlier version of your question? Perhaps you can edit the question to avoid confusion. — mklement0, Oct 20 '20 at 20:11

azro · Accepted Answer · 2020-10-20T17:40:18.023

3

If you use \" it means the enclosing quotes are double quotes too. On windows the enclosing simple quotes doesn't work so the 2 options are

# Linux + Windows
python script.py "{\"name\":\"Jon Snow\",\"kingdom\":\"Winterfell\",\"slogan\":\"King in theNorth\"}"

# Linux only
python script.py '{"name":"Jon Snow","kingdom":"Winterfell","slogan":"King in theNorth"}'

Powershell case

python script.py '{""name"":""Jon Snow"",""kingdom"":""Winterfell"",""slogan"":""King in theNorth""}'

edited Oct 20 '20 at 17:40

answered Oct 20 '20 at 16:56

azro

53,056
7
34
70

Good info re syntax for shells _other_ than PowerShell, but note that it's not about Linux (more generally, Unix-like platforms) vs. Windows per se, especially now that PowerShell is now cross-platform. Your first command applies on Windows only to `cmd.exe`. – mklement0 Oct 20 '20 at 19:38
Re PowerShell: Your `""` approach works and is the best choice, but only because the generally equivalent `\"`-escaping - in the context of passing arguments to _external programs_ only - is buggy in Windows PowerShell: the addition of a _single space character_ would have made the OP's command work (and in PowerShell v6+ it works as-is). The larger point worth stressing however: that PowerShell requires escaping at all in this case is a blatant, unfortunately very longstanding bug. – mklement0 Oct 20 '20 at 19:39
@snowcoder: As a simple workaround you can simply _append a space char._ to the `\"`-escaped JSON argument you receive - this will have no ill effects on Python's JSON parsing while avoiding the Windows PowerShell bug. – mklement0 Oct 20 '20 at 20:14

mklement0 · Answer 2 · 2020-10-20T19:51:05.733

In PowerShell:

There is de facto nothing fundamentally wrong with your attempt to escape the embedded " characters as \" for a call to an external program (such as Python), despite their being enclosed in single-quoted string ('...'):
- Due to an additional bug in Windows PowerShell (in addition to the fundamental bug discussed below) that has since been fixed in PowerShell [Core] v6+, a string with \" escapes only works if it contains at least one space character.
- In other words: In PowerShell [Core] v6+, your command would work as-is; in Windows PowerShell, it would only work if your string contained at least one space character.
- Since in this particular context (only!) \" and "" are interchangeable and use of "" also works with space-less strings in Windows PowerShell, ""-escaping, as shown in azro's helpful answer, is the more robust choice; the caveat is that most, but not all CLIs on Windows recognize "" as an escaped " char., whereas those that don't do recognize \".

# OK even in Windows PowerShell, due to the string containing spaces.
# "" instead of \" works too and avoids the need for spaces in Windows PowerShell
#  - but neither should be necessary (see below).
python myTest.py '{\"name\": \"Jon Snow\", \"kingdom\": \"Winterfell\", \"slogan\": \"King in the North\"}'

There should be something wrong, however, because you shouldn't have to escape " chars. inside a '...'-delimited string literal, which is a verbatim string literal in PowerShell.
- In fact, you don't need to do it if you call PowerShell-native commands; e.g.:
  ConvertFrom-Json '{ "foo": "bar" }' works just fine.
- Due to a longstanding bug, you do have to do it when calling external programs, unfortunately - something that hasn't been fixed yet so as not to break backward compatibility - see this answer for background information and a possible future fix.

Passing Python JSON argument with whitespace

2 Answers2

Related