Python: Split a string, respect and preserve quotes

Question

Using python, I want to split the following string:

a=foo, b=bar, c="foo, bar", d=false, e="false"

This should result in the following list:

['a=foo', 'b=bar', 'c="foo, bar"', 'd=false', 'e="false'"']

When using shlex in posix-mode and splitting with ", ", the argument for cgets treated correctly. However, it removes the quotes. I need them because false is not the same as "false", for instance.

My code so far:

import shlex

mystring = 'a=foo, b=bar, c="foo, bar", d=false, e="false"'

splitter = shlex.shlex(mystring, posix=True)
splitter.whitespace += ','
splitter.whitespace_split = True
print list(splitter) # ['a=foo', 'b=bar', 'c=foo, bar', 'd=false', 'e=false']

I believe this is a duplicate of http://stackoverflow.com/a/79985/1540600, which provides a superior answer. (Use `shlex.split()`.) — sferencik, Dec 02 '15 at 12:28
this is not a duplicate, as `shlex.split` can't specify delimitor. — RNA, Feb 22 '16 at 20:27
I agree with RNA. For using a custom delimiter see https://stackoverflow.com/questions/56791723/split-a-string-with-custom-delimiter-respect-and-preserve-quotes-single-or-dou/ — luca, Jun 29 '19 at 11:41

Janne Karila · Accepted Answer · 2016-10-04T09:31:49.127

27

>>> s = r'a=foo, b=bar, c="foo, bar", d=false, e="false", f="foo\", bar"'
>>> re.findall(r'(?:[^\s,"]|"(?:\\.|[^"])*")+', s)
['a=foo', 'b=bar', 'c="foo, bar"', 'd=false', 'e="false"', 'f="foo\\", bar"']

The regex pattern "[^"]*" matches a simple quoted string.
"(?:\\.|[^"])*" matches a quoted string and skips over escaped quotes because \\. consumes two characters: a backslash and any character.
[^\s,"] matches a non-delimiter.
Combining patterns 2 and 3 inside (?: | )+ matches a sequence of non-delimiters and quoted strings, which is the desired result.

edited Oct 04 '16 at 09:31

answered May 23 '13 at 09:55

Janne Karila

24,266
6
53
94

4

explain how it works and I'll upvote – Jason S Oct 06 '15 at 19:38
1

@JasonS I agree, but [regex101.com](https://regex101.com/) has a useful explanation section that may help, e.g.: https://i.imgur.com/SJMAQPI.png – blah238 Oct 03 '16 at 22:08
1

that's useful, but that's not the point, the point is that a good answer explains why it works. – Jason S Oct 04 '16 at 02:14
2

@JasonS Added some explanation. – Janne Karila Oct 04 '16 at 12:57
@JanneKarila But this is ignoring the null strings too. How can I retain those? – Muhammad Zeeshan Jun 05 '17 at 03:39
Strictly, this answers the question, but beware that escaping the quote character does not work perfectly for the above regex. An input like `a=\"foo, b="bar"` would not be parsed correctly. – Zorawar Jan 06 '19 at 21:13
@Zorawar Good remark. The encoder I imagined would escape the quote when it adds the quotes around, so that input would be `a="\"foo", b="bar"` – Janne Karila Jan 08 '19 at 06:29

Python: Split a string, respect and preserve quotes

1 Answers1

Linked