python subprocess newlines are escaped

Question

I'm running an mysql query in the command line using subprocess.Popen

process = subprocess.Popen(conarray, shell=False, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)

r = process.stdout.readlines()

stdout = [x.decode('utf8').rstrip() for x in r]

later I write the output in a file

f = open(file_name, 'w+', encoding='utf8')
f.write(templates[idx])

It works fine but for some reason all newlines(\n) and tabs(\t) are escaped.

\t<div id="container">\n\t\t<a name="top"

Any idea on how I can fix that ?

the value is already escaped at that point, so i'm guessing it has to be something from the query `[mysql, '--default-character-set=utf8', '-u', user, '-h', host, dbname, "-e %s" % query]` — Mihai Vilcu, Aug 09 '14 at 11:50
it's basically stdout the full code is here https://github.com/ionutvmi/SublimeMybbTplEditor/blob/master/mybb-tpl.py#L97 — Mihai Vilcu, Aug 09 '14 at 11:56
with .decode('unicode_escape') it seems to parse the new lines correctly but it doesn't convert chars like `ńńńńń` — Mihai Vilcu, Aug 09 '14 at 12:01
I meant print each lines, not the whole one: `for line in r: print(line)` — falsetru, Aug 09 '14 at 12:09
it prints something like this `b'\\t\\t\\t\\n\\t\\t\\t\\t{$lang->active_warnings}\\n\\t\\t\\t\r\n'` — Mihai Vilcu, Aug 09 '14 at 12:17
It seems like the escaped string is from the table you are quering on. You'd better check the part that insert that data into the table. — falsetru, Aug 09 '14 at 12:21
i already checked the data is correct there, I'm guessing they are escaped like that when I run subprocess.Popen() since the query result is one per line, to revert that I was doing `.decode('unicode_escape')` — Mihai Vilcu, Aug 09 '14 at 12:26
Do you mean if you run the command in command line (not through Python subprocess), does it print non-escaped version? — falsetru, Aug 09 '14 at 12:27
yes in the command line they are showed properly http://puu.sh/aL9nN/cba8d374b7.png — Mihai Vilcu, Aug 09 '14 at 12:30
Please run the following program: http://ideone.com/ueiCzE (once with the given command, and once with your mysql command). and let me know what is printed. — falsetru, Aug 09 '14 at 12:36

score 1 · Answer 1 · answered Aug 09 '14 at 12:15

If you print the whole sequence object, repr(list_object) is printed. That's the way Python represent it.

>>> lst = ['\t<div id="container">\n\t\t<a name="top"']
>>> print(lst)
['\t<div id="container">\n\t\t<a name="top"']
>>> print(lst[0])
        <div id="container">
                <a name="top"
>>>

score 1 · Answer 2 · edited May 23 '17 at 12:31

1

It turns out that in order to make them work I also needed to apply unicode-escape on that string.
But unicode-escape does NOT work in general.

>>> s = 'naïve \\t test'
>>> print(s.encode('utf-8').decode('unicode_escape'))
naÃ¯ve   test

The best solution I could find is described in this answer

edited May 23 '17 at 12:31

Community

1
1

answered Aug 10 '14 at 06:16

Mihai Vilcu

1,947
18
24

Your string appears to have actual backslashes in it, rather than special characters merely represented with backslashes in the escape sequences. It probably went through an extra layer of escaping at some point. You'll need to figure out why that's happening and fix it. – user2357112 Aug 10 '14 at 06:23
here is the code to try it out on your computer https://gist.github.com/ionutvmi/a63e84c0402db7a266fd – Mihai Vilcu Aug 10 '14 at 06:49

python subprocess newlines are escaped

2 Answers2