python regex: regex fails when reading from file

Question

My regex

table_names = ['check_channel_types', 'qualitycheckresult']
tables_group = '|'.join(table_names)
pattern = re.compile('(CREATE TABLE "({0})"(.*?);)'.format(tables_group), re.DOTALL)
match = pattern.findall(s)

works fine with this test-string:

s ="""CREATE TABLE "check_boundary_nodes" (
    "id" serial NOT NULL PRIMARY KEY,
    "test_name" varchar(120),
    "field_name" varchar(120),
    "original_pk" varchar(15),
    "check_result" varchar(255),
    "constraint" varchar(120),
    "the_geom" geometry(GEOMETRY,28992)
)
;
CREATE TABLE "check_channel_types" (
    "id" serial NOT NULL PRIMARY KEY,
    "original_pk" integer CHECK ("original_pk" >= 0) NOT NULL,
    "channel_inp_id" integer CHECK ("channel_inp_id" >= 0),
    "type" integer CHECK ("type" >= 0),
    "suggested_type" integer CHECK ("suggested_type" >= 0),
    "the_geom" geometry(LINESTRING,28992)
)
;
CREATE TABLE "qualitycheckresult" (
    "id" serial NOT NULL PRIMARY KEY,
    "qualitycheck" varchar(512) NOT NULL,
    "created" timestamp with time zone NOT NULL,
    "result" integer NOT NULL,
    "resultvalue" varchar(256) NOT NULL,
    "message" varchar(512) NOT NULL,
    "object_id" integer,
    "object_type" varchar(512) NOT NULL,
    "old_value" text NOT NULL
)
;"""

Once I read the text from a file-like object, the regular expression fails (does not find any matches). I assume it has to do with the quote characters but find it hard to debug as the string I'm reading from the 'file' is very long. What really feels strange about this is that internally it should not make a difference if it is a triple quoted string or not. Any help is highly appreciated. This is how I retrieve the data in my app:

from StringIO import StringIO
content = StringIO()
call_command('sql', 'my_app', database=self.alias,
             stdout=content)
content.seek(0)
a = content.getvalue()
type(a)
>>> <type 'str'>

it is very long. but on the command it looks just like the excerpt I posted above (without the triple quotes of course) — LarsVegas, Apr 03 '15 at 13:07
Have you tried putting the string you know works into a file and reading that in? — devinformatics, Apr 03 '15 at 13:14
Also, try using `cStringIO.StringIO` instead. What is your Python version, BTW? — Wiktor Stribiżew, Apr 03 '15 at 13:15
python 2.7. @devinformatics I will also put the string into a file and try reading it from there — LarsVegas, Apr 03 '15 at 13:19
@devinformatics, that does indeed work. Though I have no clue to as why. But going the extra way to write to and read from a file is no option, unfortunately... — LarsVegas, Apr 03 '15 at 13:27
If I put your `s` into a file, and read it with `f.read()`, it works for me... Are you sure `content` contains the same string? — Thom Wiggers, Apr 03 '15 at 13:50
@LarsVegas, I was not suggesting that option as a fix; rather as a way to try to narrow down where in the process things are breaking. — devinformatics, Apr 03 '15 at 13:54
How hard would it be to set up the data so the this command retrieves your known good string from the same source? Then we would know if it is the call or the string itself that is problematic. — devinformatics, Apr 03 '15 at 14:14
when I intercept a line given back by `StringIO` I get `\x1b[33mBEGIN;\x1b[0m\n'`. Some encoding problem than? — LarsVegas, Apr 03 '15 at 14:19
Can you intercept a line that has the CREATE TABLE command in it? — devinformatics, Apr 03 '15 at 14:34
I can but not before tuesday I am afraid, had to head home now...thanks for helping out by the way! — LarsVegas, Apr 03 '15 at 14:55

score 0 · Answer 1 · 2015-04-03T15:26:10.800

0

Try to write the output as utf8 encoding. And add the flags:

re.MULTILINE|re.DOTALL

To the regex.

f_stream = open( “my_dumpfile.txt”,     ‘w’, encoding=”utf-8″ )
call_command( “dumpdata”, indent=4, stdout=f_stream )

Or:

content = StringIO(content.read().decode('utf8'))

Or:

a = content.read().decode('utf8')

edited Apr 03 '15 at 15:26

answered Apr 03 '15 at 14:51

score 0 · Answer 2 · edited May 23 '17 at 12:22

0

Ok, let's wrap this up.

The reason the regex fails when reading from the StringIO object is that the strings contain ANSI escape sequences (apparently to give certain lines a different color). This answer shows how to remove the escape sequences using a regular expression. Then everything works just like expected.

edited May 23 '17 at 12:22

Community

1
1

answered Apr 07 '15 at 06:32

LarsVegas

6,522
10
43
67

python regex: regex fails when reading from file

2 Answers2