Python Nested List Comprehension with If Else

Question

I was trying to use a list comprehension to replace multiple possible string values in a list of values.

I have a list of column names which are taken from a cursor.description;

['UNIX_Time', 'col1_MCA', 'col2_MCA', 'col3_MCA', 'col1_MCB', 'col2_MCB', 'col3_MCB']

I then have header_replace;

{'MCB': 'SourceA', 'MCA': 'SourceB'}

I would like to replace the string values for header_replace.keys() found within the column names with the values.

I have had to use the following loop;

headers = []
for header in cursor.description:
    replaced = False
    for key in header_replace.keys():
        if key in header[0]:
            headers.append(str.replace(header[0], key, header_replace[key]))
            replaced = True
            break

    if not replaced:
        headers.append(header[0])

Which gives me the correct output;

['UNIX_Time', 'col1_SourceA', 'col2_SourceA', 'col3_SourceA', 'col1_SourceB', 'col2_SourceB', 'col3_SourceB']

I tried using this list comprehension;

[str.replace(i[0],k,header_replace[k]) if k in i[0] else i[0] for k in header_replace.keys() for i in cursor.description]

But it meant that items were duplicated for the unmatched keys and I would get;

['UNIX_Time', 'col1_MCA', 'col2_MCA', 'col3_MCA', 'col1_SourceA', 'col2_SourceA', 'col3_SourceA', 
'UNIX_Time', 'col1_SourceB', 'col2_SourceB', 'col3_SourceB', 'col1_MCB', 'col2_MCB', 'col3_MCB']

But if instead I use;

[str.replace(i[0],k,header_replace[k]) for k in header_replace.keys() for i in cursor.description if k in i[0]]

@Bakuriu fixed syntax

I would get the correct replacement but then loose any items that didn't need to have an string replacement.

['col1_SourceA', 'col2_SourceA', 'col3_SourceA', 'col1_SourceB', 'col2_SourceB', 'col3_SourceB']

Is there a pythonesque way of doing this or am I over stretching list comprehensions? I certainly find them hard to read.

I’d stick with the loops, and avoid the list comprehensions. My rule of thumb is that if you’re doing more than one thing in the comprehension, it’s probably best expanded into a loop with `.append`. — sneeu, Feb 18 '14 at 10:20
`cursor.description` holds a list of strings. `header[0]` holds a single character. Is that on purpose? — Bach, Feb 18 '14 at 10:21
@HansZauber yes sorry I will use them to write the headers of a csv file using `csv.writer()`, normally I just do `writer.writerow([i[0] for i in cursor.description])` — Dave Anderson, Feb 18 '14 at 10:21
Nested list comprehensions (or generator expressions) are indeed hard to get right and hard to read - well, not "rocket science hard", but confusing enough to make the code harder to understand than a plain for loop. TL;DR : list comprehensions are great for simple, straighforward cases, for loops are better for the complex cases. — bruno desthuilliers, Feb 18 '14 at 10:22
You do not need the `if k in i[0]` check at all. If the key is not in, then `replace` will just do nothing. — tobias_k, Feb 18 '14 at 10:24
@tobias_k but without some form of `if` you end up with a duplicate of each element as you iterate through the keys. — Dave Anderson, Feb 18 '14 at 10:26
@sneeu: or a generator that `yield`s the things that should end up in the list, even simpler than creating an empty list and appending to it — RemcoGerlich, Feb 18 '14 at 10:37

Bakuriu · Accepted Answer · 2014-02-18T10:33:56.053

[str.replace(i[0],k,header_replace[k]) if k in i[0] for k in header_replace.keys() for i in cursor.description]

this is a SyntaxError, because if expressions must contain the else part. You probably meant:

[i[0].replace(k, header_replace[k]) for k in header_replace for i in cursor.description if k in i[0]]

With the if at the end. However I must say that list-comprehension with nested loops aren't usually the way to go. I would use the expanded for loop. In fact I'd improve it removing the replaced flag:

headers = []
for header in cursor.description:
    for key, repl in header_replace.items():
        if key in header[0]:
            headers.append(header[0].replace(key, repl))
            break
    else:
        headers.append(header[0])

The else of the for loop is executed when no break is triggered during the iterations.

I don't understand why in your code you use str.replace(string, substring, replacement) instead of string.replace(substring, replacement). Strings have instance methods, so you them as such and not as if they were static methods of the class.

Thanks, didn't know about the `else` in a `for` loop that can be really useful. What is the purpose of the `_`? — Dave Anderson, Feb 18 '14 at 10:30
@DaveAnderson I use tuple unpacking to avoid using `header[0]` all the time. But I just changed it because that would work only if `cursor.description` was a pair, but since it can have any number of elements I decided to roll that back. The symbol `_` doesn't have any particular meaning. It's just often used to mean "a variable that I'm not going to use, but is required". This convention was probably taken from Haskell where it *is* a special pattern that means "any value matches". — Bakuriu, Feb 18 '14 at 10:34

score 1 · Answer 2 · answered Feb 18 '14 at 10:29

1

If your data is exactly as you described it, you don't need nested replacements and can boil it down to this line:

l = ['UNIX_Time', 'col1_MCA', 'col2_MCA', 'col3_MCA', 'col1_MCB', 'col2_MCB', 'col3_MCB']
[i.replace('_MC', '_Source')  for i in l]

>>> ['UNIX_Time',
>>>  'col1_SourceA',
>>>  'col2_SourceA',
>>>  'col3_SourceA',
>>>  'col1_SourceB',
>>>  'col2_SourceB',
>>>  'col3_SourceB']

answered Feb 18 '14 at 10:29

Michael

7,316
1
37
63

Unfortunately the common `MC` and `Source` isn't guaranteed it could be anything from `M00` through to `MFF` and the replacement values can be almost anything. – Dave Anderson Feb 18 '14 at 10:33
in that case I would advice to write specialized string replace function M_replace(source_string) – aisbaa Feb 18 '14 at 10:40
In that case I'd go with regular expressions instead of a huge dictionary of replacements. – Michael Feb 18 '14 at 10:40

score 0 · Answer 3 · answered Feb 18 '14 at 10:21

I guess a function will be more readable:

def repl(key):
    for k, v in header_replace.items():
        if k in key:
            return key.replace(k, v)
    return key

print map(repl, names)

Another (less readable) option:

import re
rx = '|'.join(header_replace)
print [re.sub(rx, lambda m: header_replace[m.group(0)], name) for name in names]

Python Nested List Comprehension with If Else

3 Answers3

Linked