LIst Comprehensions: References to the Components

Question

In sum: I need to write a List Comprehension in which i refer to list that is being created by the List Comprehension.

This might not be something you need to do every day, but i don't think it's unusual either.

Maybe there's no answer here--still, please don't tell me i ought to use a for loop. That might be correct, but it's not helpful. The reason is the problem domain: this line of code is part of an ETL module, so performance is relevant, and so is the need to avoid creating a temporary container--hence my wish to code this step in a L/C. If a for loop would work for me here, i would just code one.

In any event, i am unable to write this particular list comprehension. The reason: the expression i need to write has this form:

[ some_function(s) for s in raw_data if s not in this_list ]

In that pseudo-code, "this_list" refers to the list created by evaluating that list comprehension. And that's why i'm stuck--because this_list isn't built until my list comprehension is evaluated, and because this list isn't yet built by the time i need to refer to it, i don't know how to refer to it.

What i have considered so far (and which might be based on one or more false assumptions, though i don't know exactly where):

• doesn't the python interpreter have to give this list-under-construction a name? i think so

• that temporary name is probably taken from some bound method used to build my list ('sum'?)

• but even if i went to the trouble to find that bound method and assuming that it is indeed the temporary name used by the python interpreter to refer to the list while it is under construction, i am pretty sure you can't refer to bound methods directly; i'm not aware of such an explicit rule, but those methods (at least the few that i've actually looked at) are not valid python syntax. I'm guessing one reason why is so that we do not write them into our code.

so that's the chain of my so-called reasoning, and which has led me to conclude, or at least guess, that i have coded myself into a corner. Still i thought i ought to verify this with the Community before turning around and going a different direction.

Answer 1

I don't see why you need to do this in one go. Either iterate through the initial data first to eliminate duplicates - or, even better, convert it to a set as KennyTM suggests - then do your list comprehension.

Note that even if you could reference the "list under construction", your approach would still fail because s is not in the list anyway - the result of some_function(s) is.

Answer 2

There used to be a way to do this using the undocumented fact that while the list was being built its value was stored in a local variable named _[1].__self__. However that quit working in Python 2.7 (maybe earlier, I wasn't paying close attention).

You can do what you want in a single list comprehension if you set up an external data structure first. Since all your pseudo code seemed to be doing with this_list was checking it to see if each s was already in it -- i.e. a membership test -- I've changed it into a set named seen as an optimization (checking for membership in a list can be very slow if the list is large). Here's what I mean:

raw_data = [c for c in 'abcdaebfc']

seen = set()
def some_function(s):
    seen.add(s)
    return s

print [ some_function(s) for s in raw_data if s not in seen ]
# ['a', 'b', 'c', 'd', 'e', 'f']

If you don't have access to some_function, you could put a call to it in your own wrapper function that added its return value to the seen set before returning it.

Even though it wouldn't be a list comprehension, I'd encapsulate the whole thing in a function to make reuse easier:

def some_function(s):
    # do something with or to 's'...
    return s

def add_unique(function, data):
    result = []
    seen = set(result) # init to empty set
    for s in data:
        if s not in seen:
            t = function(s)
            result.append(t)
            seen.add(t)
    return result

print add_unique(some_function, raw_data)
# ['a', 'b', 'c', 'd', 'e', 'f']

In either case, I find it odd that the list being built in your pseudo code that you want to reference isn't comprised of a subset of raw_data values, but rather the result of calling some_function on each of them -- i.e. transformed data -- which naturally makes one wonder what some_function does such that its return value might match an existing raw_data item's value.

Answer 3

As far as I know, there is no way to access a list comprehension as it's being built.

As KennyTM mentioned (and if the order of the entries is not relevant), then you can use a set instead. If you're on Python 2.7/3.1 and above, you even get set comprehensions:

{ some_function(s) for s in raw_data }

Otherwise, a for loop isn't that bad either (although it will scale terribly)

l = []
for s in raw_data:
    item = somefunction(s)
    if item not in l:
        l.append(item)

Answer 4

Why don't you simply do:[ some_function(s) for s in set(raw_data) ]

That should do what you are asking for. Except when you need to preserve the order of the previous list.