A pythonic way of packing loops into a function to mute variables?

Question

I'm not sure how verbose I should go so please ask for elaboration if this is too terse.

Is there a way to pack the for a,b,c in product(d['a'],d['b'],d['c']): in some syntactical sugar so I would only need to type mute variables a,b,c only once for this loop itself?

Something like this may be?

my_for_loop('a','b','c'): 
  API_Call1(a)
  API_Call2(b,c)

instead of

for a,b,c in product(d['a'],d['b'],d['c']):
  API_Call1(a)
  API_Call2(b,c)

How could my_for_loop look like? I am a bit lost at how to approach this conceptually.

More detail:

I have an API that requires calling it for each cell of a cartesian product of some lists. I am using the product function to avoid nested loops. Let's say we have a list1 and a list2 it can be done in a following way

from itertools import product
for a,b in product(list1,list2):
  API_Call(a,b)

I have created a dictionary_of_lists={'a':list1,'b':list2,'c':list3...} to be able to write it like this

for a,b in product(dictionary_of_lists['a'],dictionary_of_lists['b']):
  API_Call(a,b)

for c,b in product(dictionary_of_lists['c'],dictionary_of_lists['b']):
  API_Call(c,b)

for e,f,g,h in product(dictionary_of_lists['e'],dictionary_of_lists['f'],dictionary_of_lists['g'],dictionary_of_lists['h'],):
  API_Call1(e,f,g,h)
  API_Call2(e,h)
...

So basically the variables that the loop creates are used in that API calls and they are mute otherwise, their name doesn't matter. There are many of these calls and there is some convoluted logic around them. So I would like to keep the loop itself simple and should I need to change the variables I won't have to to change them at three places for each such loop.

my_for_loop('a','b'):
  API_Call(a,b)

my_for_loop('b','c'):
  API_Call(c,b)

my_for_loop('e','f','g','h'):
  API_Call1(e,f,g,h)
  API_Call2(e,h)
...

ADDITION: I have simplified a few things but was taken by surprise where exactly ambiguity was lurking :-)

Thanks for all the answers so far!

It's a good suggestion to have the dproduct wrapper. I have one indeed, just did not want to preempt your suggestions.

The variable names are mute for the code logic but they have some meaning for the sake of maintenance of the code. So they can not consist of a single letter each.

In an attempt to clarify further: I would like to avoid using the variable names three times - in the "for ..." part, in the call to dproduct wrapper and in the API calls. Two times - in the call to the wrapper and in the API calls is OK because it reflects the logic.

Below is a more elaborated example of the code I have now.

def dproduct(d, keys):
  subset_d = dict((k, d[k]) for k in keys if k in d)
  return product(*[subset_d.values()])

for foo, bar in dproduct(d, ['foo','bar',]):
  some logic here
    if API_Call1(foo,bar) == 123: 
       some other stuff, API_Call6(quux,baz,)         
  some more stuff and a call to another dproduct
  for quux, sl, baz in dproduct(d, ['quux','sl','baz',]):
    blah blah, API_Call2(quux,sl,baz) 
    other stuff

 for pa, mf in dproduct(d, ['pa','mf',]):
  API_Call4(pa,mf)

 for quux, sl, baz in dproduct(d, ['quux','sl','baz',]):
  further logic
    if API_Call1(quux, sl, baz) == 342: some other stuff
  some more stuff and a call to another dproduct
  for pa,mf in dproduct(d, ['pa','mf',]):
    API_Call3(pa,mf)

Answer 1

First, you can write a product wrapper like this:

def dproduct(d, keys):
    return product(*(d[key] for key in keys))

for a, b, c in dproduct(d, 'abc'):
    API_Call1(a)
    API_Call2(b, c)

(Notice that you can write the same thing with operator.itemgetter instead of a genexpr; it depends on which you find more readable.)

Of course I'm taking advantage of the fact that all of your keys have single-character names, and a string is an iterable of its characters. If your real names aren't single-character, you have to be slightly more verbose:

for a, b, c in dproduct(d, ('spam', 'eggs', 'beans')):

… at which point you might want to consider taking *args instead of args in the parameters—or, if you don't mind being hacky:

def dproduct(d, keys):
    return product(*(d[key] for key in keys.split()))

for a, b, c in dproduct(d, 'spam eggs beans'):

---

You can then go farther and write wrappers around your calls, too. For example, if you yield dicts of values instead of tuples of values, you can use those dicts the same way we used the original dict of lists:

def dproduct(d, keys):
    for vals in product(*(d[key] for key in keys)):
        yield dict(zip(keys, vals))
def call1(d, keys):
    return API_Call1(*(d[key] for key in keys))
def call2(d, keys):
    return API_Call2(*(d[key] for key in keys))

for vals in dproduct(d, 'abc'):
    call1(vals, 'a')
    call2(vals, 'bc')

---

Or, if your API_Call* functions can take keyword instead of positional arguments, you can make things a lot simpler and cleaner:

def dproduct(d, keys):
    for vals in product(*(d[key] for key in keys)):
        yield dict(zip(keys, vals))
def dselect(d, keys):
    return {key: d[key] for key in keys}
for vals in dproduct(d, 'abc'):
    API_Call1(**dselect(vals, 'ab'))
    API_Call2(**dselect(vals, 'c'))

---

If you can't use keyword arguments, and you have a lot of those API_Call* functions, you can go farther and generate the wrappers dynamically:

def apiwrap(apicall):
    @functools.wraps(apicall)
    def wrapper(d, keys):
        return apicall(*(d[key] for key in keys))
    return wrapper
apicall1 = apiwrap(API_Call1)
apicall2 = apiwrap(API_Call2)
# etc.

Although if you have lots of these, you probably want to stick them in a list or a dict in the first place…

---

If you want to get way too clever, you can even split the tuples up dynamically based on the signatures of the API functions:

def dispatch(d, keys, *calls):
    for vals in product(*(d[key] for key in keys)):
        it = iter(vals)
        for call in calls:
            args = islice(it, len(signature(call).parameters))
            call(*args)
dispatch(d, 'abc', API_Call1, API_Call2)

(If your function bodies are pretty minimal, you probably want to speed things up by doing argcounts = [len(signature(call).parameters for call in calls] at the top of the function and then using zip(calls, argcounts) rather than using inspect each time in the inner loop.)

---

Anyway, without knowing more about your program, it's hard to say exactly what you can do, and what you should do—most of these ideas are not very Pythonic in general, even if they might be useful in some particular unusual case.

But either way, they should serve as examples of the kinds of things you can do pretty easily without having to get into horrible hacks involving locals and globals or getframe.

Answer 2

You can't easily insert variables into the local namespace (probably can but shouldn't without good cause). So use a dictionary to hold your named values.

I've used the operator.itemgetter function to grab the ones you want for the various API calls, and wrapped your product function in a generator.

from operator import itemgetter
from itertools import product

class DictCaller(dict):
    def __call__(self, fn, *args):
        fn(*map(itemgetter(self), args))

def my_product(d, *args):
    for xs in product(*map(itemgetter(d), args)):
        yield DictCaller(zip(args, xs))

for caller in my_product(*'abc'):
    caller(API_CALL, *'ab')
    caller(API_CALL1, *'bc')