Nesting loops to arbitrary depth by passing method objects

Question

I am trying to scan over iterable properties of n objects. I am looking for a pythonic way to perform functions in nested loops of arbitrary depth by passing functions to method calls of the loop one level up. I haven't been able to get more than the most inner loop to run when the depth is 3. Here is a non-working python pseudo code where I am querying a different value at each point in the loops. The other difficulty is I am trying to capture the output and pass it to the next outer loop

class Parent(object):
    def __init__(self):
        self.iterable = [None] * 2
        self.result = self.iterable[:]
    def loop(self, query_func):
        def innerloop():
            for i, x in enumerate(self.iterable):
                self.result[i] = query_func(x)
            return self.result[:]
        return innerloop
class ChildA(Parent):
    def __init___(self, A, object_to_queryA):
        self.iterableA = [valueA for valueA in range(A)]
        self.resultA = self.iterableA[:]
        self.object_to_query = object_to_queryA
    def query_valueA(self, x):
        return self.object_to_query.some_query_function(x)
class ChildB(Parent):
    def __init___(self, B, object_to_queryB):
        self.iterableB = [valueB for valueB in range(B))]
        self.resultB = self.iterableB[:]
        self.object_to_query = object_to_queryB
    def query_valueB(self, x):
        return self.object_to_query.some_other_query_function(x)
class ChildC(Parent):
    def __init___(self, C, , object_to_queryC):
        self.iterableC = [valueC for valueC in range(C))]
        self.resultC = self.iterableC[:]
        self.object_to_query = object_to_queryC
    def query_valueC(self, x):
        return self.object_to_query.yet_another_query_function(x)

I want to be able to call these loops as follows:

import numpy
query_objA, query_objB, query_objC = (SomeObjA(), SomeObjB(), SomeObjC())
A, B, C = (len(query_objA.data), len(query_objB.data), len(query_objC.data))
instA = ChildA(A, query_objA)
instB = ChildB(B, query_objB)
instC = ChildC(C, query_objC)
my_scanning_func = ChildA.loop(ChildB.loop(ChildC.loop))
my_queries = numpy.array(my_scanning_func()).reshape(A,B,C)
# Equally valid call example below:
my_scanning_func2 = ChildB.loop(ChildC.loop(ChildA.loop))
my_queries2 = numpy.array(my_scanning_func2()).reshape(B,C,A)

The ultimate functionality im looking for would be similar to below, but for arbitrary depth and order:

for i, x in enumerate(query_objA.data):
    response[i] = instA.some_query_function(x)
    for j, y in enumerate(query_objB.data):
        response[i][j] = instB.some_other_query_function(y)
        for k, z in enumerate(query_objC.data):
            response[i][j][k] = instC.yet_another_query_function(z)

Bonus points if this can be done via an inherited recursive function, rather than defining separate looping methods for each child, as I tried to do above. Last Note: I am trying to write Python 2.7 compatible code. Thanks in advance!

Answer 1

I'm not sure if this answers your question but I think it is at least relevant, if you want to generate a numpy array such that array[tup] = func(tup) where tup is a tuple of integer indices you could use itertools.product in combination with numpy.fromiter like this:

import itertools
#from itertools import imap as map #for python 2
import numpy

def array_from_func(dimensions, func, dtype=float):
    ranges = (range(i) for i in dimensions) #ranges of indices for all dimensions
    all_indices = itertools.product(*ranges) #will iterate over all locations regardless of # of dimensions
    value_gen = map(func, all_indices) #produces each value for each location
    array = numpy.fromiter(value_gen, dtype=dtype)
    array.shape = dimensions #modify the shape in place, .reshape would work but makes a copy.
    return array

This is useful to me to see how indices relate to the actual array output, here are three demos to show basic functionality (second one I figured out recently)

from operator import itemgetter
>>> array_from_func((2,3,4), itemgetter(1),int) #second index
array([[[0, 0, 0, 0],
        [1, 1, 1, 1],
        [2, 2, 2, 2]],

       [[0, 0, 0, 0],
        [1, 1, 1, 1],
        [2, 2, 2, 2]]])

>>> def str_join(it):
        return ",".join(map(str,it))
#the '<U5' in next line specifies strings of length 5, this only works when the string will actually be length 5
#changing to '<U%d'%len(str_join(dims)) would be more generalized but harder to understand
>>> print(array_from_func((3,2,7), str_join, '<U5')) 
[[['0,0,0' '0,0,1' '0,0,2' '0,0,3' '0,0,4' '0,0,5' '0,0,6']
  ['0,1,0' '0,1,1' '0,1,2' '0,1,3' '0,1,4' '0,1,5' '0,1,6']]

 [['1,0,0' '1,0,1' '1,0,2' '1,0,3' '1,0,4' '1,0,5' '1,0,6']
  ['1,1,0' '1,1,1' '1,1,2' '1,1,3' '1,1,4' '1,1,5' '1,1,6']]

 [['2,0,0' '2,0,1' '2,0,2' '2,0,3' '2,0,4' '2,0,5' '2,0,6']
  ['2,1,0' '2,1,1' '2,1,2' '2,1,3' '2,1,4' '2,1,5' '2,1,6']]]

>>> array_from_func((3,4), sum) #the sum of the indices, not as useful but another good demo
array([[ 0.,  1.,  2.,  3.],
       [ 1.,  2.,  3.,  4.],
       [ 2.,  3.,  4.,  5.]])

I think this is along the lines of what you are trying to accomplish but I'm not quite sure... please give me feedback if I can be more specific about what you need.

Answer 2

After much discussion with the OP I have a better idea of how you could generalize the construction of these arrays, first it seems that your objects would be designed to both iterate over predefined states or query the present state (possibly with only one of these being valid) so the iterface for object would be abstracted to something like this:

class Apparatus_interface:
    def __init__(self,*needed_stuff):
        #I have no idea how you are actually interacting with the device
        self._device = SET_UP_OBJECT(needed_stuff)
        
        #when iterating over this object we need to know how many states there are
        #so we can predefine the shape (dimensions) of our arrays
        self.num_of_states = 5
        
        #it would make sense for each object to define
        #the type of value that .query() returns (following spec of numpy's dtype)
        self.query_type = [('f1', float), ('f2', float)]

    def __iter__(self):
        """iterates over the physical positions/states of the apperatus
        the state of the device is only active in between iterations

        * calling list(device) doesn't give you any useful information, just a lot of mechanical work
        """
        for position in range(self.num_of_states):
                         # ^ not sure what this should be either, you will have a better idea
            self._device.move_to(position) #represents a physical change in the device
            yield position #should it generate different information?

    def query(self):
        return self._device.query()

with this interface you would generate your array by iterating (nested loop) over a number of devices and at each combination of states between them you query the state of another device (and record that value into an array)

---

Normally you'd be able to use itertools.product to generate the combinations of states of the devices however due to optimizations itertools.product would run the iteration code that affects the physical device before it is used in iteration, so you will need an implementation that does not apply this kind of optimization:

#values is a list that contains the current elements generated
#the loop: for values[depth] in iterables[depth] basically sets the depth-th element to each value in that level of iterable
def _product(iterables, depth, values):
    if len(iterables)-depth == 1:
        for values[depth] in iterables[depth]:
            yield tuple(values)
    else:
        for values[depth] in iterables[depth]:
            #yield from _product(iterables, depth+1, values)
            for tup in _product(iterables, depth+1, values):
                yield tup

def product(*iterables):
    """
    version of itertools.product to activate side-effects of iteration
    only works with iterables, not iterators.
    """
    values = [None]*len(iterables)
    return _product(iterables, 0, values)
    

---

Now for actually generating the array - first a process that iterates through the product of all states and makes a query at each one, note that states variable is unused as I'm going to assume the placement in the numpy array will be determined by the order the states get iterated not the values produced

def traverse_states(variable_devices, queried_device):
    """queries a device at every combination of variable devices states"""
    for states in product(*variable_devices):
        yield queried_device.query()

then the function to put the array together is quite strait forward:

def array_from_apparatus(variable_devices, queried_object, dtype=None):
    
    # the # of states in each device <==> # of elements in each dimension
    arr_shape = [device.num_of_states for device in variable_devices]

    iterator = traverse_states(variable_devices, queried_object)
    if dtype is None:
        dtype = queried_object.query_type

    array = numpy.fromiter(iterator, dtype=dtype)
    array.shape = arr_shape #this will fail if .num_of_states doesn't match the actual number of iterations
    return array

I'm not sure how I could make a decent test of this but I believe it would work or at least be close.