Python iterate over each 100 elements

Question

I don't know if this is a good way to optimize, but basically I am using python inside a 3D app to create random colors per object. And the code I have works well with objects within 10k polygons. But it crashes in 100k polygons. Is there a way to do it by chunks in the loop, basically I have the for loop and using an if statement to filter the first 100. But then I need another 100, and another 100, etc. How can I write that? Maybe with a time sleep between each. It's not going to be faster but at least won't possible crash the program. Thanks.

for i, n in enumerate(uvShellIds):


#code can only perform well within sets of 100 elements


limit = 100 #?

if 0 <= i <= 100:

    #do something
    print(n)

# now I need it to work on a new set of 100 elements
#if 101 <= i <= 200:

#(...keep going between sets of 100...)

My current code :

import maya.OpenMaya as om
import maya.cmds as cmds
import random
 
def getUvShelList(name):
    selList = om.MSelectionList()
    selList.add(name)
    selListIter = om.MItSelectionList(selList, om.MFn.kMesh)
    pathToShape = om.MDagPath()
    selListIter.getDagPath(pathToShape)
    meshNode = pathToShape.fullPathName()
    uvSets = cmds.polyUVSet(meshNode, query=True, allUVSets =True)
    allSets = []
    for uvset in uvSets:
        shapeFn = om.MFnMesh(pathToShape)
        shells = om.MScriptUtil()
        shells.createFromInt(0)
        # shellsPtr = shells.asUintPtr()
        nbUvShells = shells.asUintPtr()
 
        uArray = om.MFloatArray()   #array for U coords
        vArray = om.MFloatArray()   #array for V coords
        uvShellIds = om.MIntArray() #The container for the uv shell Ids
 
        shapeFn.getUVs(uArray, vArray)
        shapeFn.getUvShellsIds(uvShellIds, nbUvShells, uvset)
 
        # shellCount = shells.getUint(shellsPtr)
        shells = {}
        for i, n in enumerate(uvShellIds):
            
            
            #print(i,n)

            limit = 100

            if i <= limit:
            
                if n in shells:
                    # shells[n].append([uArray[i],vArray[i]])
                    shells[n].append( '%s.map[%i]' % ( name, i ) )
                else:
                    # shells[n] = [[uArray[i],vArray[i]]]
                    shells[n] = [ '%s.map[%i]' % ( name, i ) ]

        allSets.append({uvset: shells})
    
    for shell in shells:
        
        selection_shell = shells.get(shell)
        cmds.select(selection_shell)
        #print(shells.get(shell))
        facesSel = cmds.polyListComponentConversion(fromUV=True, toFace=True)
        cmds.select(facesSel)
        r = [random.random() for i in range(3)]
        cmds.polyColorPerVertex(facesSel,rgb=(r[0], r[1], r[2]), cdo=1 )
        cmds.select(deselect=1)
 
getUvShelList( 'polySurface359' )

Have a look at recursive functions. https://www.programiz.com/python-programming/recursion https://realpython.com/python-thinking-recursively/ — nigel239, Sep 07 '22 at 13:40
chop up your big list into lists of lists that are each 100 long and then loop through them — SuperStew, Sep 07 '22 at 13:42
https://stackoverflow.com/questions/312443/how-do-i-split-a-list-into-equally-sized-chunks — ZabielskiGabriel, Sep 07 '22 at 13:45
Just added my current code, hopefully it will help you guys understand better the problem. — johancc, Sep 07 '22 at 13:47
Do you know why it crashed? Perhaps it is crashing because some of the 3d scene changed and you are setting color on objects that are gone. Here is a question related to chunking lists in python https://stackoverflow.com/questions/312443/how-do-i-split-a-list-into-equally-sized-chunks — toppk, Sep 07 '22 at 13:49
@toppk it's just too much geometry, and it works fine in less dense objects, do I don't believe I am applying color to objects that are gone. — johancc, Sep 07 '22 at 13:56

tobihans · Accepted Answer · 2022-09-07T16:13:59.610

You can use islice from itertools to chunk.

from itertools import islice

uvShellIds = list(range(1000))
iterator = iter(uvShellIds)

while True:
    chunk = list(islice(iterator, 100))

    if not chunk:
        break

    print(chunk) # chunk contains 100 elements you can process

I don't know how well it fits in your current code but, below is how you can process the chunks:

from itertools import islice

uvShellIds = list(range(1000))
iterator = iter(uvShellIds)
offset = 0

while True:
    chunk = list(islice(iterator, 100))

    if not chunk:
        break

    # Processing chunk items
    for i, n in enumerate(chunk):
        # offset + i will give you the right index referring to the uvShellIds variable
        # Then , perform your actions
        if n in shells:
            # shells[n].append([uArray[i],vArray[i]])
            shells[n].append( '%s.map[%i]' % ( name, offset + i ) )
        else:
            # shells[n] = [[uArray[i],vArray[i]]]
            shells[n] = [ '%s.map[%i]' % ( name, offset + i ) ]

    offset += 100
    # Your sleep can come here

The snippet above should replace your for i, n in enumerate(uvShellIds): block.

As @David Culbreth's answer stated, I'm not sure the sleep will be of help, but I left a comment on where you can place it.

I edited the question with your code, but somehow it's only applying the color to the first chunk. Can you please have a look? btw, I believe the offset var was missing. Thanks — johancc, Sep 07 '22 at 15:31
Yes, thanks. I updated my snippet to include the initlization of `offset`. But I don't think it's a fine idea to mix everything in your original post. Or at least, you should provde some context for people coming here in the future. — tobihans, Sep 07 '22 at 15:48
Thanks, just edited the question. Now is working, will accept the solution. — johancc, Sep 07 '22 at 15:53

score 1 · Answer 2 · answered Sep 07 '22 at 13:47

I use this generator to "chunkify" my long-running operations in python into smaller batches:

def chunkify_list(items, chunk_size):
    for i in range(0, len(items), chunk_size):
        yield items[i:i+chunk_size]

With this defined, you can write your program something like this:

items = [1,2,3,4,5 ...]
for chunk in chunkify_list(items, 100):
    for item in chunk:
        process_item(item)
    sleep(delay)

Now, I'm not going to guarantee that sleep will actually solve your problems, but this lets you handle your data one chunk at a time.

Thanks, this looks that might work, just not sure out to incorporate the for loop("for chunk in chun...") in my current loop. Can you please have a look at my full code added to the question? Thank you — johancc, Sep 07 '22 at 13:54

Python iterate over each 100 elements

2 Answers2