Dynamically split lists in python

Question

I have a list with more than 5000 items and this number can change daily (can go up to 10's of thousands). The length of list changes on almost a daily basis. So I want to split the list into smaller lists of length of 300 but I want the name of these smaller lists to be defined dynamically.

I.e.:

main_list = ['1029', '2314', '466754', '6345', '3456' ....]

to

list1 = ['1029', '2314' ... first 300 elements]
list2 = ['342343', '3425' ...  next 300 elements]

and so on.

Python is able to handle lists of that size. What's the main purpose of this splitting? — dlask, Jul 07 '15 at 10:32
This data goes to an external system which can handle at max 400 requests at one time. thats why. — Manish Gupta, Jul 07 '15 at 10:34
@ManishGupta: the linked question about the chunks should answer your question, to process this you just need a list of lists. No need for named smaller lists, that just adds unneeded complications. — Wolph, Jul 07 '15 at 10:35
@Wolph, the dupe is only half the issue, the OP wants to access by name which the dupe does not cover — Padraic Cunningham, Jul 07 '15 at 10:38
@PadraicCunningham: I've got the feeling he doesn't actually need that. But the question isn't overly clear in that regard. Regardless, I've reopened the question :) — Wolph, Jul 07 '15 at 10:39

score 3 · Accepted Answer · answered Jul 07 '15 at 10:59

This data goes to an external system which can handle at max 400 requests at one time.

def process_sublist(i, sublist):
    """Deliver this i-th sublist to the external system."""
    pass

def process_list(main_list, sublist_size):
    """Split main_list into sublists and call process_sublist."""
    for i in range(0, len(main_list), sublist_size):
        index = i / sublist_size
        sublist = main_list[i:i+sublist_size]
        process_sublist(index, sublist)

# Use the function process_list.
main_list = ['1029', '2314', '466754', '6345', '3456', ...]
process_list(main_list, 300)

Padraic Cunningham · Answer 2 · 2015-07-07T10:49:52.120

1

Split into chunks and use a dict to access each by name/key:

d = {}
chnks = (main_list[i:i+300]for i in range(0, len(main_list), 300))

for ind, chnk in enumerate(chnks,1):
    d["list_{}".format(ind)] = chnk

print(d)

Using a smaller input size as an example:

main_list = ['1029', '2314', '466754', '6345', '3456',"4456"]
d = {}
chnks = (main_list[i:i+2]for i in range(0, len(main_list), 2))

for ind, chnk in enumerate(chnks,1):
    d["list_{}".format(ind)] = chnk

print(d)
{'list_3': ['3456', '4456'], 'list_2': ['466754', '6345'], 'list_1': ['1029', '2314']}

You can use a list of lists and access by index but if you just want to split the list into chunks and use each chunk once you don't need a dict or a list of lists, just iterate over the generator and pass each chunk.

edited Jul 07 '15 at 10:49

answered Jul 07 '15 at 10:34

Padraic Cunningham

176,452
29
245
321

1

You may want to point out to the OP that they just as well use a list of lists... can't see any gain using a `dict` here... – Jon Clements Jul 07 '15 at 10:44
@JonClements, probably not but it is the only way they should be creating variables dynamically, if they are only using the sublists just once then the gen exp and iteration over it will be sufficient. – Padraic Cunningham Jul 07 '15 at 10:47
If you really wanted, you could make this `d = {'list_{}'.format(idx):val for idx, val in enumerate(iter(lambda i=iter(main_list): list(islice(i, 2)), []), 1)}` - then it'd work on any arbitrary iterable as well. – Jon Clements Jul 07 '15 at 10:50

The6thSense · Answer 3 · 2015-07-07T13:03:06.033

0

gbl is nothing but a dictionary of variables and values you could split the list using range and define a new variable by defining key in dictionary

gbl=globals()
for j,i in enumerate(range(0,len(main_list),2),1):
    gbl["list"+str(j)]=main_list[i:i+2]

Input

['1029', '2314', '466754', '6345', '3456']

Output

list1=['1029','2134']
list2=['466754','6345']
list3=['3456']

edited Jul 07 '15 at 13:03

answered Jul 07 '15 at 12:56

The6thSense

8,103
8
31
65

Dynamically split lists in python

3 Answers3