How can I loop a generator in this situation?

Question

Here's the question.

I want to use django's bulk_create to save more datas at once. But the original result I get from API is a generator with amount data in it. So I want to loop this generator and bulk save data.

My trial was as below：


#  a generator with amount data   
l = (
    item for item in range(1,100230, 1)
)

#  base model table
class ModelDemo(models.Model):
    ...


# main logic code

limit = 2000
while l:
    bulk_list = []
    for index, item in enumerate(l):
        bulk_list.append(
            ModelDemo(
                ...
            )
        )
        if index == limit:
            ModelDemo.objects.bulk_create(bulk_list)
            break

It's obviously I would lose last 230 data, but I couldn't find the solution by now.

Any commentary is very welcome. great thanks.

Where do you get the generator? usually just doing list(generator) forces it to compute. — Neil, Jun 18 '19 at 10:07
Have you tried with list comprehension? `bulk_list = [ ModelDemo(...) for i,n in enumerate(range(1,100230, 1))]` then `ModelDemo.objects.bulk_create(bulk_list)` — alec_djinn, Jun 18 '19 at 10:09
@Neil I'm using facebook graph api, and it 's a genarator with over 7w objects — jia Jimmy, Jun 18 '19 at 10:18

score 1 · Accepted Answer · answered Jun 18 '19 at 10:25

This might work.

Ex:

from itertools import chain, islice
l = (
    item for item in range(1,10230, 1)
)
limit = 2000

def chunks(iterable, size=10):
    iterator = iter(iterable)
    for first in iterator:
        yield chain([first], islice(iterator, size - 1))

data = chunks(l, limit)
while data:
    try:
        bulk_list = []
        for index, item in enumerate(next(data)):
            bulk_list.append(
                ModelDemo(
                    ...
                )
            )
        ModelDemo.objects.bulk_create(bulk_list)
    except:
        break

Reference

score 0 · Answer 2 · answered Jun 18 '19 at 10:28

Ah! I understand now what your problem is.

All you need to do is make a final bulk_create call after the loop has terminated to handle the final list which was too small.

main logic code

limit = 2000

bulk_list = []
for index, item in enumerate(l):
    bulk_list.append(
        ModelDemo(
            ...
        )
    )
    if index == limit:
        ModelDemo.objects.bulk_create(bulk_list)
        bulk_list.clear() # reset the batch

ModelDemo.objects.bulk_create(bulk_list)

I'm not sure what the while loop was there for. And you shouldn't break.

How can I loop a generator in this situation?

2 Answers2

main logic code