Django storing a lot of data in table

Question

Right now, I use this code to save the data to the database-

for i in range(len(companies)):
    for j in range(len(final_prices)):
      linechartdata = LineChartData()
      linechartdata.foundation = company //this refers to a foreign-key of a different model
      linechartdata.date = finald[j]
      linechartdata.price = finalp[j]
      linechartdata.save()

Now len(companies) can vary from [3-50] and len(final_prices) can vary from somewhere between [5000-10000]. I know its a very inefficient way to store it in the database and takes a lot of time. What should I do to make it effective and less time consuming?

Possible duplicate of [Bulk create model objects in django](http://stackoverflow.com/questions/3608487/bulk-create-model-objects-in-django) — spectras, Jul 21 '16 at 13:19

Tiny Instance · Accepted Answer · 2016-07-21T13:40:52.860

2

If you really need to store them in the database you might check bulk_create. From the documents:

This method inserts the provided list of objects into the database in an efficient manner (generally only 1 query, no matter how many objects there are):

Although, I never personally used it for that many objects, docs say it can. This could make your code more efficient in terms of hitting the database and using multiple save().

Basically to try; create list of objects (without saving) and then use bulk_create. Like this:

arr = []
for i in range(len(companies)):
    for j in range(len(final_prices)):
        arr.append(
           LineChartData(
              foundation = company,
              date = finald[j],
              price = finalp[j]
            )
        )
LineChartData.objects.bulk_create(arr)

edited Jul 21 '16 at 13:40

answered Jul 21 '16 at 13:26

Tiny Instance

2,351
17
21

I know there exists a bulk_create method but I can not find the right example to understand how to use it – Sushant Jul 21 '16 at 13:30
If you're going to just rewrite the answer linked as a duplicate, at least improve upon it. Add sample code, for instance. – spectras Jul 21 '16 at 13:32
@spectras sorry man, I didn't really see your comment. Should I delete this? – Tiny Instance Jul 21 '16 at 13:34
No, but it would be much better if you could add a short code sample showing how to use `bulk_create`. Especially as ThatBird is struggling with that. – spectras Jul 21 '16 at 13:38
1

Just added an example for your code. Hope it helps @ThatBird – Tiny Instance Jul 21 '16 at 13:42
1

Thanks, it helped ! – Sushant Jul 21 '16 at 14:25

Django storing a lot of data in table

1 Answers1