0

Say you have 10 million rows in a db table represented by a django model. How do you efficiently load all of these, run a function on them and then write some changes back to the db.

I am trying to find the correct course of action when dealing with such a problem

Ahmed Salah
  • 27
  • 7
  • What is the function? Provide more details. The answer will be different if you need to add 1 to some field and different if you need to do something complicated – Bartosz Stasiak Feb 25 '23 at 21:04
  • Does this answer your question? [Django Table with Million of rows](https://stackoverflow.com/questions/2051481/django-table-with-million-of-rows) – azzam Feb 25 '23 at 21:29
  • Updating in batches can be done efficiently as mentioned [here.](https://blog.labdigital.nl/working-with-huge-data-sets-in-django-169453bca049) – azzam Feb 25 '23 at 21:39
  • That is a too broad question. To make it more focused, could you please clarify some details? For instance, what type of function do you intend to apply? Will it be a one-time execution or will it be executed multiple times? Is it going to be part of an API or is it more of a backfilling task? Providing this information will help to better understand your needs and get an accurate response. – Jonathan Quispe Feb 26 '23 at 04:40

0 Answers0