Clear cache or memory in Python

Question

I have a really large table need to calculate (10 billion + rows). It's too long to calculate all of them at one time. Therefore, I create a list of break points, and I calculate and store at the end of each step. However, each step costs longer time. I think the reason is the memory or cache, do you have any idea how to deal with in this situation or just how to clear cache or memory?

I reuse the variable for output in the loop, so the variable will not become bigger and bigger.

Thank you so much.

Cache is irrelevant, no software outside *maybe* the OS directly messes with that (cache is hardware-driven, not software). — Delioth, Apr 17 '17 at 21:20
How does the mentioned large table implemented? Is it a database table or a dictionary? — Imre Piller, Apr 17 '17 at 21:20
It's retrieved from SQL server, and transformed to pandas.DataFrame, And then calculated and stored into SQL server. @ImrePiller — Carl Zheng, Apr 17 '17 at 21:24

Camilo · Answer 1 · 2021-09-23T14:53:54.533

6

I use to solve this issue using a line that reset the variable at the end of the process, cleaning the cache:

MyVariable = None

edited Sep 23 '21 at 14:53

answered Feb 21 '19 at 13:46

Camilo

335
5
7

1

How is that different from `del MyVariable` in terms of cleaning the cache? – user30850 Dec 08 '21 at 10:17

Clear cache or memory in Python

1 Answers1