1

I have a python script which basically take the minute wise electrical load data, solar production data and does some energy flow analyses. It's fairly complex algorithm which involves a lot of loops and huge DataFrame, matrices, and arrays. When I work on the script for sometime it takes, on an average 35s to execute. All the fans of my machine start to blow.

But suppose I leave my PC for 3-4 hours and come back, the same code takes more than 120s to execute.

Why is this so and is there a way I could avoid this?

Any help would be appreciated.

Rajat Sharma
  • 21
  • 2
  • Thanks @Samwise for your comment. This was very interesting phenomenon. Is there a way around to start the program with the same speed, or have a warm cache at the beginning? – Rajat Sharma Nov 16 '20 at 22:30

1 Answers1

1

This is happening because of Caching. When you load data from the same place for the first time the data is stored in an easier accessible place (cache) because of the belief that you will soon use that data again. When you leave your computer that space is filled with other data that other processes use. This is a fundamental part of how a computer is built and the various optimization chip designers add in order to make a computer (CPU) faster. This phenomenon is also known as "cold run".

Edit: Loading data includes loading the actual code segments (heap, stack, etc) into memory

Tamir
  • 1,224
  • 1
  • 5
  • 18
  • Thank you for your informative answer. Is there way I could "warm run" my code for the first time? Because it take another 2-3 hours for it to get into the original speed. I do not want to wait that long. – Rajat Sharma Nov 16 '20 at 22:29
  • There isn't really a quick fix. if there is a file that you open at each run then maybe opening it before the script can help. You need to understand the concept behind caching first, it loads data and stores it based on your usage, so there isn't a quick trick to overcome this – Tamir Nov 16 '20 at 22:30
  • 1
    It might be possible to optimize the script itself. For example, if the script currently loads the same files repeatedly, then it speeds up after the filesystem warms the cache with that content, but you could avoid the need for FS caching completely by having the script load the files into memory only once. – Samwise Nov 16 '20 at 22:41
  • as @Samwise said you might be able to optimize some aspects of the script but there is no way to bring a "cold run" to a starting condition that is the same as the future runs. There is a small "price" to pay at the beginning unfournately – Tamir Nov 16 '20 at 22:43