5

I have a shell script which calls around 20 python scripts each reading atleast couple of files of size 500MB each. After a couple of python scripts are run, there is like 40MB out of 8GB ram. I tried the below command and it worked and left me with 5.8 GB memory free.

echo 1 > /proc/sys/vm/drop_caches

Is there a programmatic way to do this exact thing from python? Or How can I force python to free up the memory occupied by it?

eumiro
  • 207,213
  • 34
  • 299
  • 261
Boolean
  • 14,266
  • 30
  • 88
  • 129
  • 4
    Why do you want to do that? Linux is usually smart enough to discard cache pages when it needs memory for something else. Unless you can *prove* that you can improve performance this way, it's safer to assume that you don't need to do this. – Joachim Sauer Mar 30 '11 at 08:03

4 Answers4

5

Why don't you simply open the file and write to it?

with open('/proc/sys/vm/drop_caches', 'w') as stream
    stream.write('1\n')

These are really Python basics.

3

The answers by @lunaryorn and @P2bM are in the right direction, but are security hazards, because they require you to run the entire python script with root privileges. If your script is an experiment under development, and it goes berserk, then it can destroy the system.

Another side-effect is that all files created by your script (logs etc.) will be created by root -- not by you. This is not very convenient for later processing.

It is much safer to restrict the sudo to where required only:

import os

os.system('sudo sh -c "sync; echo 3 > /proc/sys/vm/drop_caches"')
os.system('sudo sh -c "sync; echo 3 > /proc/sys/vm/drop_caches"')
with open('sample.txt', 'w') as f:
    f.write('a')

This script will prompt you only once for the root password (so it can run on its own after that), and you can verify that the file sample.txt is created by you, not by root.

0 _
  • 10,524
  • 11
  • 77
  • 109
1

The drop_caches-file is a linux-specific thing and not something python does. Python already freed the memory, it's just the operating system that hasn't released the memory from cache yet.

I don't know enough python to throw the code in here out of my head, but why not simply do a system call with above "echo" statement or write to the proc filesystem directly? (after making sure the script is running on a linux machine, of course.)

grimmig
  • 1,391
  • 2
  • 14
  • 24
  • I agree that its linux specific, but there should be some way to do similar thing in python. – Boolean Mar 30 '11 at 07:29
  • Python already freed the memory, it's just the operating system that hasn't released the memory from cache yet. – grimmig Mar 30 '11 at 07:29
  • got it. As you said, I can execute the system call or command, is there a best practice way of doing that. – Boolean Mar 30 '11 at 07:31
  • sysctl is also a command that can be ran with subprocess. That can be used to manage this. But that's the best you're going to get. I saw this using ctypes: http://stackoverflow.com/questions/759892/python-ctypes-and-sysctl – Mike Ramirez Mar 30 '11 at 07:32
  • @Bala: personally I would go with the `os.system('echo 1 >/proc/sys/vm/drop_caches')` mentioned in the other answer. It's the shortest and most readable variant of doing it. – grimmig Mar 30 '11 at 07:44
  • @grimmig, I need to be sudo user to execute echo command. But, sudo asks me for password. How can I pass password to command. – Boolean Apr 04 '11 at 14:18
0

Edit : lunaryorn is even more straight forward and pythonic, but it could be imporved to:

os.system('sync')
open('/proc/sys/vm/drop_caches','w').write("1\n")
P2bM
  • 1,038
  • 6
  • 9
  • Do I need to run drop caches, if I called sync? – Boolean Mar 30 '11 at 07:36
  • drop_caches will not clear the dirty objects because it is a non-destructive operation, however by using sync all the dirty pages can be acted upon and cleared – P2bM Mar 30 '11 at 07:41
  • sync initiates the writing of buffered file system data from memory to (physical) disks. Usually used before ejecting USB drives. I doubt that it would help in this situation. – grimmig Mar 30 '11 at 07:42
  • 1
    sync does not clear the memory it just checks the dirty paged objects and checks if they are still used otherwise diclares them as not accessed by any process, sync should be run before drop_caches so that the latter can clear the most objects possible as it cannot clear used files – P2bM Mar 30 '11 at 07:47
  • `with` though takes care of closing the file in all circumstances. – 0 _ Feb 07 '15 at 08:03