0

I have created a naive bayes classifier and has used pickle to store it.
Now everytime i excute my script, pickle is loaded taking almost 10-14 seconds.
Is there any way so that i can keep my pickle object loaded into the ram and can access it from another script?

def load_classifier(self):
    f=open("segment_to_topic.pickle","rb")
    classifier=pickle.load(f)
    f.close()
    return classifier
Harshit_Rana
  • 59
  • 2
  • 7
  • If you load your script many times in a small fraction of time, maybe you should just run a for loop instead of just re-executing. – Israel Zinc Nov 01 '17 at 07:23
  • I am using it from another file and everytime parameters are different ,therefore can,t use it – Harshit_Rana Nov 01 '17 at 07:27
  • I'm almost sure you can refactor your code to make it take advantage of the pre-loaded model. Otherwise, I don't think you can control RAM banks through python (I may be wrong). – Israel Zinc Nov 01 '17 at 07:37
  • Please show how you are currently using `load_classifier()` and give more info about this "other file" and how the "other file" is used/called/run. – Wodin Nov 01 '17 at 08:19

1 Answers1

0

You can use a named pipe. UNIX has os.mkfifo, and the windows alternative is probably win32pipe.CreateNamedPipe.

A UNIX solution would be like this:

a.py 

import os
import pickle
obj = {1:2, 3:4} # any object
fname = "segment_to_topic.pickle"
os.mkfifo(fname)
with open(fname, 'w') as f:
    pickle.dump(obj, f)

b.py

import pickle
while True:
    try:
        with open('segment_to_topic.pickle','r') as f:
            print pickle.load(f)
    except:
        pass

Then you run:

python ./b.py &
python ./a.py

And b.py prints the result:

{1: 2, 3: 4}

If you are on Windows, this may be helpful.

If the scripts are both in python, another solution is to run the second script as a new thread (or process) from the first one. Threads can share their memory in a Queue object.

xhancar
  • 687
  • 1
  • 6
  • 14