0

Is there a way to completely flush memmap from memory in Python and somehow just store a pointer? I notice memmap_object.flush() and del memmap_object have different effects.

Complete Code:

temp_train_data=X_train[1000:]
temp_labels=y[1000:] 
out = np.empty((200001, 3504), np.int64)
for index,row in enumerate(temp_train_data):
    actual_index=index+1000
    data=X_train[actual_index-1000:actual_index+1].ravel()
    __,cd_i=pywt.dwt(data,'haar')
    out[index] = cd_i
out.flush()
pca_obj=IncrementalPCA()
clf = pca_obj.fit(out)

Edit 1:(@stark suggestion) Main problems relating to the question:

  1. clf = pca_obj.fit(out) trashes my RAM.
  2. I want more RAM freed up so I can increase the batch-size of iterative PCA.
Abhishek Bhatia
  • 9,404
  • 26
  • 87
  • 142
  • 1
    Using del will flush and delete the object, flush will obviously flush – Padraic Cunningham Aug 27 '15 at 17:28
  • How memory should the object occupy? Does it act as a pointer to the file? Sorry if the questions are naive, I am new to python and numpy. – Abhishek Bhatia Aug 27 '15 at 17:31
  • memap returns an array like object, if you del the object you won't be able to use it again – Padraic Cunningham Aug 27 '15 at 17:41
  • What size does the array occupy in memory. I keen to just have a pointer to the memmap on disk to save my memory. And later pass this to IncrementalPCA can batch process, saving my RAM from crashing. – Abhishek Bhatia Aug 27 '15 at 17:50
  • 1
    Mmap allocates virtual memory, not physical memory. The OS manages how much physical memory is in use. RAM will not "crash" unless you are doing ACCESS_COPY but you may run into process limits on your virtual address space. – stark Aug 27 '15 at 18:40
  • @stark Thanks for the comment! What do you suggest is the way to free up all the RAM and use it for `clf = pca_obj.fit(out)`. – Abhishek Bhatia Aug 27 '15 at 18:55
  • 2
    Why do you need to do anything? If you are getting an error you should have posted it. – stark Aug 27 '15 at 19:38
  • Sorry should have posted before: 1.) `clf = pca_obj.fit(out)` trashes my RAM. 2.) I want more RAM freed up so I can increase the batch-size of iterative PCA. – Abhishek Bhatia Aug 27 '15 at 19:40
  • 1
    You need to publish Tracebacks, rather than just saying "my RAM is crashing". Because you're not just showing the traceback and asking why am I getting a `MemoryError`, your questions have the [XY problem](http://meta.stackexchange.com/a/66378/261369). In this question, for instance, you ask about `memmap` but have no `memmap` in the code you post here. I think this is basically a dupicate of [your other question](http://stackoverflow.com/questions/32242911/using-memmap-files-for-batch-processing), too. – J Richard Snape Aug 28 '15 at 14:31

0 Answers0