1

I have a script where I am trying to append data to an existing key/value pair. The problem is that whenever I get the existing data using txn.get(), it uses RAM that it doesn't free afterwards. When the limit of RAM approaches the computer gets extremely slow and the swap space is used more and more. For now I did not reached the limit of the swap space but I am afraid my script will stop running if it reaches the limit. But either way, without that RAM usage my script will run smoothly even with a small RAM.

Here is a snippet of my script causing me this problem:

for p in xrange(strings_number):  
    data=txn.get(struct.pack('i',900000+p))  #I want to reorganize this bytestring into columns
    for j in xrange(col_number):
        to_be_appended=""
        for l in xrange(j*4,len(data),size):
            to_be_appended+=data[l:l+4]
        temp=txn.get(struct.pack('i',j))  #whenever I call txn.get() memory usage gets bigger
        txn.put(struct.pack('i',j),temp+to_be_appended,dupdata=False,overwrite=True)         
        txn.delete(struct.pack('i',900000+p))
marc_s
  • 732,580
  • 175
  • 1,330
  • 1,459
wydadman
  • 36
  • 5
  • Related but not a dupe - https://stackoverflow.com/questions/1316767/how-can-i-explicitly-free-memory-in-python . I think thats relevant here in that you could call `gc.collect()` after each loop iteration and see if it helps. Note that GC will use CPU time while it GCs. I'm not optimistic that this will help your situation though. – Freiheit Oct 10 '19 at 14:59
  • Sadly gc.collect() doesn't fix the issue. – wydadman Oct 11 '19 at 00:05
  • In that case you need to change your code to keep less data in memory – Freiheit Oct 11 '19 at 03:22

0 Answers0