Memory issue in Python. Dictionary v.s. Database (or can you combine both?)

Question

I'm trying to find patterns of words for a huge input. I was using a dictionary for this purpose, and after some hours the program crashed with MemoryError.

I modified the program. I created a database via MySQLdb and I inserted there the values of the pattern-index. So for every word I check if it is in the index and if not I write it into the index with a value. Problem is that the database approach is too slow.

I was wondering if there is any way to combine dictionaries and database for example:

if ram <90% usage:
    seek into dict
    append to dict
else:
    if not (seek into dict):
        seek into database
        append to database

Using a dictionary for the same purpose of inputting 100 kb of data takes ~1.5 sec

Using a database for the same input takes ~84 sec

Original input is 16 GB . I do not know yet how much it will take to process.

why not just load the input into a database and let it create the index on its own? — Jayanth Koushik, Mar 18 '14 at 12:38
the *index* is a modified LZW pattern finding so I dont know if the db can do that — Βασιλης Ιωσηφιδης, Mar 18 '14 at 12:39
To get free RAM and/or CPU load in python check this: http://stackoverflow.com/questions/276052/how-to-get-current-cpu-and-ram-usage-in-python — Roberto, Mar 18 '14 at 12:46
what is the approximate size of input ? What is your approximate expected time of script execution ? — Xavier Combelle, Mar 18 '14 at 13:49
First of all if you're using your script on the same machine as MySQL running, you need to set unix socket connection to DB. Then you need to check your queries with EXPLAIN statement in MySQL. Look at an indexes, never use '*' for selecting data from big tables in. Set up partitioning for tables. If you got small amount of data that will help you with caching data in memory. Did you try numpy for storing your data in memory? 16G is not so big if you got more than 32G of RAM. Also you can try handlersocket for MySQL. — Alexander Gryanko, Mar 19 '14 at 03:55
I do not have such much memory. The select * for the table will always return 1 row. — Βασιλης Ιωσηφιδης, Mar 19 '14 at 06:21
Ok I used @RobertoSánchez idea to check memory and if memory is 90%+ I store the items in the db. Now I will change mysqldb to redis (as GOD user proposed) to see if it goes faster. thanks for your advice! — Βασιλης Ιωσηφιδης, Mar 19 '14 at 10:44

Xavier Combelle · Answer 1 · 2014-03-18T16:24:57.240

0

short answer (detailed answer to come):

your use of MySQL was poor, you don't want to commit at all as you just use the database as an extension of memory. Just removing the commmit should give you a big improvement

better than using MySQL use leveldb (pip install leveldb) with sync = false

adjust the following values with your memory you want to use

block_cache_size = 512*1024*1024 #512Mo #the more important
write_buffer_size = 10*1024*1024 #10Mo

as you have a MemoryError that means you have a 32bits system it means that the total memory enable for a process can't be more than 4 Go so adjust the values to fit in min(your system memory,4Go)

edited Mar 18 '14 at 16:24

answered Mar 18 '14 at 16:09

Xavier Combelle

10,968
5
28
52

I use 64bit system because if I used 32bit I could not take a 16gb input. I get memory error because if I use the dictionary my ram and swap space get full. – Βασιλης Ιωσηφιδης Mar 18 '14 at 19:19

Memory issue in Python. Dictionary v.s. Database (or can you combine both?)

1 Answers1