35

I'm doing some queries in Python on a large database to get some stats out of the database. I want these stats to be in-memory so other programs can use them without going to a database.

I was thinking of how to structure them, and after trying to set up some complicated nested dictionaries, I realized that a good representation would be an SQL table. I don't want to store the data back into the persistent database, though. Are there any in-memory implementations of an SQL database that supports querying the data with SQL syntax?

Claudiu
  • 224,032
  • 165
  • 485
  • 680

6 Answers6

57

SQLite3 might work. The Python interface does support the in-memory implementation that the SQLite3 C API offers.

From the spec:

You can also supply the special name :memory: to create a database in RAM.

It's also relatively cheap with transactions, depending on what you are doing. To get going, just:

import sqlite3
conn = sqlite3.connect(':memory:')

You can then proceed like you were using a regular database.

Depending on your data - if you can get by with key/value (strings, hashes, lists, sets, sorted sets, etc) - Redis might be another option to explore (as you mentioned that you wanted to share with other programs).

Tim Post
  • 33,371
  • 15
  • 110
  • 174
  • 4
    One should mention that currently - as of 2020 - this doesn't work when using concurrent access to the sqlite3 object. So this fails e.g. if you plan to use it as a simple backend for a small webservice that supports concurrent access (which most web frameworks do behind the scene). – omni Nov 17 '20 at 11:58
8

It may not seem obvious, but pandas has a lot of relational capabilities. See comparison with SQL

Pierre Carbonnelle
  • 2,305
  • 19
  • 25
2

Extremely late to the party, but pyfilesystem2 (with which I am not affiliated) seems to be a perfect fit:

https://pyfilesystem2.readthedocs.io

pip install fs

from fs import open_fs
mem_fs = open_fs(u'mem://')
...
jtlz2
  • 7,700
  • 9
  • 64
  • 114
2

In-memory databases usually do not support memory paging option (for the whole database or certain tables), i,e, total size of the database should be smaller than the available physical memory or maximum shared memory size.

Depending on your application, data-access pattern, size of database and available system memory for database, you have a few choices:

a. Pickled Python Data in File System
It stores structured Python data structure (such as list of dictionaries/lists/tuples/sets, dictionary of lists/pandas dataframes/numpy series, etc.) in pickled format so that they could be used immediately and convienently upon unpickled. AFAIK, Python does not use file system as backing store for Python objects in memory implicitly but host operating system may swap out Python processes for higher priority processes. This is suitable for static data, having smaller memory size compared to available system memory. These pickled data could be copied to other computers, read by multiple dependent or independent processes in the same computer. The actual database file or memory size has higher overhead than size of the data. It is the fastest way to access the data as the data is in the same memory of the Python process, and without a query parsing step.

b. In-memory Database
It stores dynamic or static data in the memory. Possible in-memory libraries that with Python API binding are Redis, sqlite3, Berkeley Database, rqlite, etc. Different in-memory databases offer different features

  • Database may be locked in the physical memory so that it is not swapped to memory backing store by the host operating system. However the actual implementation for the same libray may vary across different operating systems.
  • The database may be served by a database server process.
  • The in-memory may be accessed by multiple dependent or independent processes.
  • Support full, partial or no ACID model.
  • In-memory database could be persistent to physical files so that it is available when the host operating is restarted.
  • Support snapshots or/and different database copies for backup or database management.
  • Support distributed database using master-slave, cluster models.
  • Support from simple key-value lookup to advanced query, filter, group functions (such as SQL, NoSQL)

c. Memory-map Database/Data Structure
It stores static or dynamic data which could be larger than physical memory of the host operating system. Python developers could use API such as mmap.mmap() numpy.memmap() to map certain files into process memory space. The files could be arranged into index and data so that data could be lookup/accessed via index lookup. This is actually the mechanism used by various database libraries. Python developers could implement custom techniques to access/update data efficiency.

yoonghm
  • 4,198
  • 1
  • 32
  • 48
1

I guess, SQLite3 will be the best option then.

If possible, take a look at memcached. (for key-value pair, lighting fast!)

UPDATE 1:

HSQLDB for SQL Like tables. (no python support)

zengr
  • 38,346
  • 37
  • 130
  • 192
  • Coming back to this a couple of years later, Redis is also a very viable option with a lot more flexibility than memcache for this sort of thing (unless SQL is a must have). – Tim Post Jul 02 '12 at 08:23
0

You could possibly use a database like SQLite. It's not strictly speaking in memory, but it is fairly light and would be completely separate from your main database.

Matthew Vines
  • 27,253
  • 7
  • 76
  • 97