Best way to store data temporarily?

Question

I'm building a sort of monitoring tool in Python where I want to keep certain stats for a short period of time. I only want to keep a maximum of, say, 30 entries for a stat, and for the older entries to be overwritten as new ones come in. This way, only the 30 most recent entries are kept. What sort of file should I use for this (I'll have multiple different stats all of which I would like to only keep their recent history. The stats are updated at regular intervals ~15 seconds).

I want this to be in a file as the data will be handled in another program.

Use memory. If you have only a little data and frequent updates, you don't need a file. — Mad Physicist, Jun 21 '18 at 15:52
This will depend on the size, format, how critical the data is, etc. Please provide some more information as to what you are trying to do. — Olivier Melançon, Jun 21 '18 at 15:53
Do you necessarily need a file? If so, do you need it updated every single operation? And sorted as in a FIFO queue? — Attersson, Jun 21 '18 at 15:53
Does it have to be stored in a file? Why not just maintain a list in memory? — Jacobm001, Jun 21 '18 at 15:55
I'd use Python built-in data structure. Since we have no information on the data you want to store, [this](https://stackoverflow.com/a/3489078/7692463) might be of interest for choosing the right structure. — scharette, Jun 21 '18 at 15:55
Buy a license and support contract or MySQL, hire a full time employee to administer the MySQL instance. Store the data in MySQL. — Jeffrey Blattman, Jun 21 '18 at 15:58
Maybe using a [file-based ringbuffer](https://stackoverflow.com/questions/6293868/a-text-file-circular-buffer-in-python) might be useful? — Jan Christoph Terasa, Jun 21 '18 at 17:09

score 2 · Answer 1 · answered Jun 21 '18 at 15:56

If you're only keeping a small number of samples (and you don't care about historic data), then the simplest solution is keeping data in memory. You can use a collections.deque object, as described here, to create a fixed-length list that will automatically drop older items as you add newer items.

For situations in which you want to keep data for longer periods (or you simply want it to persist in the event your application restarts, or you want to be able to access the data from multiple applications, etc), people often use a dedicated time series database, such as InfluxDB, Prometheus, Graphite, or any of a number of other solutions.

score 0 · Answer 2 · answered Jun 21 '18 at 15:55

You probably want to keep it all in memory. But if you need to keep a file that mimics and a data structure (say a dictionary) I've had great success with pickle. It's easy, and it's fast.

https://pythontips.com/2013/08/02/what-is-pickle-in-python/

Alternatively a more enterprise solution would be to simply store your stats in a database.

Best way to store data temporarily?

2 Answers2