Data buffering/storage - Python

Question

I am writing an embedded application that reads data from a set of sensors and uploads to a central server. This application is written in Python and runs on a Rasberry Pi unit.

The data needs to be collected every 1 minute, however, the Internet connection is unstable and I need to buffer the data to a non volatile storage (SD-card) etc. whenever there is no connection. The buffered data should be uploaded as and when the connection comes back.

Presently, I'm thinking about storing the buffered data in a SQLite database and writing a cron job that can read the data from this database continuously and upload.

Is there a python module that can be used for such feature?

Answer 1

Is there a python module that can be used for such feature?

I'm not aware of any readily available module, however it should be quite straight forward to build one. Given your requirement:

the Internet connection is unstable and I need to buffer the data to a non volatile storage (SD-card) etc. whenever there is no connection. The buffered data should be uploaded as and when the connection comes back.

The algorithm looks something like this (pseudo code):

# buffering module
data = read(sensors)
db.insert(data)

# upload module
# e.g. scheduled every 5 minutes via cron
data = db.read(created > last_successful_upload)
success = upload(data)
if success:
    last_successful_upload = max(data.created)

The key is to seperate the buffering and uploading concerns. I.e. when reading data from the sensor don't attempt to immediately upload, always upload from the scheduled module. This keeps the two modules simple and stable.

There are a few edge cases however that you need to concern yourself with to make this work reliably:

1. insert data while uploading is in progress
2. SQLlite doesn't support being accessed from multiple processes well

To solve this, you might want to consider another database, or create multiple SQLite databases or even flat files for each batch of uploads.

Answer 2

If you mean a module to work with SQLite database, check out SQLAlchemy.

If you mean a module which can do what cron does, check out sched, a python event scheduler.

However, this looks like a perfect place to implemet a task queue --using a dedicated task broker (rabbitmq, redis, zeromq,..), or python's threads and queues. In general, you want to submit an upload task, and worker thread will pick it up and execute, while the task broker handles retries and failures. All this happens asynchronously, without blocking your main app.

UPD: Just to clarify, you don't need the database if you use a task broker, because a task broker stores the tasks for you.

Answer 3

This is only database work. You can create a master and slave databases in different locations and if one is not on the network, will run with the last synched info.

And when the connection came back hr merge all the data.

Take a look in this answer and search for master and slave database