Configure SQLite for real time operations

Question

In short, this post would like to answer the following question : how (if possible) can we configure a SQLite database to be absolutely sure that any INSERT command will return in less than 8 milliseconds?

By configure, I mean: compiling options, database pragma options, and run-time options.

To give some background, we would like to apply the same INSERT statement at 120 fps. (1000 ms / 120 fps ≃ 8 ms)

The Database is created with the following strings:

"CREATE TABLE IF NOT EXISTS MYTABLE ("
      "int1 INTEGER PRIMARY KEY AUTOINCREMENT, "
      "int2 INTEGER, "
      "int3 INTEGER, "
      "int4 INTEGER, "
      "fileName TEXT);

and the options:

"PRAGMA SYNCHRONOUS=NORMAL;"
"PRAGMA JOURNAL_MODE=WAL;"

The INSERT statement is the following one:

INSERT INTO MYTABLE VALUES (NULL, ?, ?, ?, ?)

The last ? (for fileName) is the name of a file, so it's a small string. Each INSERT is thus small.

Of course, I use precompiled statements to accelerate the process.

I have a little program that makes one insert every 8 ms, and measures the time taking to perform this insert. To be more precise, the program makes one insert, THEN wait for 8 ms, THEN makes another insert, etc... At the end, 7200 inserts were pushed, so the program runs for about 1 minutes.

Here are two links that show two charts:

This image shows how many milliseconds were spent to make an insert as a function of the time expressed in minutes. As you can see, most of the time, the insert time is 0, but there are spikes than can go higher than 100 ms.

This image is the histogram representation of the same data. All the values below 5 ms are not represented, but I can tell you that from the 7200 inserts, 7161 are below 5 milliseconds (and would give a huge peak at 0 that would make the chart less readable).

The total program time is real 1m2.286s user 0m1.228s sys 0m0.320s.

Let's say it's 1 minute and 4 seconds. Don't forget that we spend 7200 times 8 milliseconds to wait. So the 7200 inserts take 4 seconds ---> we have a rate of 1800 inserts per seconds, and thus an average time of 0.55 milliseconds per insert. This is really great, except that in my case, i want ALL THE INSERTS to be below 8 milliseconds, and the chart shows that this is clearly not the case.

So where do these peaks come from?

When the WAL file reaches a given size (1MB in our case), SQLite makes a checkpoint (the WAL file is applied to the real database file). And because we passed PRAGMA SYNCHRONOUS=NORMAL, then at this moment, SQLite performs a fsync on the hard drive. We suppose this is this fsync that makes the corresponding insert really slow. This long insert time does not depend on the WAL file size. We played with the pragma WAL_AUTOCHECKPOINT (1000 by default) linked to the WAL file, and we could not reduce the height of the peaks.

We also tried with PRAGMA SYNCHRONOUS=OFF. The performances are better but still not enough. For information, the dirty_background_ratio (/proc/sys/vm/dirty_background_ratio) on my computer was set to 0, meaning that all dirty pages in the cache must be flushed immediately on the hard drive.

Does anyone have an idea and how to "smooth" the chart, meaning that all inserts time will not overpass 8 ms ?

Why do you need such timing? Speaking from the perspective of having worked on a system requiring 40 ms time slices for RS485 comm, I don't think you're going to get there on a non-real-time OS. What OS are you using? — Dark Falcon, Apr 02 '14 at 13:33
I'm no fan of multithreading in general, but this appears to be a case where execution of your INSERT statement should be in a separate thread, with both threads safely sharing a std::deque of records to be inserted. — Steger, Apr 02 '14 at 13:39
@Steger: This is exactly what we did on the above project, and gave our fast thread real-time priority, but we still could not achieve the desired timing always. We got to where it was "good enough" though. — Dark Falcon, Apr 02 '14 at 13:41
You could try some of these http://stackoverflow.com/questions/1711631/how-do-i-improve-the-performance-of-sqlite optimizations. — AnthonyLambert, Apr 02 '14 at 13:42

score 2 · Answer 1 · answered Apr 02 '14 at 14:53

By default, pretty much everything in SQLite is optimized for throughput, not latency.

WAL mode moves most delays into the checkpoint, but if you don't want those big delays, you have to use more frequent checkpoints, i.e., do a checkpoint after each transaction. In that case, WAL mode does not make sense; better try journal_mode=persist. (This will not help much because the delay comes mostly from the synchronization, not from the amount of data.)

If the WAL/journal operations are too slow, and if even synchronous=off is not fast enough, then your only choice is to disable transaction safety and try journal_mode=memory or even =off.

Configure SQLite for real time operations

1 Answers1