MPI locking for sqlite (python)

Question

I am using mpi4py for a project I want to parallelize. Below is very basic pseudo code for my program:

Load list of data from sqlite database
Based on COMM.Rank and Comm.Size, select chunk of data to process

Process data...

use MPI.Gather to pass all of the results back to root

if root:
    iterate through results and save to sqlite database

I would like to eliminate the call to MPI.Gather by simply having each process write its own results to the database. So I want my pseudo code to look like this:

Load list of data
Select chunk of data
Process data
Save results

This would drastically improve my program's performance. However, I am not entirely sure how to accomplish this. I have tried to find methods through google, but the only thing I could find is MPI-IO. Is it possible to use MPI-IO to write to a database? Specifically using python, sqlite, and mpi4py. If not, are there any alternatives for writing concurrently to a sqlite database?

EDIT:

As @CL pointed out in a comment, sqlite3 does not support concurrent writes to the database. So let me ask my question a little differently: Is there a way to lock writes to the database so that other processes wait till the lock is removed before writing? I know sqlite3 has its own locking modes, but these modes seem to cause insertions to fail rather than block. I know I've seen something like this in Python threading, but I haven't been able to find anything online about doing this with MPI.

Thanks, that section explains pretty clearly that multiple processes won't be able to write to a database _at the same time_. However, does MPI have an efficient way to allow processes to take turns writing to a database on a first-come-first-served basis? This would eliminate the need for using Gather in my program, and it would allow the processes that finish processing first to be able to write their data without waiting on slower ones to finish. — Harrison, Jul 08 '14 at 11:33

score 1 · Answer 1 · answered Dec 09 '14 at 22:17

1

I would suggest you pass your results back to the root process, and let the root process write them to the SQLite database. The pseudocode would look something like this:

load list of data
if rank == 0:
    for _ in len(data):
        result = receive from any worker
        save result
else:
    select chunk of data
    process data
    send result(s) to rank 0

The advantage over gathering is that rank 0 can save the results as soon as they are ready. There is an mpi4py example that shows how to spread tasks out over multiple workers when there are lots of tasks and the processing time varies widely.

answered Dec 09 '14 at 22:17

Don Kirkby

53,582
27
205
286

Thanks for the answer, but as you can see in the question, my original approach was to pass the data back to the root process. In the question I wanted to see if there was a way to avoid this by having each process write concurrently to a database. – Harrison Dec 16 '14 at 02:40
@Harrison, you asked, "does MPI have an efficient way to allow processes to take turns writing to a database on a first-come-first-served basis?" I'm suggesting that the way to do that is to let the root process coordinate the writes to the database. By receiving from any worker, it will be first come first served. The only down side is that the root process will be idle, but you can just allocate an extra worker process to compensate. – Don Kirkby Dec 16 '14 at 05:44

MPI locking for sqlite (python)

1 Answers1