Haskell Conduit and masking async exceptions

Question

I have the following line of code that uses aeson to serialize an IntMap and save the JSON to disk, all on a child thread:

    import Data.Aeson (encode, toJSON)
    import Data.Conduit (($$), (=$), yield)
    import qualified Data.ByteString.Lazy as BL (toStrict)
    import qualified Data.Conduit.Binary as CB (sinkFile)
    import qualified Data.Conduit.List as CL (map)

    -- ...

    forkIO . runResourceT $ yield (toJSON intMap) $$ CL.map (BL.toStrict . encode) =$ CB.sinkFile file

I would like to ensure that this code cannot be interrupted by any asynchronous exceptions. My fear is that an interruption could result in incomplete/corrupt data on disk.

What can I do in this case to ensure immunity from async exceptions? Is it possible to ensure that the child thread will be allowed to finish even if main wants to terminate?

Thanks!

To ensure that the child thread will finish, create an `MVar`; have the child thread put something into it and the main thread take from it. Don't know enough to answer the other half, though. — Daniel Wagner, Mar 30 '15 at 00:38

score 3 · Accepted Answer · edited May 23 '17 at 12:15

I'd suggest another technique, unrelated to conduit:

Create a new, temporary file,
write your data there,
fsync the file, so that everything is really written to disk,
atomically rename the temporary file to the target one; on POSIX systems this is easy, as rename is designed for it, on Windows this question should help.

With this sequence, whatever happens, the target file will either remain intact or will contain the new data, fully written to disk. If the sequence is abruptly interrupted for some reason, it'll just leave a stale temporary file.

The unix Haskell library introduces fsync only in version 2.7.1.0, but it's easy to add the call yourself. See this module in the Ganeti project (licensed under BSD2). In particular

foreign import ccall "fsync" fsync :: CInt -> IO CInt

fsyncFile :: FilePath -> IO ()
fsyncFile path =
    bracket (openFd path ReadOnly Nothing defaultFileFlags) closeFd callfsync
  where
    callfsync (Fd fd) = throwErrnoPathIfMinus1_ "fsyncFile" path $ fsync fd

Thanks for the detailed answer with links! I wouldn't have thought of taking this approach. Fascinating stuff! — the-konapie, Mar 31 '15 at 02:51

Haskell Conduit and masking async exceptions

1 Answers1