Serializing values to JSON array with pipes

Question

I'd like to serialize incoming values to JSON. Every value has a toJSON instance. The end result should be a list. The current code is the following:

import Pipes
import qualified Pipes.Prelude as P

-- assume a source of elements
main :: IO ()
main = runEffect $ source >-> P.map encode >-> P.stdoutLn

The problem is that in this way every line contains a valid JSON object, but I want the whole result to be parseable. I'd like that before the first object a [ character is outputted, then every element followed by a comma, and finally another ]. How can I do this with pipes?

Current output:

$ prog
{"key": "value"}
{"key": "value"}

Desired output:

$ prog
[{"key": "value"},
{"key": "value"}]

I found pipes-aeson but I don't understand how should I use the functions it provides.

EDIT: I modified ErikR's answer to get a Consumer, but it does not output the closing bracket:

jsonExporter :: Consumer (FilePath, AnalysisResult) IO ()
jsonExporter = do
    lift $ putStr "["
    P.map encode >-> insertCommas
    lift $ putStr "]"

I cannot understand why.

score 1 · Accepted Answer · answered Nov 01 '15 at 13:54

1

This pipe segment:

for cat $ \x -> lift $ do { putStr ", "; putStrLn x }

will emit a comma before each element in the pipe.

To give the first element special treatment, we just unroll the loop once:

insertCommas = do
  x1 <- await
  lift $ putStrLn x1      -- print first element w/o a comma
  for cat $ \x -> lift $ do { putStr ", "; putStrLn x }

Now you can write your streaming JSON pipeline as:

putStr "["
runEffect $ source >-> P.map encode >-> insertCommas
putStrLn "]"

answered Nov 01 '15 at 13:54

ErikR

51,541
9
73
124

Great! I'm still getting used to `lift` and sometimes I find it hard to use it correctly. – rubik Nov 01 '15 at 15:41
Since Aeson's `encode` returns a ByteString I think I can use the [`intersperse`](https://hackage.haskell.org/package/pipes-bytestring-2.1.1/docs/Pipes-ByteString.html#v:intersperse) function from the pipes-group package somehow? Instead of `insertCommas` I mean. – rubik Nov 01 '15 at 15:48
You can try it, but I think that intersperses a character between every pair of characters in the Bytestring. – ErikR Nov 01 '15 at 15:55
I needed a `Consumer`, so I modified your code (I put it in the question). But I cannot understand why the closing `]` is never outputted. – rubik Nov 01 '15 at 16:18
This should really be a new question. As a pipe segment `insertCommas` never returns, so that's why I print the brackets outside of the runEffect. – ErikR Nov 01 '15 at 16:34
You are right, the issue is a bit more involved. I asked it in the new question here: http://stackoverflow.com/questions/33464543/forking-the-streaming-flow-in-haskell-pipes Thank you for your patience and help. – rubik Nov 01 '15 at 16:43

Serializing values to JSON array with pipes

1 Answers1