2

I'm loading a table data into a dataframe and creating multiple JSON part files. The structure of the data is good, but the elements in JSON are not separated by commas.

This is the output:

{"time_stamp":"2016-12-08 01:45:00","Temperature":0.8,"Energy":111111.5,"Net_Energy":1111.3}
{"time_stamp":"2016-12-08 02:00:00","Temperature":21.9,"Energy":222222.5,"Net_Energy":222.0}

I'm supposed to get something like this:

{"time_stamp":"2016-12-08 01:45:00","Temperature":0.8,"Energy":111111.5,"Net_Energy":1111.3},
{"time_stamp":"2016-12-08 02:00:00","Temperature":21.9,"Energy":222222.5,"Net_Energy":222.0}

How do I do this?

the Tin Man
  • 158,662
  • 42
  • 215
  • 303
venkatesh
  • 23
  • 3
  • 1
    It's actually correct because it's a multi line file, not an array. If you look at the JSON standard it requieres no comma. – Vincent Nov 18 '19 at 18:26

1 Answers1

0

Your output is correct JSONlines output: one JSON record per line, separated by newlines. You do not need commas between the lines. In fact, that would be invalid JSON.

If you absolutely need to turn the entire output of a Spark job into a single JSON array of objects, there are two ways to do this:

  1. For data that fits in driver RAM, df.as[String].collect.mkString("[", ",", "]").

  2. For data that does not fit in driver RAM... you really shouldn't do it... but if you absolutely have to, use shell operations to begin with [, add a comma to every line of output and end in ].

Sim
  • 13,147
  • 9
  • 66
  • 95
  • Third possibility, if data fits in an executor memory you can reduce number of partitions to 1 then apply a foreachPartition on it – baitmbarek Nov 21 '19 at 21:13
  • @baitmbarek `foreachPartition` doesn't return a result so this leaves the problem if figuring out how to get the data out. Perhaps you were thinking of `mapPartitions` to return a single string and then write a massive single row text file using the ideas in https://stackoverflow.com/a/48712134/622495 (to control the file name). – Sim Nov 21 '19 at 21:35