Export JSON from Spark and input into R

Question

I've written some data from Spark to a JSON file and I am struggling to import it into R.

I cannot import it with any of the traditional JSON packages in R:

library("jsonlite")
bids <- fromJSON("win_rate_sample.json")

I get the following error:

Error in feed_push_parser(readBin(con, raw(), n), reset = TRUE) : 
 parse error: trailing garbage 
      X","domain":"ifunny_premium"}{"win":0,"bid_price":0.75,"size
                 (right here) ------^

How can I get this file into R?

Possible duplicate of [Error parsing JSON file with the jsonlite package](http://stackoverflow.com/questions/26519455/error-parsing-json-file-with-the-jsonlite-package) — miken32, Aug 22 '16 at 22:40

score 1 · Accepted Answer · edited May 23 '17 at 12:22

1

It turns out that Spark exports streaming json files like those discussed in the following question: Error parsing JSON file with the jsonlite package

The solution is to use jsonlite's streaming function:

library(jsonlite)
json_file <- stream_in(file("win_rate_sample.json"))

edited May 23 '17 at 12:22

Community

1
1

answered Jan 06 '16 at 21:32

Michael Discenza

3,240
7
30
41

Export JSON from Spark and input into R

1 Answers1

Linked