Need load following json file into spark

Asked Feb 13 '18 at 11:11

Active Feb 13 '18 at 11:11

Viewed 27 times

Am new to Spark and Scala, I have some scenario like need to pass below file into Spark sql context, I don't know how to process this file into spark.

If any one help me it would be very helpful

{
 "source": [
{
  "id": "source1",
  "type": "csv",
  "path": "/home/siva/f1.txt",
  "delimiter": ",",
  "tableName": "table1",
  "schema": "f1,f2,f3,f4,f5"
}
],
"pipe": [
{
  "id": "pipe1",
  "sql": "select f1,f2,f5 from table1"
}
],
"sinks": [
{
  "id": "output1",
  "type": "hdfs",
  "path": "/home/siva/out",
  "format": "json"
},

]
}

Please help me to proceed.

Thanks in Advance

asked Feb 13 '18 at 11:11

PANDIA LAKSHMANAN

Do you mean you have json files where each record looks like the one you've displayed (on several lines)? – Xavier Guihot Feb 13 '18 at 11:13
Yes you are right – PANDIA LAKSHMANAN Feb 13 '18 at 11:15
Is it possible for you to get this file to have one json record per line? Otherwise, we'll have to use more complex solutions. – Xavier Guihot Feb 13 '18 at 11:25
No we cannot create like that because its generated from Gson source from another application so we cannot – PANDIA LAKSHMANAN Feb 13 '18 at 11:35

Need load following json file into spark

0 Answers0