I am new to Spark. I am trying to read a json file and convert it to a DataFrame.
public void groupJsonString(SparkSession spark) {
Dataset<Row> studentRecord =
spark.read().json("/home/sparkJsonFiles/etlFinalJson.json");
studentRecord.printSchema();
}
The json file contains the below json
{"name":"Hello"}
When I run this code. I get the following exception. I am not able to understand what is the issue.
Exception in thread "main" java.lang.IllegalArgumentException: Illegal pattern component: XXX at org.apache.commons.lang3.time.FastDatePrinter.parsePattern(FastDatePrinter.java:282) at org.apache.commons.lang3.time.FastDatePrinter.init(FastDatePrinter.java:149) at org.apache.commons.lang3.time.FastDatePrinter.(FastDatePrinter.java:142) at org.apache.commons.lang3.time.FastDateFormat.(FastDateFormat.java:384) at org.apache.commons.lang3.time.FastDateFormat.(FastDateFormat.java:369) at org.apache.commons.lang3.time.FastDateFormat$1.createInstance(FastDateFormat.java:91) at org.apache.commons.lang3.time.FastDateFormat$1.createInstance(FastDateFormat.java:88) at org.apache.commons.lang3.time.FormatCache.getInstance(FormatCache.java:82) at org.apache.commons.lang3.time.FastDateFormat.getInstance(FastDateFormat.java:165) at org.apache.spark.sql.catalyst.json.JSONOptions.(JSONOptions.scala:81) at org.apache.spark.sql.catalyst.json.JSONOptions.(JSONOptions.scala:43) at org.apache.spark.sql.execution.datasources.json.JsonFileFormat.inferSchema(JsonFileFormat.scala:53) at org.apache.spark.sql.execution.datasources.DataSource$$anonfun$7.apply(DataSource.scala:177) at org.apache.spark.sql.execution.datasources.DataSource$$anonfun$7.apply(DataSource.scala:177) at scala.Option.orElse(Option.scala:289) at org.apache.spark.sql.execution.datasources.DataSource.getOrInferFileFormatSchema(DataSource.scala:176) at org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:366) at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:178) at org.apache.spark.sql.DataFrameReader.json(DataFrameReader.scala:333) at org.apache.spark.sql.DataFrameReader.json(DataFrameReader.scala:279) at Spark.spark_demo.JSONGrouping.groupJsonString(JSONGrouping.java:24) at Spark.spark_demo.JSONGrouping.main(JSONGrouping.java:18) 17/12/05 17:28:55 INFO SparkContext: Invoking stop() from shutdown hook 17/12/05 17:28:55 INFO SparkUI: Stopped Spark web UI at http://192.168.1.30:4040 17/12/05 17:28:55 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped! 17/12/05 17:28:55 INFO MemoryStore: MemoryStore cleared 17/12/05 17:28:55 INFO BlockManager: BlockManager stopped 17/12/05 17:28:55 INFO BlockManagerMaster: BlockManagerMaster stopped 17/12/05 17:28:55 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped! 17/12/05 17:28:55 INFO SparkContext: Successfully stopped SparkContext 17/12/05 17:28:55 INFO ShutdownHookManager: Shutdown hook called 17/12/05 17:28:55 INFO ShutdownHookManager: Deleting directory /tmp/spark-2aeffac1-9dcd-4014-b821-a7737a4c5678