1

What I am trying to do is to convert data in json format from kafka into dataframe. I can easily do this by creating a schema and use from_json as shown here: How to use from_json with kafka connect 0.10 and Spark Structured Streaming?

But is there anyway I can do the same thing using Java instead of Scala? As far as I know there is no "from_json" API in java.

Thanks!

teddy
  • 413
  • 3
  • 8
  • 24
  • [There is Java-specific `from_json`](https://spark.apache.org/docs/latest/api/scala/index.html#org.apache.spark.sql.functions$@from_json(e:org.apache.spark.sql.Column,schema:org.apache.spark.sql.types.StructType,options:java.util.Map[String,String]):org.apache.spark.sql.Column) – nonsleepr Jul 27 '17 at 15:37
  • @nonsleepr Are you sure? The link you posted is from scala API. And java program cannot even find the import: import org.apache.spark.sql.functions.from_json – teddy Jul 28 '17 at 01:56
  • Oh, I'm sorry got used to use Scala docs. Here's Java one: https://spark.apache.org/docs/2.1.0/api/java/org/apache/spark/sql/functions.html#from_json(org.apache.spark.sql.Column,%20org.apache.spark.sql.types.StructType) – nonsleepr Jul 28 '17 at 03:39
  • @nonsleepr I see. This looks good to me, I'll dig into it. Thanks!! – teddy Jul 28 '17 at 03:46

1 Answers1

0

I still cannot find from_json in Java even though it is in the javadoc. I feel like this could be a fix for apache spark in the future. Or it could be just spark streaming does not support java that well.

teddy
  • 413
  • 3
  • 8
  • 24