Infer safe schema from a structured streaming dataframe in spark

Asked Jul 03 '19 at 05:52

Active Jul 03 '19 at 05:52

Viewed 721 times

I wanted to infer a safe schema from a JSON data coming from Kafka.

df = spark.readStream.format("kafka").option("kafka.bootstrap.servers", "localhost:9092").option("subscribe", "input").option("auto.offset.reset", "latest").load()
jsonDF = df.selectExpr("CAST(value AS STRING) jsonData")

There are a couple of solutions on stackoverflow:

link says to save a small batch into a file, infer the schema and then use that schema for the streaming dataframe. Although the question is a year old.
link uses schema_of_json and lit ,although I'm not able to get it to work with streaming DFs.

I know inferring of schema can be dangerous, but the spark application I'm developing has multiple sources with varied schemas(multiple columns). Is there a way I can create a schema based on the columns in the json data and force cast them to String to prevent data loss.

asked Jul 03 '19 at 05:52

firecast

If `schema_of_json` is not working, you could create an udf with [genson](https://github.com/wolverdude/GenSON/). – cronoik Jul 03 '19 at 14:48
@cronoik can you give an example of this. The dataframe I'm working with is a streaming dataframe. – firecast Jul 04 '19 at 05:43
Sure, but please add some an example of json messages to your question before. – cronoik Jul 04 '19 at 19:14

Infer safe schema from a structured streaming dataframe in spark

0 Answers0