2

I am trying to create schema to parse json into spark dataframe

I have column value in json which could be either struct or string

"value": {
    "entity-type": "item",
    "id": "someid",
    "numeric-id": 30
  }

"value": "SomePicture.jpg",

How can i specify that in the schema

Neha Zaveri
  • 21
  • 5

2 Answers2

1
{
  "type": ["object", "string"],
  "properties": { ... }
}

https://json-schema.org/understanding-json-schema/index.html

Ether
  • 53,118
  • 13
  • 86
  • 159
  • In json we can do the way you specified. But while defining spark schema it doesnt work So for Spark schema I had to fetch value in string and then determine if value is going to be of structtype, based on certain conditions and then use from_json(value, new StructType()) to convert string back to json – Neha Zaveri Apr 20 '22 at 05:42
0

Solved it using below approach

In JSON, we can do the way you specified above. But while defining spark schema it doesn't work So for Spark schema I had to fetch value in String and then determine if value is going to be of structtype, based on certain conditions and then use from_json(value, new StructType()) to convert string back to JSON

Neha Zaveri
  • 21
  • 5