Spark dataframe write to JSON showing NULL ourput

Question

I am extracting JSON data from a API and trying to write on Azure container path. I am able to display data correctly in notebook, but when i write JSON most of the values are NULL. Any help on where i am going wrong?

headers = {
"accept" : "application/json",
"Content-Type": "application/json",
 "Authorization": "Bearer " + str(token)
}

 response_get= requests.get(getURL, headers=headers)
 response_final=response_get.json()
 print("Type:", type(response_final))
 data = json_normalize(response_final)
 df = spark.createDataFrame(data)
 ##df.coalesce(1).write.parquet(stagingpath,mode='overwrite')
 df.coalesce(1).write.json(stagingpath,mode='overwrite')

Answer 1

I have reproduced in my environment and followed below process and got expected results as below and followed Microsoft-Document and SO-Thread:

import requests

response = requests.get('https://reqres.in/api/users?page=3')
rdd = spark.sparkContext.parallelize([response.text])
df = spark.read.json(rdd)
df.show()
dbutils.fs.mount( source = "wasbs://mycontainer@myblobstorageaccount.blob.core.windows.net", mount_point = "/mnt/mymountpoint", extra_configs = {"fs.azure.sas.mycontainer.myblobstorageaccount.blob.core.windows.net": "SAS"})

The run below script to write json:

df.coalesce(1).write.json( "/mnt/mymountpoint/vamo.json")

Output:

Click on folder Vammo.json:

Click on part-00xxx:

Then Click on View/Edit: