When trying to create a Spark SQL table reading from a JSON file, it fails with the below error
Error in SQL statement: AnalysisException: Found duplicate column(s) in the data schema: filename
As I understand this issue is related to Spark runtime and an alternative approach suggested for dataframes is Duplicate column in json file throw error when creating PySpark dataframe Databricks after upgrading runtime 7.3LTS(Spark3.0.1) to 9.1LTS(Spark3.1.2)
Looking for a similar alternative for SparkSQL table. I tried to emulate it for spark table creation but not sure how to call out schema while creating table, any help is highly appreciated.
Note: My need is to create an external table in databricks pointing to JSON file which has duplicate column(s) issue.
I am creating a sql table as below which is resulting in the error because the json file has some duplicate column code_snippet
