0
A^B^C^D^E^F
1^b^3^z^m^6
7^w^9^55^0^D
k^^f^^

How can we remove the first line of the file and make it as header & provide schema, we don't have any schema/DDL provided to create a table. Should be done without Databricks package. It's an overview of the dataset, actual number of columns are around 100-150. Need to automate the script after reading them in spark.

Final output:

**A**   **B**   **C**   **D**   **E**   **F**
  1       b       3       z       m       6
  7       w       9       55      0       D
  k               f
maduri
  • 11
  • 5
  • 3
    Really? It's mentioned so many times - use `option("header", "true")` – T. Gawęda Sep 25 '17 at 20:27
  • @ T.Gaweda I'm unable to pull the databricks packages while starting the spark-shell from the central repositories. And I'm trying to pull the first line as header and provide it as schema, as i don't have the schema inferred. – maduri Sep 26 '17 at 02:57

0 Answers0