-1
# File location and type
file_location = "/FileStore/tables/FileName.csv"
file_type = "csv"

#CSV options
infer_schema = "true"
first_row_is_header = "true"
delimiter = ","

# The applied options are for CSV files. For other files types, these will be ignored.
df = spark.read.format(file_type) \
  .option("inferSchema", infer_schema) \
  .option("header", first_row_is_header) \
  .option("sep", delimiter) \
  .load(file_location)

display(df)

This is generic code to read the data from csv file. In this code, what is the use of ".option("inferSchema", infer_schema) " and what "" will do in this code?

Prashant Kashyap
  • 5
  • 3

2 Answers2

1

The use of the backslash at the end of the line is considered as a line continuation, which means the following to the backslash will be considered as one line to the previous. In your case, those 5 lines are considered as one line.

The reason why you need "", first, whatever you put in quotes is considered as a string, for these functions "header", "inferShema", and others are part of the syntax and you will need to keep them as they are.

This answer https://stackoverflow.com/a/56933052/6633728 might help you more.

Egidius
  • 971
  • 1
  • 10
  • 22
0

Backslash '' is used at the end of line to denote that the code after backslash is considered to be in the same line. This is mostly done is long code where code expands over single line.

inferSchema is used to infer the data types of the columns in dataframe. If we make inferSchema as true, then spark reads all the data in dataframe while loading data to infer the data types of the columns.

"" is used with .option function. It is used to add different parameter while reading a file. There can be many parameters added using option function such as header, inferSchema, sep, schema etc.

pyspark.sql.DataFrameReader.csv

You can refer the above link for further help.

Aman Lakhani
  • 51
  • 4