0

This is certainly a very stupid question but i really spent some time on this and i can't find.

I'm reading a csv file with pyspark and this produces an error because my floating separator is a ",".

Before replacing it in the strings, i'd like to verify that there is no option to specify it (something like .option("floatingDecimal", ",")) but i can't find any complete list of available option. Many tutorials giving some but nothing complete. Is one existing ? Thanks

Jice
  • 191
  • 4
  • 16

1 Answers1

2

In the official documentation of the DataFrameReader.csv:

You can set the following CSV-specific options to deal with CSV files:

  • sep (default ,): sets the single character as a separator for each field and value.
  • encoding (default UTF-8): decodes the CSV files by the given encoding type.
  • quote (default "): sets the single character used for escaping quoted values where the separator can be part of the value. If you would like to turn off quotations, you need to set not null but an empty string. This behaviour is different form com.databricks.spark.csv.
  • escape (default ): sets the single character used for escaping quotes inside an already quoted value.
  • comment (default empty string): sets the single character used for skipping lines beginning with this character. By default, it is disabled.
  • header (default false): uses the first line as names of columns.
  • inferSchema (default false): infers the input schema automatically from data. It requires one extra pass over the data.
  • ignoreLeadingWhiteSpace (default false): defines whether or not leading whitespaces from values being read should be skipped.
  • ignoreTrailingWhiteSpace (default false): defines whether or not trailing whitespaces from values being read should be skipped.
  • nullValue (default empty string): sets the string representation of a null value.
  • nanValue (default NaN): sets the string representation of a non-number" value.
  • positiveInf (default Inf): sets the string representation of a positive infinity value.
  • negativeInf (default -Inf): sets the string representation of a negative infinity value.
  • dateFormat (default null): sets the string that indicates a date format. Custom date formats follow the formats at java.text.SimpleDateFormat. This applies to both date type and timestamp type. By default, it is null which means trying to parse times and date by java.sql.Timestamp.valueOf() and java.sql.Date.valueOf().
  • maxColumns (default 20480): defines a hard limit of how many columns a record can have.
  • maxCharsPerColumn (default 1000000): defines the maximum number of characters allowed for any given value being read.
  • maxMalformedLogPerPartition (default 10): sets the maximum number of malformed rows Spark will log for each partition. Malformed records beyond this number will be ignored.
  • mode (default PERMISSIVE): allows a mode for dealing with corrupt records during parsing.
    • PERMISSIVE : sets other fields to null when it meets a corrupted record. When a schema is set by user, it sets null for extra fields.
    • DROPMALFORMED : ignores the whole corrupted records.
    • FAILFAST : throws an exception when it meets corrupted records.
user9969053
  • 44
  • 1
  • Thank you for your fast answer. i find this documentation very labyrinthine. I find it very hard to find what i'm looking for in it... So no option for float decimal separator. – Jice Jun 20 '18 at 17:54