1

I have used below code to convert the spark data frame to pandas data frame.

trainDF=df.toPandas()

where df is spark data frame and trainDF is the pandas dataframe.

but the problem is, all the data types of each column in pandas data frame becomes object.

44640 non-null object
44640 non-null object
44640 non-null object
44640 non-null object
44640 non-null object
1440 non-null object
44640 non-null object
44640 non-null object
44640 non-null object
44640 non-null object
44640 non-null object

tried using infer objects function to convert to actual data types, still no luck.

trainDF=trainDF.infer_objects()

is there any way to change the data type based on actual content of the column.

Sociopath
  • 13,068
  • 19
  • 47
  • 75
Anantha
  • 135
  • 1
  • 2
  • 8
  • Please post a small [reproducible example](https://stackoverflow.com/questions/48427185/how-to-make-good-reproducible-apache-spark-examples). – pault May 30 '19 at 14:04

0 Answers0