Change the Datatype of columns in PySpark dataframe

Question

I have an input dataframe(ip_df), data in this dataframe looks like as below:

id            col_value
1               10
2               11
3               12

Data type of id and col_value is String

I need to get another dataframe(output_df), having datatype of id as string and col_value column as decimal**(15,4)**. THere is no data transformation, just data type conversion. Can i use it using PySpark. Any help will be appreciated

aclowkay · Answer 1 · 2017-08-02T06:58:43.880

11

Try using the cast method:

from pyspark.sql.types import DecimalType
<your code>
output_df = ip_df.withColumn("col_value",ip_df["col_value"].cast(DecimalType()))

edited Aug 02 '17 at 06:58

answered Aug 02 '17 at 06:50

aclowkay

3,577
5
35
66

It is giving error-name 'DecimalType' is not defined – Arunanshu P Aug 02 '17 at 06:57
You need to import it – aclowkay Aug 02 '17 at 06:58
from pyspark.sql.types import DecimalType – Rodney Apr 16 '19 at 00:22

score 4 · Answer 2 · edited Jan 21 '19 at 17:24

4

try below statement.

output_df = ip_df.withColumn("col_value",ip_df["col_value"].cast('float'))

edited Jan 21 '19 at 17:24

TMichel

4,336
9
44
67

answered Aug 02 '17 at 13:21

Neeraj Bhadani

2,930
16
26

score 1 · Answer 3 · answered Jul 02 '21 at 07:02

You can change multiple column types

Using withColumn() -

from pyspark.sql.types import DecimalType, StringType

output_df = ip_df \
  .withColumn("col_value", ip_df["col_value"].cast(DecimalType())) \
  .withColumn("id", ip_df["id"].cast(StringType()))

Using select()

from pyspark.sql.types import DecimalType, StringType

output_df = ip_df.select(
  (ip_df.id.cast(StringType())).alias('id'),
  (ip_df.col_value.cast(DecimalType())).alias('col_value')
)

Using spark.sql()

ip_df.createOrReplaceTempView("ip_df_view")

output_df = spark.sql('''
SELECT 
    STRING(id),
    DECIMAL(col_value)
FROM ip_df_view;
''')

Change the Datatype of columns in PySpark dataframe

3 Answers3

Linked