Parameterise spark partition by clause

Question

I am trying to insert a DataFrame in na existing Hive partitioned table.

I would like to parameterize by the partition columns but my current approach is not working:

var partitioncolumn="\"deletion_flag\",\"date_feed\""
df.repartition(37).write.
  mode(SaveMode.Overwrite).
  partitionBy(partitioncolumn).
  insertInto("db.table_name")

How can I make this work?

score 2 · Answer 1 · answered Jan 31 '18 at 12:07

2

As partitionBy is defined with variadic arguments:

def partitionBy(colNames: String*): DataFrameWriter[T]

It should be:

var partitioncolumn= Seq("deletion_flag", "date_feed")
df.repartition(37).write.mode(SaveMode.Overwrite).partitionBy(
   partitioncolumn: _*
).insertInto("db.table_name")

where you provide expanded list of column names.

answered Jan 31 '18 at 12:07

user9294355

21
1

I have same answer from both the persons so I can only choose one is that ok if I take stefanobaghino as answer – sri hari kali charan Tummala Jan 31 '18 at 12:24

score 1 · Accepted Answer · answered Jan 31 '18 at 12:09

partitionBy takes a variable number of arguments (namely, Strings).

def partitionBy(colNames: String*): DataFrameWriter[T]
//                              ^ this stands for variadic arguments

In Scala, you can pass postfix a sequence with : _* to pass it as an argument list.

So you could do something like the following:

var partitioncolumn= Seq("deletion_flag", "date_feed")
df.repartition(37).write.
  mode(SaveMode.Overwrite).
  partitionBy(partitioncolumn: _*).
  insertInto("db.table_name")

Passing a sequence as variadic arguments is also described in this Q&A.

Parameterise spark partition by clause

2 Answers2