Convert Spark Scala join statement to Python

Question

I need to convert the following scala statement to python:

scala> a.join(b, a("key") === b("key"), "left").select(a.columns.map(a(_)) : _*).show

Having trouble with the : and _* I'm not familiar with Scala.

I need to implement the solution in this answer in python: https://stackoverflow.com/a/43340781/2453657

score 1 · Accepted Answer · answered Jun 25 '18 at 09:03

In the Scala DataFrame API, the select method takes a variable number of arguments. In the method signature, this is denoted by a * as in the following example:

// Greet many people
def greet(who: String*): String = ???

// all valid calls
greet()
greet("world")
greet("alice", "bob")

The : token is used to give a hint to the compiler regarding the type of the argument and the _* in this case is used to specify that we are passing a collection as a list of arguments:

def people: Seq[String] = getPeopleToGreet()

greet(people)     // won't compile
greet(people: _*) // passes the collection of people as a list of arguments -- works

Not sure of how the Python API works, but from my experience I imagine you can either pass a single value or an array, so I believe the problem just doesn't exist in Python.

Thanks. Thats what I needed to know. – Gerrie van Wyk Jun 25 '18 at 09:07 — Gerrie van Wyk, Jun 25 '18 at 09:07

Convert Spark Scala join statement to Python

1 Answers1