Pass variables in pyspark SQL statement

Question

I am trying to insert multiple variables in a Spark SQL statement and found a similar question here: How to pass variables in spark SQL, using python? My question is how to do this with a list of multiple variables that are string variables (Department variable below)? It's working for Floats/ integers. I've tried several variations of syntax, but I get a "Mismatched input 'From' expecting EOF " error.

configs = {"lim":10,
           "codes":"A",
           "department": " 'A', 'B', 'C'", ## this is what's not working
           "salary": "100.00, 200.00"
}

df = spark.sql("""SELECT col1, col2 from table
                  WHERE employee_id IN ({department})
AND salary IN ({salary})
                 LIMIT 10
               """.format(**configs))

score 0 · Answer 1 · answered Mar 03 '21 at 00:21

0

Try escaping the single quotes:

"department": " \'A\', \'B\', \'C\'"

answered Mar 03 '21 at 00:21

braebdeb

219
2
11

Have you also tried double backslash, e.g. \\'A\\' or doubling up your single quotes e.g. ' ' A ' ' – braebdeb Mar 03 '21 at 21:33

Pass variables in pyspark SQL statement

1 Answers1