2

I want to delete a record in a dataframe object in snowflake table .

Similarly I want to perform update on basis of "key" in dataframe in snowflake table.

My research indicates that the utils method can perform the DDL operation but i am unable to find the some example to refer to.

plr108
  • 1,201
  • 11
  • 16
Rahul Jain
  • 21
  • 3

1 Answers1

0

As you mentioned, you can use the runQuery() method of the Utils object to execute DDL/DML SQL statements:

https://docs.snowflake.net/manuals/user-guide/spark-connector-use.html#executing-ddl-dml-sql-statements

If you want to do it based on some keys, then you can iterate items on DataFrame, and run an SQL for each item:

how to loop through each row of dataFrame in pyspark

But this will be a performance killer. Snowflake is a data warehouse, so you should always prefer "batch updates" instead of single row updates.

I would suggest you to write your dataframe to a staging table in Snowflake, and then call a SQL to update the rows in target table based on the staging table.

Gokhan Atil
  • 9,278
  • 2
  • 11
  • 24
  • How can you import it from a python script? Could you pl provide some example? – xyz Sep 21 '20 at 04:47
  • 1
    @xyz It seems the answer is not useful to anyone (although it was correct and useful), so I prefer not to spend so much time on same subject. – Gokhan Atil Sep 23 '20 at 14:39