0

I'm currently using Snowflake Python connector (2.6.2) to move data from onPremise server to Snowflake. My files are big, so I don't use write_pandas(), I prefer use PUT command to put parquet files in Snowflake internal stage, then use COPY command to move data from stage to Snowflake table.

(All Synchronous call)

# Put parquet files to stage
copy_into_stage = f"put file://{parquet_path} @{stage_name};"
cur.execute(copy_into_stage)

# Copy Snowflake stage to table
copy_into_table = self._copy_into_statement(table, stage_name, parquet_schema)
cur.execute(copy_into_table)

Everything works fine, but I really need to show some progress bar for my users. Some dataset can take more than 1hours to upload ...

I saw that SnowSQL display some nice progress bar when you are moving data in/out to Snowflake, and I try to do the exact same thing.

Snowflake cursor object's execute method have some private parameters about progress bar : _show_progress_bar or even _get_callback

https://github.com/snowflakedb/snowflake-connector-python/blob/19474a11d5e2f71f03cff49e639fff9b0eb16ad0/src/snowflake/connector/cursor.py#L575

  • Is it possible to use build-in SnowSQL progress bar with Snowflake Python connector ?

  • Does Snowflake return some metadata regarding the actual query's progress ? To use with callback function ?

Thank you

Gohmz
  • 1,256
  • 16
  • 31

1 Answers1

0

For the first query, you may be able to do that using some of the Python functions but I don't think there is any sample for this use case from SF as such but this reference might be a good starting point: Python Progress Bar

For the 2nd requirement, you can check the Query progress as detailed here: https://docs.snowflake.com/en/user-guide/python-connector-example.html#checking-the-status-of-a-query

Srinath Menon
  • 1,479
  • 8
  • 11