0

During the communication with Snowflake (via Python), there are intermittent interruptions occurring. Sometimes, these interruptions occur after 10+ hours. It appears that the error occurs during the data download process on the client side.

To provide an understanding, our process works as follows: The content of an SQL statement is downloaded in batches and loaded into a Pandas DataFrame. Each batch of 1000 rows is divided into 7 parts and sent in parallel to the PTV server for geocoding. The resulting data is then persisted in the target table on Snowflake. I tried using snowflake.connector and snowflake-sqlalchemy, both had the same issue.

numpy==1.24.2
pandas==1.5.3
snowflake-connector-python[pandas]==3.0.4
snowflake-sqlalchemy==1.4.7
sqlalchemy==1.4.48

log file: enter image description here

Asis
  • 1
  • 1
  • What is the error message on Snowflake side? Could you post result of [Query History](https://docs.snowflake.com/en/user-guide/ui-snowsight-activity#label-snowsight-activity-query-history) for this process? – Lukasz Szozda Jul 03 '23 at 12:15
  • If you are trying to download result-sets that take longer than 6 hours then it may be that the token has expired. See the note from [here](https://docs.snowflake.com/en/user-guide/querying-persisted-results): `Note that the security token used to access large persisted query results (i.e. greater than 100 KB in size) expires after 6 hours.` – Sergiu Jul 03 '23 at 12:21
  • Obviously with the limited information provided, I don't know why you've architected the solution in the way you have - but it doesn't look to be very efficient or use Snowflake in an optimal way. It might make more sense to copy the result of the SELECT statement into an external location in one go, process that file using your PTV server and write the output to another file (or files) and then upload the output to Snowflake in one go – NickW Jul 03 '23 at 15:52

0 Answers0