0

I upload files to data bricks file system every two day, i want to know if there's a place or log to see if some file is processed or not?

Thanks

peace
  • 299
  • 2
  • 16

1 Answers1

1

Please refer this link:https://docs.databricks.com/ingestion/auto-loader/production.html#querying-files-discovered-by-auto-loader

For python code refer this link:Get the list of loaded files from Databricks Autoloader

sample code SELECT * FROM cloud_files_state('path/to/checkpoint');

Sharma
  • 303
  • 2
  • 15
  • Hi @Sharma, do you know why i will got error 'could not resolve `CLOUD_FILES_STATE` to a table-valued function; line 1 pos 17' after running the sql? I run the sql in a notebook. – peace Jan 16 '23 at 06:00
  • Can you please send the code which you ran – Sharma Jan 16 '23 at 06:01
  • this the sql. SELECT * FROM CLOUD_FILES_STATE('/FileStore/test/checkpoint/web'); – peace Jan 16 '23 at 06:02
  • No idea, even i was trying to recreate but i was not able to do that. – Sharma Jan 16 '23 at 08:48
  • it works fine if your checkpoint is in storage, e.g. `select * from CLOUD_FILES_STATE('abfss://mycontainer@mystorageaccount.dfs.core.windows.net/mytable/_checkpoint_folder')` – adrien Jul 01 '23 at 14:20