1

I have an FTP server from which I need to upload files to a blob storage in Azure. I cannot touch or move any of the files. Is there anyway to do an incremental scan (through an id or similar), or do you have to save the modified date of the newest file and then scan all files newer than that? All input are welcome.

Martin Prikryl
  • 188,800
  • 56
  • 490
  • 992
Thomas Segato
  • 4,567
  • 11
  • 55
  • 104
  • 1
    you need to save the state (last processed) somewhere / somehow – Thiago Custodio Jan 17 '20 at 16:10
  • So only the modified date can be used as incremental parameter. – Thomas Segato Jan 17 '20 at 16:11
  • Can you keep a log of the files on your end? Say in a database? Each time you do a scan, get the list of files and compare it to your log. If a file is not on the log, it's new. Then of course, add the new files to your log. – Casey Crookston Jan 17 '20 at 16:15
  • Yes I can do that. But we are talking 100000 of files. So I am looking for a way to only scan from new files since last run. I could properly do on modified date but then I need to be carefull about time zones. So if there is a way to get an id or something from the most new item. – Thomas Segato Jan 17 '20 at 16:39

1 Answers1

0

All you can do with FTP is to get a list of all files in a directory. Nothing else.

Some FTP servers will allow you retrieving the list sorted by a timestamp. This is a proprietary non-standard extension. See How to get files in FTP folder sorted by modification time. But it still won't save you from requesting a listing of all files (though you may abort the list download prematurely, once you get all files you need).

There's no way to ask an FTP server to return only "new files", no matter what's your definition of that.

If you need anything fancy, you will have to build a service on the server (server as a machine, not server as an FTP server) itself with an API that will do what you need.

Martin Prikryl
  • 188,800
  • 56
  • 490
  • 992
  • Thanks. I thought this was a walk in the park task, but after some reading I can see your right. I can see Fluent FTP has a recursive function, so I will use that and get all files each time and compare with files in DB. Not a fancy solution though. Thanks for your input. – Thomas Segato Jan 17 '20 at 17:22