5

Is there a .NET library that I can use to sync a folder to S3? I am not interested in full apps or command line tools, just the API. I need to use it from my own app directly.

If there isn't, has anyone written their own, and if so, can you share some of the key considerations when writing one? Anything specific I need to watch out for?

By the way, I will be using this for backups so that I don't have to zip an entire folder and upload the zip file to S3 each time. I am hoping that doing it this way will take a shorter time for very large folders (something like SyncBack).

Uwe Keim
  • 39,551
  • 56
  • 175
  • 291
stepanian
  • 11,373
  • 8
  • 43
  • 63
  • @Ralph Stevens I use the Amazon .NET Library to place files directly to the S3, it's a super simple Library and you can get it from NuGet as well. – balexandre Jun 03 '11 at 16:48
  • I currently do that. I want to be able to sync rather than upload the entire directory. See my definition of sync in my responses to user608576 below. – stepanian Jun 03 '11 at 16:53
  • but, this will act as a service or a desktop application that you choose both folder (S3 + Desktop) and it syncs? – balexandre Jun 03 '11 at 18:20
  • @Ralph Stevens Just curious . what solution you ended up using ? – Pit Digger Jun 06 '11 at 22:03
  • @user608576 - Funny you asked! I will use S3 for the application's file storage. That way I don't have to store them on my server and have to back them up daily. S3 has something like 99.999 durability. These are just user uploaded images - nothing mission critical. I don't think I need to back them up anymore if they are sitting on S3. – stepanian Jun 07 '11 at 03:10
  • My question was for sync what you used ? FolderWatcher , S3Tools or any other solution ? – Pit Digger Jun 07 '11 at 18:09
  • I am not doing sync anymore. I am using S3 for the storage directly. I use the Amazon DotNet SDK for all the web service calls. – stepanian Jun 13 '11 at 07:02

1 Answers1

2

You can periodically simply loop through file list and use these libraries to call API http://weblogs.asp.net/israelio/archive/2004/06/23/162913.aspx OR Try http://msdn.microsoft.com/en-us/library/system.io.filesystemwatcher.aspx ? You can simply watch OnDeleted , OnCreated and OnChanged events . And call S3 API accordingly.

You can even a temporary zip it and than put it on S3 as suggested here. How do I create 7-Zip archives with .NET? . And delete once its uploaded on S3.

Community
  • 1
  • 1
Pit Digger
  • 9,618
  • 23
  • 78
  • 122
  • I will look into these. As far as the first link you sent, I use the Amazon .NET library, which that link says we should be using going forward, and I don't believe, it has sync functionality. The second link doesn't seem to be applicable to S3, but I will look at it anyway. Thank you very much for your help. – stepanian Jun 03 '11 at 16:46
  • 2nd link has a way to loop through all files in directory and than you can call your amazon library to upload it . You can find plenty libraries to create a 7zip file. I have a solution for this and will see if i can post a link in few hours. Check my edited answer . You can see here a way to create a 7Zip and than upload it instead of uploading whole directory . http://stackoverflow.com/questions/222030/how-do-i-create-7-zip-archives-with-net – Pit Digger Jun 03 '11 at 16:48
  • Zipping the entire folder and uploading to S3 is what I am doing now, but I want to move to the sync method to make the backup faster. – stepanian Jun 03 '11 at 16:50
  • 1
    Sync means I only upload a file if the same version does not exist on S3. Also, to delete files that are on S3 that don't exist in the folder. – stepanian Jun 03 '11 at 16:52
  • Did you try http://msdn.microsoft.com/en-us/library/system.io.filesystemwatcher.aspx ? You can simply watch OnDeleted , OnCreated and OnChanged events . And call S3 API accordingly.I am assuming you have a directory mapping known to local directory. – Pit Digger Jun 03 '11 at 17:01
  • @user608576 This is a very nice class that I hadn't used before. If I don't use it for this, I will definitely use it somewhere else. Thanks for pointing it out. I'm still waiting to see if someone has a solution for the sync functionality. But I will surely award you a point for this. And this CAN be used as a workaround to what I am asking for. – stepanian Jun 03 '11 at 17:32
  • Check if this can help http://s3tools.org/s3cmd-sync . If you not have to do it using Amazon library and .NET check this . – Pit Digger Jun 03 '11 at 19:28