0

I have a requirement where in large size zipped files (size in GBs) are coming in a directory on a unix server (lets say server1) and I have to write application which will poll that directory and copy the files to another unix server (lets say server2) as they come . I have a way to know when one file is completely copied in a directory (using corresponding meta data file which will only come when copy operation of a single file is complete) . Since there are hundreds of files, we dont want to wait for all the files to be copied. Once files are copied to server2 , I have to do unzipping and some validations before I land up those files in my final repository.

Questions

  1. What would be the appropriate tech to use for this scenario,shell scripting or java or something else in terms of speed ?
  2. Since we will be doing the transfer operation file by file , how do we achieve parallelism (other than multithreading if we use java) ?

Any existing lib/package/tool available which can fit this scenario .

KBR
  • 464
  • 1
  • 7
  • 24
  • [scp](https://linux.die.net/man/1/scp)? It can handle multi-GB files provided it is either 64bit or 32bit and compiled with large file support. [rsync](https://linux.die.net/man/1/rsync) too. – Michael Mar 31 '17 at 15:49
  • Thanks Michael. so when we say scp , I am sure we are talking about writing a shell script here , how do we parallelize the execution avoiding the race conditions ? – KBR Mar 31 '17 at 15:54
  • See [this question](http://stackoverflow.com/questions/38160/parallelize-bash-script-with-maximum-number-of-processes). I can't foresee any race conditions you would need to consider. – Michael Mar 31 '17 at 15:57
  • Also, your bottleneck is likely going to be the network so I don't know how much parallelism is going to help you. In a highly parallelised setup, the receiver may receive a lot of files all at once which may become somewhat of a bottleneck. It may be preferable for them to receive the files sequentially, depending on how involved the processing is at that end. – Michael Mar 31 '17 at 16:03

0 Answers0