Running the same function simultaneously

Question

I am building a Windows console application that, after logic is performed on files they are transferred over FTP individually (we are dealing with thousands of files here) These files are relatively small (~50kb), so the login, transfer, logout of the FTP function does take a considerable amount of time, especially when multiplied against 1000's of files.

The FTP server cannot accept more than (for example) 10 simultaneous connections.

I have had a couple of ideas what would be the best course of action:

1. I am a Unity3D programmer and Coroutines sound like a perfect way to handle this. Simply loop through starting coroutines (increasing a variable ActiveConnections), then waiting for ActiveConnections to become >10 and repeating the process. Are coroutines possible in VS? I am not having any luck with it...
2. Starting another process/app solely for FTP connections, creating and destroying this console app for each file. The problem is A, it isn't very efficient (of course) and B. how am I going to manage the amount of Active connections stored in the ActiveConnections var in the main program?

Can anyone help me build on either of these ideas or possible suggest something better?

Thanks in advance!

Answer 1

Parallel linq will probably work for you.

List<FileInfo> filesToTransfer = GetFiles().AsParallel().WithDegreeOfParallelism(10);
filesToTransfer.ForAll(Transfer());

This is just some code I threw together, the Transfer() function is where you stick the login, transfer, and logout functionality. More information in the link below.

https://msdn.microsoft.com/en-us/library/dd460688(v=vs.110).aspx

Answer 2

You can do this with threading. The ThreadPool class is a good place to start.

https://msdn.microsoft.com/en-us/library/6kac2kdh(v=vs.110).aspx

https://msdn.microsoft.com/en-us/library/system.threading.threadpool(v=vs.110).aspx

Answer 3

You can accomplish this through threads. If you've never done any multi-threading, I recommend the C# yellow book: www.csharpcourse.com. There is a good intro to multi-threading in C# on pp 166.

I am unfamiliar with the specifics of how a coroutine is implemented, but in general storing a bunch of function calls and the associated local variables in memory does not scale well and is not a good idea for applications where you will be making thousands of calls.

The real issue here is the overhead introduced by having to login each time you run an individual file transfer. Even if you successfully open ten simultaneous connections, you'll just be doing ten times as many inefficient operations at once.

What I suggest you do is come up with a way to schedule your file transfers and do a batch transfer where you send many files during one connection. If you are using System.Net, there is a flag you can set on each request as detailed here: Upload Multiple files to FTP in c#

If you really want to go all out, this might be a perfect candidate to learn and apply the producer-consumer design pattern. In your application you could have one active thread running through and doing whatever it needs to do to the files, and another that is watching for those completed files and uploading them to the server.

https://en.wikipedia.org/wiki/Producer%E2%80%93consumer_problem