I am trying to download multiple files to memory with only one open connection.
I know from this documentation: https://learn.microsoft.com/en-us/dotnet/api/system.net.ftpwebrequest.keepalive – that FtpWebRequest defaults KeepAlive to true.
if the connection to the server should not be destroyed; otherwise, false. The default value is true.
I have created this piece of code:
using System;
using System.Collections.Generic;
using System.IO;
using System.Linq;
using System.Net;
using System.Text;
namespace ConsoleApp23
{
class Program
{
static void Main(string[] args)
{
String[] files = FTPListTree("ftp://server.com", "user", "password");
for (int i = 0; i <= files.Count() - 1; i++)
{
string path = files[i].ToString();
var request = (FtpWebRequest)WebRequest.Create(path);
request.Method = WebRequestMethods.Ftp.DownloadFile;
request.Credentials = new NetworkCredential("user", "password");
request.Timeout = 100000;
request.KeepAlive = true;
request.ConnectionGroupName = "DownloadFilesGroup";
request.ServicePoint.ConnectionLimit = 8;
using (var response = (FtpWebResponse)request.GetResponse())
{
using (var stream = response.GetResponseStream())
{
var ms = new MemoryStream();
stream.CopyTo(ms);
var a = ms.ToArray();
string ss = Encoding.UTF8.GetString(a);
Console.WriteLine(ss);
}
}
}
}
public static String[] FTPListTree(String ftpUri, String user, String pass)
{
try
{
List<String> files = new List<String>();
Queue<String> folders = new Queue<String>();
folders.Enqueue(ftpUri);
while (folders.Count > 0)
{
String fld = folders.Dequeue();
List<String> newFiles = new List<String>();
FtpWebRequest ftp = (FtpWebRequest)FtpWebRequest.Create(fld);
ftp.Timeout = 20000;
ftp.Credentials = new NetworkCredential(user, pass);
ftp.Method = WebRequestMethods.Ftp.ListDirectory;
using (StreamReader resp = new StreamReader(ftp.GetResponse().GetResponseStream()))
{
String line = resp.ReadLine();
while (line != null)
{
newFiles.Add(line.Trim());
line = resp.ReadLine();
}
}
ftp = (FtpWebRequest)FtpWebRequest.Create(fld);
ftp.Timeout = 20000;
ftp.Credentials = new NetworkCredential(user, pass);
ftp.Method = WebRequestMethods.Ftp.ListDirectory;
using (StreamReader resp = new StreamReader(ftp.GetResponse().GetResponseStream()))
{
String line = resp.ReadLine();
while (line != null)
{
if (!Path.HasExtension(line))
{
String dir = newFiles.First(x => line.EndsWith(x));
newFiles.Remove(dir);
folders.Enqueue(fld + dir + "/");
}
line = resp.ReadLine();
}
}
files.AddRange(from f in newFiles select fld + f);
}
return files.ToArray();
}
catch (Exception e)
{
throw;
}
}
}
}
Is this a proper way of doing dispose? I can't seem to find any documentation on if disposing the response or the response stream, will close the control connection, that the keepalive will keep open.
Is there any way I can log this when running this, so I can see the FTP calls? I don't have access to the FTP server myself.
UPDATE
I have actually tested this with a test FTP and the above code, even though it has KeepAlive=true, it uses new connection on every request. I have created an PoC with Fluent API: https://github.com/robinrodricks/FluentFTP/blob/master/FluentFTP.CSharpExamples/DownloadManyFiles.cs, that does it correctly: connect once, and downloads files. I had a feeling that it was the case with FtpWebRequest. I don't understand the KeepAlive and what it does. Maybe it reuses the TCP connection, which is a good thing in itself, but it uses the TCP connection to login on each request.
Does anybody know if this is even possible with FtpWebRequest, not logging in on each new request, like the Fluent FTP API? Am I missing something?
An example of the FTP log can be seen here
FTP is FileZilla Server 0.9.60 beta.
