list files recursive from a webserver

Question

I currently try to implement a simple webdownloader, which downloads files recursive throughout the only directory.

What i got to list the files on the server: Updater.cs:

    public static List<string> remote_filecheck()
    {
        List<string> rfiles = new List<string>();
        string url = "http://********/patchlist.txt";
        WebClient client = new WebClient();
        client.DownloadFile(url, @"patchlist.txt");

        string line;
        StreamReader reader = new StreamReader("patchlist.txt");

        while ((line = reader.ReadLine()) != null)
        {
            rfiles.Add(line);
        }
        reader.Close();
        return rfiles;
    }

I currently work with a patchlist, which consists of all direct links to my http files.

I tried nearly every single snippet on the web concerning recursive download e.g. RegEx, WebRequests and stuff.

Now i want to know if you got a good way to go recursive through my HTTP Server and list all the filenames, which is all i want to know.

When i have a List<string> of filenames, then i am able to do the rest.

Answer 1

Has the server that you are trying to get the files from got indexing switched on?

If so then it's probably a matter of scraping this page that comes back and then visiting each url one by one.

If not then I'm not sure it can be done very easily.

Ok based on comments below I think you'll want to do something like this:

        string indexUrl = "http://www.stackoverflow.com";

        WebBrowser browser = new WebBrowser();
        browser.Navigate(indexUrl);

        do
        {
            Application.DoEvents();
        } while (browser.ReadyState != WebBrowserReadyState.Complete);



        var listOfFilePaths = new List<string>();


        foreach (HtmlElement linkElement in browser.Document.GetElementsByTagName("a"))
        {
            var pagePath = linkElement.GetAttribute("href");
            listOfFilePaths.Add(pagePath);
        }

Note that the WebBrowser control needs to be run in a Windows forms app to get it work (easily). The indexPath variable I used should be changed to the path of the index page of the server (I just used stackoverflow as an example).

The foreach loop extracts all anchor (a) tags out of the site and gets the path they are pointing to and adds them to the listOfFilePaths collection.

Once this code has finished executing the listOfFilePaths collection will contain an entry for every link on the index page and hence a link to every file on the server.

From here it's a matter of looping round the listOfFilePaths collection and downloading each file one by one. Perhaps even using some rules not to download certain types of files that you're not interested in. I believe from what you've said you should be able to do this.

Hope this helps.