C# downloading a csv file from https site

Question

I am trying to download a file into my app. From what I can see if I put this link into a browser I get a good file downloaded so I know the file is available for me to get. I have other files to download from different sites and they all work well but this one will not download for me in a usable manner. I guess I do not understand something, do you know the key fact I am missing?

After many attempts I have now coded the following

 private string url =
        "https://coronavirus.data.gov.uk/api/v1/data?filters=areaType=nation&structure={\"Name\":\"areaName\",\"date\":\"date\",\"FirstDose\":\"cumPeopleVaccinatedFirstDoseByPublishDate\",\"SecondDose\":\"cumPeopleVaccinatedSecondDoseByPublishDate\"}&format=csv";

    private void btn_wc1_Click(object sender, EventArgs e)
    {
        WebClient wc = new WebClient();

        wc.Encoding = Encoding.UTF8;
        wc.DownloadFile(url, "wc1_uk_data.csv");
    }

    private void btn_wc2_Click(object sender, EventArgs e)
    {
        using (var webClient = new WebClient())
         {
             webClient.Encoding = Encoding.UTF8;
             string s = webClient.DownloadString(url);
             File.WriteAllText(@"wc2_uk_data.csv", s);
         }
    }

    private async void btn_https_Click(object sender, EventArgs e)
    {
        HttpClient _client = new HttpClient();
        byte[] buffer = null;
        try
        {
            HttpResponseMessage task = await _client.GetAsync(url);
            Stream task2 = await task.Content.ReadAsStreamAsync();
            using (MemoryStream ms = new MemoryStream())
            {
                await task2.CopyToAsync(ms);
                buffer = ms.ToArray();
            }
            File.WriteAllBytes("https_uk_data.csv", buffer);
        }
        catch
        {

        }
    }

    private void btn_wc3_Click(object sender, EventArgs e)
    {
        using (var webClient = new WebClient())
        {
            webClient.Encoding = Encoding.UTF8;
            byte[] myDataBuffer = webClient.DownloadData(url);
            MemoryStream ms = new MemoryStream(myDataBuffer);
            FileStream f = new FileStream(Path.GetFullPath(Path.Combine(Application.StartupPath, "wc3_uk_data.csv")),
            FileMode.OpenOrCreate);
            ms.WriteTo(f);
            f.Close();
            ms.Close();
        }
    }

Using the following UI

All the different functions above will download a file but none of the downloaded files is usable. It seems like it is not the file but maybe something to do with information regarding the file. As my app does not know what to do with this I never reply with what ever the other end wants. I guess if I replied the next set of data I got would be the file.

If I put the URL into a browser then I get a file that is good. This link is good at the moment.

https://coronavirus.data.gov.uk/api/v1/data?filters=areaType=nation&structure={"Name":"areaName","date":"date","FirstDose":"cumPeopleVaccinatedFirstDoseByPublishDate","SecondDose":"cumPeopleVaccinatedSecondDoseByPublishDate"}&format=csv

Anyone got any idea on what I need to do in my app to get the file like the browser does?

So https://learn.microsoft.com/en-us/dotnet/api/system.net.webclient.downloadfile?view=net-5.0 is not working? — Tony Stark, Jan 11 '21 at 18:19
Does this answer your question? [Download Csv file with WebClient in C# but the size of file is less than when download with browser](https://stackoverflow.com/questions/59594169/download-csv-file-with-webclient-in-c-sharp-but-the-size-of-file-is-less-than-wh) — user1672994, Jan 11 '21 at 18:28
WebClient uses `Encoding.Default` if no other encoding is specified. Unfortunately, you have to specify the Encoding yourself. Most of the time, since it's a Web resource, the Encoding will be UTF-8, but it's not necessarily so. If you use the HttpClient class instead, the Encoding is *auto-detected*, so you don't need to provide one beforehand, hoping it's the right one. If you **have to** use WebClient, for some reason, you can do something as described here: [Kanji characters from WebClient html different from actual Kanji in website](https://stackoverflow.com/a/49848091/7444103) — Jimi, Jan 11 '21 at 20:02

score 0 · Answer 1 · answered Jan 11 '21 at 19:26

0

You need to set the WebClient.Encoding before calling DownloadString

string url = "https://coronavirus.data.gov.uk/api/v1/data?filters=areaType=nation&structure={\"Name\":\"areaName\",\"date\":\"date\",\"FirstDose\":\"newPeopleReceivingFirstDose\",\"SecondDose\":\"newPeopleReceivingSecondDose\"}&format=csv";
using (var webClient = new WebClient())
{
    webClient.Encoding = Encoding.UTF8;
    string s = webClient.DownloadString(url);
}

Here is a related question: WebClient.DownloadString results in mangled characters due to encoding issues, but the browser is OK

answered Jan 11 '21 at 19:26

Ryan

7,835
2
29
36

Thanks but still no good. I tried the link https://coronavirus.data.gov.uk/api/v1/data?filters=areaType=nation&structure={"Name":"areaName","date":"date","FirstDose":"cumPeopleVaccinatedFirstDoseByPublishDate","SecondDose":"cumPeopleVaccinatedSecondDoseByPublishDate"}&format=csv but is still does not bring back the csv file. I am not really sure what it brings back as it is unreadable as any UTF or Unicode. – user3884423 Jan 11 '21 at 22:25

C# downloading a csv file from https site

1 Answers1