Why does the WebClient.DownloadFile method not work and say "JSON not found" while the URL is working using a Web browser?

Question

I am trying to download a file with C# using this code:

using (var client = new WebClient())
{
    System.Net.ServicePointManager.SecurityProtocol = System.Net.SecurityProtocolType.Tls12;
    string downloadUrl = "https://data.3dbag.nl/cityjson/v21031_7425c21b/3dbag_v21031_7425c21b_8044.json";
    string destinationFile = "test.json";
    client.DownloadFile(downloadUrl, destinationFile);
}

An example url is this: https://data.3dbag.nl/cityjson/v21031_7425c21b/3dbag_v21031_7425c21b_8044.json

This URL is working in my browser, but in C# I'm getting an (404) Not Found error. Does anyone know how to fix this?

Wild guess: the server expects a User-Agent header. Anyway, there's a simple solution: use a tool like Telerik's Fiddler to compare the raw request made by your browser and that made by C#. — ProgrammingLlama, Aug 12 '21 at 09:35
I quickly played around with "Postman" and noticed that the header field "Accept-Encoding" with the value "gzip" is required. --> client.Headers.Add("Accept-Encoding", "gzip"); — MartinM43, Aug 12 '21 at 09:39
Press F12 in your browser right now.. Then click on the link. Then look in the "Network" tab on the right at "headers" -> "request headers" (and maybe, in future, cookies esp if its a resource available after youre logged in). That "request headers" list is the list of things that your browser sends that makes the server respond successfully i.e. "what works", so start out by replicating that.. Use a program like Insomnia to issue an identical request, then delete headers one by one til it stops, put the one back and resume deleting otehrs to get to the minimum (if you care) — Caius Jard, Aug 12 '21 at 09:39

score 1 · Accepted Answer · edited Aug 12 '21 at 09:53

1

It looks like this server requires the header Accept-Encoding: gzip:

using (var client = new WebClient())
{
    System.Net.ServicePointManager.SecurityProtocol = System.Net.SecurityProtocolType.Tls12;
    string downloadUrl = "https://data.3dbag.nl/cityjson/v21031_7425c21b/3dbag_v21031_7425c21b_8044.json";
    string destinationFile = "test.json";
    client.Headers.Add("Accept-Encoding", "gzip"); //<-- add Header!
    client.DownloadFile(downloadUrl, destinationFile);
}

You will get an "compressed response".

So you will have to decompress the response!

Compression/Decompression string with C#

edited Aug 12 '21 at 09:53

answered Aug 12 '21 at 09:49

MartinM43

101
1
1
8

2

Thanks, adding this header does the trick! – gleeuwdrent Aug 12 '21 at 09:56
1

There are much better ways to decompress the result: https://stackoverflow.com/questions/2973208/automatically-decompress-gzip-response-via-webclient-downloaddata – canton7 Aug 12 '21 at 10:38

Why does the WebClient.DownloadFile method not work and say "JSON not found" while the URL is working using a Web browser?

1 Answers1