0

I'm trying to get the HTML source as string from the following website:

https://physics.nist.gov/cgi-bin/cuu/CCValue?mal|ShowSecond=Browse&First=malc2

But when I'm using WebClient in C#:

ServicePointManager.Expect100Continue = true;
ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls12;
WebClient client = new WebClient();
string s = client.DownloadString("https://physics.nist.gov/cgi-bin/cuu/CCValue?mal|ShowSecond=Browse&First=malc2");

I'm getting back you have supplied an ill-formed request from the website and not the correlation coefficient I'm looking for.

I took as well a look at the following question How to post data to specific URL using WebClient in C# but it didn't helped me.

I think the problem has to do with the | character in the URL. If you click on the link above it doesn't work because the | is replaced by %7C.

System.Diagnostics.Process.Start(url) opens the correct page in a browser.

See https://dotnetfiddle.net/wAYWEu

Update

Using Wireshark and switching to http I can see that the | character gets replaced by %7C.

144 2.241432    10.131.36.167   195.176.26.193  HTTP    253 GET http://physics.nist.gov/cgi-bin/cuu/CCValue?mal%7CShowSecond=Browse&First=malc2 HTTP/1.1

How can I avoid that?

Wollmich
  • 1,616
  • 1
  • 18
  • 46
  • what options you selected in the form that gives that url? – tttony Jun 22 '22 at 12:03
  • https://physics.nist.gov/cuu/Correlations/index.html then `alpha particle mass energy equivalent` and `alpha particle mass`. – Wollmich Jun 22 '22 at 12:08
  • There's a difference between the `UploadString` referred to in the question you linked to and the `DownloadString` method you're calling in your code. – phuzi Jun 22 '22 at 12:16
  • @phuzi I tried it as well with `UploadString` and separating the URL and parameters. No success. – Wollmich Jun 22 '22 at 12:17
  • 1
    @Wollmich You should use [HttpClient](https://learn.microsoft.com/en-us/dotnet/api/system.net.http.httpclient?view=net-6.0) nowdays instead of WebClient. Even Microsoft recommends [this](https://learn.microsoft.com/en-us/dotnet/api/system.net.webclient?view=net-6.0#remarks). Anyway, I think you are out of luck, it seems it is the service that is not doing a proper job handling the querystring-parameters. The request https://physics.nist.gov/cgi-bin/cuu/CCValue?mal%7CShowSecond=Browse&First=malc2 should work if they did a proper decoding on their end. – Esko Jun 23 '22 at 05:54

0 Answers0