C# Retrieve data from a Post with parameters

Question

I know that my question looks like a duplicated question, but I could not find a helpful solution for my issue.

So I am trying to scrape data from a cargo ships data providing website Link (It's a Korean website. The black button on the right is the search button)

but in order to obtain data from it, some radio buttons have to be set up then hit search.

I thought I would be able to just pass parameters values through FormUrlEncodedContent then simply use PostAsync, but somehow I could not be able to get them pass through.

Here is my codes so far

       using (var client = new HttpClient())
       {
            client.DefaultRequestHeaders.TryAddWithoutValidation("User-Agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.99 Safari/537.36");
            client.DefaultRequestHeaders.TryAddWithoutValidation("Content-Type", "application/x-www-form-urlencoded");

            var doc = new HtmlAgilityPack.HtmlDocument();
            var content = new FormUrlEncodedContent(structInfo.ScriptValues);
            var response = await client.PostAsync(structInfo.PageURL, content);
            var responseString = await response.Content.ReadAsStringAsync();
            Console.WriteLine(responseString);
           
       }

        using (WebClient client = new WebClient())
        {
            var reqparm = new System.Collections.Specialized.NameValueCollection();
            reqparm.Add("v_time", "month");
            reqparm.Add("ROCD", "ALL");
            reqparm.Add("ORDER", "item2");
            reqparm.Add("v_gu", "S");
            byte[] responsebytes = client.UploadValues("http://info.bptc.co.kr:9084/content/sw/frame/berth_status_text_frame_sw_kr.jsp", "POST", reqparm);
            string responsebody = Encoding.UTF8.GetString(responsebytes);

            Console.WriteLine(responsebody);
        }

Values I put in the StructInfo Class

            PageURL = "http://info.bptc.co.kr:9084/content/sw/frame/berth_status_text_frame_sw_kr.jsp",
            ScriptValues = new Dictionary<string, string>
            {
                {"v_time", "month"},
                {"ROCD", "ALL"},
                {"ORDER", "item2"},
                {"v_gu", "S"}
            },

What I have tried so far are HttpClient, WebClient, WebBrowser but I had no luck.

But a strange thing is when I try to send a post with Burp Suite, data comes out just fine like the way in I wanted.

I've been searching a solution for last 4 hours, didn't have any luck.

Would you guys mind help me?

Thanks

Answer 1

Generated code for C# - RestSharp by Postman

    var client = new RestClient("http://info.bptc.co.kr:9084/Berth_status_text_servlet_sw_kr");
    client.Timeout = -1;
    var request = new RestRequest(Method.POST);
    request.AddHeader("Content-Type", "application/x-www-form-urlencoded");
    request.AddParameter("v_time", "3days");
    request.AddParameter("ROCD", "ALL");
    request.AddParameter("ORDER", "item2");
    request.AddParameter("v_gu", "S");
    IRestResponse response = client.Execute(request);
    Console.WriteLine(response.Content);

HttpClient version

    using var client = new HttpClient();
    var content = new FormUrlEncodedContent(new[]
    {
        new KeyValuePair<string, string>("v_time", "3days"),
        new KeyValuePair<string, string>("ROCD", "ALL"),
        new KeyValuePair<string, string>("ORDER", "item2"),
        new KeyValuePair<string, string>("v_gu", "S"),
    });
    string url = "http://info.bptc.co.kr:9084/Berth_status_text_servlet_sw_kr";
    var response = await client.PostAsync(url, content);
    var bytes = await response.Content.ReadAsByteArrayAsync();
    string responseString = Encoding.UTF8.GetString(bytes);
    Console.WriteLine(responseString);

The issue

If we talk about the HttpClient version, assuming you are using .net core.

The exception is thrown on ReadAsStringAsync call. More specifically down below: https://github.com/microsoft/referencesource/blob/aaca53b025f41ab638466b1efe569df314f689ea/System/net/System/Net/Http/HttpContent.cs#L95

The response has ContentType: text/html; charset=euc-kr.

And the problem is .net core is not supporting Korean charset out of the box. My workaround is using ReadAsByteArrayAsync instead and then using supported UTF8 encoder later. It screws Korean characters though. The better way would be to reference the System.Text.Encoding.CodePages package and then use Encoding.RegisterProvider. Something like this Encoding.GetEncoding can't work in UWP app