How to fetch Text Block from dynamic SSL page via c# only

Question

In my C# code, I want to use only server-side code to fetch the content of the page here:

https://1962ordo.today

Problem is, that the dynamic part is appended upon page load, so today it looks like this, but tomorrow it will look different.

https://1962ordo.today/day/sts-cosmas-and-damian-2-2-4/

I want to fetch the text only of the container tag, and nothing else, but would like to keep the <br> tags in place.

I've tried using a streamreader with SSL capabilities, but it doesn't seem to work correctly.

In the end, I need a simple html string that I can load into my page.

So far, this is what I've come up with; it tries to load the content, but keeps trying to load and never resolves (I have it in the asp.net page_load part):

protected void Page_Load(object sender, EventArgs e) {

    string ordourl;
    ordourl = "https://1962ordo.today";

    ServicePointManager.SecurityProtocol = (SecurityProtocolType)3072;

    using (WebClient webClient = new WebClient()) {
        webClient.Headers["User-Agent"] = "Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.9.2.6) Gecko/20100625 Firefox/3.6.6 (.NET CLR 3.5.30729)";
        webClient.Headers["Accept"] = "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8";
        webClient.Headers["Accept-Language"] = "en-us,en;q=0.5";
        webClient.Headers["Accept-Encoding"] = "gzip,deflate";
        webClient.Headers["Accept-Charset"] = "ISO-8859-1,utf-8;q=0.7,*;q=0.7";
        StreamReader sr = new StreamReader(webClient.OpenRead(ordourl));
        string results = sr.ReadToEnd();
        div_ordo.InnerHtml = results;
    };

}

If I use InnerText instead of InnerHtml, it loads into the page's <div> the underlying html, but amoungst that, none of it is the actual data for the day. The Streamreader isn't getting the data which is the result of the actual page's final load. It's just getting the actual page's html. I need the code to wait for the database info to be loaded into that page and display the final results, not the initial page.

I've tried this, and it does exactly the same thing:

    protected void Page_Load(object sender, EventArgs e) {


        string ordourl;
        ordourl = "https://1962ordo.today";
        string ordotext;
        ordotext = "";

        ServicePointManager.SecurityProtocol = (SecurityProtocolType)3072;

        WebRequest request = WebRequest.Create(ordourl);
        WebResponse response = request.GetResponse();
        using (StreamReader reader = new StreamReader(response.GetResponseStream())) {
            ordotext = reader.ReadToEnd();
            div_ordo.InnerText = ordotext;
            reader.Close();

            }
        response.Close();
}

What's happening within the Strip() method? I tested your code and it works without hanging but I don't know what the Strip() method logic is doing. — quaabaam, Aug 21 '19 at 23:10
oh, that... actually, that was accidentally posted. I was trying to strip some html and display just the text that got fetched. I'll remove it from my posted code. You say yours works? Mine just keeps trying to load and load and the page keeps flashing and no text ever appears - just the image. It seems like it doesn't wait for the data to load - I have to use the debugger, I think. — bgmCoder, Aug 21 '19 at 23:49
The code you posted works fine for me. Are you sure your client is actually connecting to the server and not getting hung-up trying to start the connection? You can use something like Fiddler to see the traffic and the response to determine if you are in fact making the connection. — quaabaam, Aug 22 '19 at 00:54
Since your're trying to do this from within a webpage.. .check this out... https://stackoverflow.com/questions/302775/call-a-webpage-from-c-sharp-in-code — quaabaam, Aug 22 '19 at 00:57
Actually, I have it working. I figured out a url parameter and can specify a date. If I do that, it load perfectly. Now I just have to filter out the correct html section for display. — bgmCoder, Aug 22 '19 at 01:01
@quaabaam How can you put asp.net coed in fiddler? This is all server side code, not javascript. — bgmCoder, Aug 23 '19 at 03:57
This fiddler, https://www.telerik.com/fiddler, it's a known and free tool that allows you to see the communication between your application and the server. So you would see the call from your app.. the wait and maybe see where the waiting happens — quaabaam, Aug 23 '19 at 16:15

score 0 · Answer 1 · answered Aug 22 '19 at 14:23

Now, I think the webclient couldn't deal with the page-load itself, but I figured out how to load the current day via url parameters, and now it works perfectly. Just grab today's date and custom format it and add it to the query string.

        string ordohtml = "Ordo here...";
        string ordotext = "";

        string ordodate;
        DateTime dt = new DateTime(System.DateTime.Now.Year, System.DateTime.Now.Month, System.DateTime.Now.Day);
        ordodate = DateTime.Now.ToString("yyyyMMd");

        string ordourl = "https://1962ordo.today?date=" + ordodate;

        ServicePointManager.SecurityProtocol = (SecurityProtocolType)3072;
        using (WebClient webClient = new WebClient()) {
            webClient.Headers["User-Agent"] = "Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.9.2.6) Gecko/20100625 Firefox/3.6.6 (.NET CLR 3.5.30729)";
            webClient.Headers["Accept"] = "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8";
            webClient.Headers["Accept-Language"] = "en-us,en;q=0.5";
            webClient.Headers["Accept-Encoding"] = "gzip,deflate";
            webClient.Headers["Accept-Charset"] = "ISO-8859-1,utf-8;q=0.7,*;q=0.7";
            StreamReader sr = new StreamReader(webClient.OpenRead(ordourl));
            ordohtml = sr.ReadToEnd();
            div_ordo = ordohtml.InnerHtml;

        };

How to fetch Text Block from dynamic SSL page via c# only

1 Answers1