C# WebBrowser: cannot get latest web page HTML after HTTP POST?

Question

I have the following:

   protected static void GetWebPageWorker()
   {
       using (WebBrowser browser = new WebBrowser())
       {
           //  browser.ClientSize = new Size(_width, _height);
           browser.ScrollBarsEnabled = false;
           browser.ScriptErrorsSuppressed = true;
           browser.Navigate(_url);

           // Wait for control to load page
           while (browser.ReadyState != WebBrowserReadyState.Complete)
               Application.DoEvents();

           //Insert the search term in to the input textfield
           browser.Document.GetElementById(search_div_id).OuterText = search_term;

           //Click on the search button (POST request)
           browser.Document.GetElementById(search_button_id).InvokeMember("Click");

           //This didn't work
           //browser.Update();

           while (browser.ReadyState != WebBrowserReadyState.Complete)
               Application.DoEvents();

           //PROBLEM this is returning the original HTML before POST
           //and not the HTML from the web page after the POST.
           html = browser.DocumentText;
       }
   }

it inserts a value to search for in a HTML element and then clicks the search button and the POST is done. However the WebBrowser object doesn't seem to refresh its DocumentText property so that I can return the HTML of the page after the search button has been clicked.

How can I fix this?

Checks this answer and the links listed there: http://stackoverflow.com/a/22262976/1768303. On a side note, consider to stop spinning a busy-waiting loop with `DoEvents`, it usually does nothing but troubles. — noseratio, May 13 '14 at 03:08
So I need webBrowser.DocumentCompleted += handler;, then to wait and then have the handler decide when to stop the waiting and progress, collecting the new HTML? — user997112, May 13 '14 at 03:33
It's more than that. The page might have some dynamic AJAX logic, in which case you need some kind of polling, and even then you might not be able to capture what you want. But you can get close, the code I linked does that. — noseratio, May 13 '14 at 04:21
Ok have taken your code- where would you recommend I enter my search term and click the search button? within NavigateAsync()? — user997112, May 13 '14 at 14:56
I recommend doing it like this, look for `InvokeMember` there: http://stackoverflow.com/a/19063643/1768303 — noseratio, May 13 '14 at 21:19

score 0 · Accepted Answer · answered May 13 '14 at 01:59

WebBrowser is most likely the wrong tool for this task. You would use WebBrowser when you want to actually be able to see the web page on the screen. It looks like you are instead just trying to POST something and get the result.

Try WebClient instead, it is much simplier and works much better for POST:

using (var client = new WebClient())
{
    client.Headers["User-Agent"] = "Mozilla/4.0 (Compatible; Windows NT 5.1; 
           MSIE 6.0) (compatible; MSIE 6.0; Windows NT 5.1; 
          .NET CLR 1.1.4322; .NET CLR 2.0.50727)";

    const string url = "http://www.example.com";    

     var inputs = new NameValueCollection {
         {"search_div_id", search_term},
     }

    var response = client.UploadValues(url, inputs);
    string html = Encoding.UTF8.GetString(response);
}

*"WebBrowser is most likely the wrong tool for this task."* - it depends on the requirements and the final goal. `WebClient` doesn't execute JavaScript and may even end up with a completed different page delivered, if the server checks User Agent string. — noseratio, May 13 '14 at 03:03
WebClient is a pain because the POST request isn't entirely simple and then I have to go and replicate every single parameter and debugging is a pain. I just want to effectively automate a web browser: enter the text and then click submit. — user997112, May 13 '14 at 03:30

score 0 · Answer 2 · answered Jun 22 '18 at 12:04

Please use the following Statements after Invoking Click Member Function. It will surely work.

`HtmlElement hleGetData = (HtmlElement)hdoc.GetElementById("getButton");
hleGetData.InvokeMember("click");
while (webBrowser1.ReadyState != WebBrowserReadyState.Complete)
{
    System.Windows.Forms.Application.DoEvents();
};
//Some DoEvents....
System.Windows.Forms.Application.DoEvents();
System.Windows.Forms.Application.DoEvents();
System.Windows.Forms.Application.DoEvents();
System.Windows.Forms.Application.DoEvents();
System.Windows.Forms.Application.DoEvents();`

Łukasz Włodarczyk · Answer 3 · 2020-10-14T09:22:15.377

webBrowser1.Refresh(); and webBrowser1.Update(); don't work after POST request.

while (webBrowser1.ReadyState != WebBrowserReadyState.Complete)
        {
            System.Windows.Forms.Application.DoEvents();
           //System.Threading.Thread.Sleep(500);
        }

don`t work also.

Only way at the moment is to use webBrowser1.Navigate("URL"); once again after every POST request.

Microsoft Visual Studio Community 2019 Version 16.7.5 VisualStudio.16.Release/16.7.5+30523.141 Microsoft .NET Framework Version 4.8.03752

C# WebBrowser: cannot get latest web page HTML after HTTP POST?

3 Answers3