Waiting for javascript-rendered page loading in CefSharp

Question

I started to learn CefSharp. It is a convinient tool for web scraping. But I have a problem. I want to wait for a page to be loaded. But the page loading is completed by ajax. And I'm waiting for some html tags. I wrote like this.

public ChromiumWebBrowser brser;
Thread main_thread;
public Form1()
{
    brser = new ChromiumWebBrowser("some_url_here");
    panelBrowser.Controls.Add(brser);
    brser.Dock = DockStyle.Fill;
    main_thread = new Thread(mainStart);
    main_thread.Start();
}
private void mainStart()
{
    brser.LoadingStateChanged += LoadingStateChanged;
    brser.Load("other_url_here");
}
private async void LoadingStateChanged(object sender, LoadingStateChangedEventArgs e)
{
//... Omitted Code ...
    Context.LogInfo("[LOG]\t----");
    frame = brser.GetBrowser().GetFrame("iframe1");
    if(frame == null)
    {
        Context.LogInfo("[LOG]\tiframe1 is null. return");
        return;
    }
    int nretry = 100;
    try
    {
        while (nretry > 0)
        {
            html = await Task.Run(frame.GetSourceAsync);
            if (html.Contains("<ul class=\"some_class_to_be_loaded\">"))
            {
                break;
            }
            nretry--;
            Thread.Sleep(100);
        }
        if (nretry == 0)
        {
            MessageBox.Show("Connection Error");
            try
            {
                main_thread.Abort();
                return;
            }
            catch (Exception ex)
            {
                return;
            }
        }
    }
    catch
    {
        return;
    }
}

But it does not work. I debugged it. But when the thread sleeps, the page loading sleeps too. As a result, waiting 'loading complete event' failed. Please help me.

What is it you are actually trying to achieve? Are you just after the html source after your Ajax operation has finished? — amaitland, Dec 09 '19 at 21:37

score 0 · Accepted Answer · answered Dec 09 '19 at 18:33

0

You can wrap your waiting part by delegate. Like this.

Invoke(new Action(async delegate ()
{
//waiting code
    int nretry = 100;
    while (nretry > 0){
//...
        Thread.Sleep(100);
    }
}

answered Dec 09 '19 at 18:33

devcrazy

505
5
14

I don't understand. Where should I put this snippet? – glinda93 Dec 09 '19 at 18:34
1

You can put your `while {...}` waiting part into the delegate function. – devcrazy Dec 09 '19 at 18:54
I'm surprised this works, sleeping on the UI thread is asking for trouble. You would be blocking the calling thread as well as you are using Invoke which is a synchronous operation. – amaitland Dec 09 '19 at 21:36
Nice to meet you. I know you are a great CefSharp contributor. In my answer, i think thread sleeping is not on UI thread. `brser.Load` is async function, isn't it? @amaitland – devcrazy Dec 10 '19 at 05:57
Your code suggests calling `Invoke` which will invoke on the `UI` thread in a `sync` fashion. `Load` is a non blocking function, has no bearing on the behaviour in this instance. Using `await Task.Delay(100);` would be a slight improvement. https://learn.microsoft.com/en-us/dotnet/api/system.threading.tasks.task.delay?view=netframework-4.8 Overall the polling for a piece of html to appear and searching through a potentially large block of `html` every `100ms` is far far far from ideal in my opinion. – amaitland Dec 10 '19 at 06:09
It helps me. Thank you. But I mean `Invoke` is called in `LoadingStateChanged`. I test and it worked. Maybe it looks boring but it worked. And i guess, the asker wants to wait dozens of seconds and `100ms` is not enough for him. @amaitland – devcrazy Dec 10 '19 at 06:39
Calling `Thread.Sleep` in the `Invoke` method will sleep the `UI Thread` and because `Invoke` is blocking you are blocking the thread that called `Invoke`. Using `BeginInvoke` will at least stop the calling thread from being blocked, and `await Task.Delay` will not put your applications `UI` to sleep. – amaitland Dec 10 '19 at 06:55
@bravemaster modifying your original code and replacing Thread.Sleep with await Task.Delay would at least avoid blocking one of the threads. This answer may work it, though will have unexpected side effects. Calling Thread.Sleep on the UI thread will make your application less responsive to user input and potentially unresponsive. – amaitland Dec 12 '19 at 22:13
@amaitland Thank you for your suggestion. Task.Delay may be a workaround of Thread.Sleep. Btw https://stackoverflow.com/questions/58241504/cefsharp-winfoms-page-load-and-wait/59276733#59276733 doesn't seem to work. – glinda93 Dec 12 '19 at 22:48
@bravemaster I'd suggest you read some of the answers at https://stackoverflow.com/questions/20082221/when-to-use-task-delay-when-to-use-thread-sleep to get a better understanding. This is not the place to discuss a answer to a totally unrelated question. – amaitland Dec 12 '19 at 22:54
Ask a new question if you have a problem you need help with. – amaitland Dec 13 '19 at 02:03

Waiting for javascript-rendered page loading in CefSharp

1 Answers1