ScrapyJS - How to properly wait for page load?

Question

I am using ScrapyJS and Splash to simulate a form submit button click

def start_requests(self):
        script = """
        function main(splash)
            assert(splash:autoload("https://ajax.googleapis.com/ajax/libs/jquery/2.1.3/jquery.min.js"))
            assert(splash:go(splash.args.url))

            local js = [[
                var $j = jQuery.noConflict();
                $j('#USER').val('frankcastle');
                $j('#password').val('punisher');
                $j('.button-oblong-orange.button-orange a').click();
            ]]

            assert(splash:runjs(js))

            local resumeJs = [[
                function main(splash) {
                    var $j = jQuery.noConflict();
                    $j(document).ready(function(){
                        splash.resume();
                    })
                }
            ]]

        assert(splash:wait_for_resume(resumeJs))

            return {
                html = splash:html()
            }
        end
        """
        splash_meta = {'splash': {'endpoint': 'execute', 'args': {'wait': 0.5, 'lua_source': script}}}

        for url in self.start_urls:
            yield scrapy.Request(url, self.after_login, meta=splash_meta)

def after_login(self, response):
        print response.body
        return

After doing splash:runjs(js), ~~I am resorting to splash:wait(5)~~ tried splash:wait_for_resume to get the result. This might not always work ( network latency ), so is there a better way?

Pretty sure splash allows you to wait for an element to be visible — Padraic Cunningham, Apr 04 '16 at 10:40
@PadraicCunningham Please check my edit, tried splash:wait_for_resume ( not sure if I am doing it correctly ) but no luck — Krishnaraj, Apr 04 '16 at 11:15
found a similar issue - https://github.com/scrapinghub/splash/issues/200 — Krishnaraj, Apr 04 '16 at 11:30

score 8 · Accepted Answer · answered Apr 04 '16 at 12:47

It turns out the only way is to use splash:wait() but do it in a loop and check for availability of some element (like footer).

def start_requests(self):
        script = """
        function main(splash)
            assert(splash:autoload("https://ajax.googleapis.com/ajax/libs/jquery/2.1.3/jquery.min.js"))
            assert(splash:go(splash.args.url))

            local js = [[
                var $j = jQuery.noConflict();
                $j('#USER').val('frankcastle');
                $j('#password').val('punisher');
                $j('.button-oblong-orange.button-orange a').click();
                $j('body').empty() // clear body, otherwise the wait_for footer will always be true
            ]]

            assert(splash:runjs(js))

            function wait_for(splash, condition)
                while not condition() do
                    splash:wait(0.05)
                end
            end

            wait_for(splash, function()
                return splash:evaljs("document.querySelector('#footer') != null")
            end)

            return {
                html = splash:html()
            }
        end
        """
        splash_meta = {'splash': {'endpoint': 'execute', 'args': {'wait': 0.5, 'lua_source': script}}}

        for url in self.start_urls:
            yield scrapy.Request(url, self.after_login, meta=splash_meta)

score 0 · Answer 2 · answered Oct 13 '17 at 00:40

So I haven't played with this yet (got to Lua and some successful attempts with Splash only today).

if you do something like this:

recheck = True

html = splash:html()
splash:wait(0.5)
while recheck = True:
    splash:wait(0.5)
    html2 = splash:html()
    if html != html2:
       pass
    elif:
       recheck = False
       return {
          html = splash:html(),
         }

Gonna be using the similar thing for infinite scroll pages that populate list items in response to scrolls (or Page_downs)

Sorry for unfamiliarity with the Lua/Splash syntax

score 0 · Answer 3 · answered Jan 17 '18 at 02:32

There is a bit better way to check for it, but nevertheless you need a loop with waitings. The idea is to use splash:on_response(response) as a callback when page is updated. Note that response callback will be called async so main loop must wait for all page modifyings, that is why we have a 'wait-for' loop (e.g. given by @Krishnaraj).

Below given an example with pressing button button_id 10 times, for downloading additional content.

function main(splash)
    assert(splash:go(splash.args.url))

    function wait_for(splash, condition)
        while not condition() do
            splash:wait(0.2)
        end
    end

    local clicks = 0

    splash:on_response(function(res)
        clicks = clicks + 1

        if clicks < 10 then
            assert(splash:runjs("document.getElementById(\"button_id\").click();"))
        end
    end)

    assert(splash:runjs("document.getElementById(\"button_id\").click();"))

    wait_for(splash, function()
        return clicks >= 10
    end)

    return splash:html()
end

ScrapyJS - How to properly wait for page load?

3 Answers3