0

I'm using Scrapy-splash but this question is about Lua script. I need to wait until one element is visible. The problem is that this element can be navigated only using it's text.

The equivalent XPATH is: //*[@id="tab-nav-main"]//span[text()="desc"]')

I tried:

#tab-nav-main span:contains(desc)

which works in chrome but not in lua

What would you do?

function main(splash, args)
            splash:set_user_agent(args.ua)
            assert(splash:go(splash.args.url))

            local i=0
            local maxwait=5

            while not splash:select("#tab-nav-main span:contains(description)") do
                if i==maxwait then
                    break     --times out at maxwait secs
                end
                i=i+1
                splash:wait(1)      --each loop has duration 1sec
            end
            return {
                html = splash:html(),
                }
        end

ERROR

2019-03-14 17:07:05 [scrapy_splash.middleware] WARNING: Bad request to Splash: {'description': 'Error happened while executing Lua script', 'error': 400, 'info': {'splash_method': 'select', 'line_number': 9, 'error': 'cannot select the specified element {\'js_error\': \'Error: SyntaxError: DOM Exception 12\', \'js_error_type\': \'SyntaxError\', \'js_error_message\': \'SyntaxError: DOM Exception 12\', \'type\': \'JS_ERROR\', \'message\': "JS error: \'Error: SyntaxError: DOM Exception 12\'"}', 'source': '[string "..."]', 'message': '[string "..."]:9: cannot select the specified element {\'js_error\': \'Error: SyntaxError: DOM Exception 12\', \'js_error_type\': \'SyntaxError\', \'js_error_message\': \'SyntaxError: DOM Exception 12\', \'type\': \'JS_ERROR\', \'message\': "JS error: \'Error: SyntaxError: DOM Exception 12\'"}', 'type': 'SPLASH_LUA_ERROR'}, 'type': 'ScriptError'}
Milano
  • 18,048
  • 37
  • 153
  • 353
  • `:contains` is not actually a CSS selector (see: https://stackoverflow.com/questions/1520429/is-there-a-css-selector-for-elements-containing-certain-text). Can you post the real URL? If the element has no other identifiers, does it at least have a fixed position so you can use an index? – malberts Mar 14 '19 at 16:35
  • are you sure the css worked in chrome with `:contains()`? – supputuri Mar 14 '19 at 17:04
  • I've figured out another way but I'm still curious if there is a way to do this. @supputuri Yes (Ubuntu 18.04 - Chrome). Try to put this into the console $('span:contains(worked in chrome)').text() - maybe it works because of JQuery... not sure... – Milano Mar 14 '19 at 18:21
  • What is that another way? And I am getting the following error when tried in windows10, chrome console `Uncaught DOMException: Failed to execute '$' on 'CommandLineAPI': 'span:contains(Text)' is not a valid selector. at :1:1`. – supputuri Mar 14 '19 at 18:26
  • Yes, `:contains()` only works when JQuery is present (e.g. here on SO). Does your other way involve using an index or nth-child? – malberts Mar 14 '19 at 20:45
  • @malberts Ok, no index no child, I just found another element which is present only when this element is present too so I can use any of them to stop waiting... – Milano Mar 15 '19 at 01:20

0 Answers0