2

I know about using /execute endpoint and passing LUA script to it which will load the page and scroll it via executing JavaScript code. However, this is "one and off" type of action, where I have no fine control about how or when to scroll the page within my Python program.

In other words, once Splash loads the page, additional actions on that page (e.g. executing JavaScript code) cannot be performed. I'd like to be able to interact with current page session within my Python program, Selenium-style. Is this possible in Splash, and how?

Alternative is to subclass QWebEnginePage and build a browser on top of it, but this is a whole new world of work which I'd like to avoid if possible.

rsrx
  • 1,433
  • 1
  • 13
  • 25
  • What if you first created a [javascript profile](https://splash.readthedocs.io/en/stable/api.html#javascript-profiles) that places both an `iframe` and a script that interacts with that `iframe` in your page and allows you to execute whatever you send to that `iframe`? Then setup a local server that sends your interaction to your website etc... (it's possible to [deactivate security features](https://splash.readthedocs.io/en/stable/api.html#javascript-security)) that usually prevent you from looking inside an `iframe`.... – Nearoo Oct 31 '18 at 00:06
  • @Nearoo how will this let me execute JavaScript code on an existing page? iiuc, your suggestion will just redirect the problem to my custom page than the real one. – rsrx Nov 05 '18 at 00:14
  • No, not if I understood the docs correctly. It's a good chunk of work though. I just meant that you could create a javascript profile that attaches an iframe to the dom of the external website which loads code from _your_ server that repeatedly fetches code that repeatedly fetches _more_ code from your local server using ajax, so you can interact with the website. Abstracted sufficiently this could be a simple `server.init()`, `server.send_js_code(code)`. Just an idea off the top of my head in case you're really desperate for a solution. Might be easier to edit the source for scrapy though. – Nearoo Nov 05 '18 at 17:01
  • Does this answer your question? [how does scrapy-splash handle infinite scrolling?](https://stackoverflow.com/questions/40325657/how-does-scrapy-splash-handle-infinite-scrolling) – Gallaecio Nov 26 '19 at 12:28

0 Answers0