Selenium Alternatives for Google Cloud VM instance?

Question

Are there any alternatives to Selenium that don't require a web driver or browser to operate? I recently moved my code over to a Google Cloud VM instance, and when I run it there are multiple errors. I've been trying to get it to work for hours but just can't (no luck with PhantomJS, Chrome and GeckoDriver - tried re-downloading browsers, editing the sources.list file e.c.t.).

The page I'm web scraping uses JavaScript to load in numbers, which I was I initially chose Selenium. Everything else works perfectly though!

Answer 1

You could simply use the request library.

https://requests.readthedocs.io/en/master/

https://anaconda.org/anaconda/requests

You would then need to send a GET or POST request to the server.

If you do not know how to generate a proper POST request, simply try to "record" it.

If you have chrome, got to the page you want to navigate, press F12, navigate to the "Network" section and write method:POST into the filter.

Further info here: https://stackoverflow.com/a/39661536/11971785

At first it is a bit more confusing than selenium, but once you understand it its waaaay better in my opinion.

Also the Java values shown on the page can usually be simply read out of the java code which is returned by your request.

No web driver or anything required and a lot more stable and customizable.