running selenium on python gae?

Question

I have a flask app with a endpoint that uses selenium and chrome driver.I use this code to scrape webpages of a angular website,create a json and serve it to the client.

Earlier,this site was easily scraped by using BeautifulSoup and I used to store the json in datastore along with current time of the put operation.This was done so that I don't scrape the website with every client request and a function made sure that it has been good 5 hours before the website is scraped again.

But now I have to use selenium and I cant think how selenium can open browser on the server(as it needs on my local machine to do anything).I also researched about headless state of chrome but currently it works only for node.js server.

The only option I am seeing right now is to scrape the site on my local machine and upload json to the gae datastore every time new data is produced(which is precise day of the month) on the website. Is there a way to completely automate the process?

Alex · Answer 1 · 2018-12-27T16:52:23.440

0

I was able to get headless selenium running on GAE python, but I had to do it in App Engine Flex. See this answer:

Python Headless Browser for GAE

What do you mean by "it needs on my local machine to do anything"?

edited Dec 27 '18 at 16:52

answered Dec 27 '18 at 16:44

Alex

5,141
12
26

Thanks but I dont have any docker file. – code_tinkerer Jan 01 '19 at 06:54
I was not able to do this in the standard environment. I setup a custom run time. https://cloud.google.com/appengine/docs/flexible/custom-runtimes/ – Alex Jan 01 '19 at 16:43

running selenium on python gae?

1 Answers1