Python Selenium to script Medium rss feed

Question

I'm trying to script some blogs with python and selenium.

However, the source page is limited to a few articles, thus I need to scroll down to load the AJAX..

Is there a way to get the full source in one call with selenium?

The code would be something like:

        # url and page source generating
        url = url_constructor_medium_news(blog_name)
        content = social_data_tools.selenium_page_source_generator(driver, url)
        try:
            # construct soup
            soup = BeautifulSoup(content, "html.parser").rss.channel
            # break condition
            divs = soup.find_all('item')
        except AttributeError as e:
            print(e.__cause__)

        # friendly
        time.sleep(3 + random.randint(1, 5))

Answer 1

I don't believe there is a way to populate the driver with unloaded data that would otherwise be obtained by scrolling.

An alternative solution to getting the data would be driver.execute_script("windows.scrollTo(0, document.body.scrollHeight);")

I've previously used this as a reference.

I hope this helps!