How to crawl IMDB? [Read more] button not being pressed

Question

I'm Extracting IMDB movie reviews.

There is a problem To bring up the movie reviews, The [read-more] button must be pressed.

But after the review is over, I don't know how to end this.

It is currently being handled in a 'polling' way. How can you handle this more intelligently?

when there is more to read:

when there is nothing more to read:

Thank you!

Answer 1

If you are doing it in Python, you can use xpath to extract the xpath from html page Example of retrieving reviews is given below. You can use try except case so that if if there is no information in the page, loop will end. Look at below example, it might help you - -

reviews = driver.find_elements_by_xpath('//article[@itemprop = "review"]')
            for review in reviews:

                # Initialize an empty dictionary for each review
                review_dict = {}

                # Find xpaths of the fields desired as columns in future data frame
                # We use the try/except statements to account for the fact that the reviews are not required to have
                # all the fields listed below, and if a review does not have a certain field we wish to make the
                # corresponding field blank in that particular row, rather than quit upon receiving an error.
                try:
                    airline = review.find_element_by_xpath(
                        '//div[@class = "review-heading"]//h1[@itemprop = "name"]').text
                except:
                    airline = page
                try:
                    overall = review.find_element_by_xpath('.//span[@itemprop = "ratingValue"]').text
                except:
                    overall = ""

In same way, you can use xpath element for your IMDB case and use try except so that no error pops up if there is nothing to read.