2

I'm trying to extract information from Goodreads. The problem is if I go into a url like:

https://www.goodreads.com/shelf/show/programming?page=2

with Selenium chrome webdriver or with BeautifulSoup, it still shows the first page, instead of the second one.
Example with the chrome webdriver:

Example with the chrome webdriver

While on a normal browser, it displays those books instead:

enter image description here

Mohamed Oun
  • 561
  • 1
  • 9
  • 24

1 Answers1

1

Looks like that happen because you're not logged-in in your selenium session, you will have to login and save the cookies between restarts.

Take a look at this stackoverflow answers to understand how to extract cookies.

Or Duan
  • 13,142
  • 6
  • 60
  • 65