-1

So I want to use beautifulsoup4 to query information about waterloo professors in ratemyprof. The data needs to be taken from a navbar where you continually need to press Load More to load more data.

Furthermore, on inspecting the load more button, it links to a javascript function. How do I write code to continuously load more till it cannot? Here is a snippet of what it looks like upon inspection of button: Load More Inspection

Can someone please help me write the loop to extract info from this page: "http://www.ratemyprofessors.com/search.jsp?queryBy=schoolId&schoolName=University+of+Waterloo&schoolID=1490&queryoption=TEACHER"

Sajid Mahmood
  • 11
  • 3
  • 2
    "_Can someone please help me write the loop_" Yes, but only if you show us what you'd done to solve this problem thus far. Please know that SO is not a code writing service. We help you debug your code, not write it for you. – cs95 Jul 20 '17 at 07:34
  • Hello sorry I'm new to programming in general. https://i.imgur.com/5QgVIgr.png This is as far as I've written. But the output's a bit weird. – Sajid Mahmood Jul 20 '17 at 08:25
  • https://i.imgur.com/hKWOsN0.png This is when I noticed the javascript on the load more button – Sajid Mahmood Jul 20 '17 at 08:26
  • 1
    Please. Information goes in the question, not comments. – cs95 Jul 20 '17 at 08:26
  • so as you can see, it's giving me a template, not what I wanted at all – Sajid Mahmood Jul 20 '17 at 08:39
  • you shouldn't be posting links, you should be updating the question with your actual code. – wpercy Jul 20 '17 at 19:41

1 Answers1

0

It's not easy.

using evaluateJavaScript grab more data which js load.

or selenium may help you.

see also other answer

Jiun Bae
  • 550
  • 3
  • 14