0

I'm trying to scrap some monster infobox table from rswiki.

Some specific monster have multiple levels, for example:

https://oldschool.runescape.wiki/w/Dwarf

You can switch through the different levels by clicking on boxes on top of the infobox: "Level 7","Level 10"...

Once you click on the level box it changes the url to match the level.

So when i request the url https://oldschool.runescape.wiki/w/Dwarf#Level_10, it's bringing data from the first level only, in case: https://oldschool.runescape.wiki/w/Dwarf#Level_7, and i can't get to scrap other levels.

import requests
from bs4 import BeautifulSoup

url = 'https://oldschool.runescape.wiki/w/Dwarf#Level_20'
response = requests.get(url, headers = {'User-Agent':'Mozilla/5.0'})
soup = BeautifulSoup(response.content, 'html.parser')
soup_minfobox = soup.find_all('table', class_ ="infobox infobox-switch no-parenthesis-style infobox-monster")

print(soup_minfobox[0].text)

Output: Level 7Level 10Level 11Level 20DwarfReleased6 April 2001 (Update)MembersNoCombat level7Size1x1 ...

Excuse me the makeshift code, but in the output you can see that it is the data from the lv 7 in the end, although the url is for the lv 20.

Tawfir
  • 3
  • 2

1 Answers1

0

If you manually trigger the events (from the browser's console), you'll see that the infobox changes:

$("span[data-switch-anchor='#Level_7']").click();
$("span[data-switch-anchor='#Level_10']").click();
$("span[data-switch-anchor='#Level_11']").click();
$("span[data-switch-anchor='#Level_20']").click();

So you can use the above selectors and consult the answers provided in the following topic on how to invoke an event using BeautifulSoup:

invoking onclick event with beautifulsoup python

Nick Papadiochos
  • 329
  • 3
  • 8
  • I'm not totally sure if i understood what the following topic solution is, but i could solve the problem using selenium, although i find it slow and it force opens the browser. I'm a beginner at this, maybe I'm doing something wrong. – Tawfir May 25 '22 at 18:00
  • Take 1 of the above commands and paste it into the Console tab of the Developer Tools of your web browser. Once you hit enter, you will see that the infobox changes - this way you've successfully manually created an event. You now need to automate that event creation using beautifulsoup. – Nick Papadiochos May 25 '22 at 21:39
  • Thank you very much for your help, I'm trying very hard to understand the topic you linked, from what i get he is using selenium to solve the problem, because we can't invoke events with BeautifulSoup. Some answer in that topic talks about some evaluateJavascript method from BeautifulSoup, but i can't find it anywhere. – Tawfir May 25 '22 at 22:23