I have the following code in python, which extracts only the introduction of the article on "Artificial Intelligence", while instead I would like to extract all sub-sections (History, Goals ...)
import requests
def get_wikipedia_page(page_title):
endpoint = "https://en.wikipedia.org/w/api.php"
params = {
"format": "json",
"action": "query",
"prop": "extracts",
"exintro": "",
"explaintext": "",
"titles": page_title
}
response = requests.get(endpoint, params=params)
data = response.json()
pages = data["query"]["pages"]
page_id = list(pages.keys())[0]
return pages[page_id]["extract"]
page_title = "Artificial intelligence"
wikipedia_page = get_wikipedia_page(page_title)
Someone proposed to use another approach that parses html and uses BeautifulSoup to convert to text:
from urllib.request import urlopen
from bs4 import BeautifulSoup
url = "https://en.wikipedia.org/wiki/Artificial_intelligence"
html = urlopen(url).read()
soup = BeautifulSoup(html, features="html.parser")
# kill all script and style elements
for script in soup(["script", "style"]):
script.extract() # rip it out
# get text
text = soup.get_text()
# break into lines and remove leading and trailing space on each
lines = (line.strip() for line in text.splitlines())
# break multi-headlines into a line each
chunks = (phrase.strip() for line in lines for phrase in
line.split("
"))
# drop blank lines
text = '\n'.join(chunk for chunk in chunks if chunk)
print(text)
This is not a good-enough solution, as it includes all text that appears on the website (like image text), and it includes citations in the text (e.g. [1]), while the first script removes them.
I suspect that the api of wikipedia should offer a more elegant solution, it would be rather weird if one can get only the first section?
