Python, extract text from webpage

Question

I am working on a project where I am crawling thousands of websites to extract text data, the end use case is natural language processing.

EDIT * since I am crawling 100's of thousands of websites I cannot tailor a scraping code for each one, which means I cannot search for specific element id's, the solution I am looking for is a general one *

I am aware of solutions such as the .get_text() function from beautiful soup. The issue with this method is that it gets all the text from the website, much of it being irrelevant to the main topic on that particular page. for the most part a website page will be dedicated to a single main topic, however on the sides and top and bottom there may be links or text about other subjects or promotions or other content.

With the .get_text() function it return all the text on the site page in one go. the problem is that it combines it all (the relevant parts with the irrelevant ones. is there another function similar to .get_text() that returns all text but as a list and every list object is a specific section of the text, that way it can be know where new subjects start and end.

As a bonus, is there a way to identify the main body of text on a web page?

Maybe you could try to use regex to get the links you'd need. — anveshjhuboo, May 18 '20 at 04:04
@MustardTiger, have u tried using ```find_all``` which allows to search elements by tag & attributes then call ```text``` — sushanth, May 18 '20 at 04:33

score 3 · Answer 1 · answered May 18 '20 at 05:22

Below I have mentioned snippets that you could use to query data in desired way using BeautifulSoup4 and Python3:

import requests
from bs4 import BeautifulSoup

response = requests.get('https://yoursite/page')
soup = BeautifulSoup(response.text, 'html.parser')
# Print the body content in list form
print(soup.body.contents[0])
# Print the first found div on html page
print(soup.find('div'))
# Print the all divs on html page in list form
print(soup.find_all('div'))
# Print the element with 'required_element_id' id
print(soup.find(id='required_element_id'))
# Print the all html elements in list form that matches the selectors
print(soup.select(required_css_selectors))
# Print the attribute value in list form
print(soup.find(id='someid').get("attribute-name"))
# You can also break your one large query into multiple queries
parent = soup.find(id='someid')
# getText() return the text between opening and closing tag
print(parent.select(".some-class")[0].getText())

For your more advance requirement, you can check Scrapy as well. Let me know if you face any challenge in implementing this or if your requirement is something else.

Hi i made an edit to the question to make things clearer – Mustard Tiger May 18 '20 at 17:29 — Mustard Tiger, May 18 '20 at 17:29

score 0 · Answer 2 · answered Mar 02 '23 at 08:48

The closest real-life example of what you're looking for could be the Reader View aka Reading Mode found in Firefox, Safari and others.

Here's one question on that topic here on StackOverflow: How does Firefox reader view operate

Firefox is said to be relying on github.com/mozilla/readability, which they thankfully open-sourced.

Python, extract text from webpage

2 Answers2