How to print paragraphs and headings simultaneously while scraping in Python?

Question

I am a beginner in python. I am currently using Beautifulsoup to scrape a website.

str='' #my_url
source = urllib.request.urlopen(str);
soup = bs.BeautifulSoup(source,'lxml');
match=soup.find('article',class_='xyz');
for paragraph in match.find_all('p'):
    str+=paragraph.text+"\n"

My tag Structure -

<article class="xyz" >
<h4>dr</h4>
<p>efkl</p>
<h4>dr</h4>
<p>efkl</p>
<h4>dr</h4>
<p>efkl</p>
<h4>dr</h4>
<p>efkl</p>         
</article>

I am getting output like this (as I am able to extract the paragraphs) -

 efkl
 efkl
 efkl
 efkl

Output I want ( I want the headings as well as the paragraphs) -

 dr
 efkl
 dr
 efkl
 dr
 efkl
 dr
 efkl

I want my output to also contains headings along with paragraphs.How to modify code in such a way that it contains header before paragraphs (Like in original HTML) .

Possible duplicate of [Python BeautifulSoup give multiple tags to findAll](https://stackoverflow.com/questions/20648660/python-beautifulsoup-give-multiple-tags-to-findall) — Keyur Potdar, Mar 03 '18 at 11:05

score 4 · Accepted Answer · answered Mar 04 '18 at 13:29

You can peel the same apple in different ways to serve the purpose. Here are few of them:

Using .find_next():

from bs4 import BeautifulSoup

content="""
<article class="xyz" >
<h4>dr</h4>
<p>efkl</p>
<h4>dr</h4>
<p>efkl</p>
<h4>dr</h4>
<p>efkl</p>
<h4>dr</h4>
<p>efkl</p>         
</article>
"""
soup = BeautifulSoup(content,"lxml")

for items in soup.find_all(class_="xyz"):
    data = '\n'.join(['\n'.join([item.text,item.find_next("p").text]) for item in items.find_all("h4")])
    print(data)

Using .find_previous_sibling():

for items in soup.find_all(class_="xyz"):
    data = '\n'.join(['\n'.join([item.find_previous_sibling("h4").text,item.text]) for item in items.find_all("p")])
    print(data)

Commonly used approach: multiple tags used within list:

for items in soup.find_all(class_="xyz"):
    data = '\n'.join([item.text for item in items.find_all(["h4","p"])])
    print(data)

All the three approaches produce the same result:

dr
efkl
dr
efkl
dr
efkl
dr
efkl

How to print paragraphs and headings simultaneously while scraping in Python?

1 Answers1

Linked