How to scrape all the pages in the site Python

Question

I am trying to web scrape a forum using python which has multiple pages.

Example :https://www.f150forum.com/f118/would-you-buy-f150-again-463954/

How can I go to multiple pages of the website to extract comments from each user? Thank you in advance

url = "https://www.f150forum.com/f118/would-you-buy-f150-again-463954/"
page = requests.get(url, headers=headers)
soup = BeautifulSoup(page.text, 'html.parser')
domains = soup.find_all("div")
posts = soup.find(id = "posts")
comments_class = soup.findAll('div',attrs={"class":"ism-true"})   
comments = [row.get_text() for row in comments_class]

Check out this library: https://scrapy.org/ – kendriu Jan 07 '20 at 21:27 — kendriu, Jan 07 '20 at 21:27

αԋɱҽԃ αмєяιcαη · Accepted Answer · 2020-01-08T15:33:06.553

1

import requests
from bs4 import BeautifulSoup
import re

data = []
with requests.Session() as req:
    for item in range(1, 11):
        print(f"Extracting Page# {item}")
        r = req.get(
            f"https://www.f150forum.com/f118/would-you-buy-f150-again-463954/index{item}/")
        soup = BeautifulSoup(r.text, 'html.parser')
        result = [item.get_text(strip=True, separator=" ") for item in soup.findAll(
            "div", id=re.compile("^post_message_"))]
        data.append(result)

print(data)

edited Jan 08 '20 at 15:33

answered Jan 07 '20 at 21:55

αԋɱҽԃ αмєяιcαη

11,825
3
17
50

Thank you it worked great. But I am confused as why do we have to use "print('*' * 30)" this as print? – anonymous13 Jan 08 '20 at 14:02
Feel free to remove it – αԋɱҽԃ αмєяιcαη Jan 08 '20 at 14:03
Thank you. The result I am trying to extract is coming as one single text. Is there a way I can extract each comment as a list item? – anonymous13 Jan 08 '20 at 14:48

How to scrape all the pages in the site Python

1 Answers1