Webscraping with BS4 NoneType object has no attribute find

Question

I'm not sure why my code isn't working. I get AttributeError: 'NoneType' object has no attribute 'find'

My code is as follows:

import requests
from bs4 import BeautifulSoup
import csv

root_url = "https://urj.org/urj-congregations?congregation=&distance_address_field=&distance_num_miles=5.0&worship_services=All&community=All&urj_camp_affiliations=All&page=0"
html = requests.get(root_url)
soup = BeautifulSoup(html.text, 'html.parser')

paging = soup.find("nav",{"aria-label":"pagination-heading-3"}).find("li",{"class":"page-item"}).find_all("a")
start_page = paging[1].text
last_page = paging[len(paging)-2].text


outfile = open('congregationlookup.csv','w', newline='')
writer = csv.writer(outfile)
writer.writerow(["Name", "Address", "Phone"])


pages = list(range(1,int(last_page)+1))
for page in pages:
    url = 'https://urj.org/urj-congregations?congregation=&distance_address_field=&distance_num_miles=5.0&worship_services=All&community=All&urj_camp_affiliations=All&page=%s' %(page)
    html = requests.get(url)
    soup = BeautifulSoup(html.text, 'html.parser')

    #print(soup.prettify())
    print ('Processing page: %s' %(page))

    name_list = soup.findAll("div",{"class":"views-field views-field-congregation"})
    for element in name_list:
        name = element.find('h3').text
        address = element.find('field-content mb-2').text.strip()
        phone = element.find("i",{"class":"fa fa-phone mr-1"}).text.strip()

        writer.writerow([name, address, phone])

outfile.close()
print ('Done')

I'm trying to scrape the name, address, and phone number from the URJ Congregations website.

Thank you

There are a lot of `find()` calls... which one is throwing the error? In any case, it's likely that a previous find has returned `None` because the thing you're searching for wasn't found. — kindall, Feb 01 '22 at 20:25
@kindall paging = soup.find("div",{"Class":"pagination-heading-3"}).find("li",{"class":"page-item"}).find_all("a") is throwing the error. — Alex, Feb 01 '22 at 20:26
There is a typo in your find statement. Actual value of `arial-label` attribute is `
` (note that there are 2 dashes not 1) — Rustam Garayev, Feb 01 '22 at 20:39

sudden_appearance · Accepted Answer · 2022-02-01T21:38:20.387

Final code

import csv
import requests
from bs4 import BeautifulSoup

# root_url = "https://urj.org/urj-congregations?congregation=&distance_address_field=&distance_num_miles=5.0&worship_services=All&community=All&urj_camp_affiliations=All&page=0"
# html = requests.get(root_url)
# soup = BeautifulSoup(html.text, 'html.parser')
# paging = soup.find("nav", {"aria-label": "pagination-heading--3"}).find("ul", {"class": "pagination"}).find_all("a")
# start_page = paging[1].text
# last_page = paging[len(paging) - 3].text

outfile = open('congregationlookup.csv', 'w', newline='')
writer = csv.writer(outfile)
writer.writerow(["Name", "Address", "Phone"])

pages = list(range(1, 1000))
for page in pages:

    url = 'https://urj.org/urj-congregations?congregation=&distance_address_field=&distance_num_miles=5.0&worship_services=All&community=All&urj_camp_affiliations=All&page=%s' % (
        page)
    html = requests.get(url)
    soup = BeautifulSoup(html.text, 'html.parser')

    # print(soup.prettify())
    print('Processing page: %s' % (page))
    elements = soup.find_all("div", {"class": "views-row"})
    if len(elements) == 0:
        break
    for element in elements:
        name = element.find("div", {"class": "views-field views-field-congregation"}).text.strip()
        address = element.find("div", {"class": "views-field views-field-country"}).text.strip()
        phone = element.find("div", {"class": "views-field views-field-website"}).text.strip().split("\n")[0]
        writer.writerow([name, address, phone])

outfile.close()
print('Done')

Wow this worked like a charm! Thank you so much for your help! The only issue now is that the loops stop at 9. So even though there are more pages, it doesn't capture them all. — Alex, Feb 01 '22 at 21:06
That's because you are only capturing page numbers from first page — sudden_appearance, Feb 01 '22 at 21:19
@Alex, i updated code, now it doesn't search for pagination, it just iterates until fails. — sudden_appearance, Feb 01 '22 at 21:32
Upd: it parsed 821 rows. So i guess this is what you wanted) — sudden_appearance, Feb 01 '22 at 21:48

score 0 · Answer 2 · answered Feb 01 '22 at 20:34

Most likely, your name_list contains a None type. So, when you attempt to run element.find(), you are performing a string operation on a None, hence your error.

https://docs.python.org/3/library/stdtypes.html#str.find

Also as an FYI, findAll() is bs3 syntax. You should use find_all() Difference between "findAll" and "find_all" in BeautifulSoup

score 0 · Answer 3 · answered Feb 01 '22 at 20:41

0

There is a load of problems

The first problem is

"pagination-heading--3"

istead of

"pagination-heading-3"

Next i changed

paging = soup.find("nav",{"aria-label":"pagination-heading-3"}).find("li",{"class":"page-item"}).find_all("a")

To

paging = soup.find("nav", {"aria-label": "pagination-heading--3"}).find("ul", {"class": "pagination"}).find_all("a")

This was the line where i swapped first problematic string. And also i changed the second search to find ul. You were trying to find 1 li and searching inside of it. This would have reproduced empty list Next

last_page = paging[len(paging) - 3].text

as you are trying to get 3rd element from the end

It still doesn't work, i will keep updating

answered Feb 01 '22 at 20:41

sudden_appearance

1,968
1
4
15

Thank you. This fixed the pagination issue. However, there are two issues with my scraping. First, it seems like I am only getting the first element of each page; that is, the first congregation on each page rather than all of the congregations on each page. I also, can't seem to get the address and phone number. – Alex Feb 01 '22 at 20:56
@Alex, added completed code. Ask anything if needed expalation – sudden_appearance Feb 01 '22 at 21:05

Webscraping with BS4 NoneType object has no attribute find

3 Answers3