beautiful soup returns none from sainsbury's

Question

Similar to beautiful soup find returns none from rightmove, but for a different site: https://www.sainsburys.co.uk/gol-ui/SearchDisplayView?filters[keyword]=milk

I try running:

url='https://www.sainsburys.co.uk/gol-ui/SearchDisplayView?filters[keyword]=banana'

# configure driver
chrome_options = webdriver.ChromeOptions()
chrome_options.add_argument("--headless")
chrome_driver = os.getcwd() + "\\chromedriver.exe"  # IF NOT IN SAME FOLDER CHANGE THIS PATH
driver = webdriver.Chrome(options=chrome_options, executable_path=chrome_driver)
driver.get(url)

page = driver.page_source
page_soup = soup(page,'html.parser')

container_tag1='pt__content'
containers = page_soup.findAll("div",{"class":container_tag1})
# print(containers)
print(len(containers))

to no avail.

I tried without selenium, and failed as well.

Any suggestions?

score 0 · Answer 1 · answered Dec 11 '20 at 00:00

You have to wait for the page to fully render before passing the HTML to BeautifulSoup. One option is to use the .sleep method from the built-in time module.

from time import sleep
from selenium import webdriver
from bs4 import BeautifulSoup

URL = "https://www.sainsburys.co.uk/gol-ui/SearchDisplayView?filters[keyword]=banana"

driver = webdriver.Chrome(r"c:\path\to\chromedriver.exe")
driver.get(URL)
sleep(5)  # <-- Wait for the page to fully render

soup = BeautifulSoup(driver.page_source, "html.parser")
print(soup.find_all("div", {"class": "pt__content"}))

I added in a 15s sleep as well, and it still did not work for me — frank, Dec 11 '20 at 16:01

beautiful soup returns none from sainsbury's

1 Answers1