Selenium won't load all the content

Question

I've been lookin at this webpage, https://www.tractorsupply.com/tsc/product/welded-wire-48-in-x-100-ft to try and get the price using Beautiful Soup and Selenium with python. However, when I use selenium to access the page with this code:


    url = 'https://www.tractorsupply.com/tsc/product/welded-wire-48-in-x-100-ft'
    headers = 'Chrome/101.0.4951.41 (Windows NT 10.0; Win64; x64)'
    opts = Options()
    opts.add_argument(f"user-agent={headers}")

    driver = webdriver.Chrome(chrome_options=opts)

    driver.get(url)

    time.sleep(10)

I get this resulting page:

When I should be getting this page:

I've done a little research and it seems they use JavaScript to populate those fields based on my zip code, but i'm not entirely sure becuase I don't know much about JS.

What I really need is text value of that price and i'm not sure how to pursue that.

Thanks

It seems to be displaying a cookies section so I am assuming it's using your google profile and then displaying items related to that. Try looking into https://stackoverflow.com/questions/31062789/how-to-load-default-profile-in-chrome-using-python-selenium-webdriver — Arundeep Chohan, May 18 '22 at 00:55

score 0 · Answer 1 · answered May 19 '22 at 18:07

I was able to figure out a solution, the code is as follows:

headers = 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/101.0.4951.67 Safari/537.36'
opts = Options()
opts.add_argument(f"user-agent={headers}")

opts.add_argument("--disable-blink-features=AutomationControlled") # must be included

opts.add_argument("--enable-javascript") # must be included

opts.headless = True

driver = webdriver.Chrome(options=opts)

driver.get(url)

time.sleep(5)

Selenium won't load all the content

1 Answers1