My web crawler gets stuck on the Dell website. Is it fixable?

Question

First off, I'm new to coding and Python. This is the first project that I came up with to try. When I run this code there are no errors and I get to the point of opening the Dell website, inputting the service tag and pressing the enter key. At that point the Dell website gives me a pop up that says "Please wait while we validate this action. Once validated, please submit your request again." and gives a 30 second countdown.

import openpyxl as xl
from selenium import webdriver
from selenium.webdriver.common.keys import Keys

wb = xl.load_workbook('/home/user/Code/learning/inventory.xlsx')
sheet = wb['Sheet1']

for row in range(2, sheet.max_row + 1):
    service_tag_cell = sheet.cell(row, 4).value
    warranty_cell: str
    warranty_cell = sheet.cell(row, 5).value
    if service_tag_cell != '': # and warranty_cell == '':
        driver = webdriver.Firefox()
        driver.get('https://www.dell.com/support/home/en-us')

            # Find the search box, enter the service tag number and press the enter key
        driver.find_element_by_id('inpEntrySelection').send_keys(service_tag_cell + Keys.ENTER)
        
            # Find warranty field
        warranty_date = driver.find_element_by_class_name('warrantyExpiringLabel')
        
        warranty_cell = warranty_date.value_of_css_property
        driver.close()
    sheet.cell(row, 5).value = warranty_cell

wb.save('inventory2.xlsx')

I've tried searching google to understand what prompts this message from Dell. I get the sense it just doesn't want bots like mine searching their website. But is the message a result of my poor implementation that could be corrected? Or is my goal of taking a spreadsheet of service tags and returning the expiration date dead in water?

score 0 · Accepted Answer · answered Mar 25 '22 at 03:31

If they're using methods of detected automated actions then you'll be playing cat-and-mouse.

I can suggest that you try setting a random User-Agent with a library like Random User Agents:

from random_user_agent.user_agent import UserAgent
from random_user_agent.params import SoftwareName

user_agent_rotator = UserAgent(software_names=[SoftwareName.CHROME.value], limit=100)
user_agent = user_agent_rotator.get_random_user_agent()

options.add_argument(f'user-agent={user_agent}')
driver = webdriver.Chrome(chrome_options=options)

But if that doesn't work, there's plenty of other ways in which Selenium can be detected. This and This have good information that may helpful.

My web crawler gets stuck on the Dell website. Is it fixable?

1 Answers1