0

I'm interacting with an online site to get a dataset to filter offline. I'm using Selenium webdriver to input values into a javascript date selector form, submitting the form and downloading the csv file. Through my tests, I've had intermittent issues with errors from the webdriver saying: "Unable to locate the element through xpath, element does not exist". It works about half the time, while I'm using the debugger, but when I run the program it seems to try and grab it before it's available. I can inspect the source and see that the element is available, but the webdriver doesn't seem to find it.

I created a loop to wait 3 seconds before trying again, and I only let it run about 15 seconds before it times out and kills the program. I previously input a counter to see if the program looped the correct amount of times. On occasion, it terminated the program execution on the first iteration. The last handful of tests seem to be working fine, but I'm wondering if there isn't a better way to accomplishing the same task?

This is the function that is causing a problem. In the for loop, download = driver.find_element_by_xpath('//*[@id=\"pagination-element-top\"]/div/div[1]/a') seems to throw an error, but only on occasion. 

def search_dates(url, date):

    options = webdriver.ChromeOptions()
    prefs = {"download.default_directory": os.getcwd()}
    options.add_experimental_option("prefs", prefs)
    driver = webdriver.Chrome(options=options)
    driver.get(url)
    date_selector = driver.find_element_by_class_name("date-range-container ")
    start_date = date_selector.find_element_by_id("startDateId")
    if datetime.now().year == 2019:
        start_date.send_keys("1//1/2019")
    else:
        start_date.send_keys("1/1/" + str(datetime.now().year))
    end_date = date_selector.find_element_by_id("endDateId")
    end_date.send_keys(date)
    button = driver.find_element_by_class_name("msl-button.msl-button-green")
    button.send_keys(Keys.ENTER)
    csv_fn = ''
    for i in range(5):
        time.sleep(3)
        download = driver.find_element_by_xpath('//*[@id=\"pagination-element-top\"]/div/div[1]/a')
        download.send_keys(Keys.ENTER)
        # Gets the name of the csv file
        csv_fn = download.get_attribute("download")
        try:
            if os.path.exists(csv_fn):
                driver.close()
                break
            else:
                continue
        except Exception as e:
            print(str(e))
            sys.exit(0)
    return csv_fn

This is my function to parse the csv file. I'm using python pandas to parse the csv file into a DataFrame.

def process_csv(csv_file):
    csv_dict = {}
    if os.path.exists(csv_file):
        try:
            with open(csv_file, 'r') as f:
                numbers = pd.read_csv(f, delimiter=',')
                for i, row in numbers.iterrows():
                    if (datetime.strptime(row['Draw Date'], '%m/%d/%Y').strftime('%A')) == 'Saturday':
                        csv_dict[row['Draw Date']] = row['Winning Numbers'].replace(',', '')
                    else:
                        pass
        except IOError as e:
            print("I/O error({0}): {1}".format(e.errno, e.strerror))
        except Exception as e:
            print(str(e))
    return csv_dict

The intent is to input a start and end date into the webform, press the submit button, and press the download button to download the csv file into the current working directory. Once the csv is downloaded, I parse the data to align with the dates that I'm looking for to run my test.

Is using xpath the best way to grab what I'm looking for? I tried using element_by_name, and element_by_class_name, but it didn't seem to work. Would this be easier or better using python requests? I'm looking to creating a single executable, and would like some opinion on if this is correct, or if requests would help fix some issues I'm having.

Thanks!

erroneous
  • 1
  • Have you checked [this](https://stackoverflow.com/questions/34548041/selenium-give-file-name-when-downloading/56570364#56570364)? It will make sure the download is completed and you will get the donwloaded file name when you call the method. – supputuri Aug 08 '19 at 12:35
  • It looks like the script you mentioned just waits a longer amount of time, about 3 minutes. They do implement a download percent checker, which I haven't thought about, but in my case the webdriver says that the element is not even there. This is before I attempt to download. At first, I thought it was trying to grab the file before it was downloaded, but after debugging, the error states that the webdriver cannot find the element by xpath. It does work, but also throws this error a handful of times. Just looking for another solution, or reason to this issue. – erroneous Aug 08 '19 at 16:59
  • can you please share the element which it's not able to find intermittently. – supputuri Aug 08 '19 at 17:28
  • My goal is to find this element: ```download = driver.find_element_by_xpath('//*[@id=\"pagination-element-top\"]/div/div[1]/a')``` – erroneous Aug 09 '19 at 08:05
  • The element I am trying to grab is bold. I mostly want to grab the download string and the href.
    ****
     – erroneous Aug 09 '19 at 08:28
  • Have you tried with Explicit wait? Using EC and webdriverwait. – supputuri Aug 11 '19 at 21:41
  • I did. Thank you. I still get weird errors, as it sometimes does not want to grab the element. Weirdest thing is that it never finds the element if I am running webdriver in headless state with ```options.headless = True```. I tried using this option and it raised an error every time. – erroneous Aug 12 '19 at 04:15
  • ```download = WebDriverWait(driver, 5).until(EC.presence_of_element_located(( By.CSS_SELECTOR, '#pagination-element-top > div > div.csv-download > a')))``` – erroneous Aug 12 '19 at 04:19

0 Answers0