Appending Dataframe Inside Loop

Question

I'm trying to get a list of links by looping through elements and then clicking through pagination. I'm not sure how to append each loop in the pandas dataframe after it goes through pagination shown below so I can call the dataframe outside the loop to list all the links.

It always overwrites and prints out the last line.

while True:

    links = [link.get_attribute('href') for link in driver.find_elements_by_class_name('view-detail-link')]

    for link in links:

        df_links = pd.DataFrame([[link]], columns=['link'])

    try:

        NextPage = driver.find_element_by_xpath('//a[@class="ui-pagination-next ui-goto-page"]')
        driver.execute_script("arguments[0].click();", NextPage)

        time.sleep(3)

    except NoSuchElementException:
        break

print(df_links.link[0])

This should be avoided See [this solution](https://stackoverflow.com/a/37009561/4333359) Append DataFrames to a list, concatenate one at the end of the loop — ALollz, Mar 15 '19 at 23:54

score 1 · Accepted Answer · answered Mar 15 '19 at 22:35

You need to create your DataFrame outside the loop. Then each time you create a new DataFrame in the loop you append it to the main one:

df = pd.DataFrame()

while True:

    links = [link.get_attribute('href') for link in driver.find_elements_by_class_name('view-detail-link')]

    for link in links:

        df_links = pd.DataFrame([[link]], columns=['link'])
        df = df.append(df_links)

    try:

        NextPage = driver.find_element_by_xpath('//a[@class="ui-pagination-next ui-goto-page"]')
        driver.execute_script("arguments[0].click();", NextPage)

        time.sleep(3)

    except NoSuchElementException:
        break

print(df.link[0])

Toby - let me know if you can answer this question as well: [https://stackoverflow.com/questions/55192162/iterate-through-dataframe-rows-in-loop] — Bronson77, Mar 16 '19 at 00:10

Appending Dataframe Inside Loop

1 Answers1