list index out of range scraping

Question

 soup = BeautifulSoup(driver.page_source)

        for each_div in soup.findAll("div", { "class" : "trapac-form-view-results tpc-results" }):

            if each_div.findAll("table", {"class": "sticky-enabled table-select-processed tableheader-processed sticky-table"})[1]:
                for child0 in each_div.findAll("table", {"class": "sticky-enabled table-select-processed tableheader-processed sticky-table"})[1]:

                    if child0.name == "table":
                         print("2")
                         child = child0.findChildren()

                         for child in child0:
                             if child.name == "tbody":
                                 child1 = child.findChildren()

the code above perfectly fine but when the table[1] tag is not available it gives me IndexError

IndexError: list index out of range

i tried try n catch not succeeded

how i can put condition in such a way that if table[1] does not exist i should be out of complete loop and look for next variable

help appreciated

score 0 · Answer 1 · answered Dec 29 '16 at 10:30

The method findAll returns a list containing all matching elements, meaning that if the length of the list is 0 then no tables could be matched. By checking that the length is greater than 1 you can be certain that there's at least two table elements that matched (since you use that second element found by findAll).

By storing the result in a variable you can also avoid doing duplicate work (i.e. calling findAll twice).

soup = BeautifulSoup(driver.page_source)

for each_div in soup.findAll("div", { "class" : "trapac-form-view-results tpc-results" }):
    tables = each_div.findAll("table", {"class": "sticky-enabled table-select-processed tableheader-processed sticky-table"})
    if len(tables) > 1:
        for child0 in tables[1]:

            if child0.name == "table":
                 print("2")
                 child = child0.findChildren()

                 for child in child0:
                     if child.name == "tbody":
                         child1 = child.findChildren()

score 0 · Accepted Answer · edited May 23 '17 at 11:46

Pythonic way to do this is using try/except (but not if and of course not len!)

soup = BeautifulSoup(driver.page_source)

    for each_div in soup.findAll("div", { "class" : "trapac-form-view-results tpc-results" }):

        try:
            node = each_div.findAll("table", {"class": "sticky-enabled table-select-processed tableheader-processed sticky-table"})[1]
        except IndexError:
            continue

        for child0 in node:

            if child0.name == "table":
                 print("2")
                 child = child0.findChildren()

                 for child in child0:
                     if child.name == "tbody":
                         child1 = child.findChildren()

There are several reasons why this way is the best one.

Thanks a lot with couple of changes code is working fine, the idea of try n catch is what i was looking for, thank again — GAURAV CHHABRA, Dec 29 '16 at 11:39

list index out of range scraping

2 Answers2