BeautifulSoup iterating over a table and insert each row value into list variable TypeError: list indices must be integers, not Tag

Question

I am iterating over a table in a for loop and I would like to store the value in a list variable. If i store it into a variable I just get the first value out when i return the value in my function call. In each iteration of the for loop I have several values. Storing it in a variable is not good. I would need to store it into a list so I can capture all the values.

The error I get is:

list1[div] = div.text.strip().encode('utf-8'), div.find_next("a").text.strip().encode('utf-8')
TypeError: list indices must be integers, not Tag

My code is:

def extract_testcases_from_report_htmltestrunner():
filename = (r"C:\temp\selenium_report\ClearCore501_Automated_GUI_TestReport.html")
html_report_part = open(filename,'r')
soup = BeautifulSoup(html_report_part, "html.parser")

for div in soup.select("#result_table tr div.testcase"):
    print(div.text.strip().encode('utf-8'), div.find_next("a").text.strip().encode('utf-8'))
    list1 = []
    for div in soup.select("#result_table tr div.testcase"):
        var = div.text.strip().encode('utf-8'), div.find_next("a").text.strip().encode('utf-8')
        list1[div] = div.text.strip().encode('utf-8'), div.find_next("a").text.strip().encode('utf-8')
        return var

If i comment out return var My output from the print statement is:

('test_000001_login_valid_user', 'pass')
('test_000002_select_a_project', 'pass')
('test_000003_verify_Lademo_CRM_DataPreview_is_present', 'pass')
('test_000004_view_data_preview_Lademo_CRM_and_test_scrollpage', 'pass')
('test_000005_sort_data_preview_by_selecting_column_header', 'pass')
# etc.  More tests

If i call the function with return var my output is:

('test_000001_login_valid_user', 'pass')

I would like my function call to return all the testcases. I think I would need to return it as a list. I can then call this function and iterate over the list and print it in my email code for the email message.

Thanks, Riaz

I have it returning as a list now. When i call the function and print it's return value it prints all the values in 1 line. I would like to separate it into separate lines.

def extract_testcases_from_report_htmltestrunner():
    filename = (r"C:\temp\selenium_report\ClearCore501_Automated_GUI_TestReport.html")
    html_report_part = open(filename,'r')
    soup = BeautifulSoup(html_report_part, "html.parser")

    list1 = []
    for div in soup.select("#result_table tr div.testcase"):
        #print(div.text.strip().encode('utf-8'), div.find_next("a").text.strip().encode('utf-8'))
        list1.append((div.text.strip().encode('utf-8'), div.find_next("a").text.strip().encode('utf-8')))
    return list1

if __name__ == "__main__":

    print extract_testcases_from_report_htmltestrunner()

The output is:

[('test_000001_login_valid_user', 'pass'), ('test_000002_select_a_project', 'pass'), ('test_000003_verify_Lademo_CRM_DataPreview_is_present', 'pass') etc.

I would like the output to be like:

[('test_000001_login_valid_user', 'pass')
('test_000002_select_a_project', 'pass')
('test_000003_verify_Lademo_CRM_DataPreview_is_present', 'pass')
etc.

Thanks, Riaz

score 2 · Accepted Answer · edited May 23 '17 at 10:29

2

You want to yield, you can only return from a function once so your function ends as soon as you hit the return on the first iteration:

from bs4 import BeautifulSoup
def extract_testcases_from_report_htmltestrunner():
    filename = (r"C:\temp\selenium_report\ClearCore501_Automated_GUI_TestReport.html")
    html_report_part = open(filename,'r')
    soup = BeautifulSoup(html_report_part, "html.parser")
    for div in soup.select("#result_table tr div.testcase"):
          yield div.text.strip().encode('utf-8'), div.find_next("a").text.strip().encode('utf-8')


all_data = list(extract_testcases_from_report_htmltestrunner())

edited May 23 '17 at 10:29

Community

1
1

answered Aug 13 '16 at 23:10

Padraic Cunningham

176,452
29
245
321

I'm not sure how I can use this function. If I call the function if __name__ == "__main__": print extract_header_from_summary_from_report_htmltestrunner() the Output I get is – Riaz Ladhani Aug 14 '16 at 14:06
I tried to put a return at the end of the function return all_data it says return outside of this function – Riaz Ladhani Aug 14 '16 at 14:06
I would like the function to return all of the values so when i call the function I can put all the values into my email code in the message body part of the email. Thanks – Riaz Ladhani Aug 14 '16 at 14:14
Ah I've got it now. if __name__ == "__main__": all_data = list(extract_testcases_from_report_htmltestrunner()) print all_data – Riaz Ladhani Aug 14 '16 at 14:36
Thanks very much for your help. – Riaz Ladhani Aug 14 '16 at 14:36
One other thing can I separate each value out by a new line. The output is all in 1 line seperated by a comma. E.g. [('test_000001_login_valid_user', 'pass'), ('test_000002_select_a_project', 'pass') Where the comma is can i put a new line there so the next value goes down onto a new line – Riaz Ladhani Aug 14 '16 at 14:37
I've worked it out. import pprint pprint.pprint(all_data). It prints each value onto a new line now. I will see how I can put this into my email code now. – Riaz Ladhani Aug 14 '16 at 15:21

BeautifulSoup iterating over a table and insert each row value into list variable TypeError: list indices must be integers, not Tag

1 Answers1