using loop to iterate the html table values python

Question

I'm trying to make python script that gets all the grades of the student using requests and bs4. Now i have a problem looping the values

   for rows in tr:
        td = tbody.find_all('td')
        subject.append(td[0].get_text())
        fq.append(td[1].get_text())
        sq.append(td[2].get_text())
        ave.append(td[3].get_text())
            
    for i in subject:
        print(f"Subject: {i}")

    for i in fq:
        print(f"First Quarter: {i}")
    
    for i in sq:
        print(f"Second Quarter: {i}")

    for i in ave:
        print(f"Average: {i}")
# here my goal is there are 4 list and are all connected like all the first value of the subject list, f_quar, s_quar and the average are linked together, like gen math(subject), 90(f_qaur), 90(s_qaur), and 90(average)

Output:

Subject:  GENERAL MATHEMATICS
Subject:  GENERAL MATHEMATICS 
Subject:  GENERAL MATHEMATICS
Subject:  GENERAL MATHEMATICS
Subject:  GENERAL MATHEMATICS
Subject:  GENERAL MATHEMATICS
Subject:  GENERAL MATHEMATICS
Subject:  GENERAL MATHEMATICS
First Quarter:   ##.00
First Quarter:   ##.00
First Quarter:   ##.00
First Quarter:   ##.00
First Quarter:   ##.00
First Quarter:   ##.00
First Quarter:   ##.00
First Quarter:   ##.00 
Second Quarter:   ##.00
Second Quarter:   ##.00
Second Quarter:   ##.00
Second Quarter:   ##.00
Second Quarter:   ##.00
Second Quarter:   ##.00
Second Quarter:   ##.00
Average:   ##.00
Average:   ##.00
Average:   ##.00
Average:   ##.00
Average:   ##.00
Average:   ##.00
Average:   ##.00
Average:   ##.00

Expected Output:

Subject: Gen Math
Subject: Stats
...

First Quarter: 90.00
First Quarter: 90.00
...
Second Quarter: 90.00
Second Quarter: 90.00
...
Average: 90.00
Average: 90.00
...

Im new at pyton so loops is my weakness. Also the code seems so wrong since i need the subject, 1stQ grade, 2ndQ grade and the average. Thanks!. This is the html code of the table:

<table cellspacing="0" class="table table-bordered table-striped" id="tblss1" width="100%">
<thead>
<tr class="success">
<th style="text-align:center">SUBJECT</th>
<th style="text-align:center">1ST</th>
<th style="text-align:center">2ND</th>
<th style="text-align:center">AVE</th>
</tr>
</thead>
<tbody>
<tr>
<td style="color:purple"> GENERAL MATHEMATICS </td>
<td align="center"> <strong> ##.00 </strong></td>
<td align="center"> <strong> ##.00 </strong></td>
<td align="center" style="color:blueviolet"> <strong> ##.00  </strong></td>
</tr>
<tr>
<td style="color:purple"> EARTH SCIENCE </td>
<td align="center"> <strong> ##.00 </strong></td>
<td align="center"> <strong> ##.00 </strong></td>
<td align="center" style="color:blueviolet"> <strong> ##.00  </strong></td>
</tr>
<tr>
<td style="color:purple"> PHYSICAL EDUCATION AND HEALTH </td>
<td align="center"> <strong> ##.00 </strong></td>
<td align="center"> <strong> ##.00 </strong></td>
<td align="center" style="color:blueviolet"> <strong> ##.50  </strong></td>
</tr>
<tr>
<td style="color:purple"> GENERAL CHEMISTRY 1 </td>
<td align="center"> <strong> ##.00 </strong></td>
<td align="center"> <strong> ##.00 </strong></td>
<td align="center" style="color:blueviolet"> <strong> ##.00  </strong></td>
</tr>
<tr>
<td style="color:purple"> 21ST CENTURY LITERATURE </td>
<td align="center"> <strong> ##.00 </strong></td>
<td align="center"> <strong> ##.00 </strong></td>
<td align="center" style="color:blueviolet"> <strong> ##.00  </strong></td>
</tr>
<tr>
<td style="color:purple"> READING AND WRITING </td>
<td align="center"> <strong> ##.00 </strong></td>
<td align="center"> <strong> ##.00 </strong></td>
<td align="center" style="color:blueviolet"> <strong> ##.00  </strong></td>
</tr>
<tr>
<td style="color:purple"> GENERAL BIOLOGY 1 </td>
<td align="center"> <strong> ##.00 </strong></td>
<td align="center"> <strong> ##.00 </strong></td>
<td align="center" style="color:blueviolet"> <strong> ##.00  </strong></td>
</tr>
<tr>
<td style="color:purple"> ENTREPRENEURSHIP </td>
<td align="center"> <strong> ##.00 </strong></td>
<td align="center"> <strong> ##.00 </strong></td>
<td align="center" style="color:blueviolet"> <strong> ##.50  </strong></td>
</tr>
</tbody>
</table>

What have you tried? I didn't understand your comment, what do you mean by linked? — 0xRyN, Jul 11 '22 at 13:11
@0xRyN mb im english is not my first language and also bad at explaining. btw i updated the code above i think you can undestand what i am trying to do... — ciumai, Jul 11 '22 at 13:19
Can you edit your question and show your **exact** expected output? — Jack Fleeting, Jul 11 '22 at 13:23
@JackFleeting edited also with the output of my current code — ciumai, Jul 11 '22 at 13:28

4lexKidd · Answer 1 · 2022-07-11T13:36:25.930

You use i as index twice (outer and inner loop).

I am not sure if the interpreter can handle that "override" of the variable so easily, because it might do it but after returning to the outer loop the object/iterator-cursor in i could be gone.

Try changing the inner loop index variable name to not override i from the outer loop.

If this does not solve your issue please describe in more detail what you try to achieve or what the seen behavior is.

*Post Edit: This way you will only get the same results for all entrys. You need to build a double loop doing the following steps:

find all tr blocks and iterating over them

for tr_block in tbody.find_all('tr')

in each tr_block append the corresponding td blocks to their lists

td = tr_block.find_all('td')

subject.append(td[0].get_text()) #[...]

after that you should have lists filled with all data from the html which you then can zip together to sets if needed.

score 0 · Answer 2 · answered Jul 11 '22 at 13:29

Based on my understanding, here is what you're trying to acheive. I'm assuming the first for loops actually adds all the data properly.

subjects = [] 
fq = []
sq = []
avgs = []
for rows in tr:
    td = tbody.find_all('td')
    subjects.append(td[0].get_text())
    fq.append(td[1].get_text())
    sq.append(td[2].get_text())
    avgs.append(td[3].get_text())

for subject in subjects:
    print(subject)

for f in fq:
    print(f)

for s in sq:
    print(s)

for a in avgs:
    print(a)

score 0 · Answer 3 · answered Jul 11 '22 at 14:08

0

In cases like this, it's simpler and faster to read the table into a dataframe:

import pandas as pd
table = """[your html above]"""
print(pd.read_html(table)

Output:

SUBJECT    1ST    2ND    AVE
 0            GENERAL MATHEMATICS  ##.00  ##.00  ##.00
 1                  EARTH SCIENCE  ##.00  ##.00  ##.00
 2  PHYSICAL EDUCATION AND HEALTH  ##.00  ##.00  ##.50
 3            GENERAL CHEMISTRY 1  ##.00  ##.00  ##.00

etc.

answered Jul 11 '22 at 14:08

Jack Fleeting

24,385
6
23
45

thanks it worked but i have another question, when i tried to dataframe like this it showes "..." >>>4 21ST CENTURY LITERATURE FROM THE PHILIPPINES A... 90.0 90.0 90.0 – ciumai Jul 12 '22 at 00:15
@King Not sure why that happens, but the output in the answer is based on the html in the question and there's nothing there about literature from the Philippines... – Jack Fleeting Jul 12 '22 at 00:21
mb the whole name of the subject is 21st century literature from the philippines and the world, anyways i found an answer [https://stackoverflow.com/questions/21028819/how-to-remove-ellipsis-from-a-row-in-a-python-pandas-series-or-data-frame-shown] – ciumai Jul 12 '22 at 06:17

using loop to iterate the html table values python

3 Answers3