Python: How to scrape the table data from webpage

Question

I tried to scrap the data of the following website: https://www.sbs.gob.pe/app/pp/EstadisticasSAEEPortal/Paginas/TIActivaTipoCreditoEmpresa.aspx?tip=B in the date 29/01/2010

But I cant find an easy way to get these table data. Can you help with it? Just to make sure, my problem is whith that date because the table has no id

I tried:

table_area = driver.find_element(By.ID,'ctl00_cphContent_pnContenidoAnteriorMnASP')
inicial=table_area.find_elements(By.XPATH,".//td") 
data=[item.text for item in inicial]

score 1 · Accepted Answer · answered Jun 20 '23 at 20:45

To scrape the table data from the website you need to induce WebDriverWait for the visibility_of_element_located() and using DataFrame from Pandas you can use the following locator strategy:

Code Block:

from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
import pandas as pd

driver.get("https://www.sbs.gob.pe/app/pp/EstadisticasSAEEPortal/Paginas/TIActivaTipoCreditoEmpresa.aspx?tip=B")
data=WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.CSS_SELECTOR, "table.rpgTable[id$='DataZone_DT']"))).get_attribute("outerHTML")
df = pd.read_html(data)
print(df)

Console Output:

[     BBVA Comercio Crédito Pichincha    BIF Scotiabank  ... Ripley  Alfin   ICBC Bank of China    BCI Promedio
0    9.11    12.50    8.95     10.15   8.81       8.62  ...      -      -   8.97          s.i.  10.50     9.03
1   10.58    12.50    9.27         -   5.95       9.32  ...      -      -      -          s.i.      -    10.25
2    8.92        -    8.76      9.59   9.70       8.82  ...      -      -      -          s.i.   9.50     8.85
3    9.92        -    9.37     10.37   9.94       9.09  ...      -      -  10.00          s.i.  11.50     9.48
4    9.40        -    9.36         -   9.61          -  ...      -      -   8.55          s.i.      -     9.56
5    8.71        -    8.86         -  11.15       8.66  ...      -      -      -          s.i.      -     8.79
6    8.89        -    8.64         -  10.50       8.51  ...      -      -      -          s.i.      -     8.68
7   11.27    15.82   10.52     10.90  10.85      10.00  ...      -      -   8.99          s.i.   9.57    10.75
8   12.84    20.35   10.87     10.97  11.05      10.03  ...      -      -      -          s.i.      -    11.32
9   11.75    12.67   11.22     11.76  10.54      11.09  ...      -      -      -          s.i.      -    10.97
10  10.88    12.50   11.00     10.58  11.36       9.78  ...      -      -   8.96          s.i.      -    10.66
11  10.93    11.00   10.14     11.31  11.18      10.69  ...      -      -   9.00          s.i.   9.57    10.57
12   9.93        -    9.77     12.55   9.65       9.91  ...      -      -      -          s.i.      -     9.92
13  11.18        -    9.81         -  10.27      10.19  ...      -      -      -          s.i.      -    10.43
14  15.40    11.89   14.62     12.52  13.07      13.81  ...      -      -      -          s.i.      -    14.64
15  15.58    15.99   14.22     13.44  13.04      13.00  ...      -      -      -          s.i.      -    13.95
16  17.13    13.66   14.52     10.27  10.79      13.94  ...      -      -      -          s.i.      -    14.81
17  14.31    10.42   12.95     12.22  13.62      13.39  ...      -      -      -          s.i.      -    13.48
18  14.78    11.70   12.90     12.81  12.82      12.87  ...      -      -      -          s.i.      -    13.34
19  14.69    13.31   11.51     12.49  15.43      13.20  ...      -      -      -          s.i.      -    13.70
20  16.22    14.00   16.88      8.50  12.31      14.80  ...      -      -      -          s.i.      -    16.39
21  20.77    14.14   21.72     12.65  15.41      16.64  ...      -      -      -          s.i.      -    23.24
22  20.94        -   18.34         -  20.69      14.59  ...      -      -      -          s.i.      -    18.37
23  22.73        -   18.48         -      -      11.90  ...      -      -      -          s.i.      -    22.66
24  22.10        -   14.51         -  16.57      13.75  ...      -      -      -          s.i.      -    26.46
25  20.72        -   12.28     12.30      -      15.21  ...      -      -      -          s.i.      -    27.69
26  21.50    14.14   16.39         -  14.10      17.06  ...      -      -      -          s.i.      -    31.87
27  20.63        -   21.94     12.77  12.42      16.70  ...      -      -      -          s.i.      -    22.04
28  25.54        -   30.17     14.34  10.49      15.72  ...      -      -      -          s.i.      -    38.59
29  33.72        -   30.83     29.60      -          -  ...      -      -      -          s.i.      -    32.99
30      -        -   17.93         -      -      14.80  ...      -      -      -          s.i.      -    17.83
31  21.79        -       -         -  10.49          -  ...      -      -      -          s.i.      -    45.13
32  25.75        -   19.00         -      -          -  ...      -      -      -          s.i.      -    14.78
33  21.96        -   11.97         -      -          -  ...      -      -      -          s.i.      -    31.08
34  22.80        -   18.00         -      -          -  ...      -      -      -          s.i.      -    60.35
35  21.73        -   15.52         -      -      19.20  ...      -      -      -          s.i.      -    53.79
36  21.69        -   31.45     13.58      -      15.72  ...      -      -      -          s.i.      -    30.18
37  39.78    15.55   48.37     38.94  31.30      38.15  ...  71.36  64.27      -          s.i.      -    52.33
38  51.34    28.32   56.47     40.34  54.29      44.64  ...  80.16      -      -          s.i.      -    61.91
39  13.22        -       -         -      -          -  ...      -      -      -          s.i.      -    15.11
40  13.26        -   14.03         -  14.69      11.40  ...      -  25.50      -          s.i.      -    13.93
41  20.08    42.90   64.84     23.54  10.64      21.58  ...      -  18.50      -          s.i.      -    50.48
42  19.10    15.33   19.78     35.04  16.80      19.70  ...  33.99  65.94      -          s.i.      -    23.64
43      -    48.65       -         -      -          -  ...      -      -      -          s.i.      -    48.65
44   8.97    10.55    9.51     10.46  10.18       9.35  ...      -      -      -          s.i.      -     9.64
45   8.97    10.55    9.51     10.46  10.18       9.35  ...      -      -      -          s.i.      -     9.64

[46 rows x 18 columns]]

Thank you , but my problem is when dates are below 2015-10-01, for example, if you try your code with the date 29/01/2010, your code fails — Edu2694, Jun 20 '23 at 21:09
@Edu2694 That's because the the `` name is different in _2015-10-01_ and _2023-06-19_, but the workflow remains the same. So only the `
` locator would change. — undetected Selenium, Jun 20 '23 at 21:16
thank you, but I think that in the date 29/01/2010 (for example), there is no table id to do it, how can we did that without it? — Edu2694, Jun 20 '23 at 22:16
@Edu2694 Let's discuss the issue in [Selenium](https://chat.stackoverflow.com/rooms/223360/selenium) room. — undetected Selenium, Jun 20 '23 at 22:19

Python: How to scrape the table data from webpage

1 Answers1