Unable to locate element under dynamic-comp

Question

There are dozens of questions here on SO with a title very similar to this one - but most of them seem related to some iFrame, which prevents Selenium to access the intended tag, node, or whatever.

In my case, I'm trying to access this site. All I want is to read the data on a table - it is easy to identify, given there it is inside a div with a very particular ID. The table is also simple to read. Despite that, there is this dynamic-comp tag, which seems to be my stumbling block - I can access all elements outside of it, and no element inside at all - be it by ID, class, tag name, whatever.

How do I handle this? Is this some kind of special IFrame? I'd have tried the .switchTo approach, but the dynamic-comp elements have no ID or class, just the tag alone.

EDIT: I also tried adding wait = WebDriverWait(driver,20), just in case. Didn't work. My goal is to iterate through the dates using the date selector, so I intend to read the table multiple times.

Answer 1

The table you need is the last one inside #oReportCell. To get it you can use (//td[@id='oReportCell']//table)[last()] xpath or #oReportCell table css selector and get the last one.

How to get table with requests and beautifulsoup, @PedroLobito's solution proposal. You can use pandas to collect and save data:

import requests
from bs4 import BeautifulSoup

params = (
    ('path', 'conteudo/txcred/Reports/TaxasCredito-Consolidadas-porTaxasAnuais-Historico.rdl'),
    ('parametros', ''),
    ('exibeparametros', 'true'),
)

response = requests.get('https://www.bcb.gov.br/api/relatorio/pt-br/contaspub', params=params)
page = BeautifulSoup(response.json()['conteudo'], 'lxml')
table = page.select('#oReportCell table')[-1]
for tr in table.find_all('tr'):
    row_values = [td.text.strip() for td in tr.find_all('td')]
    print(row_values)

Output:

['', '', '', '']
['', '', 'Taxas de juros']
['Posição', 'Instituição', '% a.m.', '% a.a.']
['1', 'SINOSSERRA S/A - SCFI', '0,47', '5,76']
['2', 'GRAZZIOTIN FINANCIADORA SA CFI', '0,81', '10,13']
['3', 'BCO CATERPILLAR S.A.', '0,91', '11,44']
['4', 'BCO DE LAGE LANDEN BRASIL S.A.', '0,91', '11,54']
['5', 'BCO VOLKSWAGEN S.A', '0,93', '11,76']
['6', 'BCO KOMATSU S.A.', '1,02', '12,92']
['7', 'BCO SANTANDER (BRASIL) S.A.', '1,13', '14,43']
['8', 'BCO VOLVO BRASIL S.A.', '1,16', '14,80']
['9', 'BCO DO ESTADO DO RS S.A.', '1,32', '17,07']
['10', 'BV FINANCEIRA S.A. CFI', '1,39', '18,05']
['11', 'FINANC ALFA S.A. CFI', '1,42', '18,43']
['12', 'AYMORÉ CFI S.A.', '1,44', '18,75']
['13', 'BCO RIBEIRAO PRETO S.A.', '1,46', '19,05']
['14', 'BCO BRADESCO S.A.', '1,47', '19,15']
['15', 'TODESCREDI S/A - CFI', '1,72', '22,75']
['16', 'CAIXA ECONOMICA FEDERAL', '2,46', '33,84']
['17', 'SIMPALA S.A. CFI', '2,50', '34,48']
['18', 'LEBES FINANCEIRA CFI SA', '3,12', '44,60']
['19', 'BCO RENDIMENTO S.A.', '3,15', '45,06']
['20', 'BECKER FINANCEIRA SA - CFI', '3,52', '51,47']
['21', 'BCO DO BRASIL S.A.', '3,61', '53,08']
['22', 'BCO CETELEM S.A.', '3,70', '54,65']
['23', 'LECCA CFI S.A.', '3,87', '57,65']
['24', 'HS FINANCEIRA', '3,98', '59,65']
['25', 'CREDIARE CFI S.A.', '4,17', '63,32']
['26', 'KREDILIG S.A. - CFI', '4,42', '68,06']
['27', 'CENTROCRED S.A. CFI', '4,60', '71,61']
['28', 'SENFF S.A. - CFI', '4,79', '75,31']
['29', 'ZEMA CFI S/A', '4,81', '75,68']
['30', 'VIA CERTA FINANCIADORA S.A. - CFI', '5,32', '86,31']
['31', 'OMNI BANCO S.A.', '5,35', '86,93']
['32', 'OMNI SA CFI', '5,42', '88,47']
['33', 'LUIZACRED S.A. SCFI', '5,55', '91,16']
['34', 'BCO HONDA S.A.', '5,67', '93,89']
['35', 'BCO LOSANGO S.A.', '5,71', '94,70']
['36', 'BANCO SEMEAR', '6,00', '101,13']
['37', 'NEGRESCO S.A. - CFI', '6,24', '106,69']
['38', 'GAZINCRED S.A. SCFI', '6,60', '115,24']
['39', 'PORTOCRED S.A. - CFI', '7,03', '125,93']
['40', 'AGORACRED S/A SCFI', '7,27', '132,10']

Answer 2

To locate and print the items from Posição, Instituição, % a.m. and % a.a. columns you need to induce WebDriverWait for the visibility_of_all_elements_located() and you can use either of the following Locator Strategies:

• Using XPATH:

  driver.get('https://www.bcb.gov.br/estatisticas/reporttxjuros?path=conteudo%2Ftxcred%2FReports%2FTaxasCredito-Consolidadas-porTaxasAnuais-Historico.rdl&nome=Hist%C3%B3rico%20Posterior%20a%2001%2F01%2F2012&exibeparametros=true')
  print([my_elem.text for my_elem in WebDriverWait(driver, 60).until(EC.visibility_of_all_elements_located((By.XPATH, "//div[text()='Instituição']//following::tr[@valign='top']//td/div")))])
  

• Note : You have to add the following imports :

  from selenium.webdriver.support.ui import WebDriverWait
  from selenium.webdriver.common.by import By
  from selenium.webdriver.support import expected_conditions as EC
  

• Console Output:

  ['1', 'SINOSSERRA S/A - SCFI', '   0,47', '     5,76', '2', 'GRAZZIOTIN FINANCIADORA SA CFI', '   0,81', '    10,13', '3', 'BCO CATERPILLAR S.A.', '   0,91', '    11,44', '4', 'BCO DE LAGE LANDEN BRASIL S.A.', '   0,91', '    11,54', '5', 'BCO VOLKSWAGEN S.A', '   0,93', '    11,76', '6', 'BCO KOMATSU S.A.', '   1,02', '    12,92', '7', 'BCO SANTANDER (BRASIL) S.A.', '   1,13', '    14,43', '8', 'BCO VOLVO BRASIL S.A.', '   1,16', '    14,80', '9', 'BCO DO ESTADO DO RS S.A.', '   1,32', '    17,07', '10', 'BV FINANCEIRA S.A. CFI', '   1,39', '    18,05', '11', 'FINANC ALFA S.A. CFI', '   1,42', '    18,43', '12', 'AYMORÉ CFI S.A.', '   1,44', '    18,75', '13', 'BCO RIBEIRAO PRETO S.A.', '   1,46', '    19,05', '14', 'BCO BRADESCO S.A.', '   1,47', '    19,15', '15', 'TODESCREDI S/A - CFI', '   1,72', '    22,75', '16', 'CAIXA ECONOMICA FEDERAL', '   2,46', '    33,84', '17', 'SIMPALA S.A. CFI', '   2,50', '    34,48', '18', 'LEBES FINANCEIRA CFI SA', '   3,12', '    44,60', '19', 'BCO RENDIMENTO S.A.', '   3,15', '    45,06', '20', 'BECKER FINANCEIRA SA - CFI', '   3,52', '    51,47', '21', 'BCO DO BRASIL S.A.', '   3,61', '    53,08', '22', 'BCO CETELEM S.A.', '   3,70', '    54,65', '23', 'LECCA CFI S.A.', '   3,87', '    57,65', '24', 'HS FINANCEIRA', '   3,98', '    59,65', '25', 'CREDIARE CFI S.A.', '   4,17', '    63,32', '26', 'KREDILIG S.A. - CFI', '   4,42', '    68,06', '27', 'CENTROCRED S.A. CFI', '   4,60', '    71,61', '28', 'SENFF S.A. - CFI', '   4,79', '    75,31', '29', 'ZEMA CFI S/A', '   4,81', '    75,68', '30', 'VIA CERTA FINANCIADORA S.A. - CFI', '   5,32', '    86,31', '31', 'OMNI BANCO S.A.', '   5,35', '    86,93', '32', 'OMNI SA CFI', '   5,42', '    88,47', '33', 'LUIZACRED S.A. SCFI', '   5,55', '    91,16', '34', 'BCO HONDA S.A.', '   5,67', '    93,89', '35', 'BCO LOSANGO S.A.', '   5,71', '    94,70', '36', 'BANCO SEMEAR', '   6,00', '   101,13', '37', 'NEGRESCO S.A. - CFI', '   6,24', '   106,69', '38', 'GAZINCRED S.A. SCFI', '   6,60', '   115,24', '39', 'PORTOCRED S.A. - CFI', '   7,03', '   125,93', '40', 'AGORACRED S/A SCFI', '   7,27', '   132,10']