Python Selenium Timeout Exception when scraping from dynamically loaded table within an iframe

Question

I am trying to scrape from a dynamically loaded table using Selenium

Since it is dynamically loaded by javascript, I need to use Webdriverwait but I am keep getting

Timeout exception errors

My code is as below :

from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
driver = webdriver.Chrome("my_path")
import time


URL = r'https://www.customs.go.jp/toukei/srch/indexe.htm?M=09&P=1,2,,,,,,,,1,0,2020,0,10,0,2,440131,,,,,,,,,,1,,,,,,,,,,,,,,,,,,,,,,200'

driver.get(URL)



data = WebDriverWait(driver, 15).until(EC.element_to_be_clickable((By.CSS_SELECTOR, ".value")))
print(data.text)

I have searching through all possible ways to do the WebDriverWait, but all the below options did not work

WebDriverWait(driver, 15).until_not(EC.title_is(title))
driver.set_script_timeout(10)
time.sleep(10)
element = wait.until(EC.element_to_be_clickable((By.XPATH, '//div[@class="test"]')))

My ultimate goal is to scrape the table from the URL. Can anyone help?

undetected Selenium · Accepted Answer · 2022-02-18T11:56:16.060

The element #_ctl12_btnExportCSV is a __doPostBack enabled element.

As the element is within an <iframe> so you have to:

Induce WebDriverWait for the desired frame to be available and switch to it.
Induce WebDriverWait for the desired element to be clickable.

You can use either of the following Locator Strategies:

Using CSS_SELECTOR:

driver.get('https://www.customs.go.jp/toukei/srch/indexe.htm?M=09&P=1,2,,,,,,,,1,0,2020,0,10,0,2,440131,,,,,,,,,,1,,,,,,,,,,,,,,,,,,,,,,200')
WebDriverWait(driver, 20).until(EC.frame_to_be_available_and_switch_to_it((By.NAME,"FR_M_INFO")))
print(WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.CSS_SELECTOR, "table.value"))).text)

Using XPATH:

driver.get('https://www.customs.go.jp/toukei/srch/indexe.htm?M=09&P=1,2,,,,,,,,1,0,2020,0,10,0,2,440131,,,,,,,,,,1,,,,,,,,,,,,,,,,,,,,,,200')
WebDriverWait(driver, 20).until(EC.frame_to_be_available_and_switch_to_it((By.NAME,"TopPage")))
print(WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.XPATH, "//table[@class='value']"))).text)

Note : You have to add the following imports :

 from selenium.webdriver.support.ui import WebDriverWait
 from selenium.webdriver.common.by import By
 from selenium.webdriver.support import expected_conditions as EC

Console Output:

CUSTOMS UNIT1 UNIT2 CURRENT MONTH CUMULATIVE YEAR TO DATE
QUANTITY1 QUANTITY2 VALUE QUANTITY1 QUANTITY2 VALUE
4401.31-000
(105 CHINA)TOTAL   MT   46 2516   350 17871
100 TOKYO   MT   - -   26 1458
200 YOKOHAMA   MT   - -   5 285
260 SENDAISHIOGAMA   MT   5 268   27 1544
370 MATSUYAMA   MT   7 382   43 2363
400 OSAKA   MT   17 945   67 4135
402 OSAKA SAKAI   MT   - -   102 3603
500 NAGOYA   MT   11 610   57 3211
803 TOMAKOMAI   MT   - -   17 961
804 OTARU   MT   6 311   6 311
(110 VIETNAM)TOTAL   MT   52761 834897   819727 14059241
100 TOKYO   MT   78 1824   1066 18003
200 YOKOHAMA   MT   155 2123   3579 53269
243 KASHIMA   MT   - -   60782 988066
244 HITACHI   MT   14300 219359   142892 2425105
250 ONAHAMA   MT   - -   53262 890457
252 SOOMA   MT   - -   23196 407325
260 SENDAISHIOGAMA   MT   - -   3058 33513
262 ISHINOMAKI   MT   - -   30570 585228
300 KOBE   MT   - -   156 2753
342 FUKUYAMA   MT   - -   341 7919
352 HAMADA   MT   - -   20 362
370 MATSUYAMA   MT   - -   23 432
400 OSAKA   MT   - -   14 307
500 NAGOYA   MT   14 209   14 209
505 KINUURA   MT   - -   49001 751832
520 SHIMIZU   MT   - -   10 278
540 YOKKAICHI   MT   - -   94539 1649611
600 MOJI   MT   21820 348327   57148 921286
603 TOBATA   MT   9580 158030   208229 3741854
604 HAKATA   MT   - -   60 996
620 SHIMONOSEKI   MT   - -   20 222
660 OOITA   MT   - -   45979 864566
670 HOSOJIMA   MT   - -   10433 148152
750 KAGOSHIMA   MT   6300 96702   31200 496288
754 SHIBUSHI   MT   154 2562   560 9606
755 SENDAI   MT   260 3986   1972 31489
803 TOMAKOMAI   MT   52 908   547 10816
807 KUSHIRO   MT   - -   618 11181
822 HACHINOHE   MT   - -   78 1276
842 KAMAISHI   MT   - -   168 3231
900 OKINAWA CHIKU   MT   48 867   192 3609
(111 THAILND)TOTAL   MT   - -   11091 220690
244 HITACHI   MT   - -   10552 200467
370 MATSUYAMA   MT   - -   29 446
624 TOKUYAMA   MT   - -   510 19777
(113 MALYSIA)TOTAL   MT   486 6944   77339 1366094
243 KASHIMA   MT   - -   22000 326139
322 MIZUSHIMA   MT   - -   25 371
603 TOBATA   MT   - -   11319 218308
623 UBE   MT   - -   1751 30598
660 OOITA   MT   - -   34766 689086
670 HOSOJIMA   MT   - -   3544 44067
754 SHIBUSHI   MT   - -   76 1107
803 TOMAKOMAI   MT   - -   51 778
822 HACHINOHE   MT   486 6944   3807 55640
(117 PHILPIN)TOTAL   MT   - -   77 656
345 HIROSHIMA   MT   - -   77 656
(118 INDNSIA)TOTAL   MT   332 5448   13731 218740
200 YOKOHAMA   MT   50 747   555 8707
260 SENDAISHIOGAMA   MT   - -   312 4894
300 KOBE   MT   - -   25 410
322 MIZUSHIMA   MT   - -   234 3739
500 NAGOYA   MT   128 2120   1162 20451
505 KINUURA   MT   - -   10090 157615
803 TOMAKOMAI   MT   101 1747   348 6182
822 HACHINOHE   MT   53 834   1005 16742
(203 SWEDEN)TOTAL   MT   - -   6 1070
400 OSAKA   MT   - -   6 1070
(224 RUSSIA)TOTAL   MT   - -   25 441
803 TOMAKOMAI   MT   - -   25 441
(235 ESTONIA)TOTAL   MT   - -   23 641
500 NAGOYA   MT   - -   23 641
(302 CANADA)TOTAL   MT   67190 1365696   492185 9981576
100 TOKYO   MT   - -   221 6918
140 SAKATA   MT   - -   69000 1426484
250 ONAHAMA   MT   - -   20200 422807
252 SOOMA   MT   21000 421282   83027 1672320
262 ISHINOMAKI   MT   - -   72876 1558082
300 KOBE   MT   - -   585 13187
400 OSAKA   MT   - -   211 4208
430 MAIZURU   MT   7637 141124   7637 141124
505 KINUURA   MT   - -   133377 2522015
623 UBE   MT   - -   41403 881231
624 TOKUYAMA   MT   10000 206451   30498 620841
628 IWAKUNI   MT   28000 582954   28000 582954
754 SHIBUSHI   MT   - -   50 1327
803 TOMAKOMAI   MT   527 13184   4997 125268
900 OKINAWA CHIKU   MT   26 701   103 2810
(304 USA)TOTAL   MT   - -   31 2678
202 KAWASAKI   MT   - -   2 255
300 KOBE   MT   - -   0 230
372 IMABARI   MT   - -   29 2193
(601 AUSTRAL)TOTAL   MT   - -   70 5362
352 HAMADA   MT   - -   10 742
700 NAGASAKI   MT   - -   60 4620
(606 NEWZELD)TOTAL   MT   - -   37 1650
200 YOKOHAMA   MT   - -   9 925
300 KOBE   MT   - -   28 725

Reference

You can find a couple of relevant discussions in:

Thank you so much!! It has really been useful. If you have the time, can I ask you one further question from here? Is there any possible way to convert the text to a data frame so that it is easy to handle? — Hanalia, Dec 21 '20 at 12:52
@Hanalia Apologies, rarely I deal with _data frame_. Feel free to raise a new question on SO. — undetected Selenium, Dec 21 '20 at 12:54

Python Selenium Timeout Exception when scraping from dynamically loaded table within an iframe

1 Answers1

Reference