1

I am trying to crawl this currency rate website: https://banking.nonghyup.com/servlet/PGEF0011I.view

(Just click the exchange rate check and you'll see the pic like below)

enter image description here

I want to crawl this dynamical table and I find its url via "Inspect->Network is : https://banking.nonghyup.com/servlet/PGEF0012R.frag

However, when I start to request it by selenium, it returns error due to abnormal requests, which comes up "We're sorry for causing you any inconvenience. You will not be able to use the service due to abnormal request. Please try again after confirmation." I am not sure how to proceed to the next step. Could you help? Many thanks!

driver.get('https://banking.nonghyup.com/servlet/PGEF0012R.frag') 
soup = BeautifulSoup(driver.page_source, 'html.parser')
undetected Selenium
  • 183,867
  • 41
  • 278
  • 352
ShanChe Wu
  • 315
  • 2
  • 8

1 Answers1

1

Seems you were close. Accessing the url https://banking.nonghyup.com/servlet/PGEF0011I.view manually for the first time also shows the following error:

We're sorry for causing you any inconvenience. You will not be able to use the service due to abnormal request. Please try again after confirmation.

Guide Code : 90304, 16.49-03

Snapshot:

error

However, as per your mentioned steps through clicking on the element with text as Exchange Rate Check also you can crawl the currency rates using the following Locator Strategies:

  • Code Block:

    from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

options = webdriver.ChromeOptions()
options.add_argument("start-maximized")
options.add_experimental_option("excludeSwitches", ["enable-automation"])
options.add_experimental_option('useAutomationExtension', False)
driver = webdriver.Chrome(options=options, executable_path=r'C:\Utility\BrowserDrivers\chromedriver.exe')
driver.get("https://banking.nonghyup.com/servlet/PGEF0011I.view")
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.LINK_TEXT, "Exchange Rate Check"))).click()
WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.XPATH, "//table[starts-with(@summary, 'Country,Currency Namem,Cash')]//td")))
print(driver.find_element_by_xpath("//table[starts-with(@summary, 'Country,Currency Namem,Cash')]").get_attribute("outerHTML"))

  • Console Output:

    <table class="tb_col t_center" summary="Country,Currency Namem,Cash(When Buying,When Selling),Remittance(When Sending,When Receiving),Base Exchange Rate,Compared to D-1">
                    <caption>??????2</caption>
                    <colgroup>
                        <col style="width:12%;">
                        <col style="width:auto;">
                        <col style="width:13%;">
                        <col style="width:13%;">
                        <col style="width:13%;">
                        <col style="width:13%;">
                        <col style="width:13%;">
                        <col style="width:13%;">
                    </colgroup>
                    <thead>
                        <tr>
                            <th rowspan="2">Country</th>
                            <th rowspan="2">Currency Name</th>
                            <th colspan="2">Cash</th>
                            <th colspan="2">Remittance</th>
                            <th rowspan="2">Base Exchange Rate</th>
                            <th rowspan="2" style="border-right: none;">Compared to the previous day</th>
                        </tr>
                        <tr>
                            <th>When Buying</th>
                            <th>When Selling</th>
                            <th>When Sending</th>
                            <th>When Receiving</th>
                        </tr>
                    </thead>
                    <tbody>

                        <tr>
                            <td>USA</td>

                                <td>USD</td>


                                <td class="t_right">1,188.74</td>
                                <td class="t_right">1,147.86</td>
                                <td class="t_right">1,179.60</td>
                                <td class="t_right">1,157.00</td>

                            <td class="t_right">1,168.30</td>
                            <td style="border-right: none;">
                            <span class="raise"><em>??</em>5.80</span>
                            </td>
                        </tr>


                        <tr>
                            <td>Japan</td>

                                <td>JPY(100)</td>


                                <td class="t_right">1,090.24</td>
                                <td class="t_right">1,052.74</td>
                                <td class="t_right">1,081.99</td>
                                <td class="t_right">1,060.99</td>

                            <td class="t_right">1,071.49</td>
                            <td style="border-right: none;">
                            <span class="raise"><em>??</em>7.03</span>
                            </td>
                        </tr>


                        <tr>
                            <td>EU</td>

                                <td>EUR</td>


                                <td class="t_right">1,312.25</td>
                                <td class="t_right">1,260.79</td>
                                <td class="t_right">1,299.38</td>
                                <td class="t_right">1,273.66</td>

                            <td class="t_right">1,286.52</td>
                            <td style="border-right: none;">
                            <span class="raise"><em>??</em>4.76</span>
                            </td>
                        </tr>


                        <tr>
                            <td>China</td>

                                <td>CNY</td>


                                <td class="t_right">176.35</td>
                                <td class="t_right">156.39</td>
                                <td class="t_right">168.01</td>
                                <td class="t_right">164.73</td>

                            <td class="t_right">166.37</td>
                            <td style="border-right: none;">
                            <span class="raise"><em>??</em>0.43</span>
                            </td>
                        </tr>


                        <tr>
                            <td>England</td>

                                <td>GBP</td>


                                <td class="t_right">1,529.50</td>
                                <td class="t_right">1,469.52</td>
                                <td class="t_right">1,514.35</td>
                                <td class="t_right">1,484.67</td>

                            <td class="t_right">1,499.51</td>
                            <td style="border-right: none;">
                            <span class="raise"><em>??</em>9.42</span>
                            </td>
                        </tr>


                        <tr>
                            <td>Swiss</td>

                                <td>CHF</td>


                                <td class="t_right">1,203.57</td>
                                <td class="t_right">1,156.39</td>
                                <td class="t_right">1,191.66</td>
                                <td class="t_right">1,168.30</td>

                            <td class="t_right">1,179.98</td>
                            <td style="border-right: none;">
                            <span class="raise"><em>??</em>11.06</span>
                            </td>
                        </tr>


                        <tr>
                            <td>Canada</td>

                                <td>CAD</td>


                                <td class="t_right">898.34</td>
                                <td class="t_right">863.12</td>
                                <td class="t_right">889.53</td>
                                <td class="t_right">871.93</td>

                            <td class="t_right">880.73</td>
                            <td style="border-right: none;">
                            <span class="raise"><em>??</em>3.55</span>
                            </td>
                        </tr>


                        <tr>
                            <td>Hong Kong</td>

                                <td>HKD</td>


                                <td class="t_right">152.19</td>
                                <td class="t_right">146.23</td>
                                <td class="t_right">150.68</td>
                                <td class="t_right">147.74</td>

                            <td class="t_right">149.21</td>
                            <td style="border-right: none;">
                            <span class="raise"><em>??</em>0.67</span>
                            </td>
                        </tr>


                        <tr>
                            <td>Sweden</td>

                                <td>SEK</td>


                                <td class="t_right">123.05</td>
                                <td class="t_right">117.05</td>
                                <td class="t_right">121.23</td>
                                <td class="t_right">118.87</td>

                            <td class="t_right">120.05</td>
                            <td style="border-right: none;">
                            <span class="raise"><em>??</em>0.21</span>
                            </td>
                        </tr>


                        <tr>
                            <td>Australia</td>

                                <td>AUD</td>


                                <td class="t_right">814.61</td>
                                <td class="t_right">782.67</td>
                                <td class="t_right">806.54</td>
                                <td class="t_right">790.74</td>

                            <td class="t_right">798.64</td>
                            <td style="border-right: none;">
                            <span class="raise"><em>??</em>3.32</span>
                            </td>
                        </tr>


                        <tr>
                            <td>Denmark</td>

                                <td>DKK</td>


                                <td class="t_right">176.47</td>
                                <td class="t_right">167.87</td>
                                <td class="t_right">173.87</td>
                                <td class="t_right">170.47</td>

                            <td class="t_right">172.17</td>
                            <td style="border-right: none;">
                            <span class="raise"><em>??</em>0.64</span>
                            </td>
                        </tr>


                        <tr>
                            <td>Norway</td>

                                <td>NOK</td>


                                <td class="t_right">130.28</td>
                                <td class="t_right">123.94</td>
                                <td class="t_right">128.36</td>
                                <td class="t_right">125.86</td>

                            <td class="t_right">127.11</td>
                            <td style="border-right: none;">
                            <span class="raise"><em>??</em>0.09</span>
                            </td>
                        </tr>


                        <tr>
                            <td>Saudi Arabia</td>

                                <td>SAR</td>


                                <td class="t_right">342.67</td>
                                <td class="t_right">280.37</td>
                                <td class="t_right">314.60</td>
                                <td class="t_right">308.44</td>

                            <td class="t_right">311.52</td>
                            <td style="border-right: none;">
                            <span class="raise"><em>??</em>1.54</span>
                            </td>
                        </tr>


                        <tr>
                            <td>Kuwait</td>

                                <td>KWD</td>


                                <td class="t_right">4,230.03</td>
                                <td class="t_right">3,460.95</td>
                                <td class="t_right">3,883.56</td>
                                <td class="t_right">3,807.42</td>

                            <td class="t_right">3,845.49</td>
                            <td style="border-right: none;">
                            <span class="raise"><em>??</em>19.34</span>
                            </td>
                        </tr>


                        <tr>
                            <td>Bahrain</td>

                                <td>BHD</td>


                                <td class="t_right">3,408.69</td>
                                <td class="t_right">2,788.93</td>
                                <td class="t_right">3,129.48</td>
                                <td class="t_right">3,068.14</td>

                            <td class="t_right">3,098.81</td>
                            <td style="border-right: none;">
                            <span class="raise"><em>??</em>15.35</span>
                            </td>
                        </tr>


                        <tr>
                            <td>UAE</td>

                                <td>AED</td>


                                <td class="t_right">349.86</td>
                                <td class="t_right">286.26</td>
                                <td class="t_right">321.20</td>
                                <td class="t_right">314.92</td>

                            <td class="t_right">318.06</td>
                            <td style="border-right: none;">
                            <span class="raise"><em>??</em>1.56</span>
                            </td>
                        </tr>


                        <tr>
                            <td>Singapore</td>

                                <td>SGD</td>


                                <td class="t_right">878.86</td>
                                <td class="t_right">836.00</td>
                                <td class="t_right">865.91</td>
                                <td class="t_right">848.95</td>

                            <td class="t_right">857.43</td>
                            <td style="border-right: none;">
                            <span class="raise"><em>??</em>3.35</span>
                            </td>
                        </tr>


                        <tr>
                            <td>Malaysia</td>

                                <td>MYR</td>


                                <td class="t_right">298.22</td>
                                <td class="t_right">256.02</td>
                                <td class="t_right">284.12</td>
                                <td class="t_right">278.56</td>

                            <td class="t_right">281.34</td>
                            <td style="border-right: none;">
                            <span class="raise"><em>??</em>0.58</span>
                            </td>
                        </tr>


                        <tr>
                            <td>New Zealand</td>

                                <td>NZD</td>


                                <td class="t_right">762.00</td>
                                <td class="t_right">732.12</td>
                                <td class="t_right">754.45</td>
                                <td class="t_right">739.67</td>

                            <td class="t_right">747.06</td>
                            <td style="border-right: none;">
                            <span class="raise"><em>??</em>10.62</span>
                            </td>
                        </tr>


                        <tr>
                            <td>Thailand</td>

                                <td>THB</td>


                                <td class="t_right">41.13</td>
                                <td class="t_right">36.11</td>
                                <td class="t_right">39.00</td>
                                <td class="t_right">38.24</td>

                            <td class="t_right">38.62</td>
                            <td style="border-right: none;">
                            <span class="raise"><em>??</em>0.31</span>
                            </td>
                        </tr>


                        <tr>
                            <td>Indonesia</td>

                                <td>IDR(100)</td>


                                <td class="t_right">9.11</td>
                                <td class="t_right">7.47</td>
                                <td class="t_right">8.37</td>
                                <td class="t_right">8.21</td>

                            <td class="t_right">8.29</td>
                            <td style="border-right: none;">
                            <span class="raise"><em>??</em>0.03</span>
                            </td>
                        </tr>


                        <tr>
                            <td>Taiwan</td>

                                <td>TWD</td>


                                <td class="t_right">40.77</td>
                                <td class="t_right">35.81</td>
                                <td class="t_right">38.66</td>
                                <td class="t_right">37.92</td>

                            <td class="t_right">38.29</td>
                            <td style="border-right: none;">
                            <span class="raise"><em>??</em>0.10</span>
                            </td>
                        </tr>


                        <tr>
                            <td>Philippine</td>

                                <td>PHP</td>


                                <td class="t_right">24.44</td>
                                <td class="t_right">21.46</td>
                                <td class="t_right">23.17</td>
                                <td class="t_right">22.73</td>

                            <td class="t_right">22.95</td>
                            <td style="border-right: none;">
                            <span class="raise"><em>??</em>0.07</span>
                            </td>
                        </tr>


                        <tr>
                            <td>India</td>

                                <td>INR</td>


                                <td class="t_right">17.72</td>
                                <td class="t_right">14.64</td>
                                <td class="t_right">16.42</td>
                                <td class="t_right">16.10</td>

                            <td class="t_right">16.26</td>
                            <td style="border-right: none;">
                            <span class="raise"><em>??</em>0.03</span>
                            </td>
                        </tr>


                        <tr>
                            <td>Russia</td>

                                <td>RUB</td>


                                <td class="t_right">19.40</td>
                                <td class="t_right">15.97</td>
                                <td class="t_right">18.31</td>
                                <td class="t_right">17.97</td>

                            <td class="t_right">18.14</td>
                            <td style="border-right: none;">
                            <span class="decline"><em>??</em>0.01</span>
                            </td>
                        </tr>


                        <tr>
                            <td>South Africa</td>

                                <td>ZAR</td>


                                <td class="t_right">78.10</td>
                                <td class="t_right">78.10</td>
                                <td class="t_right">79.03</td>
                                <td class="t_right">77.17</td>

                            <td class="t_right">78.10</td>
                            <td style="border-right: none;">
                            <span>0.00</span>
                            </td>
                        </tr>


                        <tr>
                            <td>Mexico</td>

                                <td>MXN</td>


                                <td class="t_right">60.38</td>
                                <td class="t_right">60.38</td>
                                <td class="t_right">60.98</td>
                                <td class="t_right">59.78</td>

                            <td class="t_right">60.38</td>
                            <td style="border-right: none;">
                            <span class="decline"><em>??</em>0.37</span>
                            </td>
                        </tr>


                        <tr>
                            <td>Vietnam</td>

                                <td>VND(100)</td>


                                <td class="t_right"> - </td>
                                <td class="t_right"> - </td>
                                <td class="t_right"> - </td>
                                <td class="t_right"> - </td>                                

                            <td class="t_right">5.03</td>
                            <td style="border-right: none;">
                            <span class="raise"><em>??</em>0.03</span>
                            </td>
                        </tr>


                    </tbody>    
                </table>
undetected Selenium
  • 183,867
  • 41
  • 278
  • 352