Selenium & BeautifulSoup cannot find Fields table elements within #shadow_root

Question

I have been trying to scrape data from the following site: https://developer.salesforce.com/docs/atlas.en-us.netzero_cloud_dev_guide.meta/netzero_cloud_dev_guide/sforce_api_objects_airtravelemssnfctr.htm#maincontent.

I want to obtain the Fields table but nothing seems to be working. Anytime I try to get any element from the site I get the following error: 'NoSuchElementException: Message: no such element: Unable to locate element'

I have tried everything!

Some things I have tried:

driver.get('https://developer.salesforce.com/docs/atlas.en-us.netzero_cloud_dev_guide.meta/netzero_cloud_dev_guide/sforce_api_objects_airtravelemssnfctr.htm')
driver.find_element("name", "featureTable sort_table")

driver.get('https://developer.salesforce.com/docs/atlas.en-us.netzero_cloud_dev_guide.meta/netzero_cloud_dev_guide/sforce_api_objects_airtravelemssnfctr.htm')
driver.find_element("xpath", '//*[@id="maincontent"]/doc-xml-content//doc-content-layout/doc-content//div/div[1]/div[3]/div/table')

And many more. I have also tried WebDriverWait to let the page load and attempted to change to iframe but I don't think there is one as I cannot find an iframe name or ID.

Anything helps! Thanks in advance!

score 0 · Answer 1 · answered Aug 12 '22 at 07:01

0

Element you are trying to access is inside the shadow-rot.
There are several techniques to access such elements.
You can find here several questions dealing with it, like this:

answered Aug 12 '22 at 07:01

Prophet

32,350
22
54
79

score 0 · Answer 2 · answered Aug 12 '22 at 12:11

shadow_root

The shadow_root attribute returns a shadow root of the element if there is one or an error. Only works from Chromium 96 onwards. Previous versions of Chromium based browsers will throw an assertion exception.

Solution

Using google-chrome v96 (and above) and selenium4 to access the Fields table, as the table element is within multiple #shadow-root (open) you can use the following locator strategies:

Code Block:

driver = webdriver.Chrome(service=s, options=options)
driver.execute("get", {'url': 'https://developer.salesforce.com/docs/atlas.en-us.netzero_cloud_dev_guide.meta/netzero_cloud_dev_guide/sforce_api_objects_airtravelemssnfctr.htm#maincontent'})
shadow_host = driver.find_element(By.CSS_SELECTOR, 'doc-xml-content')
shadow_root = shadow_host.shadow_root
shadow_child = shadow_root.find_element(By.CSS_SELECTOR, 'doc-content')
shadow_grand_child = shadow_child.shadow_root
element = shadow_grand_child.find_element(By.CSS_SELECTOR, 'table.featureTable')
print(element.get_attribute("outerHTML"))
driver.quit()

Console Output:

<table class="featureTable sort_table" summary="">


               <thead class="thead sorted" align="left">
                  <tr>
                     <th class="featureTableHeader  " id="d51659e96">Field</th>

                     <th class="featureTableHeader  " id="d51659e99">Details</th>

                  </tr>

               </thead>

               <tbody class="tbody">
                  <tr>
                     <td class="entry" headers="d51659e96" data-title="Field"><span class="keyword parmname">Ch4PsgrKmLongHaulInKgCo2e</span></td>

                     <td class="entry" headers="d51659e99" data-title="Details">
                        <dl class="dl detailList">

                              <dt class="dt dlterm">Type</dt>

                              <dd class="dd">double</dd>



                              <dt class="dt dlterm">Properties</dt>

                              <dd class="dd">Create, Filter, Nillable, Sort, Update</dd>



                              <dt class="dt dlterm">Description</dt>

                              <dd class="dd"> The CH4 emissions per passenger-kilometer in CO2e from long-haul
                                 flights. </dd>


                        </dl>

                     </td>

                  </tr>

                  <tr>
                     <td class="entry" headers="d51659e96" data-title="Field"><span class="keyword parmname">Ch4PsgrKmMediumHaulInKgCo2e</span></td>

                     <td class="entry" headers="d51659e99" data-title="Details">
                        <dl class="dl detailList">

                              <dt class="dt dlterm">Type</dt>

                              <dd class="dd">double</dd>



                              <dt class="dt dlterm">Properties</dt>

                              <dd class="dd">Create, Filter, Nillable, Sort, Update</dd>



                              <dt class="dt dlterm">Description</dt>

                              <dd class="dd"> The CH4 emissions per passenger-kilometer in CO2e from
                                 medium-haul flights. </dd>


                        </dl>

                     </td>

                  </tr>
                  <tr>
                     <td class="entry" headers="d51659e96" data-title="Field"><span class="keyword parmname">DistanceUnit</span></td>

                     <td class="entry" headers="d51659e99" data-title="Details">
                        <dl class="dl detailList">

                              <dt class="dt dlterm">Type</dt>

                              <dd class="dd">picklist</dd>



                              <dt class="dt dlterm">Properties</dt>

                              <dd class="dd">Create, Defaulted on create, Filter, Group, Nillable, Restricted
                                 picklist, Sort, Update</dd>



                              <dt class="dt dlterm">Description</dt>

                              <dd class="dd"> The unit of measure for the distance. </dd>

                              <dd class="dd">Possible values are: <ul class="ul bulletList">
                                    <li class="li"><samp class="codeph nolang">Kilometers</samp></li>

                                    <li class="li"><samp class="codeph nolang">Miles</samp></li>

                                 </ul>

                              </dd>

                              <dd class="dd">The default value is 'Kilometers'.</dd>


                        </dl>

                     </td>

                  </tr>

                  <tr>
                     <td class="entry" headers="d51659e96" data-title="Field"><span class="keyword parmname">EmissionFactorDataSource</span></td>

                     <td class="entry" headers="d51659e99" data-title="Details">
                        <dl class="dl detailList">

                              <dt class="dt dlterm">Type</dt>

                              <dd class="dd">textarea</dd>



                              <dt class="dt dlterm">Properties</dt>

                              <dd class="dd">Create, Nillable, Update</dd>



                              <dt class="dt dlterm">Description</dt>

                              <dd class="dd"> The source of the emissions factor reference data. </dd>


                        </dl>

                     </td>

                  </tr>

                  <tr>
                     <td class="entry" headers="d51659e96" data-title="Field"><span class="keyword parmname">EmissionFactorUpdateYear</span></td>

                     <td class="entry" headers="d51659e99" data-title="Details">
                        <dl class="dl detailList">

                              <dt class="dt dlterm">Type</dt>

                              <dd class="dd">picklist</dd>



                              <dt class="dt dlterm">Properties</dt>

                              <dd class="dd">Create, Filter, Group, Nillable, Restricted picklist, Sort,
                                 Update</dd>



                              <dt class="dt dlterm">Description</dt>

                              <dd class="dd"> The year in which this reference data for the emissions factor
                                 was most recently updated. </dd>

                              <dd class="dd">Possible values are: <ul class="ul bulletList">

                                    <li class="li"><samp class="codeph nolang">2000</samp></li>

                                    <li class="li"><samp class="codeph nolang">2001</samp></li>

                                    <li class="li"><samp class="codeph nolang">2002</samp></li>

                                    <li class="li"><samp class="codeph nolang">2003</samp></li>

                                    <li class="li"><samp class="codeph nolang">2004</samp></li>

                                    <li class="li"><samp class="codeph nolang">2005</samp></li>

                                    <li class="li"><samp class="codeph nolang">2006</samp></li>

                                    <li class="li"><samp class="codeph nolang">2007</samp></li>

                                    <li class="li"><samp class="codeph nolang">2008</samp></li>

                                    <li class="li"><samp class="codeph nolang">2009</samp></li>

                                    <li class="li"><samp class="codeph nolang">2010</samp></li>

                                    <li class="li"><samp class="codeph nolang">2011</samp></li>

                                    <li class="li"><samp class="codeph nolang">2012</samp></li>

                                    <li class="li"><samp class="codeph nolang">2013</samp></li>

                                    <li class="li"><samp class="codeph nolang">2014</samp></li>

                                    <li class="li"><samp class="codeph nolang">2015</samp></li>

                                    <li class="li"><samp class="codeph nolang">2016</samp></li>

                                    <li class="li"><samp class="codeph nolang">2017</samp></li>

                                    <li class="li"><samp class="codeph nolang">2018</samp></li>

                                    <li class="li"><samp class="codeph nolang">2019</samp></li>

                                    <li class="li"><samp class="codeph nolang">2020</samp></li>

                                    <li class="li"><samp class="codeph nolang">2021</samp></li>

                                    <li class="li"><samp class="codeph nolang">2022</samp></li>

                                    <li class="li"><samp class="codeph nolang">2023</samp></li>

                                    <li class="li"><samp class="codeph nolang">2024</samp></li>

                                    <li class="li"><samp class="codeph nolang">2025</samp></li>

                                    <li class="li"><samp class="codeph nolang">2026</samp></li>

                                    <li class="li"><samp class="codeph nolang">2027</samp></li>

                                    <li class="li"><samp class="codeph nolang">2028</samp></li>

                                    <li class="li"><samp class="codeph nolang">2029</samp></li>

                                    <li class="li"><samp class="codeph nolang">2030</samp></li>

                                    <li class="li"><samp class="codeph nolang">2031</samp></li>

                                    <li class="li"><samp class="codeph nolang">2032</samp></li>

                                    <li class="li"><samp class="codeph nolang">2033</samp></li>

                                    <li class="li"><samp class="codeph nolang">2034</samp></li>

                                    <li class="li"><samp class="codeph nolang">2035</samp></li>

                                    <li class="li"><samp class="codeph nolang">2036</samp></li>

                                    <li class="li"><samp class="codeph nolang">2037</samp></li>

                                    <li class="li"><samp class="codeph nolang">2038</samp></li>

                                    <li class="li"><samp class="codeph nolang">2039</samp></li>

                                    <li class="li"><samp class="codeph nolang">2040</samp></li>

                                 </ul>

                              </dd>


                        </dl>

                     </td>

                  </tr>

                  <tr>
                     <td class="entry" headers="d51659e96" data-title="Field"><span class="keyword parmname">LastReferencedDate</span></td>

                     <td class="entry" headers="d51659e99" data-title="Details">
                        <dl class="dl detailList">

                              <dt class="dt dlterm">Type</dt>

                              <dd class="dd">dateTime</dd>



                              <dt class="dt dlterm">Properties</dt>

                              <dd class="dd">Filter, Nillable, Sort</dd>



                              <dt class="dt dlterm">Description</dt>

                              <dd class="dd">The timestamp for when the current user last viewed a record
                                 related to this record.</dd>


                        </dl>

                     </td>

                  </tr>

                  <tr>
                     <td class="entry" headers="d51659e96" data-title="Field"><span class="keyword parmname">LastViewedDate</span></td>

                     <td class="entry" headers="d51659e99" data-title="Details">
                        <dl class="dl detailList">

                              <dt class="dt dlterm">Type</dt>

                              <dd class="dd">dateTime</dd>



                              <dt class="dt dlterm">Properties</dt>

                              <dd class="dd">Filter, Nillable, Sort</dd>



                              <dt class="dt dlterm">Description</dt>

                              <dd class="dd">The timestamp for when the current user last viewed this record.
                                 If this value is null, this record might only have been referenced
                                 (LastReferencedDate) and not viewed. </dd>


                        </dl>

                     </td>

                  </tr>

                  <tr>
                     <td class="entry" headers="d51659e96" data-title="Field"><span class="keyword parmname">LongHaulMinimumDistance</span></td>

                     <td class="entry" headers="d51659e99" data-title="Details">
                        <dl class="dl detailList">

                              <dt class="dt dlterm">Type</dt>

                              <dd class="dd">double</dd>



                              <dt class="dt dlterm">Properties</dt>

                              <dd class="dd">Create, Filter, Nillable, Sort, Update</dd>



                              <dt class="dt dlterm">Description</dt>

                              <dd class="dd"> The minimum distance for a long-haul flight that’s adjusted
                                 according to the short-haul or medium-haul distances. </dd>


                        </dl>

                     </td>

                  </tr>

                  <tr>
                     <td class="entry" headers="d51659e96" data-title="Field"><span class="keyword parmname">MediumHaulMaximumDistance</span></td>

                     <td class="entry" headers="d51659e99" data-title="Details">
                        <dl class="dl detailList">

                              <dt class="dt dlterm">Type</dt>

                              <dd class="dd">double</dd>



                              <dt class="dt dlterm">Properties</dt>

                              <dd class="dd">Create, Filter, Nillable, Sort, Update</dd>



                              <dt class="dt dlterm">Description</dt>

                              <dd class="dd"> The maximum distance of a medium-haul flight. </dd>


                        </dl>

                     </td>

                  </tr>

                  <tr>
                     <td class="entry" headers="d51659e96" data-title="Field"><span class="keyword parmname">N2oPsgrKmLongHaulInKgCo2e</span></td>

                     <td class="entry" headers="d51659e99" data-title="Details">
                        <dl class="dl detailList">

                              <dt class="dt dlterm">Type</dt>

                              <dd class="dd">double</dd>



                              <dt class="dt dlterm">Properties</dt>

                              <dd class="dd">Create, Filter, Nillable, Sort, Update</dd>



                              <dt class="dt dlterm">Description</dt>

                              <dd class="dd"> The N2O emissions per passenger-kilometer in CO2e from long-haul
                                 flights. </dd>


                        </dl>

                     </td>

                  </tr>

                  <tr>
                     <td class="entry" headers="d51659e96" data-title="Field"><span class="keyword parmname">N2oPsgrKmMediumHaulInKgCo2e</span></td>

                     <td class="entry" headers="d51659e99" data-title="Details">
                        <dl class="dl detailList">

                              <dt class="dt dlterm">Type</dt>

                              <dd class="dd">double</dd>



                              <dt class="dt dlterm">Properties</dt>

                              <dd class="dd">Create, Filter, Nillable, Sort, Update</dd>



                              <dt class="dt dlterm">Description</dt>

                              <dd class="dd"> The N2O emissions per passenger-kilometer in CO2e from
                                 medium-haul flights.</dd>


                        </dl>

                     </td>

                  </tr>

                  <tr>
                     <td class="entry" headers="d51659e96" data-title="Field"><span class="keyword parmname">N2oPsgrKmShortHaulInKgCo2e</span></td>

                     <td class="entry" headers="d51659e99" data-title="Details">
                        <dl class="dl detailList">

                              <dt class="dt dlterm">Type</dt>

                              <dd class="dd">double</dd>



                              <dt class="dt dlterm">Properties</dt>

                              <dd class="dd">Create, Filter, Nillable, Sort, Update</dd>



                              <dt class="dt dlterm">Description</dt>

                              <dd class="dd"> The N2O emissions per passenger-kilometer in CO2e from short-haul
                                 flights.</dd>


                        </dl>

                     </td>

                  </tr>

                  <tr>
                     <td class="entry" headers="d51659e96" data-title="Field"><span class="keyword parmname">N2oPsgrMileLongHaulInKgCo2e</span></td>

                     <td class="entry" headers="d51659e99" data-title="Details">
                        <dl class="dl detailList">

                              <dt class="dt dlterm">Type</dt>

                              <dd class="dd">double</dd>



                              <dt class="dt dlterm">Properties</dt>

                              <dd class="dd">Create, Filter, Nillable, Sort, Update</dd>



                              <dt class="dt dlterm">Description</dt>

                              <dd class="dd"> The N2O emissions per passenger-mile in CO2e from long-haul
                                 flights. </dd>


                        </dl>

                     </td>

                  </tr>

                  <tr>
                     <td class="entry" headers="d51659e96" data-title="Field"><span class="keyword parmname">N2oPsgrMileMediumHaulInKgCo2e</span></td>

                     <td class="entry" headers="d51659e99" data-title="Details">
                        <dl class="dl detailList">

                              <dt class="dt dlterm">Type</dt>

                              <dd class="dd">double</dd>



                              <dt class="dt dlterm">Properties</dt>

                              <dd class="dd">Create, Filter, Nillable, Sort, Update</dd>



                              <dt class="dt dlterm">Description</dt>

                              <dd class="dd"> The N2O emissions per passenger-mile in CO2e from medium-haul
                                 flights.</dd>


                        </dl>

                     </td>

                  </tr>

                  <tr>
                     <td class="entry" headers="d51659e96" data-title="Field"><span class="keyword parmname">N2oPsgrMileShortHaulInKgCo2e</span></td>

                     <td class="entry" headers="d51659e99" data-title="Details">
                        <dl class="dl detailList">

                              <dt class="dt dlterm">Type</dt>
                              <dd class="dd">double</dd>
                              <dt class="dt dlterm">Properties</dt>
                              <dd class="dd">Create, Filter, Nillable, Sort, Update</dd>
                              <dt class="dt dlterm">Description</dt>

                              <dd class="dd"> The N2O emissions per passenger-mile in CO2e from short-haul
                                 flights.</dd>
                        </dl>

                     </td>

                  </tr>

                  <tr>
                     <td class="entry" headers="d51659e96" data-title="Field"><span class="keyword parmname">Name</span></td>

                     <td class="entry" headers="d51659e99" data-title="Details">
                        <dl class="dl detailList">

                              <dt class="dt dlterm">Type</dt>
                              <dd class="dd">string</dd>
                              <dt class="dt dlterm">Properties</dt>
                              <dd class="dd">Create, Filter, Group, idLookup, Sort, Update</dd>
                              <dt class="dt dlterm">Description</dt>
                              <dd class="dd">Name of the account.</dd>
                        </dl>
                     </td>
                  </tr>
                  <tr>
                     <td class="entry" headers="d51659e96" data-title="Field"><span class="keyword parmname">OwnerId</span></td>

                     <td class="entry" headers="d51659e99" data-title="Details">
                        <dl class="dl detailList">
                              <dt class="dt dlterm">Type</dt>
                              <dd class="dd">reference</dd>
                              <dt class="dt dlterm">Properties</dt>
                              <dd class="dd">Create, Defaulted on create, Filter, Group, Sort, Update</dd>
                              <dt class="dt dlterm">Description</dt>
                              <dd class="dd">The ID of the user who owns this record. </dd>
                              <dd class="dd">This is a polymorphic relationship field.</dd>
                              <dt class="dt dlterm">Relationship Name</dt>
                              <dd class="dd">Owner</dd>
                              <dt class="dt dlterm">Relationship Type</dt>
                              <dd class="dd">Lookup</dd>
                              <dt class="dt dlterm">Refers To</dt>
                              <dd class="dd">Group, User</dd>
                        </dl>
                     </td>
                  </tr>
                  <tr>
                     <td class="entry" headers="d51659e96" data-title="Field"><span class="keyword parmname">ShortHaulMaximumDistance</span></td>
                     <td class="entry" headers="d51659e99" data-title="Details">
                        <dl class="dl detailList">
                              <dt class="dt dlterm">Type</dt>
                              <dd class="dd">double</dd>
                              <dt class="dt dlterm">Properties</dt>
                              <dd class="dd">Create, Filter, Nillable, Sort, Update</dd>
                              <dt class="dt dlterm">Description</dt>
                              <dd class="dd"> The maximum distance of a short-haul flight. </dd>
                        </dl>
                     </td>
                  </tr>
               </tbody>
                </table>

Selenium & BeautifulSoup cannot find Fields table elements within #shadow_root

2 Answers2

shadow_root

Solution