My selenium program can't find anything, how do I fix that?

Question

I can't find any element. Even The function .page_source returns only : "" I am coding on google colab.

import sys
sys.path.insert(0,'/usr/lib/chromium-browser/chromedriver')
from selenium import webdriver
chrome_options = webdriver.ChromeOptions()
chrome_options.add_argument('--headless')
chrome_options.add_argument('--no-sandbox')
chrome_options.add_argument('--disable-dev-shm-usage')

from selenium.webdriver.common.proxy import Proxy, ProxyType

from selenium.webdriver.common.by import By
import time
proxy_ip_port = "64.235.204.107:3128"
proxy = Proxy()
proxy.proxy_type = ProxyType.MANUAL
proxy.http_proxy = proxy_ip_port
proxy.ssl_proxy = proxy_ip_port

capabilities = webdriver.DesiredCapabilities.CHROME
proxy.add_to_capabilities(capabilities)

wd = webdriver.Chrome('chromedriver',chrome_options=chrome_options, desired_capabilities = capabilities)

wd.get('http://whatismyipaddress.com')

html = wd.page_source
print(html)

and the outcome is :

<html><head></head><body></body></html>

Turn off headless option, see what happens when you navigate to that page. Are you sure proxy works? — Barry the Platipus, Jul 21 '22 at 21:01
@platipus_on_fire_333 that is the thing, I am working on google colab so I can't turn off the headless option. And I am forced to work on colab because I don't know why but my Mac won't install correctly PIP — NatGJ, Jul 21 '22 at 21:11
Are you sure your proxy works as intended? Did you try another proxy? — Barry the Platipus, Jul 21 '22 at 21:14
@platipus_on_fire_333 I just tried a few other proxies it was indeed the problem ! thanks mate — NatGJ, Jul 21 '22 at 21:32

undetected Selenium · Answer 1 · 2022-07-21T22:50:46.833

To extract the page source you need to induce WebDriverWait for the visibility_of_element_located() for a static element as an example the IPv4 element and you can use either of the following locator strategies:

Using CSS_SELECTOR:

driver.execute("get", {'url': 'https://whatismyipaddress.com/'})
WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.CSS_SELECTOR, "span.address#ipv4")))
print(driver.page_source)

Using XPATH:

driver.execute("get", {'url': 'https://whatismyipaddress.com/'})
WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.XPATH, "//span[@class='address' and @id='ipv4']")))
print(driver.page_source)

Note : You have to add the following imports :

from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

Console Output:

<html lang="en-US" class="chrome chrome103 wf-loading wf-montserrat7copensans3a3002c4002c6002c800-n4-loading" style="overflow-x: hidden;"><head>
<style id="style_whatismyipaddress_leaderboard_dynamic">
            div[name='whatismyipaddress_leaderboard_dynamic']:not(div[id*='_slot']) {
                display: flex;
                align-items: center;
                justify-content: center;
                min-height: 90px !important;
            }
        </style><script async="" src="//c.amazon-adsystem.com/aax2/apstag.js"></script><script async="" src="//confiant-integrations.global.ssl.fastly.net/gptprebidnative/202207121816/wrap.js"></script><script src="//js-sec.indexww.com/ht/p/184310-82987131453484.js"></script><script src="https://freestar-io.videoplayerhub.com/gallery.js"></script><script>window.confiant = window.confiant || {};
        window.confiant.prebidNameSpace='fsprebid';
        window.confiant.callback = function(blockingType, blockingId, isBlocked, wrapperId, tagId, impressionData) {
        postMessage({
          'message': 'CONFIANT_BLOCK',
          'prebid': impressionData.prebid,
          'dfp': impressionData.dfp,
        })};</script><script src="https://pghub.io/js/pandg-sdk.js"></script><meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<meta http-equiv="X-UA-Compatible" content="IE=edge">
<link rel="profile" href="https://gmpg.org/xfn/11">
<title>What Is My IP Address - See Your Public Address - IPv4 &amp; IPv6</title>
<link rel="canonical" href="https://whatismyipaddress.com/">

<link rel="alternate" hreflang="x-default" href="https://whatismyipaddress.com/">
<link rel="alternate" hreflang="en" href="https://whatismyipaddress.com/">
<link rel="alternate" hreflang="fr" href="https://whatismyipaddress.com/fr/mon-ip">
<link rel="alternate" hreflang="de" href="https://whatismyipaddress.com/de/meine-ip">
<link rel="alternate" hreflang="es" href="https://whatismyipaddress.com/es/mi-ip">
<link rel="alternate" hreflang="tr" href="https://whatismyipaddress.com/tr/ip-im">
<link rel="alternate" hreflang="pl" href="https://whatismyipaddress.com/pl/moj-ip">
<link rel="alternate" hreflang="zh-cn" href="https://whatismyipaddress.com/zh-cn/index">
<link rel="alternate" hreflang="zh-tw" href="https://whatismyipaddress.com/zh-tw/index">
<link rel="alternate" hreflang="zh-hk" href="https://whatismyipaddress.com/zh-tw/index">
<link rel="alternate" hreflang="ja" href="https://whatismyipaddress.com/ja/index">
<link rel="alternate" hreflang="it" href="https://whatismyipaddress.com/it/il-mio-ip">
<link rel="alternate" hreflang="vi" href="https://whatismyipaddress.com/vi-vn/index">
<link rel="alternate" hreflang="id" href="https://whatismyipaddress.com/id/ip-saya">
<link rel="alternate" hreflang="ru" href="https://whatismyipaddress.com/ru/index">
<link rel="alternate" hreflang="pt" href="https://whatismyipaddress.com/pt/meu-ip">
<link rel="alternate" hreflang="nl" href="https://whatismyipaddress.com/nl/mijn-ip">


<!-- This site is optimized with the Yoast SEO plugin v12.4 - https://yoast.com/wordpress/plugins/seo/ -->
<meta name="description" content="Find out what your public IPv4 and IPv6 address is revealing about you! My IP address information shows your location; city, region, country, ISP and location on a map. Many proxy servers, VPNs, and Tor exit nodes give themselves away.">
<meta property="og:locale" content="en_US">
<meta property="og:type" content="website">
<meta property="og:title" content="What Is My IP Address? IP Address Tools and More">
<meta property="og:description" content="IP address lookup, location, proxy detection, email tracing, IP hiding tips, blacklist check, speed test, and forums. Find, get, and show my IP address.">
<meta property="og:url" content="https://whatismyipaddress.com/">
<meta property="og:site_name" content="WhatIsMyIPAddress">
<meta name="twitter:card" content="summary_large_image">
<meta name="twitter:description" content="IP address lookup, location, proxy detection, email tracing, IP hiding tips, blacklist check, speed test, and forums. Find, get, and show my IP address.">
<meta name="twitter:title" content="What Is My IP Address? IP Address Tools and More">
<meta name="twitter:image" content="https://whatismyipaddress.com/wp-content/uploads/concern-security.svg">
.   
.   
</div></div></body></html>

My selenium program can't find anything, how do I fix that?

1 Answers1