0

I am using Github codespace for creating an automated web scraping application using Webdriver-manager webdriver-manager with Selenium.

I have tried: How can we use Selenium Webdriver in collab.research.google.com?

!pip install selenium
!apt-get update # to update ubuntu to correctly run apt install
!apt install chromium-chromedriver
!cp /usr/lib/chromium-browser/chromedriver /usr/bin
import sys
sys.path.insert(0,'/usr/lib/chromium-browser/chromedriver')
from selenium import webdriver
chrome_options = webdriver.ChromeOptions()
chrome_options.add_argument('--headless')
chrome_options.add_argument('--no-sandbox')
chrome_options.add_argument('--disable-dev-shm-usage')
wd = webdriver.Chrome('chromedriver',options=chrome_options)
wd.get("https://www.webite-url.com")

But it did not work!
Can you help me in setting up Webdriver-manager github codespaces or share some link?

Gradley bear
  • 27
  • 4
PhuongBui712
  • 11
  • 2

3 Answers3

0

Please consider rephrasing the question in a better way. The problematic is not clear.

Amadeus
  • 322
  • 1
  • 2
  • 9
0

please check https://stackoverflow.com/posts/46929945

this should work for you,

from selenium import webdriver
from selenium.webdriver.chrome.options import Options
options = Options()
options.headless = True
driver = webdriver.Chrome(CHROMEDRIVER_PATH, options=options)
Gradley bear
  • 27
  • 4
0

I had troubles getting selenium to work in a codespaces environment also. I thought I had a solution that worked, but it was actually in my local VS code.

It turns out that Codespaces does not allow Chrome to be installed, and therefore, Selenium and Webdriver are not able to find the necessary binary files for Chrome that are necessary to open the browser and scrape web information. See the link below and quoted text from VS Code website.

See information and limitations on developing in Codespaces

Extensions that rely on Chrome, such as Protractor Test Runner and Browser Preview. Chrome is not included in a Codespace. Try to find alternative experiences, or you can use these extensions on your project in local VS Code (not connected to Codespaces).

I have written code that works on a local computer within a locally installed version of VS code. As the code is written, it will work only in VS code on Linux or Windows systems.

IMPORTANT! First, download chromedriver from the link below, extract the files, and copy the file path. Try doing this instead of installing from apt. Make sure to download the correct one for your system.
chromedriver

Download the zip file to current working directory where code is located. Then unzip the file to the same location. That way, os.getcwd() gets the location where chromedriver is located easily.

Then, in your terminal, run

pip install webdriver_manager

Then run

pip install -U selenium

Here is the code that helped me open a browser with selenium and get information about elements:

import os
import sys
import time
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.chrome.service import Service
from webdriver_manager.chrome import ChromeDriverManager

driver = webdriver.Chrome(service=Service(ChromeDriverManager().install()))

opsys = os.name

if opsys == 'nt':
# you can change location value to the path you copied earlier
    location = 'C:\\ProgramFiles\\Google\\Chrome\\Application\\'
    chromeDriver = 'chromedriver.exe'

elif opsys == 'posix':
# location value can be changed to the path you copied earlier
    location = os.getcwd()
    chromeDriver = 'chromedriver'

else:
    sys.exit("cannot perform request on operating system")

options = webdriver.ChromeOptions()
options.binary_location = location

# get url from user
url = input("Enter URL: ")

driver.get(url)

# allow webpage to load contents
time.sleep(3)

# get element by class name value and print the text content
item = driver.find_element(By.CLASS_NAME, 'class_value_here')
print(item.text)

# Get links by class value
elems = driver.find_elements(By.CLASS_NAME, 'class_value_here')

for elem in elems:
    value = elem.get_attribute("href")
    print(value)

driver.close()
CatByte-io
  • 16
  • 4