Logging in automation using Selenium requests / responses- why it wont work?

Question

I need to automate logging in process to a site i visit frequently. I am using a SCRAPY SPIDER. My settings.py file looks as follow:

SELENIUM_DRIVER_NAME='chrome'
SELENIUM_DRIVER_EXECUTABLE_PATH=r"D:\SCRAPPING\chromedriver_win32\chromedriver.exe"
SELENIUM_DRIVER_ARGUMENTS=[]

BOT_NAME = 'tpad_reports'

SPIDER_MODULES = ['tpad_reports.spiders']
NEWSPIDER_MODULE = 'tpad_reports.spiders'

ROBOTSTXT_OBEY = False

DOWNLOADER_MIDDLEWARES = {
    'scrapy_selenium.SeleniumMiddleware': 800,
}

My code looks as follows:

import scrapy
import time
from scrapy_selenium import SeleniumRequest
# my directorate--->>  cd tpad_reports/tpad_reports/spiders> 
# the code to run spider---->>> scrapy runspider tsc_reports.py

def wait(driver):
    time.sleep(2)
    return True

class TscReportsSpider(scrapy.Spider):
    name = 'tsc_reports'
    allowed_domains = ['tpad2.tsc.go.ke']
    start_urls = ['https://tpad2.tsc.go.ke/auth/login']

    def start_requests(self):
        for url in self.start_urls:
            yield SeleniumRequest(url= url ,wait_time=10,wait_until=wait,callback=self.parse)

    def parse(self, response):
        driver=response.request.meta['driver']
        
        tsc_no=driver.find_element_by_xpath('//input[@type="text"]')
        tsc_no.send_keys(620127)

        id_no=driver.find_element_by_xpath('//input[@type="number"]')
        id_no.send_keys(27376193)

        password=driver.find_element_by_xpath('//input[@type="password"]')
        password.send_keys(620127)

        login=driver.find_element_by_xpath('//button[@class="btn btn-md"]')
        login.submit()

        time.sleep(2)

When I run my spider i get error

File "D:\SCRAPPING\tpad_reports\tpad_reports\spiders\tsc_reports.py", line 23, in parse tsc_no=driver.find_element_by_xpath('//input[@type="text"]') AttributeError: 'WebDriver' object has no attribute 'find_element_by_xpath'

So the browser pops up, and the site is well loaded. The problem starts when i start keying in the first input. Somebody please help solve this, I expected to be logged in but cannot be done

score 0 · Answer 1 · answered Feb 12 '23 at 10:51

I finally figured it out:

find_element_by_* commands are deprecated in the latest Selenium Python libraries.

As AutomatedTester mentions: This DeprecationWarning was the reflection of the changes made with respect to the decision to simplify the APIs across the languages and this does that.

Using xpath:

element = find_element_by_xpath("element_xpath")

Needs be replaced with:

element = driver.find_element(By.XPATH, "element_xpath")

For more details check:

score 0 · Answer 2 · answered Feb 12 '23 at 12:24

0

According to JuliusFX you also need to import By

from selenium.webdriver.common.by import By

answered Feb 12 '23 at 12:24

engels

3
1

OP has already answered the question himself/herself. – Shawn Feb 15 '23 at 11:12
yikes, but sometimes newcomers forgot about some imports and other stuff so i mentioned this for someone who would have same trouble – engels Feb 15 '23 at 14:24

Logging in automation using Selenium requests / responses- why it wont work?

2 Answers2