Python Selenium to Scrape USPS

Question

I am trying to create a script to login to USPS website to get a list of incoming packages from Informed Delivery.

I have tried two methods:

1. Requests
2. Selenium

---

Requests

I captured the Login request and imported into Postman. When I sent request, I received error:

{
    "actionErrors": [
        "We have encountered an error.  Please refresh the page and try again."
    ],
    "actionMessages": [],
    "fieldErrors": {}
}

In the request body, it sends a token value (from login form). The request headers also send a few headers starting with x-jfuguzwb-. These look to be tokens of different values.

---

Selenium

Even using a headless browser didn't work.

LOGIN_URL = "https://reg.usps.com/entreg/LoginAction_input?app=Phoenix&appURL=https://www.usps.com/"
driver.get(LOGIN_URL)
username = driver.find_element_by_name('username')
username.send_keys(USERNAME)
password = driver.find_element_by_name('password')
password.send_keys(PASSWORD)
driver.find_element_by_id('btn-submit').click()

displays an error saying "Our apologies that you are having issues with your login."

---

There was a Python Module called myusps but it has not been updated for a couple years.

Are there any suggestions as to how I can accomplish this?

Answer 1

A bit of more information about your usecase and the error Our apologies that you are having issues with your login which you are seeing would have helped us to debug the issue in a better way. However, I was able to send a character sequence to both the username and password field and invoke click() on the Sign In button using Selenium inducing WebDriverWait for the element_to_be_clickable() and you can use either of the following Locator Strategies:

• Using css-selectors:

  from selenium import webdriver
  from selenium.webdriver.support.ui import WebDriverWait
  from selenium.webdriver.common.by import By
  from selenium.webdriver.support import expected_conditions as EC
  
  options = webdriver.ChromeOptions() 
  options.add_argument("start-maximized")
  options.add_experimental_option("excludeSwitches", ["enable-automation"])
  options.add_experimental_option('useAutomationExtension', False)
  driver = webdriver.Chrome(options=options, executable_path=r'C:\WebDrivers\chromedriver.exe')
  driver.get('https://reg.usps.com/entreg/LoginAction_input?app=Phoenix&appURL=https://www.usps.com/')
  WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "input#username"))).send_keys("Bijan")
  driver.find_element_by_css_selector("input#password").send_keys("Bijan")
  driver.find_element_by_css_selector("button#btn-submit").click()
  

• Using xpath:

  from selenium import webdriver
  from selenium.webdriver.support.ui import WebDriverWait
  from selenium.webdriver.common.by import By
  from selenium.webdriver.support import expected_conditions as EC
  
  options = webdriver.ChromeOptions() 
  options.add_argument("start-maximized")
  options.add_experimental_option("excludeSwitches", ["enable-automation"])
  options.add_experimental_option('useAutomationExtension', False)
  
  driver = webdriver.Chrome(options=options, executable_path=r'C:\WebDrivers\chromedriver.exe')
  driver.get('https://reg.usps.com/entreg/LoginAction_input?app=Phoenix&appURL=https://www.usps.com/')
  WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//input[@id='username']"))).send_keys("Bijan")
  driver.find_element_by_xpath("//input[@id='password']").send_keys("Bijan")
  driver.find_element_by_xpath("//button[@id='btn-submit']").click()
  

• Note : You have to add the following imports :

  from selenium.webdriver.support.ui import WebDriverWait
  from selenium.webdriver.common.by import By
  from selenium.webdriver.support import expected_conditions as EC
  

• Browser Snapshot:

Answer 2

This answer below has helped me solve automation issues with site logins that shall go unnamed. I recommend taking a look at the user @colossatr0n answer.

You can use vim, or as @Vic Seedoubleyew has pointed out in the answer by @Erti-Chris Eelmaa, perl, to replace the cdc_ variable in chromedriver(See post by @Erti-Chris Eelmaa to learn more about that variable). Using vim or perl prevents you from having to recompile source code or use a hex-editor. Make sure to make a copy of the original chromedriver before attempting to edit it. Also, the methods below were tested on chromedriver version 2.41.578706.

Can a website detect when you are using selenium with chromedriver?