2

I wrote a Python code that automate marking my attendance on this online community. I wanted it to run everyday so I've been trying to make it run on Aws Lambda.

It worked without any problems on my computer with a UI but but when I modified the script to run headless, it fails saying "unable to locate" element.

My code is as follows:

try:
    import json
    from selenium.webdriver import Chrome
    from selenium.webdriver.chrome.options import Options
    import os
    import shutil
    import uuid
    import boto3
    from datetime import datetime
    import datetime

    print("All Modules are ok ...")

except Exception as e:

    print("Error in Imports ")



class WebDriver(object):

    def __init__(self):
        self.options = Options()

        self.options.binary_location = '/opt/headless-chromium'
        self.options.add_argument('--headless')
        self.options.add_argument('--no-sandbox')
        self.options.add_argument('--start-maximized')
        self.options.add_argument('--window-size=1920,1080')
        self.options.add_argument('--single-process')
        self.options.add_argument('--disable-dev-shm-usage')

    def get(self):
        driver = Chrome('/opt/chromedriver', options=self.options)
        return driver



def lambda_handler(event, context):

    instance_ = WebDriver()
    driver = instance_.get()
    
    url=''
    id=''
    password=''
    
    driver.get(url)
    
    driver.find_element_by_id('login_id').send_keys(id)
    driver.find_element_by_id('login_pw').send_keys(password)

    driver.find_element_by_xpath('/html/body/div[6]/div/div/div/div/div/div/div[2]/div[1]/div/div/form/button').click() # click the 'login' button using XPATH




    driver.find_element_by_xpath('//*[@id="undefined-sticky-wrapper"]/nav/div[1]/div[1]/div/div/ul/li[7]/a').click() # click the community button, move to attendance

    driver.find_element_by_xpath('//*[@id="talk_submit"]').click() # click the attendance button
    
    
    
    
    return True

and this is the error message below.

"errorMessage": "Message: no such element: Unable to locate element: {\"method\":\"id\",\"selector\":\"login_id\"}\n  (Session info: headless chrome=69.0.3497.81)\n  (Driver info: chromedriver=2.43.600233 (523efee95e3d68b8719b3a1c83051aa63aa6b10d),platform=Linux 4.14.243-194.434.amzn2.x86_64 x86_64)\n",
  "errorType": "NoSuchElementException",

and this is the target HTML

<input type="text" name="mb_id" id="login_id" required="" class="form-control input-sm" size="20" maxlength="20" placeholder="아이디">

I've been googling it and I ran into another person who struggled with similar problems on StackOverFlow. one of the solutions suggests using WebDriverWait but I got a timeout too.

Please let me know if I'm missing something or doing something wrong. thank you

Kerry Lee
  • 47
  • 5
  • if I'm not mistaken, I've had a similar project where we tried to use selenium with xpath but it won't run in headless mode. It has to load the full webpage to be able to read elements with xpath. – Huy Sep 18 '21 at 07:49
  • oh.. then would it be impossible to automate something with xpath? – Kerry Lee Sep 18 '21 at 08:19
  • thanks to your question, I revisited the topic and found this answer https://stackoverflow.com/questions/50050622/python-selenium-cant-find-element-by-xpath-when-browser-is-headless It might not solve your problem directly, but you could try dumping the HTML page in headless mode to see if it's different from the normal one. – Huy Sep 18 '21 at 08:27
  • thank you! let me do that and try other solutions on that post too – Kerry Lee Sep 18 '21 at 09:36
  • by the way, do you happen to know how to get latest Headless-Chromium for a layer on Lambda? I have the latest Chromedriver but I dunno how to install Headless-Chromium – Kerry Lee Sep 18 '21 at 09:38
  • I just used this zip file on Github that has Chromedriver, Headless-Chromium. and they are quite old maybe that's the reason but with those, at least I can import all the libraries – Kerry Lee Sep 18 '21 at 09:39
  • I've added an answer below with links & instructions included. Hope that helps! : ) – Huy Sep 18 '21 at 11:55
  • I found a better place to download both `chromedriver` and `Chromium` portable binary. Link in the EDIT section below. – Huy Sep 24 '21 at 16:41

1 Answers1

1

Possible solutions:

  1. Try dumping the HTML page while in headless mode to see if there's any difference and change your xpath accordingly.

  2. Update Chromedriver & Chrome to the latest versions

wget https://dl.google.com/linux/direct/google-chrome-stable_current_amd64.deb
sudo apt install ./google-chrome-stable_current_amd64.deb

EDIT:

I've recently started a new project that makes use of Selenium. Thus, I have taken a look at this again and found a place to download updated binaries for both chromedriver and Chromium. You can visit this page, find a section correlated with your OS, and click on Archive to download a zip package. Unpack within your project directory and it's ready to use.

Huy
  • 794
  • 6
  • 10