2

I am trying to do an automated browsing (flight search) but upon trying to submit the form, the website always detects me. I first tried slowing down my form filling procedure using Thread.sleep but that did not work either. I also tried the answers suggested in these posts link 1, link 2, link 3, which removed the notification "Chrome is being controlled by automated test software" but I am still detected upon submitting form and asked to verify am human. The website is american

Below is my automated browsing code: Fills the form but gets asked to verify on submitting.

 System.setProperty("webdriver.chrome.driver", "path\\chromedriver.exe");
    ChromeOptions options = new ChromeOptions();                

    //options.addArguments("--headless");                        
    
    //My avoid detection tries
    options.addArguments("user-agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.116 Safari/537.36");                         //specify user agent
    options.setExperimentalOption("excludeSwitches", new String [] {"enable-automation"});
    options.setExperimentalOption("useAutomationExtension", false);
    ChromeDriver driver = new ChromeDriver(options);   //create headless browser
    Map javascriptCodes = Map.of(
            "source", "Object.defineProperty(navigator, 'webdriver', {get: () => undefined }); Object.defineProperty(navigator, 'languages', {get: function() { return ['en-US', 'en']; }, }); Object.defineProperty(navigator, 'plugins', { get: function() { return [1, 2, 3, 4, 5]; }, });"
    );
    
    driver.executeCdpCommand("Page.addScriptToEvaluateOnNewDocument", javascriptCodes);

    JavascriptExecutor js = driver;

    driver.get("https://www.americanairlines.ie/intl/ie/index.jsp");

    try {
        //AUTOMATED BROWSING
        //Wait till button is clickable
        WebDriverWait wait = new WebDriverWait(driver, Duration.ofSeconds(10));
        String waitCondition = "bookingModule-submit";
        wait.until(ExpectedConditions.elementToBeClickable(By.id(waitCondition)));

        WaitFor(5);

        //if cookie popup alert, close it
        if (driver.findElements(By.xpath("//button[@class='optoutmulti_button']")).size() != 0) {
            //close popup
            WebElement popUp = driver.findElement(By.xpath("//button[@class='optoutmulti_button']"));
            popUp.click();
            WaitFor(1);
        }

        //Scroll to form
        //Find element by link text and store in variable "Element"
        WebElement Element = driver.findElement(By.id("bookingModule"));
        //This will scroll the page till the element is found
        js.executeScript("arguments[0].scrollIntoView();", Element);

        //Fill in origin
        WebElement origin = driver.findElement(By.xpath("//input[@name='origin']"));
        origin.sendKeys("MIA");
        WaitFor(2);

        //Fill in destination
        WebElement destination = driver.findElement(By.xpath("//input[@name='destination']"));
        destination.sendKeys("LAX");
        WaitFor(2);

        //Clear depart date
        driver.findElement(By.id("aa-leavingOn")).clear();
        WaitFor(1);
        //Fill in depart date
        driver.findElement(By.id("aa-leavingOn")).sendKeys("15/08/2020");
        
        WaitFor(2);

        //Clear depart date
        driver.findElement(By.id("aa-returningFrom")).clear();
        WaitFor(1);
        //Fill in return date
        driver.findElement(By.id("aa-returningFrom")).sendKeys("30/08/2020");
        
        WaitFor(2);

        //Submit form
        //class = "btn btn-fullWidth"
        WebElement submitBtn = driver.findElement(By.xpath("//input[@id='bookingModule-submit']"));
        submitBtn.click();
        //UPON CLICKING HERE IS WHERE I GET ASKED TO VERIFY MYSELF :(

        //Set web driver wait to 20 seconds. Waits till it receives the element with id = fare-select-button
        wait = new WebDriverWait(driver, Duration.ofSeconds(20));

        waitCondition = "owd-calendar-slide-previous";
        wait.until(ExpectedConditions.elementToBeClickable(By.className(waitCondition)));

        String innerHTML = driver.findElement(By.className("row upsell-bound")).getAttribute("innerHTML");
        System.out.println(innerHTML);
    }
    catch (Exception ex){
        ex.printStackTrace();
    }

And my sleep method

public static void WaitFor(int seconds) throws InterruptedException {
    seconds = seconds * 1000;
    Thread.sleep(seconds);
}

My Chrome browser version is 83. Selenium is 3.141.x, I also tried Selenium 4.0.0-alpha-4 to no avail.

Vy Do
  • 46,709
  • 59
  • 215
  • 313
gunner
  • 89
  • 4
  • Many webs have anti-crawler mechanisms. From selenium point of view there is no chance to avoid those detection. You have to ask web administrator to allow specific crawler (selenium) or some bypass (f.e. specific request header with some credentials). – pburgr Jul 14 '20 at 08:03
  • How you tried out `options.addArguments("--disable-blink-features=AutomationControlled");` as its not mentioned in your code. – Kuldeep Kamune Jul 16 '20 at 18:29
  • `executeCdpCommand("Page.addScriptToEvaluateOnNewDocument"..)` This does not help if the value is read before new document is called. I tested using a simple html+JS page that looks in `navigator` – dustytrash Jan 14 '21 at 04:54

0 Answers0