Why some websites are not automatable using Selenium

Question

I tried automating https://www.westernunion.com/global-service/track-transfer web page but couldn't figure out why website is not navigating to next page.

My script is opening page -> Entering MTCN as 2587051083 -> Clicking on Continue button but after click nothing shows up. While replicating the same steps manully works well. Is there any browser settings am I missing for these kind of sites? I am clueless

public static void main(String ar[]) {
        System.setProperty("webdriver.chrome.driver","D:\\Study\\selenium-java-2.48.2\\selenium-2.48.2\\chromedriver.exe");
        driver=new ChromeDriver();
        driver.manage().timeouts().implicitlyWait(10, TimeUnit.SECONDS);
        driver.manage().window().maximize();
        driver.get("https://www.westernunion.com/global-service/track-transfer");
        driver.findElement(By.xpath("//input[@id='trackingNumber']")).sendKeys("2587051083");
        driver.findElement(By.xpath("//button[@id='button-track-transfer']")).click();
        }

I have edited my question. Please let me know your thoughts. Also please remove your downvote. — Nakul, Dec 13 '18 at 18:25
interesting, for some reason it cannot create session to get `sessionId` try using firefox — ewwink, Dec 13 '18 at 23:44

undetected Selenium · Accepted Answer · 2020-01-28T08:28:16.260

On https://www.westernunion.com/global-service/track-transfer web page to send a character sequece within the tracking field I have made some minor modification to your own code inducing WebDriverwait for the desired element to be clickable and then invoke click() on the element with text as Continue as follows:

Code Block:

import org.openqa.selenium.By;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.chrome.ChromeDriver;
import org.openqa.selenium.chrome.ChromeOptions;
import org.openqa.selenium.support.ui.ExpectedConditions;
import org.openqa.selenium.support.ui.WebDriverWait;

public class westernunion {

    public static void main(String[] args) {

        System.setProperty("webdriver.chrome.driver","C:\\Utility\\BrowserDrivers\\chromedriver.exe");
        ChromeOptions opt = new ChromeOptions();
        opt.addArguments("start-maximized");
        opt.addArguments("disable-infobars");
        opt.addArguments("--disable-extensions");
        WebDriver driver=new ChromeDriver(opt);
        driver.get("https://www.westernunion.com/global-service/track-transfer");
        new WebDriverWait(driver, 10).until(ExpectedConditions.elementToBeClickable(By.cssSelector("input.new-field.form-control.tt-mtcn.ng-pristine.ng-valid-mask"))).sendKeys("2587051083");
        driver.findElement(By.cssSelector("button.btn.btn-primary.btn-lg.btn-block.background-color-teal.remove-margin#button-track-transfer")).click();
    }
}

It seems the click() does happens and the spinner becomes visible for a while but the search is interrupted and on inspecting the webpage you will find that some of the <script> tag and <link> tag refers to css having keyword dist. As an example:

<link rel="stylesheet" type="text/css" href="/content/wucom/dist/20181210075630/css/responsive_css.min.css">
<script src="/content/wucom/dist/20181210075630/js/js-bumblebee.js"></script>
<link ng-if="trackTransferVm.trackTransferData.newTrackTransfer || trackTransferVm.trackTransferData.isRetail" rel="stylesheet" type="text/css" href="/content/wucom/dist/20181210075630/css/main.min.css" class="ng-scope" style="">

Which is a clear indication that the website is protected by Bot Management service provider Distil Networks and the navigation by ChromeDriver gets detected and subsequently blocked.

Distil

As per the article There Really Is Something About Distil.it...:

Distil protects sites against automatic content scraping bots by observing site behavior and identifying patterns peculiar to scrapers. When Distil identifies a malicious bot on one site, it creates a blacklisted behavioral profile that is deployed to all its customers. Something like a bot firewall, Distil detects patterns and reacts.

Further,

"One pattern with Selenium was automating the theft of Web content", Distil CEO Rami Essaid said in an interview last week. "Even though they can create new bots, we figured out a way to identify Selenium the a tool they're using, so we're blocking Selenium no matter how many times they iterate on that bot. We're doing that now with Python and a lot of different technologies. Once we see a pattern emerge from one type of bot, then we work to reverse engineer the technology they use and identify it as malicious".

Reference

You can find a couple of detailed discussion in:

Thanks for your kind help, I am also able to enter text and click on button though script but issue comes after clicking Continue button. Are you able to navigate to next page? — Nakul, Dec 14 '18 at 08:40
What I am looking is for the next page.... Any ways thanks a lot for your effort and time. Have a great day. — Nakul, Dec 14 '18 at 09:02
@Nakul Checkout my answer update and let me know the status. — undetected Selenium, Dec 14 '18 at 09:29
i have gone through the links and seems we cant automate those protected websites. Any ways thanks a lot for the detailed description/links. — Nakul, Dec 14 '18 at 11:54

Why some websites are not automatable using Selenium

1 Answers1

Distil

Reference