1

I couldn't find much information on Google regarding this topic. Below, I have provided three results from the same Selenium tests. Why am I getting different results when running the tests from different places?

INFO:

So our architecture: Bitbucket, Bamboo Stage 1 (Build, Deploy to QA), Bamboo Stage 2 (start Amazon EC2 instance "Test", run tests from Test against recently deployed QA)

  • Using Chrome Webdriver.
  • For all three of the variations I am using the same QA URL that our application is deployed on.
  • I am running all tests Parallelizable per fixture
  • The EC2 instance is running Windows Server 2012 R2 with the Chrome browser installed
  • I have made sure that the test solution has been properly deployed to the EC2 "test" instance. It is indeed the exact same solution and builds correctly.

First, Local:

Local Test Run

Second, from EC2 Via SSM Script that invokes the tests: Note that the PowerShell script calls the nunit3-console.exe just like it would be utilized in my third example using the command line.

On EC2 via SSM

Lastly, RDP in on EC2 and run tests from the command line:

RDP on EC2 and run tests from command line

This has me perplexed... Any reasons why Selenium is running different on different machines?

Peter Mortensen
  • 30,738
  • 21
  • 105
  • 131
J-Roel
  • 520
  • 1
  • 4
  • 21
  • What are the failures/errors? – Florent B. Sep 29 '17 at 16:49
  • Depends, sometimes its a NoSuchElementException, sometimes it Asserts false when it should be true. The results are not consistent. EDIT: And I made sure I was running the same version of Chrome browsers. I have it in my head it is a browser issue. – J-Roel Sep 29 '17 at 16:51
  • 1
    It could be due to a timeout too low compared to the latency, or due to a missing explicit wait. It could also be that the browser is launching with a window too small. You'll have to investigate to find the reason. There's not enough information in your post to tell why. – Florent B. Sep 29 '17 at 17:02
  • And it's hard to tell on the window sizes. When running from the powershell script the chrome browser instance runs in the background. I don't even see the windows. I have waits setup to ensure the pages have fully loaded before any operations. I am using POM model. That would explain why I get NoSuchElementExceptions randomly... maybe I need to add an explicit wait after the window.maximize. I will play around with that. – J-Roel Sep 29 '17 at 17:05
  • These helped run the tests correctly when I RDP into the EC2 instances. But I am still having issues running them in the background via that script. [Window Maximize](https://github.com/seleniumhq/selenium-google-code-issue-archive/issues/7405) and [Scroll Into View](https://stackoverflow.com/questions/3401343/scroll-element-into-view-with-selenium#20487332) – J-Roel Sep 29 '17 at 17:46
  • Well, I'm back to where I was a month ago. The browser is not visible, therefore Selenium is not seeing the elements. It is how aws ssm agent is running the browsers in the background. Is there any way to force the agent to run the application like a normal app? I can run the script on the ec2 instance and it runs fine... just like it does from the command line. So it has to be the ssm agent. [I hit this dead end a month ago](https://stackoverflow.com/questions/45765769/aws-run-command-act-different-than-running-on-server-locally) – J-Roel Sep 29 '17 at 18:14
  • 1
    I would try to set a specific size for the window rather than calling maximize. You could also try with the headless command switch. – Florent B. Sep 29 '17 at 18:26

2 Answers2

1

This really should be a comment, but I can't comment yet so... I don't know enough about the application you are testing to say for sure, but this seems like something I've seen testing the application I'm working on.

I have seen two issues. First, Selenium is checking for the element before it's created. Sometimes it works and sometimes it fails, it just depends on how quickly the page loads when the test runs. There's no rhyme or reason to it. Second, the app I'm testing is pretty dumb. When you touch a field, enter data and move on to the next, it, effectively, posts all editable fields back to the database and refreshes all the fields. So, Selenium enters the value, moves to the next field and pops either a stale element error or can't find element error depending on when in the post/refresh cycle it attempts to interact with the element.

The solution I have found is moderately ugly, I tried the wait until, but because it's the same element name, it's already visible and is grabbed immediately which returns a stale element. As a result, the only thing that I have found is that by using explicit waits between calls, I can get it to run correctly consistently. Below is an example of what I have to do with the app I'm testing. (I am aware that I can condense the code, I am working within the style manual for my company)

Thread.Sleep(2000);
By nBaseLocator = By.XPath("//*[@id='attr_seq_1240']");
IWebElement baseRate = driver.FindElement(nBaseLocator);
baseRate.SendKeys(Keys.Home + xBaseRate + Keys.Tab);

If this doesn't help, please tell us more about the app and how it's functioning so we can help you find a solution.

Haendler
  • 313
  • 1
  • 2
  • 10
  • I am using Page Object Model. So the elements are found with a filter. [See POM in Selenium C#](http://executeautomation.com/blog/pom-in-selenium-c/) But I don't think that is my problem. It might be the way AWS SSM is running the tests on my EC2 instance. If I run the powershell script on my EC2, then the tests pass...... But when I run them via SSM, then they randomly throw the error(s). I have ran "Enable-PSRemoting -Force" and I am setup to receive requests. And some tests do succeed. They have to open a window, but they work just fine. So I am at a lose. – J-Roel Oct 02 '17 at 20:04
  • Using POM shouldn't cause an issue. I'm refactoring my code to use POM as well. I'm going to go out on a limb here. Selenium, as I understand it, is designed to replicate direct user interaction with the web app being tested. Basically, it's acting as a stand in for the end user. I wonder if there is something weird happening when you hand it off to SSM to run. I wonder if since SSM is an agent on your EC2 instance if it's not behaving like an end user. Now I'm wondering what would happen if you connected to the front end of the app and tested directly from your computer. – Haendler Oct 02 '17 at 20:14
  • I have. I setup a new IAM Role and ran them from my local computer. Fails. If I RDP onto the EC2, run the PowerShell script; the tests run just fine. It is either how the SSM Send Command is running, or the security feature in powershell is preventing even the headless browser from running correctly. I put Sleeps all over. That doesn't help. It still randomly throws NoElementException – J-Roel Oct 02 '17 at 20:30
  • EDIT: I put Sleeps all over. That doesn't help. It still randomly throws NoElementException EDIT2: I can Assert.IsTrue and the test will pass, it is only when I require loading up a page. – J-Roel Oct 02 '17 at 20:36
0

@Florent B. Thank you!

EDIT: This ended up not working... The tests are still running different when called remotely with a powershell script. But, the tests are running locally on both the ec2 instance and my machine correctly.

So the headless command switch allowed me to replicate my failed tests locally.

Next I found out that a headless chrome browser is used during the tests when running via script on an EC2 instance... That is automatic, so the tests where indeed running and the errors where valid.

Finally, I figured out that the screen size is indeed the culprit as it was stuck to a size of 600/400 (600/400?)

So after many tries, the only usable screen size option for Windows, C# and ChromeDriver 2.32 is to set your webDriver options when you initiate you driver:

ChromeOptions chromeOpt = new ChromeOptions();
chromeOpt.AddArguments("--headless");
chromeOpt.AddArgument("--window-size=1920,1080");
chromeOpt.AddArguments("--disable-gpu");
webDriver = new ChromeDriver(chromeOpt);

FINISH EDIT: Just to update Screen size is large enough.

Screen sizes at beginning and end of test

Still attempting to solve the issue. Anyone else ran into this? AWS SSM Command -> Powershell -> Run Selenium Tests with Start-Process -> Any test that requires an element fails because ElementNotFound or ElementNotVisible exceptions. Using POM for tests. FindsBy filter in c# is not finding elements.

Running tests locally on EC2 run fine from cmd, powershell and Powershell ISE.

The tests do not work correctly when executing with the AWS SSM Command. Cannot find any resources to fix problem.

J-Roel
  • 520
  • 1
  • 4
  • 21