5

I'm trying to get HtmlUnitDriver to work in my development environment. As a beginner, I've tried implementing the example from the following page using the latest selenium server jar: http://code.google.com/p/selenium/wiki/GettingStarted

Unfortunately, whenever I try to run this program, I get the following exception:

Exception in thread "main" org.openqa.selenium.NoSuchElementException: Unable to locate element with name: q
For documentation on this error, please visit: http://seleniumhq.org/exceptions/no_such_element.html
Build info: version: '2.35.0', revision: 'c916b9d', time: '2013-08-12 15:42:01'
System info: os.name: 'Windows 7', os.arch: 'x86', os.version: '6.1', java.version: '1.6.0_16'
Driver info: driver.version: HtmlUnitDriver
    at org.openqa.selenium.htmlunit.HtmlUnitDriver.findElementByName(HtmlUnitDriver.java:853)
    at org.openqa.selenium.By$ByName.findElement(By.java:292)
    at org.openqa.selenium.htmlunit.HtmlUnitDriver$5.call(HtmlUnitDriver.java:1404)
    at org.openqa.selenium.htmlunit.HtmlUnitDriver$5.call(HtmlUnitDriver.java:1)
    at org.openqa.selenium.htmlunit.HtmlUnitDriver.implicitlyWaitFor(HtmlUnitDriver.java:1094)
    at org.openqa.selenium.htmlunit.HtmlUnitDriver.findElement(HtmlUnitDriver.java:1401)
    at org.openqa.selenium.htmlunit.HtmlUnitDriver.findElement(HtmlUnitDriver.java:419)
    at Justin.Main.main(Main.java:30)

I've tried modifying my code to incorporate the fixes implemented here:

HtmlUnitDriver causes problems while getting an url

I've tried getting the URL of the page using driver.getCurrentUrl() after the call to driver.get("http://www.google.com"), but the string that is returned is about:blank.

Similar code in this example would definitely work if I were to run it using FirefoxDriver, but to meet requirements, I need my script to run headless with selenium (it is okay if it is run with a specific BrowserVersion so long as it is headless).

Any help would be greatly appreciated.

UPDATE:

This is the code I'm trying to run right now. I just want to see that I can get HtmlUnitDriver to work with something as simple as entering a search query into Google.

package Justin;


import org.openqa.selenium.By;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.WebElement;
import org.openqa.selenium.htmlunit.HtmlUnitDriver;


public class Main {
public static void main(final String[] args) {
    // Create a new instance of the html unit driver
    // Notice that the remainder of the code relies on the interface,
    // not the implementation.
    final WebDriver driver = new HtmlUnitDriver();
    ((HtmlUnitDriver)driver).setJavascriptEnabled(true);

    // And now use this to visit Google
    driver.get("http://www.google.com");

    try {
        Thread.sleep(30000);
    } catch (final InterruptedException e) {
        // TODO Auto-generated catch block
        e.printStackTrace();
    }

    // Find the text input element by its name
    final WebElement element = driver.findElement(By.name("q"));

    // Enter something to search for
    element.sendKeys("Cheese!");

    // Now submit the form. WebDriver will find the form for us from the element
    element.submit();

    // Check the title of the page
    System.out.println("Page title is: " + driver.getTitle());

}
}

UPDATE 2:

This issue is very bizarre. I've tried the same code on my computer at home and it works perfectly fine with BrowserVersion set to Firefox or Internet Explorer. This issue is definitely being caused by the computer configuration at my workplace, although I still have no idea why this is happening.

Community
  • 1
  • 1
Justin
  • 742
  • 5
  • 17
  • 34
  • Hey Justin, could you share your actual test class? Maybe we're missing something. – ddavison Oct 02 '13 at 14:43
  • Yeah sure. The above code is what I'm trying to execute right now. I've tried placing a hardcoded sleep there to see if it would load the page, but it doesn't seem to fix the problem. – Justin Oct 02 '13 at 14:51
  • For sake of learning.. especially being new.. you should use Chrome or Firefox. HtmlUnit seems to be very unpredictable, and besides, you can't actually **see** the things happening. If you'd like, check out [getting-started-with-selenium](http://github.com/ddavison/getting-started-with-selenium/archive/master.zip). it's a nice framework to help get you started.. something i wish i had when i started – ddavison Oct 02 '13 at 15:06
  • I literally ran [this test](https://github.com/ddavison/getting-started-with-selenium/blob/master/src/tests/java/com/company/seleniumframework/functional/SampleFunctionalTest.java) using HtmlUnit, and Firefox.. firefox worked, but HtmlUnit doesn't. that bothers me at a fundamental level. I'd advise avoiding HtmlUnit for now – ddavison Oct 02 '13 at 15:08
  • For my purposes, I don't need to see what is happening. I've already gotten the code I want to run working on Firefox, but there are some completely different issues on that end that is outside of the scope of this question. – Justin Oct 02 '13 at 15:22
  • well good luck with it then. `HtmlUnit` is not a recommended production-style regression testing option. – ddavison Oct 02 '13 at 15:25
  • I'm not quite using Selenium for testing in this case. It's for a small tool I wrote to scrape information from this website and save it to Excel spreadsheet. Since the script will be running on someone's computer periodically and they are also doing work at the same time, it would have been best if the operation was not intrusive (not visible). – Justin Oct 02 '13 at 16:34
  • Are you by any chance behind a corporate proxy? That make the test would work on firefox (as firefox gets the system's proxy settings), but wouldn't with HtmlUnit, as it is a standard application by itself. – acdcjunior Oct 02 '13 at 16:36
  • @acdcjunior: I think so. The environment that I worked in is very locked down. Is there a setting I can change to get HtmlUnit working? – Justin Oct 02 '13 at 17:24
  • If you are behind a proxy, you'll need to instruct HtmlUnit to use it. To do that you'll need the proxy's IP address and port. If it requires authentication, you'll have some other stuff to see as well. Have you got the IP/port? – acdcjunior Oct 02 '13 at 17:29
  • I'm not exactly sure how I would get the IP or port, or how I would instruct HtmlUnit to use it. – Justin Oct 02 '13 at 17:31
  • Hi Justin - i have the same problem but no solution at now. My setup is spring-boot, maven, cucumber-jvm, selenium. At my computer i think it is a config problem. Maybe you have a config problem too? I use the RunWith(Cucumber.class) and then i have the wrong behaivor for HttpUnitDriver(true) but if i use the SpringJUnit4ClassRunner then the driver works correct. Check your RunWith, maybe it will be work with a other class? (myproject: https://github.com/FunThomas424242/books.example) – Huluvu424242 Feb 28 '15 at 14:53
  • At this page you can see the settings for htmlunitdriver (maybe setHttpProxy or setProxy, or setProxySettings is helping): https://selenium.googlecode.com/git/docs/api/java/org/openqa/selenium/htmlunit/HtmlUnitDriver.html – Huluvu424242 Feb 28 '15 at 15:11
  • @Justin I am facing same situation like yours. You able to resolve the issue? – TodayILearned Jan 20 '16 at 03:40

6 Answers6

1

In the example code given, it assumes the older Google page exists which has the search field with name=q.

The page no longer is labelled that way - try changing driver.findElement(By.name("q")) to driver.findElement(By.cssSelector("input[id=gbqfq]") and see if that fixes your issue - at the very least it should be able to input to the search bar once you get the page loaded up.

If you called driver.getCurrentUrl() immediately after trying to get the page, it may not have loaded completely and instead returned about:blank.

If that doesn't work, we can keep troubleshooting - it's harder to see what's actually happening with a headless browser, so you may want to switch to a FirefoxDriver temporarily to visualize exactly what's happening.

JPierce
  • 54
  • 1
  • `` - `name` attr is still there – ddavison Oct 02 '13 at 14:41
  • Hi JPierce. I've tried adding your modifications, but I'm still getting a NoSuchElementException. It seems that HtmlUnitDriver is adamant about not loading the webpage. I've tried it with both javascript enabled and disabled. – Justin Oct 02 '13 at 14:43
  • Huh. Thanks sircapsalot, I totally missed that name attribute. – JPierce Oct 02 '13 at 14:46
  • I don't have enough rep to reply to the OP.... but removing the `((HtmlUnitDriver)driver).setJavascriptEnabled(true)` line from the edited code causes it to work for me and not explode into exceptions. The thread.sleep() part is also unnecessary, since I believe there is an implicit wait built into driver.get(). – JPierce Oct 02 '13 at 14:58
  • I tried removing the ((HtmlUnitDriver)driver).setJavascriptEnabled(true) line, but it appears that I still get the same issue. Incidentally, how is your build environment for selenium configured? – Justin Oct 02 '13 at 15:16
  • The only thing I did to set up a build environment was to change the project build path to point to selenium-server-standalone-2.33.0.jar (that version only because I use it on another project). That allowed it to build and run without any problems - the jar can be found online at https://code.google.com/p/selenium/downloads/list – JPierce Oct 02 '13 at 17:33
  • In that case, it might be the network settings for where I work that are causing the issue. – Justin Oct 02 '13 at 17:51
  • That could be it, but if you can navigate there normally it seems a little suspect. One last thing you can try is to add a few more lines before the WebDriver is loaded to ensure it doesn't try to use a proxy: `DesiredCapabilities capabilities = new DesiredCapabilities();` `Proxy proxy = new Proxy();` `proxy.setProxyType(ProxyType.DIRECT);` `capabilities.setCapability(CapabilityType.PROXY, proxy);` and then change the WebDriver declaration to `final WebDriver driver = new HtmlUnitDriver(capabilities);` That would ensure it tries to connect by clearing the proxy settings before launching. – JPierce Oct 02 '13 at 17:54
  • What lines would I add to ensure that it doesn't try to use a proxy? – Justin Oct 02 '13 at 17:55
  • Sorry, the lines got lost before the edit - see above (all of these classes are included in the selenium jar) – JPierce Oct 02 '13 at 17:57
  • @JPierce: The same issue is still occurring after the change. – Justin Oct 02 '13 at 18:04
  • Well, the next thing I would think is a possible DNS issue, as I get the same thing with an invalid URL. Try changing your driver.get() line to `driver.get("http://74.125.21.103");` – JPierce Oct 02 '13 at 18:13
  • I'm getting the same result as before when I enter driver.get("http://74.125.21.103"); – Justin Oct 02 '13 at 18:17
  • If you open internet explorer and go into the settings->connection->LAN Settings, is there a address configured in there that ends with .pac? – JPierce Oct 02 '13 at 18:57
  • No. There is a web proxy address though. – Justin Oct 02 '13 at 19:27
  • Okay - is it a url:port combination, or a link to a page/file? – JPierce Oct 02 '13 at 19:28
  • It is a url:port combination. – Justin Oct 03 '13 at 12:52
  • Ok, go ahead and remove the capabilities and proxy stuff so you're back to just the Driver declaration - then try changing that line to `HtmlUnitDriver driver = new HtmlUnitDriver();`, then add the line underneath `driver.setProxy(host, port);` where host is the url, and port is, well, the port - hopefully that will work for you. – JPierce Oct 03 '13 at 13:12
1

Ill try to give it a shot:

If it is a network problem, please download this tool to check if the port your are connecting to is in use: http://technet.microsoft.com/en-us/sysinternals/bb897437.aspx

Very handy for these kind of situations.

For further debugging, try reading out the source. Put this line before the first element selector and after the page loaded.:

System.out.println(driver.getPageSource());

I wonder if the element is there..

HerrWalter
  • 622
  • 1
  • 5
  • 13
0

Try using the Selenium IDE plugin with Firefox to see what it scrapes when it does it actions. You can export the code into java and then try to run that also. I would also step through the code on the both home and work computer to see if there are any differences. I have seen code work (when that element not found error happens) when being stepped through due to the DOM having time to fully load. Furthermore, try to instantiate and actual browser so you can visuallly see whats happening.

Also, verify version of all software your using between home and workstation.

  • The funny thing is that the code above works with FirefoxDriver. It just doesn't work with HtmlUnitDriver, but I need to be able to run this program headless. – Justin Oct 08 '13 at 12:52
  • why, do you need headless? –  Oct 08 '13 at 15:14
  • Because I'm using Selenium to scrape information from this website. It is being run as a background task to scrape information from this website every hour and place the information into an Excel spreadsheet. Unfortunately, it is being run on the same workstation that someone uses for work. It would be best if a Mozilla Firefox window wasn't being launched every hour so that the user could do her work. – Justin Oct 09 '13 at 12:45
0

1)Try to add 

driver.manage().timeouts().implicitlyWait(30, TimeUnit.SECONDS);

before 

final WebElement element = driver.findElement(By.name("q"));

2)or try

 try {
      WebElement element = driver.findElement(By.name("q"));
    } catch (Exception e) {
      Thread.sleep(1000);  
    }
Bart Wojtala
  • 1,290
  • 9
  • 7
0

Try using

HtmlUnitDriver driver = new HtmlUnitDriver();

Can you check if the page is loaded by checking 

String pageTitle = driver.getTitle();
Raging Bull
  • 18,593
  • 13
  • 50
  • 55
NoviceTechie
  • 1
0

If you require headless selenium, and firefox works with your test, try using the phantomjs webdriver.

xvan
  • 4,554
  • 1
  • 22
  • 37