The calculations in my code are well-tested, but because there is so much GUI code, my overall code coverage is lower than I'd like. Are there any guidelines on unit-testing GUI code? Does it even make sense?
For example, there are graphs in my app. I haven't been able to figure out how to automate the testing of the graphs. It takes a human eye, AFAIK, to check if the graph is correct.
(I'm using Java Swing)
Designs like MVP and MVC typically try to abstract as much logic out of the actual GUI as possible. One very popular article about this is "The Humble Dialog Box" by Michael Feathers. Personally I've had mixed experiences with trying to move logic out of the UI - sometimes it's worked very well, and at other times it's been more trouble than it's worth. It's somewhat outside my area of expertise though.
Of course, the answer is to use MVC and move as much logic out of the GUI as possible.
That being said, I heard from a coworker a long time ago that when SGI was porting OpenGL to new hardware, they had a bunch of unit tests that would draw a set of primatives to the screen then compute an MD5 sum of the frame buffer. This value could then be compared to known good hash values to quickly determine if the API is per pixel accurate.
You can try UISpec4J is an Open Source functional and/or unit testing library for Swing-based Java applications...
You can try to use Cucumber and Swinger for writing functional acceptance tests in plain english for Swing GUI applications. Swinger uses Netbeans' Jemmy library under the hood to drive the app.
Cucumber allows you to write tests like this:
Scenario: Dialog manipulation
Given the frame "SwingSet" is visible
When I click the menu "File/About"
Then I should see the dialog "About Swing!"
When I click the button "OK"
Then I should not see the dialog "About Swing!"
Take a look at this Swinger video demo to see it in action.
There is Selenium RC, which will automate testing a web based UI. It will record actions and replay them. You'll still need to walk through the interactions with your UI, so this will not help with coverage, but it can be used for automated builds.
Testing is an art form. I agree the logic should be removed GUI as much as possible. We can then focus our unit testing there. Like anything else testing is about reducing risk. You dont always need to test everything but alot of times the best thing is to break the different testing up for at different areas.
The other question is what are you really trying to test at the UI layer. UI testing is the most expensive testing because it usually takes longer to create, maintain and it is the most brittle. If you test the logic to know the coordinates are correct prior to trying to draw the line what are you testing specifically? If you want to test a graph with a red line is drawn. Can you give it a set predetermined coordinates and test if certain pixels are red or not red? As suggested above bitmap comparisons work, Selenium but my main focus would be not to over test the GUI but rather test the logic that will help create the UI and then focus on what portion of the UI breaks or is suspect and focus a handful of tests there.
Here is some tips:
Try to remove the most code you can from the GUI (have controller, and model object) this way you will be able to test them without the GUI.
For the graphic, you should test the value that you supply to the code that generate the graphic.
You can use JFCUnit to test your GUI, but graphics can be more challenging. I have on a couple of occasions taken snapshots of my GUI and automatically compared it to a previous version. While this doesn't provide an actual test, it does alert you if an automatic build fails to produce the expected output.
What I gather from your question is that you're looking for an automated way to test your GUI behavior in detail, the example you give is testing whether a curve is actually drawn correctly.
Unit testing frameworks provide a way to do automated testing, but I think the kind of tests you want to do are complicated integration tests that verify the correct behavior of a multitude of classes, among which the classes of your GUI toolkit/library, which you should not want to test.
Your options depend very much on what platforms/toolkits/frameworks you use: for example, an application using Qt as its GUI framework can use Squish to automate its tests. You verify the results of your tests once, and subsequent automatically executed tests compare results against the verified results.
My approach to GUI testing is evolving, as is the industry consensus. But I think a few key techniques are beginning to emerge.
I use one or more of these techniques, depending on the situation (e.g. what kind of GUI it is, how quickly it needs to be built, who the end-user will be, etc.).
Manual testing. You always have the GUI running while working on the code, and ensure it is in sync with the code. You manually test and re-test the part that you work on as you work on it, switching between the code and the running application. Every time you complete some significant piece of work, you give the whole screen or area of the application an overall test, to ensure there are no regressions.
Unit testing. You write tests for functions or small units of GUI behaviour. For example, your graphs may need to calculate different shades of a colour based on a 'base' colour. You can extract this calculation to a function and write a unit test for it. You can search for logic like this in the GUI (especially re-usable logic) and extract it into discreet functions, which can be more easily unit tested. Even complex behaviour can be extracted and tested in this manner – for example, a sequence of steps in a wizard can be extracted to a function and a unit-test can verify that, given an input, the correct step is returned.
Component explorer. You create an 'explorer' screen whose only role is to showcase each of the re-usable components that make up your GUI. This screen gives you a quick and easy way to visually verify that every component has the correct look & feel. The component explorer is more efficient than manually going through your whole application, because A) you only have to verify each component once, and B) you don't have to navigate deep into the application to see the component, you can just view and verify it immediately.
Automation testing. You write a test that interacts with the screen or component, simulating mouse clicks, data entry, etc., asserting that the application functions correctly given these manipulations. This can be useful as an extra backup test, to capture potential bugs that your other tests might miss. I tend to reserve automation testing for the parts of the GUI that are most prone to breaking and/or are highly critical. Parts where I want to know as early as possible if something has broken. This could include highly complex interactive components that are vulnerable to breaking or important main screens.
Diff/snapshot testing. You write a test that simply captures the output as a screenshot or as HTML code and compares it with the previous output. That way, you're alerted you whenever the output changes. Diff tests may be useful if the visual aspect of your GUI is complex and/or subject to change, in which case, you want quick and visual feedback on what impact a given change has on the GUI as a whole.
Rather than heavy-handedly using every possible kind of test, I prefer to pick and choose the testing technique based on the kind of thing I'm working on. So in one case I'll extract a simple function and unit-test it, but in another case I'll add a component to the component explorer, etc. It depends on the situation.
I haven't found code coverage to be a very useful metric, but others may have found a use for it.
I think the first measure is the number and severity of bugs. Your first priority is probably to have an application that functions correctly. If the application functions correctly, there should few or no bugs. If there are many or severe bugs, then presumably, you are either not testing or your tests are not effective.
Other than reducing bugs, there are other measures such as performance, usability, accessibility, maintainability, extensibility, etc. These will differ, depending on what kind of application you're building, the business, the end-user, etc.
This is all based on my personal experience and research as well as a great write-up on UI Tests by Ham Vocke.
From what I know, this is quite complicated, and really depends on the language - numerous languages have their own way of testing GUI, but if you really need to test the GUI (as opposed to model/gui interaction), you often have to simulate an actual user clicking buttons. For example, the SWT framework used in Eclipse provides SWTBot, JFCUnit has already been mentioned, Mozilla has their own way of simulating this in XUL (and from what I've read on their blogs, these tests seem to be quite brittle).
Sometimes you have to take a screenshot, and test for pixel-perfect rendering (I believe that Mozilla does this to check for correctly rendered pages) - this requires a lengthier setup, but might be what you need for the graphs. That way, when you update your code and a test breaks, you have to manually check the image if the failure was real, or you improved the graph rendering code to generate prettier graphs and need to update the screenshots.
If you are using Swing, FEST-Swing is useful for driving your GUI and testing assertions. It makes it pretty straightforward to test things like "if I click button A, dialog B should be displayed" or "if I select option 2 from the drop-down, all the checkboxes should become deselected".
The graph scenario that you mention is not so easy to test. It's pretty easy to get code coverage for GUI components just by creating and displaying them (and perhaps driving them with FEST). However, making meaningful assertions is the hard part (and code coverage without meaningful assertions is an exercise in self-deception). How do you test that the graph was not drawn upside-down, or too small?
I think that you just have to accept that some aspects of GUIs cannot be tested effectively by automated unit tests and that you will have to test them in other ways.
It's not your job to test the GUI library. So you can dodge the responsibility to check what is actually drawn on screen and check widgets' properties instead, trusting the library that they accurately represent what is drawn.
