Regression testing for styling and layout of web applications

Question

I realise this is a non-trivial task, but is there a way to regression test the styling and rendered layout of a web application?

I've found it straightforward to unit test and regression test functionality, both server-side and client-side.

However, a frustrating issue I've run into are CSS changes made to fix one layout issue that break the layout and styling on a different page. The only way I know how to detect these is to manually view every page in an application, and compare it with my own expectations, but obviously this can be burdensome and expensive when an application can have dozens of pages. Has there been any research using image processing or other techniques to detect these kinds of problems automatically?

+1, this would be useful, but (if you're doing it right) with the vast majority of changes, you're either going to break all of the pages, or none of them. — thirtydot, Feb 28 '11 at 15:57
Yes, but you don't have to check after each and every change, of course. — Zsub, Mar 02 '11 at 10:22

score 2 · Answer 1 · edited Aug 11 '20 at 15:58

There is a way to test the layout of a web application using Galen Framework. This tool has its own language and is very easy to learn and understand. It is a Selenium-based and you can run tests in Selenium Grid, (Sauce Labs) if you want to test your application in different browsers.

This tool gets the location of a specified element on the page and check them relatively to each other.

Example: if you want to check that a menu pane is below the header and stretches to the width of a browser and has 50 pixels height, you can do it like this:

menu
    below: header 5px
    width: 100% of screen/width
    height: 50px

This tool can also be used to test responsive designs.

You can find complete documentation on the official website, http://galenframework.com.

The best part is you can even create Java tests. The Galen JavaScript API is also available along with the sample projects on GitHub.

Again, test(s) written once can be used at multiple phases of the application life cycle.

Can you add some characteristics of Galen Framework, like if it is commercial or open source? — Peter Mortensen, Aug 11 '20 at 15:55

score 2 · Accepted Answer · edited Aug 11 '20 at 15:42

Actually, one of the hidden gems of Selenium is that you can use it to take screen captures of the browser. Then using a technique as described in Find differences between images using C#, you can highlight the differences between previous snapshots.

This example shows how to grab a screenshot of the homepage and then create a difference image. With a little extra tinkering, you can use the same technique to count the pixels that are different.

[Test]
public void CompareHomePageToPrevious()
{
    string fileName = Path.Combine(Environment.CurrentDirectory, "homepage.bmp");
    var selenium = new DefaultSelenium("localhost", 4444, "*chrome", "http://mywebsite");
    selenium.Start();
    selenium.Open("/");
    selenium.CaptureEntirePageScreenshot(fileName, "");

    // Load current and previous captures
    var current  = new Bitmap(filename);
    var previous = new Bitmap(_previousHomepageImage);

    // Find all pixels that are the same and make them pink
    Bitmap diff = ImageTool.GetDifferenceImage(current,previous,Color.Pink);
    diff.MakeTransparent(Color.Pink);

    // Save the image for viewing
    // or count the number of different
}

Selenium is a really interesting choice because it's cross-platform and cross-browser, meaning that you can capture screens of different browsers. You can use this technique to compare snapshots between builds or to compare between browsers.

score 1 · Answer 3 · edited Aug 11 '20 at 15:44

1

I guess you could generate an 'ideal' shot of each page in your web application to use as reference.

Then generate a new shot of every page after a CSS update and compare with the previous ones. A 1:1 comparison should be okay if you make sure you always keep the same resolution, etc.

You could even make it so that you can tell it that if the page differs, the other page is actually 'better' than the example and to use the 'other' as the example for the next runs.

That way, when you fix something you can see the difference for one page, and mark it as being better, and then fix a bug on another page while making sure you don't regress the bug you were trying to fix in the first place.

edited Aug 11 '20 at 15:44

Peter Mortensen

30,738
21
105
131

answered Feb 28 '11 at 15:51

Zsub

1,799
2
15
28

1

Actually I saw solution like that in my company. It was developed in AutoIt, and in general it was doing what you described. As for saying where is the difference, after determining that there is a difference,it was dividing page into 4 areas and checking if in those areas there is a difference. It was doing that recursively until some threshold. In the end you would get pretty precise area where is a difference. Additionally screenshot with area marked is provided. – yoosiba Feb 28 '11 at 16:54
That would make this even more useful :) I think the actual implementation of the diff algorithm is fairly trivial. – Zsub Feb 28 '11 at 19:11
Guess you could use http://code.google.com/p/browsershots/wiki/ShotFactoryStandalone (used by http://browsershots.org/) instead of AutoIt. I haven't tried though – user123444555621 Feb 28 '11 at 19:59

score 0 · Answer 4 · edited Aug 11 '20 at 15:47

0

You should check out Web Consistency Testing as an approach to do your regression testing.

edited Aug 11 '20 at 15:47

Peter Mortensen

30,738
21
105
131

answered Jan 22 '13 at 20:44

nirvdrum

2,319
17
26

these links are borked. please fix the links or delete the answer – activedecay Mar 07 '16 at 15:18

score 0 · Answer 5 · edited Aug 11 '20 at 15:49

If you're developing in Ruby, take a look at the rspec-page-regression gem. It's an RSpec plugin that lets you compare requested page snapshots against an expected image. It works with Poltergeist, which is a Capybara driver that uses PhantomJS and supports rendering page snapshots.

Rspec-page-regression provides an RSPec matcher that lets you do things like

context "user page" do
  before(:each) do
    visit user_path
  end

  it { page.should match_expected }

  context "popup help" do
    before(:each) do
      click_button "Help"
    end

    it { page.should match_expected }
  end
end

And it provides a mechanism to manage the expectation images.

Disclaimer: I'm the author of the gem.

Regression testing for styling and layout of web applications

5 Answers5

Linked