Monday, April 12, 2010

What is Regression Testing?

We are on a journey. The company I work for is a very large one and has suffered some lag in migrating to modern ways of software development, a situation many companies of this size often find themselves in. Fortunately the team I work with have come together to make a conscious and consistent effort to improve our area of influence. First and foremost was the introduction of a structured developer oriented testing plan. Because we are a development team we started with unit testing, integration testing and some build scripts to run these tests. From there we have improved they way we do those tests, streamlined our scripts and made CI a solid part of our process. Automated deployments have become easier too leading to automated staged deployments. We are now at the stage where we are adding in Fitness and UI testing to the equation.

Each of these step have been driven from the ground up. We have had to, carefully, maneuver these processes into our daily process and show the benefits to management. It has taken time but they have responded very well; so well that they are jumping on board! This is a great thing and has made all the hard work, and it was hard work, worthwhile.
However there are still political games to play. The language is fractured and management want to throw around terms that they don't really understand and force their idea of these terms onto the very people that introduced them to the department, sound familiar? ;)

"Unit Testing" and "Regression Testing" are two of the terms du jour. Unit testing "means" something a developer does and regression testing is an automated UI test that a non techie can watch as it magically clicks away at the screen. This, as you may know, is not correct... well not strictly correct; and in a place where the language is so commonly fractured it is hard to justify fighting for seemingly such trivial victories.
So instead I vent here.

The things that i felt i needed to express most importantly are:
Any automated test is a regression test
- and -
You do not write regression tests.
What i am referring to in that last statement is merely the fact that you write a repeatable test that tests a certain unit, module or scenario, that can be automated and then your check it in to source control. The very notion that it is part of your build process/continuous integration means it now acts as a regression test.
So our tests that
  • Test a unit, i.e. an xUnit framework which may also uses mocks and stubs etc
  • Testing seams or integration points i.e. DB tests, file systems tests, web service tests etc
  • System smoke test
  • High level acceptance tests using tools like cucumber, fitness etc
  • UI testing - i.e. a button clicking framework like QTP
are all regression test because the are repeatable, automated tests.
Please don't make the mistake of thinking you will explicitly write a regression test. The test should test that the code produces a specific outcome e.g. enforces a business rule.
The automation and maintenance of a healthy CI environment makes those tests regression tests.

I guess the key here is really your CI process. If the tests can be dodged or accidentally not run somehow then this will need to be addressed.
Make the pit of success an easy one for you team to fall in to and let them know regression tests are a side effect of good daily habits.

No comments: