xUnit Test Patterns: Refactoring Test Code by Gerard Meszaros, Martin Fowler (Foreword by)

From the Publisher

XUnit Test Patterns uses the practice of patterns to share proven techniques for writing automated tests using the XUnit family (e.g., JUnit and NUnit) of test automation frameworks. When properly applied, these patterns result in tests that are easier to write, easier to understand, more robust and repeatable, easier to maintain, and ultimately more cost-effective.

• Maximize software return on investment by learning how to refactor test code and reduce or prevent crippling test maintenance.
• All patterns are introduced with a narrative section discussing the principles behind each one.
• Includes a comprehensive glossary of testing and test automation terms.

Software testing has received renewed attention with the widespread adoption of Extreme Programming and other agile methodologies. The under-appreciated practice of testing provides a timely and accurate measurement--a reality check--so that developers know whether any new action needs to be taken. And automated software testing ensures that once the software works, it is not accidentally broken during subsequent software development or maintenance activities.

Product Details

• Table of Contents
• Forewords & Introductions

Table of Contents

Forewords & Introductions

The Value of Self-Testing Code

In Chapter 4 of Ref, Martin Fowler writes:

If you look at how most programmers spend their time, you'll find that writing code is actually a small fraction. Some time is spent figuring out what ought to be going on, some time is spent designing, but most time is spent debugging. I'm sure every reader can remember long hours of debugging, often long into the night. Every programmer can tell a story of a bug that took a whole day (or more) to find. Fixing the bug is usually pretty quick, but finding it is a nightmare. And then when you do fix a bug, there's always a chance that anther one will appear and that you might not even notice it till much later. Then you spend ages finding that bug.

Some software is very difficult to test manually. In these cases we are often forced into writing test programs. I recall a project I was working on in 1996. My task was to build an event framework that would let client software register for an event and be notified when some other software raised that event (the Observer GOF pattern.) I could not think of a way to test this without writing some sample client software. I had about twenty different scenarios I need to try so I coded up each scenario with the requisite number of observers, events and event raisers. At first, I logged what was occurring in the console and scanned it manually. Scanning it became very tedious very quickly and being quite lazy, I looked for an easier way. For each test I populated a Dictionary indexed by the expected event and the expected receiver of it with the the name of the receiver as the value. When a particular receiver was notified of the event,they looked in the Dictionary for the entry indexed by themselves and the event they just received. If it existed, they removed themselves. If it didn't, they added the entry with an error message saying it was an unexpected event notification. After running all the tests, the test program merely looked in the Dictionary and printed out its contents if it was not empty. This made running all my tests nearly zero in cost. They either passed quietly or spewed a list of test failures. Unbeknown to myself, I had discovered the concept of a Mock Object and a Test Automation Framework out of necessity!

My First XP Project

In late 1999, I attended the OOPSLA conference and I picked up a copy of Kent Beck's new book on "Extreme Programming" XPE. I was used to doing iterative and incremental development and already believed in the value of automated unit testing although I had not tried to apply it universally. I had a lot of respect for Kent since I had known him since the first PLoP¹ conference back in 1994 so I decided that it was worth trying to apply eXtreme Programming on a ClearStream Consulting project. Shortly after OOPSLA I was fortunate to come across a suitable project for trying it on, an add-on application that interacted with an existing database but which had no user interface. The client was open to developing software in a different way.

We started doing eXtreme Programming "by the book" using pretty much all the practices including pair-programming, collective ownership and test-driven development. Of course, we had a few challenges figuring out how to test some aspects of the behavior of the application but we were writing tests for most of the code. Then, as the project progressed, I started to notice a disturbing trend; it was taking longer and longer to implement seemingly similar tasks. I explained the problem to the developers and asked them to record on each task card how much time had been spent writing new tests, modifying existing tests and writing the production code. Very quickly, a trend emerged. While writing new tests and writing the production code seemed to be staying more or less constant, the amount of time spent modifying existing tests was increasing and the developers' estimates were going up as a result. When a developer asked me to pair on a task and we spent 90% of the time modifying existing tests to accommodate a relatively minor change, I knew we had to change something, and soon!

We analyzed the kinds of compile errors and test failures we were getting as we introduced the new functionality and found that many of the tests were affected by the changes to methods of the system under test (SUT). This was no surprise, but what was surprising is that most of the impact was during the fixture setup part of the test and that the changes were not impacting the core logic of the tests.

This was an important discovery because it showed us that we had the knowledge about how to create the objects of the SUT scattered across most of the tests. In other words, the tests knew too much about non-essential parts of the behavior of the SUT. I say "non-essential" because most of the affected tests did not care about how the objects in the fixture were created; they were interested in ensuring that they were in the correct state. Upon further examination, we found that many of the tests were creating identical or nearly identical objects in their test fixtures.

• When we had a bunch of tests that need identical objects, we simply created a method that returned that kind of object ready to use. We now call these Creation Methods.
• Some tests needed to specify different values for some attribute of the object. So we passed that attribute as a parameter to the Parameterized Creation Method (see Creation Method).
• Some tests wanted to create a malformed object to ensure that the SUT would reject it. Defining a separate Parameterized Creation Method for each attribute cluttered the signature of our Test Helper, so we created a valid object and then replaced the value of the One Bad Attribute.

We had discovered what would become² our first test automation patterns. Later, when tests started failing because the database did not like the fact that we were trying to insert another object with the same key that had a unique constraint, we added code to generate the unique key programmatically. We called this variant an Anonymous Creation Method (see Creation Method) to indicate the presence of this added behavior.

Identifying the problem, what we now call a Fragile Test, was an important event on this project and the subsequent definition of the solution patterns saved this project from possible failure. Without this discovery we would, at best, have abandoned the automated unit tests that we had already built. At worst, the tests would have reduced productivity so much that we would have been unable to deliver on our commitments. As it turned out, we were able to deliver what we had promised and with very good quality. Yes, the testers³ people still found bugs in our code because we were definitely missing some tests. Introducing the changes needed to fix those bugs, once we had figured out what the missing tests needed to look like, were relatively straight forward. We were hooked. Automated unit testing and test-driven development really did work and we have been using them consistently ever since.

As we applied the practices and patterns on subsequent projects, we have run into new problems and challenges. In each case, we have "peeled the onion" to find the root cause and have come up with ways to address them. As these techniques have matured, we have added them to our repertoire of techniques for automated unit testing.

We first described some of these patterns in a paper presented at XP2001. In discussion with other participants at that and subsequent conference we discovered that many of them were using the same or similar techniques. That elevated them from "practice" to "pattern" (a recurring solution to a recurring problem in a context). The first paper on test smells RTC was presented at the same conference building on the concept of code smells first described in Ref.

My Motivation

I am a great believer in the value of automated unit testing. I practiced software development without it for the better part of two decades and I know that my professional life is much better with it than without it. I truly believe that the xUnit framework and the automated tests it enables is one of the truly great advances in software development. I find it very frustrating when I see companies trying to adopt automated unit testing but being unsuccessful because of a lack of key information and skills.

As a software development consultant with ClearStream Consulting, I get to see a lot of projects. Sometimes I get called in early in a project to help the client make sure they "do things right". More often than not I only get called in when things are already off the rails. As a result, I get to see a lot of "worst practices" that result in the test smells. If I am lucky and they called early enough, I can help the client recover from the mistakes. If not, they muddle through less than satisfied with how TDD and automated unit testing worked and the word goes out that automated unit testing is a waste of time. In hindsight, most of these mistakes and best practices are easily avoidable given the right knowledge at the right time. But how do you get the knowledge without making the mistakes for yourself? At the risk of sounding self-serving, hiring someone who has the knowledge is the most time-efficient way of learning any new practice or technology. Taking a course or reading a book is a much less expensive (but less effective) alternative. I am hoping that by writing down a lot of these mistakes and how to avoid them, I can save you a lot of grief on your project whether it is fully agile or just more agile than it has been in the past. Gerry Wienberg's law of Raspberry Jam SoC not withstanding.

Whom This Book Is For

I have written this book primarily for software developers (programmers, designers and architects) who want to write better tests and for the managers and coaches who need to understand what the developers are doing and why they (the developers) need to be cut enough slack so they can learn to do it even better! The focus is on developer tests and customer tests automated using xUnit but some of the higher level patterns also apply to tests automated using other technologies than xUnit. Rick Mugridge and Ward Cunningham have written an excellent book on Fit FitB and they advocate many of the same practices. Developers will likely want to read the book cover to cover but they should focus on skimming the reference chapters rather than trying to read them word by word. The emphasis should be on getting an overall idea of which patterns exist and how they work so that they can come back to a particular pattern when a need arises. The first few elements (up to and include the "When to Use It" section) of each pattern should provide this overview.

Managers and coaches can focus on reading the narrative sections only and maybe read the Test Smells reference chapter. They can also read the Strategy patterns as these are decisions they need to understand and provide support to the developers as they work their way through them. As a minimum, managers should read Chapter 3, Goals of Test Automation.

About the Cover Photo

One of the first thought I had when Martin Fowler asked if he could "steal me for his series" (right after "I'd be honored") was "What bridge would I put on the cover?" I thought about the ability of testing to avoid catastrophic failures of software and how that related to bridges. Several famous bridge failures immediately came to mind including the "Galloping Gertie" (the Tacoma Narrows bridge), the "Iron Workers Memorial Bridge" in Vancouver (named for the iron workers who died when a part of it collapsed during construction) and several others. After further reflection, it just did not seem right to claim that testing might have prevented these failures so I chose a bridge with a more personal connection. The picture on the cover is of the New River Gorge bridge in West Virginia. It spans the New River, which cuts a deep gorge through the area. I first passed over and subsequently paddled under this bridge on a whitewater kayaking trip in the late 1980s. The style of the bridge is also relevant; the complex arch structure underneath the bridge is largely hidden from those who use it to get to the other side of the gorge. The road deck is completely level and four lanes wide resulting in a very smooth passage. In fact, at night it is quite possible to be completely oblivious to the fact that one is thousands of feet above the valley floor. A good test automation infrastructure has the same effect; writing tests is easy because most of the complexity is hidden beneath the road bed.

Colophon

This book was written using XML, which I published to HTML for previewing on my web site. I edited the XML using Eclipse and the XML Buddy plug-in. The HTML was generated using a Ruby program that I first obtained from Martin Fowler and which I then evolved quite extensively as I evolved my custom markup language. Code samples were written, compiled and executed in (mostly) Eclipse and were inserted into the HTML automatically by XML tag handlers (one of the main reasons for using Ruby instead of XSLT.) This gave me the ability to "publish early, publish often" to the web site. I could also generate a single Word or PDF document for reviewers from the source but this required some manual steps.

1. The Pattern Languages of Programs conference.
2. Technically, they are not truly patterns until they have been discovered by three independent project teams.
3. The testing function is sometimes refered to as "Quality Assurance" but this usage is, strictly speaking, incorrect.
4. The "Law of Raspberry Jam" states: "The wider you spread it, the thinner it gets."