The automation contradiction – Four reasons why automated tests need to tell the human story

Ah, test automation. Those wonderful monkeys that free us from the routine drudgery of re-running boring tests over and over! And because the computer monkeys not people execute these tests, some testers write such tests as though it's only important that the compiler understands them in the compiler's 0's and 1's kind of way.

As a proponent of ATDD or Executable Specifications this is not a view that I share. In fact, I coach people who write automated tests in Fitnesse to become story tellers. Because it's not simply a matter of getting a test to execute that's important. In order for the test to be an asset over the longer term, it's also important that other team members can read the test, and easily understand how the functionality works and what is being tested.

That means that tests need to have good names, and must be well organised in a tree structure or tagged. They need to have a short one or two sentence purpose, and clearly express the key business functionality under test as succinctly as possible. (A myriad of small details, such click there, edit this isn't enough). Finally, if possible, all tests should also express the 'why' which describes why the functionality was implemented and who the customer is.

Now, none of this should be all that new to a seasoned IT professional. Particularly one who has been worked on a legacy feature for which such information is missing. What may come as more of a surprise, is that I've come to realise that the readability of tests is actually more important for automated tests that it is for manual tests.

The main reason for this seems to be that manual tests are just that, manual. And every time they are run the person running the test picks up the implicit knowledge about what the test does, and hopefully also some tacit knowledge about how the functionality works. However, with automated tests we don't get this tacit learning. Therefore, it's more important than ever that a quick scan of a test provides correct insight into what the test covers.

Now as anyone who has tried to maintain an automated test suite knows, the tests also sometimes break. (Otherwise what would be the use of running the tests in the first place?). Without knowing what the purpose of the test was in the first place, it's pretty hard to decide what to do with the test and how to fix it. At this point you'll be grateful for any effort expended on expressing the test in a more human readable manner. So if you ever see yourself altering existing functionality in the future, then this is reason number two for spending some extra effort now writing more readable tests.

Thirdly, well written, executable specifications are an immensely powerful way for the whole team to communicate. This is because they are written in a form that everyone, from customers to BAs, testers and programmers (who like to debug things), can understand and talk about. And because they represent what the product does, and not what people think it does, they are also very real. Quite frankly I don't think this is something you can appreciate until you have experienced it in action. So if you are still sceptical at this point then I suggest read Gojko Adzic's Specification by Example which contains case studies of other people who have tried this approach.

The final reason that test organisation becomes really important in automated test systems is that due to their nature we end up with more tests. This comes about because in a well written test system the marginal cost of each new test becomes smaller. More tests mean more testing, but also more to organise, so we better make sure we do it well.

Which really brings us back to full circle. Automated tests have the ability to give software teams the courage to refactor code and release more often, but only if the tests themselves don't become an area of technical debt that no one wants to change. For this reason it is super important that tests are written to be human readable, and organised so it is easy to get a feel for test coverage in any area.

Third time Lucky - Thriving in the new normal

On September 4th last year, I woke up to a very unexpected sensation of the bed moving violently. Within seconds I was wedged in the door frame and my house was tossing me about like a small boat in choppy water.

Unfortunately for Christchurch that first shake was just the beginning. Since then we have had two more large, even more devastating quakes which have quite literally rocked our world. In this city, cosying up to your co-workers as you huddle under a desks is not an experience to be ashamed of!

And despite the fact that my family, our homes and our work places, have only been lightly effected (by Christchurch standards) by the quakes. It has still been a traumatic time. I remember feeling a sense of shock and later anger when I saw the Cathedral, such a symbol of strength after the first quake, in pieces. It may only be bricks and mortar, but definitely felt a huge sense of loss that I felt.

Given that a large part of my job as a Agile Test Consultant involves coaxing people to change habits formed over many years, it was with a sense of irony I noticed how uncomfortable it was to be on the other end of change! After all Christchurch is the city I grew up in and is my home - what gives The Earth the right to change it without my permission?

Nine months later, and life is quite different in the once sleepy little conventional city of Christchurch. We are still having periods of lots of shakes - it seems the earth underneath us just doesn't to want to go back to sleep. Added to this, is the difficulty that much of our land having sunk a vital meter or more and continues to liquefy in the bigger shakes.

As the earth continues to move regularly and randomly under our feet it is time for us to start to accept that, love it or hate it, this is the new normal and could continue for a couple more years at least. While this may at first seem like an unfair chore to have to go through, it is also an opportunity.

In order to see that it is an opportunity, at times such as this it is worth knowing a little more about the human condition and what really makes us happy. For instance, many people believe that winning Lotto is a one way trip to happiness but as Martin Seligman writes in his book Authentic Happiness, while such events initially bring us great joy, in the long term winners return to their baseline happiness level.

Seligman further writes "The good news, however, is that after disastor strikes, the [happiness] thermostat will strive to pull us out of our misery eventually....Even individuals who become paraplegic as a result of spinal cord accidents quickly begin to adapt... and within eight weeks they they report more net positive emotion than negative emotion"

Also, I have read some studies that have shown that although redundancy is hard at first, five years afterwards many people think it about it as a positive overall experience. This is partly due to the fact that once the safe option of going to the same job doing the same old stuff is taken away, people are forced to reassess where they are in life. For many this meant they changed their career in some way to align more with what they really wanted to do but would not normally risk trying.

As IT professionals in Christchurch we also find ourselves with a lot of challenges right now. We might be working in less than ideal spaces, have days a time when we must work from home, or have difficulty filling vacancies as it is hard to get people to move to Christchurch.

But I believe this cloud has a silver lining. At the moment we are being forced to reassess what is really important, and what we can or might want to change. Due to our adversity, there is also an opportunity for a lot of innovation to happen here. And that's what I am seeing in Christchurch.

I know of teams who are working in semi-permanent spaces who have acquired laptops and are discarding the concept of 'individual desks'. Thus while solving their immediate problem of having to move out of their office occasionally, they are also hoping to improve the communication within their team, by reducing the distance separating any two team mates.

And while getting qualified staff in Christchurch is a big issue, it could be an opportunity to focus on how best we can introduce lots of young, outgoing, Generation Y'ers into the work place. For while they will take some time to become experienced IT practitioners, I believe that many agile teams I've worked with could really benefit with an influx of collaborative, team orientated people - skills that Gen Y's are well known for. All this could actually help our businesses, particularly in the medium and longer terms.

So I challenge all of you in Christchurch to look at the position we find ourselves in now, not as a disaster, but as an opportunity. For while the changes we've had to endure have at times been hard and unnerving, ultimately they allow us to reinvent ourselves which in the long term, could be highly beneficial.

As for me, I plan to stay in Christchurch because it's still a great place to live. I love this city where I can cycle to most of my clients, and after work I can mountain bike in the Port Hills and feel a world away from the stress of the city.

And when the earth shakes, I choose to use it as a reminder that no matter how hard we try, life is neither predictable or completely safe - though a modern life might feel that way it is just an illusion. I use the quakes to remind me to make the most of each day, to be good to the PEOPLE around me, and to REALLY LIVE and challenge myself.

Automated tests must go green.

You might have one of hundreds of reasons why some of your tests fail and they might be very good reasons. Lack of time to get the automation working properly, functionality that is still buggy, the list goes on... But one thing I am passionate about is that the standard set of automated tests that are run regularly should all go green.

This means that the moment something breaks it is fixed immediately, or if that's not possible then it is removed from the suite and recorded as a task to do later. You don't have to delete it, just put it aside (maybe into another suite of similarly-able tests) until it becomes a lovely green healthy test again.

I just want to be clear here that I'm not saying here that the only way you should use automated testing tools is with perfectly green tests. I am a strong proponent of half manual-half automated testing. What I am recommending is that you separate out the tests that run reliably and are fully automated, from those that aren't.

Now this a simple rule but people often struggle to get their head around it. I don't know how many times I have had the following discussion in the last few weeks, but it's a fundamental. So here it is in an abridged form.

Questioner - "So Clare I've got this test and the way we are currently have it implemented in the system is wrong. I doesn't quite comply with what it should. Josie says that you've told her we've got to make our Fitnesse test pass - but that's just wrong as it's testing the wrong thing. How will we know to fix it if I make the test pass?"

Me - "Yes that's right. Automated tests must go green. Otherwise how does Bob in the other team know that when he makes a (programming) improvement to the Test System that he hasn't broken anything? He won't have the tacit knowledge that you have that this test always fails and will think he's broken the test".

"But it's just wrong. We can't have tests that test for the wrong behaviour."

"Okay. Lets look at it another way. Is there anything at the moment in your personal life something that you would like to do daily if you had more time?"

"Huh? what do you mean? "

"Well say something like flossing your teeth every day, or stretching after exercise - those are the things that I wish I could persuade myself to do"

"Uh." - Thinking to herself that that Clare has gone crazy and wondering where all this is going.

"Now do you remind yourself every morning that you're not doing it, and spend a minute or so processing that reminder every day? And would you then send a reminder out to all your co-workers frequently that says 'Clare needs to be reminded to floss her teeth'? Because that's broken tests do."

"But..."

"Actually I haven't quite finished. Broken tests are actually much worse than this because they don't typically say what the problem is. Instead the reminder is 'Something is wrong here and you now need to spend some time investigating it to see if you caused it!'

"But..."

"And anyway have we actually planned to fix this bug, or is it just something we hope to do in the future when we have more time?"

And so the conversation continues...

We need to keep in mind that the value of automated regression testing comes from it being a pragmatic tool to give us more information in how the changes we are making now are effecting our existing functionality. Automation needs to be about how our system is now, not about some magical Eden where we have fixed all the bugs we know we have.

So what do I suggest if you have a test that won't run reliably or when you know it is testing functionality that isn't right? Well I don't have any hard and fast rules, but here are some ideas.

If the functionality is buggy then use your normal software development processes to deal with the bug and not the automated testing process. Write a story about it or put in a bug in a bug report and let it be prioritised as it normally would be (as long as it not stopping lots of existing automated tests running in which case it is your job to make sure management knows seriousness of the situation). That's how you deal with the fact that you have a bug on your hands.

Then make your test pass according to what the system is doing right now. However, write in the margin (use a comment field in the table or write it in plain text above the table in Fitnesse) that 'currently the system isn't doing what is expected and the result should really be ___ '.

That way if the bug does get fixed, and the test starts to fail, no-one will waste time angsting over whether the new result is right or wrong. They will be able to quickly update the test and move on.

The reason I suggest doing this, is that the system might be a little wrong now but at least this way we make sure that the system is stable and we are not getting unexpected side effects of new code that become more incorrect over time, or may confuse our users. This way we still get cheap information (because it's automated) which we wouldn't get it we skipped the test completely.

When you have tests that don't run reliably, then I suggest you move them into their own suite. Name this suite something more appropriate such as "Manually assisted automated testing" or "Tooling assisted tests" because at this point in time that is exactly what such tests are! And make sure you use this terminology when you talk about these tests with management.

This is really important, because the return of investment (ROI) of running such tests frequently, is seldom worth the cost of running them. So it is important to separate that cost from the cost and thus ROI of running the fully automated tests regularly.

Putting such tests into their own test suite is not going to stop you running these tests before each release, but it allows you and management to separately control how often these tests are run and to focus development time on the most valuable work. And it may also help you highlight to management that more investment is needed in the automated test system.

This is a lesson that I have learnt the hard way, through my own blood, sweat and tears. Want to know what will happen if you try and cope with all the breakages yourself? Then watch this very short excerpt from my presentation at the ANZTB conference in March. I hope it lightens up your day.

Do you disagree with what I have written? Or do you have an even better strategy for dealing with such tests? Please write your comments below - I want to learn more.

Photo Acknowledgment: Sun rays @ Sonnenborgh Park Utrecht by LamberTwm
Porari River track - Fern Frond by ~ Pil ~