Thursday, January 19, 2012

Table: and Query: prefixes no longer needed - FitNesse improvements Part 2

This is the second of a three part series into some new changes in the latest release of FitNesse. The first part was about easier ways to debug Fitnesse for .NET

In this installment we look at a a small change that means that query and table tables no longer have to specify their table type each time. This is useful for improving the readability of specifications/tests written in Fitnesse.

For example, tables that used to have to be written with a "Table:" prefix like this one

can now be written as

Now, this may seem like a comparatively minor improvement and in many ways it is. However, it is small, but worthwhile improvement because it touches on one of the most important principles of Specification by Example, which led to the development of FitNesse.

The principle I am talking about is that specifications/tests should be written so that they are understandable to the whole development team, including the business people. In particular, this means test/specs should only specifies what is being checked. The details about how to get the test to run should abstracted into the fixture layer or the setup.

The old “Query:” or “Table:” text belongs very much in the later category. It's an implementation detail that tells FitNesse how to interpret the table. It's about how to run the test and that is why it should be removed where possible from the specifications. FitNesse still needs to know how to interpret the table, so we still need to put this information somewhere.

How to specify a default table type for a table

Firstly, we still need to tell FitNesse how to interpret tables which are not decision tables. This is done by using a “Define Table Type” table which has the format shown below.

|Define table type |
|Bowling | as Table |
|BowlingHistory | as Query |

This table should be listed straight after any “Import” tables and should be hidden away with other execution-only details in a SetUp page. Once this table has been defined we can use the table with out a prefix.

Now many programmers working on FitNesse may find it a little bit unreasonable to get so worked up about such a little word. You might think, “surely, business people can just learn to ignore anything that comes before the colon”. And you are right they can. But the problem is that FitNesse has lots and lots of little details that must be learnt and many people find that a big ordeal. So, in order to get business people more involved – which is after all the aim of such a tool – I think it is important we keep coming up with ways to make FitNesse simpler to learn

Friday, January 6, 2012

How to debug FitNesse .NET in Visual Studio - FitNesse improvements Part 1

This is the first of a three part series where I introduce to FitNesse users out there the changes I have recently made to FitNesse. In doing so, I hope that these improvements will not benefit more teams using FitNesse but over time FitNesse will be moulded into a tool that really allows us to fully live the vision of specification, or even documentation, by example

The first in these series covers a new way to debug FitNesse when using C# and other .Net languages. It has always been possible to debug FitNesse using RunnerW.exe instead of Runner.exe, but it was fiddly to find the right process to connect to each time when using Visual Studio.

So while working with the team at Emendo, we decided it would be much easier to debug FitNesse if we started the runner through Visual Studio.

There's a few steps you'll need to go through to set it this up, but once you have done that it works like a charm and then debugging starts with two button clicks!


Debugging set up

Step 1: Tell FitNesse that when you remote debug you want to start the runner yourself.

Add the following line of wiki mark to the same page as where you tell fitnesse which dlls to use.

!define MANUALLY_START_TEST_RUNNER_ON_DEBUG {true}

Step 2: Configure Visual Studio to debug by starting runner.exe

In your Visual Studio project, go to the properties page and edit the debug page to start the runner.exe application when debugging.


Specifying the correct command line arguments for the runner is important. Luckily, FitNesse provides a way of finding out the settings that it is using – simply run the test you want to debug, and then click on the information icon at the top of the page (shown to the right of this paragraph).

Then you should see a page with a table like the following:

Copy the “command” string into the Visual studio, but first remove the first argument (since it is the runner and you have already specified it) and secondly make sure the last number is port 8085 – the default (but configurable) port that FitNesse expects to connect to when when it is running in debug mode.

Lastly, be sure to set the working directory to the one that FitNesse uses.

Setup is now complete!


Debugging a page

Now you are ready to debug your fitNesse test.

  1. Start test in FitNesse in Debug mode by either

      1. clicking your javascript browser shortcut, or

      2. Adding ?responder=test&remote_debug=true to the URL for the page

  2. Add breakpoints

  3. Start debugging within Visual Studio


This setup should continue to work for fine for all your tests, so long as they all use the same set of dlls. However if you find you add another FitNesse dll you will need to update your command line setting.

If you have any questions or can't get this to work, please post them as a comment or send them to the FitNesse newsgroup and I will try to answer them.

Happy debugging!

Thursday, January 5, 2012

ATDD using FitNesse course now available!

I am proud to announce that I have joined together with Clarus consulting to create a fabulous ATDD course based on FitNesse. So whether you are in Australia, New Zealand or further abroad, you can now get leading edge coaching to kick start the use of ATDD or FitNesse within your organisation.

However, this 2 day course covers far more than just learning how to use FitNesse. Instead it is a team based learning session during which the whole team, (BAs, testers, developers, product owners), works together to create executable specifications for a project. This experience should in

I've already helped many teams quickly increase their automated testing coverage while at the same time improving communication using FitNesse. And I am truly excited about the huge positive effects that those teams have continued to enjoy after I left. It is these experiences that I leveraged when writing the course.

So if you are ready to find out how to build the right software, and how to build it right so that you can confidently release the software time and time again, then get in touch right now by contacting me here.

Course Overview:

Topics for the course are taken from the following.

Day 1:

Introduction to ATDD

  • ATDD – a new way of working
  • ATDD vs TDD
  • Why we should automate tests
  • Why the whole team should be involved


Specification Workshop

  • How to run a Specification Workshop


Writing up specifications (tests)

  • FitNesse Wiki Basics
  • Guidelines for writing executable specifications
  • Tricks for writing specifications in FitNesse
  • Introduction to FitNesse Automation


Day 2:

Making specifications (tests) executable

  • Coding basic tables
  • Structuring FitNesse code
  • Setting up FitNesse for the team
  • Debugging FitNesse code


Organising specifications

  • Creating a fixture toolbox
  • Organising specifications
  • Suites in FitNesse
  • Source Control and FitNesse


Advanced topics

  • Dealing with legacy systems
  • Solutions to common automation problems
  • Overview of other ATDD tools
  • Tips for successful FitNesse implementations


The course is suitable for teams who are completely new to FitNesse as well as those who are already using FitNesse but want to use it more effectively. Everyone should come away from the course with a solid understanding of how to use FitNesse to collaboratively build executable specifications. Teams familiar with FitNesse will also explore some of the more advanced topics in depth.





Thursday, August 18, 2011

The automation contradiction – Four reasons why automated tests need to tell the human story

Ah, test automation. Those wonderful monkeys that free us from the routine drudgery of re-running boring tests over and over! And because the computer monkeys not people execute these tests, some testers write such tests as though it's only important that the compiler understands them in the compiler's 0's and 1's kind of way.


As a proponent of ATDD or Executable Specifications this is not a view that I share. In fact, I coach people who write automated tests in Fitnesse to become story tellers. Because it's not simply a matter of getting a test to execute that's important. In order for the test to be an asset over the longer term, it's also important that other team members can read the test, and easily understand how the functionality works and what is being tested.

That means that tests need to have good names, and must be well organised in a tree structure or tagged. They need to have a short one or two sentence purpose, and clearly express the key business functionality under test as succinctly as possible. (A myriad of small details, such click there, edit this isn't enough). Finally, if possible, all tests should also express the 'why' which describes why the functionality was implemented and who the customer is.

Now, none of this should be all that new to a seasoned IT professional. Particularly one who has been worked on a legacy feature for which such information is missing. What may come as more of a surprise, is that I've come to realise that the readability of tests is actually more important for automated tests that it is for manual tests.

The main reason for this seems to be that manual tests are just that, manual. And every time they are run the person running the test picks up the implicit knowledge about what the test does, and hopefully also some tacit knowledge about how the functionality works. However, with automated tests we don't get this tacit learning. Therefore, it's more important than ever that a quick scan of a test provides correct insight into what the test covers.

Now as anyone who has tried to maintain an automated test suite knows, the tests also sometimes break. (Otherwise what would be the use of running the tests in the first place?). Without knowing what the purpose of the test was in the first place, it's pretty hard to decide what to do with the test and how to fix it. At this point you'll be grateful for any effort expended on expressing the test in a more human readable manner. So if you ever see yourself altering existing functionality in the future, then this is reason number two for spending some extra effort now writing more readable tests.

Thirdly, well written, executable specifications are an immensely powerful way for the whole team to communicate. This is because they are written in a form that everyone, from customers to BAs, testers and programmers (who like to debug things), can understand and talk about. And because they represent what the product does, and not what people think it does, they are also very real. Quite frankly I don't think this is something you can appreciate until you have experienced it in action. So if you are still sceptical at this point then I suggest read Gojko Adzic's Specification by Example which contains case studies of other people who have tried this approach.

The final reason that test organisation becomes really important in automated test systems is that due to their nature we end up with more tests. This comes about because in a well written test system the marginal cost of each new test becomes smaller. More tests mean more testing, but also more to organise, so we better make sure we do it well.

Which really brings us back to full circle. Automated tests have the ability to give software teams the courage to refactor code and release more often, but only if the tests themselves don't become an area of technical debt that no one wants to change. For this reason it is super important that tests are written to be human readable, and organised so it is easy to get a feel for test coverage in any area.

Sunday, July 3, 2011

Third time Lucky - Thriving in the new normal

On September 4th last year, I woke up to a very unexpected sensation of the bed moving violently. Within seconds I was wedged in the door frame and my house was tossing me about like a small boat in choppy water.

Unfortunately for Christchurch that first shake was just the beginning. Since then we have had two more large, even more devastating quakes which have quite literally rocked our world. In this city, cosying up to your co-workers as you huddle under a desks is not an experience to be ashamed of!

And despite the fact that my family, our homes and our work places, have only been lightly effected (by Christchurch standards) by the quakes. It has still been a traumatic time. I remember feeling a sense of shock and later anger when I saw the Cathedral, such a symbol of strength after the first quake, in pieces. It may only be bricks and mortar, but definitely felt a huge sense of loss that I felt.

Given that a large part of my job as a Agile Test Consultant involves coaxing people to change habits formed over many years, it was with a sense of irony I noticed how uncomfortable it was to be on the other end of change! After all Christchurch is the city I grew up in and is my home - what gives The Earth the right to change it without my permission?

Nine months later, and life is quite different in the once sleepy little conventional city of Christchurch. We are still having periods of lots of shakes - it seems the earth underneath us just doesn't to want to go back to sleep. Added to this, is the difficulty that much of our land having sunk a vital meter or more and continues to liquefy in the bigger shakes.



As the earth continues to move regularly and randomly under our feet it is time for us to start to accept that, love it or hate it, this is the new normal and could continue for a couple more years at least. While this may at first seem like an unfair chore to have to go through, it is also an opportunity.

In order to see that it is an opportunity, at times such as this it is worth knowing a little more about the human condition and what really makes us happy. For instance, many people believe that winning Lotto is a one way trip to happiness but as Martin Seligman writes in his book Authentic Happiness, while such events initially bring us great joy, in the long term winners return to their baseline happiness level.

Seligman further writes "The good news, however, is that after disastor strikes, the [happiness] thermostat will strive to pull us out of our misery eventually....Even individuals who become paraplegic as a result of spinal cord accidents quickly begin to adapt... and within eight weeks they they report more net positive emotion than negative emotion"

Also, I have read some studies that have shown that although redundancy is hard at first, five years afterwards many people think it about it as a positive overall experience. This is partly due to the fact that once the safe option of going to the same job doing the same old stuff is taken away, people are forced to reassess where they are in life. For many this meant they changed their career in some way to align more with what they really wanted to do but would not normally risk trying.


As IT professionals in Christchurch we also find ourselves with a lot of challenges right now. We might be working in less than ideal spaces, have days a time when we must work from home, or have difficulty filling vacancies as it is hard to get people to move to Christchurch.
But I believe this cloud has a silver lining. At the moment we are being forced to reassess what is really important, and what we can or might want to change. Due to our adversity, there is also an opportunity for a lot of innovation to happen here. And that's what I am seeing in Christchurch.

I know of teams who are working in semi-permanent spaces who have acquired laptops and are discarding the concept of 'individual desks'. Thus while solving their immediate problem of having to move out of their office occasionally, they are also hoping to improve the communication within their team, by reducing the distance separating any two team mates.

And while getting qualified staff in Christchurch is a big issue, it could be an opportunity to focus on how best we can introduce lots of young, outgoing, Generation Y'ers into the work place. For while they will take some time to become experienced IT practitioners, I believe that many agile teams I've worked with could really benefit with an influx of collaborative, team orientated people - skills that Gen Y's are well known for. All this could actually help our businesses, particularly in the medium and longer terms.

So I challenge all of you in Christchurch to look at the position we find ourselves in now, not as a disaster, but as an opportunity. For while the changes we've had to endure have at times been hard and unnerving, ultimately they allow us to reinvent ourselves which in the long term, could be highly beneficial.


As for me, I plan to stay in Christchurch because it's still a great place to live. I love this city where I can cycle to most of my clients, and after work I can mountain bike in the Port Hills and feel a world away from the stress of the city.

And when the earth shakes, I choose to use it as a reminder that no matter how hard we try, life is neither predictable or completely safe - though a modern life might feel that way it is just an illusion. I use the quakes to remind me to make the most of each day, to be good to the PEOPLE around me, and to REALLY LIVE and challenge myself.

Friday, May 20, 2011

Automated tests must go green.

You might have one of hundreds of reasons why some of your tests fail and they might be very good reasons. Lack of time to get the automation working properly, functionality that is still buggy, the list goes on... But one thing I am passionate about is that the standard set of automated tests that are run regularly should all go green.

This means that the moment something breaks it is fixed immediately, or if that's not possible then it is removed from the suite and recorded as a task to do later. You don't have to delete it, just put it aside (maybe into another suite of similarly-able tests) until it becomes a lovely green healthy test again.

I just want to be clear here that I'm not saying here that the only way you should use automated testing tools is with perfectly green tests. I am a strong proponent of half manual-half automated testing. What I am recommending is that you separate out the tests that run reliably and are fully automated, from those that aren't.

Now this a simple rule but people often struggle to get their head around it. I don't know how many times I have had the following discussion in the last few weeks, but it's a fundamental. So here it is in an abridged form.

Questioner - "So Clare I've got this test and the way we are currently have it implemented in the system is wrong. I doesn't quite comply with what it should. Josie says that you've told her we've got to make our Fitnesse test pass - but that's just wrong as it's testing the wrong thing. How will we know to fix it if I make the test pass?"

Me - "Yes that's right. Automated tests must go green. Otherwise how does Bob in the other team know that when he makes a (programming) improvement to the Test System that he hasn't broken anything? He won't have the tacit knowledge that you have that this test always fails and will think he's broken the test".

"But it's just wrong. We can't have tests that test for the wrong behaviour."

"Okay. Lets look at it another way. Is there anything at the moment in your personal life something that you would like to do daily if you had more time?"

"Huh? what do you mean? "

"Well say something like flossing your teeth every day, or stretching after exercise - those are the things that I wish I could persuade myself to do"

"Uh." - Thinking to herself that that Clare has gone crazy and wondering where all this is going.

"Now do you remind yourself every morning that you're not doing it, and spend a minute or so processing that reminder every day? And would you then send a reminder out to all your co-workers frequently that says 'Clare needs to be reminded to floss her teeth'? Because that's broken tests do."

"But..."

"Actually I haven't quite finished. Broken tests are actually much worse than this because they don't typically say what the problem is. Instead the reminder is 'Something is wrong here and you now need to spend some time investigating it to see if you caused it!'

"But..."

"And anyway have we actually planned to fix this bug, or is it just something we hope to do in the future when we have more time?"

And so the conversation continues...

We need to keep in mind that the value of automated regression testing comes from it being a pragmatic tool to give us more information in how the changes we are making now are effecting our existing functionality. Automation needs to be about how our system is now, not about some magical Eden where we have fixed all the bugs we know we have.

So what do I suggest if you have a test that won't run reliably or when you know it is testing functionality that isn't right? Well I don't have any hard and fast rules, but here are some ideas.

If the functionality is buggy then use your normal software development processes to deal with the bug and not the automated testing process. Write a story about it or put in a bug in a bug report and let it be prioritised as it normally would be (as long as it not stopping lots of existing automated tests running in which case it is your job to make sure management knows seriousness of the situation). That's how you deal with the fact that you have a bug on your hands.

Then make your test pass according to what the system is doing right now. However, write in the margin (use a comment field in the table or write it in plain text above the table in Fitnesse) that 'currently the system isn't doing what is expected and the result should really be ___ '.

That way if the bug does get fixed, and the test starts to fail, no-one will waste time angsting over whether the new result is right or wrong. They will be able to quickly update the test and move on.

The reason I suggest doing this, is that the system might be a little wrong now but at least this way we make sure that the system is stable and we are not getting unexpected side effects of new code that become more incorrect over time, or may confuse our users. This way we still get cheap information (because it's automated) which we wouldn't get it we skipped the test completely.

When you have tests that don't run reliably, then I suggest you move them into their own suite. Name this suite something more appropriate such as "Manually assisted automated testing" or "Tooling assisted tests" because at this point in time that is exactly what such tests are! And make sure you use this terminology when you talk about these tests with management.

This is really important, because the return of investment (ROI) of running such tests frequently, is seldom worth the cost of running them. So it is important to separate that cost from the cost and thus ROI of running the fully automated tests regularly.

Putting such tests into their own test suite is not going to stop you running these tests before each release, but it allows you and management to separately control how often these tests are run and to focus development time on the most valuable work. And it may also help you highlight to management that more investment is needed in the automated test system.

This is a lesson that I have learnt the hard way, through my own blood, sweat and tears. Want to know what will happen if you try and cope with all the breakages yourself? Then watch this very short excerpt from my presentation at the ANZTB conference in March. I hope it lightens up your day.

Do you disagree with what I have written? Or do you have an even better strategy for dealing with such tests? Please write your comments below - I want to learn more.

Thursday, August 12, 2010

Super-charge your daily stand up

Some Scrum followers believe that Scrum must be followed exactly otherwise it's utility is reduced, but since my top strength is creativity, I really struggle to do the same thing, in the same way, day after day.

And I don't believe I'm alone as someone who benefits from a little variety in the daily routine. For instance, one of the top tips for sticking to an exercise plan is to not always run around the same block, but to vary the path and companions to to keep it interesting.

In the same way,I believe that we can avoid our Scrum teams getting "stuck in a rut" by occasionally changing how we express the three core questions:
  1. What did I do yesterday?
  2. What do I plan on doing today?
  3. Do I have any impediments?

For me the questions are one specific implementation of questions that achieve two main aims:

  • Prevent duplication of effort, by
    • Sharing knowledge about what has been learnt/created that others in the team might like to know/use. (Q. 1)
    • Making sure no two people are independently working on the same task (Q. 2)
  • Looking for ways to help each other, so the team velocity increases. (Q.2, Q.3)

I believe there are many other sets of nearly equivilent questions that would result in the same information getting shared but produce a different feeling amongst the team. And by chopping and changing the questions we could inject a little spice into our standups to keep them feeling fresh and vital.

For instance, imagine a scrum team that is treating the stand up just like a progress meeting where team members feel they are reporting to the project manager. They could super charge their daily stand-up by changing the questions to:

  1. What did I learn / create yesterday that will be useful for others in the team to know? (This doesn't require an answer if all tasks went as expected but might if I have something the rest of the team will find useful or hit road blocks)
  2. What are my general plans for today?
  3. What help would get me their faster?

Hopefully these questions challenge the speaker to think more about sharing knowledge and less about justifying yesterdays work by constructing as long a list of things worked on as possible. (Which as another member of the scrum team I find excruciatingly boring....please stick to the interesting stuff).

In a similar vein, I've seen many teams where everyone is working mostly individually and not helping each other much. I guess this is common in our strongly individualistic Western culture where asking for help is often seen as a sign of weakness. I even see this in my 95 year old Grandma (and in all my Grandparents while they were alive) who struggles to have people help her, despite spending her first 85+ years in round-the-clock faithful service to her family and the community in general.

But we're missing the point! The whole point of Scrum is to find ways for the team to increase their velocity by working together. To use the strengths of the team in harmony to work more efficiently and to be more engaged with work.

But the thing is, I don't find many competent mind readers on the Scrum teams I work in. So the next best thing I can do, is to be humble and honest and let others know what I am struggling with, or even what is sapping my energy. Because you never know it might be exactly the sort of thing that floats the boat of someone else in the team.

So I wonder what would happen if for one sprint everyone in such teams committed to the challenge of completing the following statement at every stand up.

  • Today I could work a little more effectively if only someone would ...

For me, impediments bring about mental images of huge things that I've tried three different approaches to get round but I'm still stuck on. Whereas this statement focuses more on identifying opportunities for collaboration, however small. This is because good teams also help each other by removing small frustrations during the day to keep the energy up and the work moving.

For instance, the other day I got my team-mate a coffee, not because he's more senior than me, or because I'm the woman in the team. I got him a coffee because I was leaving the meeting and he wasn't, and he happened to mention he could do with a coffee (with a look on his face that confirmed he needed a pick-me-up). If the roles were reversed, I know he'd do the same for me. This is one small example of true team work.

What would happen if everyone in your team asked for a little bit of assistance everyday?


Are you honest and humble enough to start asking for help and to charge-up your Scrum team with super team work power?".

If you ask me, it's a lot easier to learn to be open and humble, than training everyone around you to read your mind!

Tuesday, March 23, 2010

Is Feedback the new Agile?

Lately I've been reading a good book on change, Influencer, the power to change anything. It was talks about the fascinating methods of Dr Mimi Silbert, who by creating an environment full of constant feedback about performance, helps career criminal change their lives forever. For instance, within hours of entering her Delancey Street Foundation, recruits will already have started working in the restaurant, setting actual tables and feeling the satisfaction from having achieved something new. (A sense of achievement is one of the best sorts of feedback there is.)

To see the power of feedback for yourself, take a graduate employee on her first day and give her a very small task to do. (Adding her name and team wiki and fixing up the layout is ideal.) Make sure that within an hour, and then several times through out the day, she gets feedback from completion of tasks. See how this sets her up for the rest of her time in the job. I've done this with many new graduates and always has good results.

After reading Influencer, I started thinking how all this applied to software development and I quickly saw some correlations with Agile methods. I started thinking to myself, "Is Agile all about feedback? Can we can simply replace the Agile Manifesto with a simple pledge , 'Thou shall give and receive feedback as much as possible?'" Well maybe that was taking it a bit far, but there certainly seemed many parallels.

For instance, continuous integration and unit tests must be obvious ones. The aim of both is to fail fast. That way you get your feedback as soon as possible that the code doesn't work.

And what about Scrum? When I was taught Scrum, the feedback cycles were strongly emphasised. First there is the daily standup meeting which gives feedback about how current sprint is going. Then there is the retrospective which gives feedback about how the sprint process is working. Bingo another one.

Pair programming (an opportunity for feedback into better ways to do whatever you are doing right now) is another. As is delivering small features regularly (feedback to the customer on how the process is going). And then there is keeping tasks small (feedback to the programmer that they have succeeded). The list just kept growing. So Agile may be more than just feedback, but nevertheless feedback seemed to be a huge part of Agile.

Then a slightly adventurous idea came to me. I thought "Lets turn the question upside down". Knowing what we know, that feedback is a powerful motivator for behaviour change, how can we improve feedback in our development processes to make them even more Agile?

Suddenly I was flooded by questions. How can we get our automated tests running faster? Have better coverage? Give each other more feedback about what behaviour we like? And behaviour we don't like? Reduce our units of work so they're finished quicker? Get our features to our customers faster? Give our programmers a better sense on completion for each card? Let our testers see the bugs they found have been fixed? Share the improvements the customer feels with the whole team?

These questions are the same questions we have always been asking. The difference is now we are emphasising giving each individual team member feedback about the results of his or her actions. With this in mind, it offers new possibilities for improving processes.

After all if feedback can tame the urges of hardened criminals, I believe it can work a power of good on our customers, programmers, testers and even our development managers.

Thursday, July 16, 2009

Half and half testing - The best of both worlds.

Automated and manual testing. With testing, it is often a strict dichotomy where at any one time you are only doing one or the other. A while back we asked the question "Why not have tests which utilise the strengths of both man-kind and machine?" and the results amazed us.

The original idea came about because we were having trouble automating our gui tests with Selenium. At that time, we just didn't have the time to solve all the issues. This meant that we only were testing our gui in an ad hoc way, which was slow, frustrating and lacking in coverage.

At the time, we had good coverage of the backend processing and number crunching, but couldn't test easily that our gui worked correctly with the backend - a classic integration test problem. With a big release coming up, where different teams had written the gui and backend parts, we realised this was something we had to tackle head on.

Our solution was to write a fixture that would temporarily halt the Fitnesse test, tell the user to do something and wait until they pushed "done", then carry on with the test.



The resulting tests, were partly manual, partly automated but incredibly efficient compared to our completely manual testing. It takes about an hour to run all the tests. To do the same amount of testing completely manually would have taken over 2 weeks and been less complete. It was a huge win.

In general our half and half tests go like this:
1 Fitnesse: Create some data for the tests
2 Human: Use the gui to create more data.
3 Fitnesse: Test the interactions between all the data is correct.

For example, our system needs both buyers and sellers so one benefit is we can now test the buyer and seller gui separately. For instance, by creating the seller data using Fitnesse first, we are able to test the buyer wizard on its own without having to test both wizards (buyer and seller) together. Another win.


We see big benefits because the data created by the user is picked up by Fitnesse and works seamlessly with the rest of the fixtures. We do this by creating an account with Fitnesse first and telling the manual tester to use that account. That way when the done button is pushed, Fitnesse can query the database to find what the objects were created. Now the tests are fully automated, we are still using this system, it has another benefit that it gets round the 'What is this word' human-check on the create account page as well.

While the initial idea behind the fixture was to use it as a temporary solution we have found this fixture to be an extremely valuable tool. With it, testers can now run tests even when one small part of the process is not yet properly automated. Previously, everything had to be automated or the test wouldn't run.

We have used it to manually run stored procedures on the database, and to write tests for critical bugs. We even use it for ad-hoc (but Fitnesse assisted) testing. The long term strategy is still to fully automate most of these tests, but this fixture allows us to get some value out of the tests before they are completely working. And it helps us to 'test the tests' to assess how much work it is to automate a test, before we decide its priority for automating. There are also a few tests, where we want to cast a human eye over the layout. Those we plan to always be half manual.

The "Ask User" fixture may a simple idea, but one that can really help you get more value from your testing system. For that reason, I think it's a great tool to have in the toolbox. Here's the Ask User java code to get you started. It's not very pretty but it does the trick. However, to get most value out of it you probably want to find some way to integrate it with the rest of your Fitnesse tests. That part, I leave as an exercise for you to do.

Saturday, June 20, 2009

Creating a QA process from out of thin air

This week I had the opportunity to present the work I have been doing over the last year to a group of testers at a Testing Professionals Network (TPN) meeting. I enjoyed sharing the changes we have made and how far we have come in a short time.

Agile Acceptance testing with Fitnesse

A little more about what I've been up too...

About a year ago, I swapped jobs within my company from being a developer to the challenging and highly stressful job (in the beginning) of QA team lead. This meant I was tasked with building a QA team to test a high load, high availability adserver.

This all happened about a year after the company started when we suddenly found we had enough customers that we had to be very careful about what we released. However, the product had been created with such speed that we had no automatic system tests and it was incredibly hard to get any test to run successfully on our test environment. We had also just hired our first tester, who was struggling to get anything working. So it was a big problem and it needed solving, quickly.

My reponse was to start building an automatic testing system, starting with the area of biggest ROI (return on investment). I chose budgeting as I guessed not going over our promised budget was pretty important for our customers, was almost impossible to test by hand but was a reasonably simple test to autotmate.

From those small beginnings the suite slowly grew. As things started to take shape more and more developers could see the power of the system and were happy to contribute to it. Others required the boss to tell them that they must do it. Now 12ish months later, thanks partly to a re-write and thus the need to test everything, we probably have about 90% functionality coverage with our automated tests. Not only that, but we also have a team of developers who are enthusiastic about getting the tests running.

Having seen several other companies take a long time to develop automated tests I am truly amazed at our process. I believe our success is due to getting everyone involved in the testing process. Making the software developers run the tests really helped. That way they got to see how their system design affects the system testability.

So one of the main themes I had hoped to share in this presentation was that in order to build a good, reliable automated system test suite you generally need to improve the testability of the system under test. Improving testability generally benefitsboth testers and developers, as the tools that are developed also help during development and debugging of the system. For example, we created an installer, tools for checking the system status and ways to easily create test data.

So creating a quality test system quickly, I believe, requires the cooperation of both testers and developers. However, putting such a system in place is tricky because it generally requires a simultaneous culture and technology change.

An example of a culture change required is that developers often believe testers are solely responsible for testing. However, developers can make a huge difference on the testability of the system depending on how they design the system. So to get everyone focused on the end goal, tested and releasable software, it is important that both managers and developers understand that developers and testers share the responsibility for testing the system.

However, if a developer has not worked on a project with a good automated system test suite, then they may not be able to see how they can help the tester. Then, if they see tests running erratically they are more likely to question the value of such tests, than to question the way that they themselves have designed the system.

Thus the first culture change requires the developer to be able to picture how they can assist the testing process. But getting them to make the first changes, requires them to believe that the tests are worth building in the first place.

My approach to tackling this chicken and egg problem, is to be agile. Start with a working tests, then ask for small improvements that make the tests more reliable, or quick, or easy to write. Give some value, ask for an improvement, give value, ask again, give value, ask again...

So my second message to all tests leads out there is don't just stick your head down and suffer with unpredictable automated tests. Think about what you need changed about the system, and make sure your developers listen to you. It may seem like a huge effort for small gains in the beginning, but they all snow ball and save you time in the long run.

Please, take a look at the presentation and send me any comments you have. In particular if there are slides that don't make sense without the audio then write a comment and I'll explain them in more detail.

What small change can you make that will make your system more testable?

Friday, February 20, 2009

A lesson from the river

When the week is over, I love nothing more than to head out of town into the clean air and freedom of the wilderness. Being a bit of an adrenalin junkie, I often choose some rather adventurous way of doing this. In particular, I love to kayak the sort of rivers that send shivers down the spine of normal people. While I kayak purely for enjoyment, I have found that the experiences from the river often alter how I think back in the office.


You see, software development is a little like kayaking in that both are team games. On a river you depend on your mates watching out for you in order to paddle something too dangerous to paddle alone. This is not much different than a software project which requires you to work with
other developers to create something too big to finish alone.

The big difference between the two situations is the level of risk you play with. With kayaking you don’t just play with your job and lifestyle, you literally play with your life. For those who can accept the danger it is a a perfect way to gain risk assessment muscles.

Kayaking has taught me that I should not study just the rapid I want to paddle, but I must also look downstream. What will happen if I take a swim? Is there a nice pool after the tricky bit or does the river flow into a deadly rock sieve? In many cases a grade V river with pools is easier and less dangerous than a continuous grade IV river with no place to rescue a swimmer.

On a river I refer to this concept of the consequences if it all goes wrong, by the unforgettable term, “F*** up factor” (FUF). In the office you may prefer to use “Consequence”, but the concept remains the same. Risk management is more than just asking “What is the likelihood of it all going to plan?” It’s also about asking “Can I live with the consequence if this doesn’t work out as planned?”

While we all think this way to some degree, I often see people behaving in ways that makes me think their FUF muscles aren’t strong. Perhaps a person will balk at trying a new idea (high risk, but low FUF as all you are losing is development time), yet would change the database on the production server without a dry run if the rest of the team didn't prevent it (medium risk, high FUF because it could affect all our customers).

Or consider the FUF of an exception in test code vs production code. While it’s equally important both code bases are maintainable, the FUF of an exception in our test code is nearly zero. Therefore when writing test code putting in exception handling code probably isn’t worth the effort. But for production code we probably want to think of the FUF of every exception and handle it appropriately.

A further benefit of FUF analysis is that it may help you feel more confident when dealing with the unknown. After all, if you have accepted that you can live with the FUF should it all go to custard, then your brain can stop worrying about the possibility of it going wrong. That should allow you to focus on the more important task of making it go right.

I recommend you give FUF analysis a try. It should make you develop a more appropiate application for your users and hopefully make you more adventurous in the process.