When a team that has relied mostly or entirely on manual testing decides to adopt Scrum, it will quickly discover how hard it is to run short sprints when there’s a lot of manual testing to be done each sprint. It will also realize that unless it does something drastic, the technical debt will continue to accumulate. Teams in this situation can follow a three-step process to extricate themselves from at least the worst of these problems:
- Stop the bleeding.
- Stay current.
- Catch up.
The first priority of a team with technical debt in the form of an over-reliance on manual testing is to stop the bleeding, stop things from getting worse. The best tourniquet is to find ways to automate some of what is being tested manually. To mix metaphors, teams should find the low-hanging fruit: tests that will be easy to automate but save a lot of manual effort. Brian Marick, a leading authority on testing and coauthor of the Agile Manifesto, has told me that “the real low-hanging fruit is often not automating some test execution but automating other testing tasks, like populating databases or automatic navigation to the page where you’ll start manual testing. You’re not reducing the number of manual tests, but you’re reducing the total time it takes to run them.”
After the bleeding has been stopped, the situation will no longer be getting worse from sprint to sprint. Manual tests still will be added each sprint but the team is finding enough low-hanging fruit each sprint to offset the time needed to run the new manual tests. At this point, it is time to move to step two: learning to stay current. During this phase, the team focuses on learning how to write and automate tests for whatever new features are added during the sprint. While doing this, no more debt is accumulating, so the situation isn’t getting any worse, but it’s not yet getting any better either. Learning to add automated tests in the same sprint as the feature will be a new skill for the team. It won’t be as hard to learn as the initial skills were during the first phase but will require new discipline.
Eventually the team enters the final phase, which is when it catches up on additional outstanding testing debt. I generally tell teams that I coach that I don’t care how quickly the outstanding testing debt is brought down as long as it is indeed coming down. Obviously, I would prefer the debt to come down as fast as possible. But, by stating it this way I am emphasizing that what I am most concerned with are the first two steps: stop the bleeding and stay current.
For more on testing with an agile process like Scrum, pick up a copy of Succeeding with Agile.
Tags: agile testing
Great post Mike. I find that non-software companies tend to acquire a large amount of technical debt for this exact reason. Along with the advise that you provide in this post, I also suggest a “this point forward” strategy of adding tests to all new code, and any code that is touched during the sprint. This will allow for continued progress and slows the accumulation of debt.
Hi Mike, any suggestion for some good automation tools, especially for that first phrase (automating part of the process such as automatic navigation)?
Do you recommend interface automation tool such as Selenium that are sensitive to breaking?
Hi Robert–
Thanks. The “From this point forward” approach you describe is essentially what I’m after with the “Stay Current” phase above. After we’ve stopped things from getting worse by learning how to automate well by doing some of the easy stuff, I next want the team in a mode where all new features come with automated tests developed in that iteration. That’s the key place to get to.
Once there, paying down the debt is simply a matter of doing it. It may take a long time but things are no longer getting worse.
Hi Vinny–
In my previous post, I commented on the importance of looking at multiple levels of testing. I stressed that we want to minimize the amount of testing through the user interface layer because of how easy it is to break such tests.
I do like Selenium, though. But, for the best discussion of testing tools, check out the Agile Testing book by Lisa Crispin and Janet Gregory. It’s fantastic.
Great post Mike
Makes perfect sense to me
Actually- the advice you gave on this can be stretched to many other areas as well.
Many a times you come across an anti-pattern spread over in your application and you feel like fixing the whole thing. The correct way to do is- “No-more-this-way-from-this-point-forward”. And then you can refactor the existing software piece at a time.
Hi Rajiv–
Great point. Thanks, Mike
Important post. Thanks Mike. The path to automated testing is definitely the path to sanity.
To extend Rajiv’s point, I had a team one time who were very tempted to fix debt when they found it, wherever in the code it was, which inevitably would have been to the detriment of meeting their commitment. Instead then, they wrote down each required fix/change on a sticky note and stuck them in a special place on the taskboard called “technical debt”.
This served two purposes, i) it made the debt visible to the organization (who, like the team, could watch it grow as more trouble was unearthed) and ii) allowed the team to make more realistic estimates on new stories that they now knew would touch that damaged code.
The rule was don’t ever touch a piece of dirty code and not clean it. No more code-arounds, no more new code without automated unit and acceptance tests. It was a powerful mechanism for transparency and change. I now recommend it to all teams I work with.
Hi Tobias–
Good story and useful advice for many teams.
I’ve had teams maintain a “Technical Debt Backlog” on a big white sheet in the team room. I mentioned that once either here or perhaps on a discussion list. I got lambasted for advocating essentially what you describe. The lean zealots were telling me we should “stop the line” any time we found any form of technical debt. As much as I like the lean principles, I still think they need to be applied to software development with caution because of the differences between manufacturing and the non-repetitive nature of software development.
Great Post. It has already pushed me to make some changes to our process. Much appreciated.
Hi Mike,
> I got lambasted for advocating essentially what you describe. The lean zealots were telling me we should “stop the line” any time we found any form of technical debt.
Lambasted for a common sense approach? That’s a little fanatical. The concept of “stop the line” when technical debt is encountered is a sure way to project failure. It is a defocus onto low-priority tasks which goes against the “focus” value in Scrum. Debt has to be handled when appropriate, not when it is first noticed. I am not a big fan of Lean as applied to software, this is another reason why.
Yes, “lambasted.” And yes I thought it was a bit of a fanatical “process over context” type situation.
A team I am coaching has come up with a similar approach to the technical debt problem.
They have a place where they have posted those annoying problems which they encounter but which they do not want to touch right away. Each time they find themselves touching this sore spot in the code, they tag the sticky with a dot. That way when they want to negotiate for a refactor or to reduce some technical debt they can easily identify what has been hurting them the most, most recently.
It’s a nice low tech way of tracking technical debt which translates into a BVC in the team room.
[...] Reduce Manual Test Debt [...]
Carlo–
I love that. I’ve made the lists before but never thought of putting the dots on them. Excellent. Thanks!!
Tobias – I really like that ‘Tag & Sweep’ approach to technical debt.
Technical Debt, is something that is the stakeholders job to address at their discretion. Thats the whole point of the metaphor, and the basis of the solutions. There are times when the stakeholder would prefer a spike approach to meet business needs (you accrue TD, to meet immediate tactical business goals) vs consolidation where the team devotes more effort to servicing the technical debt (via refactoring and building up the test suite).
A stop the line approach undermines the whole concept of the ‘Debt’ metaphor. Its a bit barmy – deciding when to accrue or service debt, real or metaphorical, is always the responsibility of the wallet holder.
The problem with TD has not been with the fact that it is accrued, or serviced. Its problem has been with its visibility.
Tobias: I like the fact that you tag features/backlog items as being ‘tainted’ by TD. It adds much needed context – a stakeholder may prefer to route around tainted features rather than service them, but they will see it accruing. A responsible stake holder can then plan their maintenance periods around them.
Vinny Ly: The Ruby community are very advanced on acceptance level automated testing (i.e. automated testing for QA professionals and stakeholders), there are a lot of very powerful tools in this field, such as Watir, and Cucumber. IME Watir is most accessible to QA professionals (and is very well documented).
Mike: I am a recent convert to your blog, and its great stuff. Don’t know how I got referred to it (probably Lisa Crispin/Brian Marick on twitter). Great stuff.
Hi RIchard–
I’m glad you’re hear and thanks for joining the conversation.
Mike,
Great post. I really liked Brian Marick’s suggestions in particular.
Two self-interested (but nevertheless very true) points:
1) An additional point to be considered is the importance of ensuring that the tests you’re running are achieving as much coverage as possible in as few tests as possible. Proven, scientific approaches (refinements on pairwise testing) to test design are now easier than ever to implement with tools like our Hexawise tool and/or James Bach’s AllPairs tool.
2) When it comes to automatically populating databases with data that has a broad mix of data ranges, data types, different combinations of multiple categories of data to be associated with an individual account, etc., the same considerations with point 1 are true. See Hexawise and/or AllPairs.
(Disclaimer: My firm created Hexawise).
Hi Justin–
Agile teams talk often about focusing on delivering the most valuable features (and not needing to develop the low-value features). I often hear things like 20% of the features deliver 80% of the value. I’ve never seen anyone quantify that but I’m sure the general idea is true and agile teams pay attention to this when developing features.
But, you’re very right that when a team is trying to catch up on all the tests they never automated before (pre-agile, I hope) then finding 20% of the tests that deliver the 80% of the value should also be that team’s goal. I haven’t seen Hexawise before but I checked out your site and the product seems very promising. Good luck with it.
Great article!
A classic bottom-up approach to solving a problem. Partial test automation as a start to save time, then build on it. I’ve done it myself without even thinking of it as being a valid approach to use in any situation.
[...] Cohn skriver om detta i sin blog och kallar det för projektets ”Manaul Test Technical Debt”. Projektet har en sk [...]