In Scrum we tell teams that at the each of each sprint they need to product a “potentially shippable product increment.” However, “potentially shippable” and “shippable” are not the same thing. Some large or complex projects will require the use of “release sprint” or “hardening sprint” at the end of a release cycle (say 6 two-week sprints then a 2-week release sprint). The release sprint is not a dumping ground for sloppy work; rather it is a place where some hardening of the system can occur.
A great example is a bank I work with. They have 12,000,000 lines of COBOL code that manages bank balances. They have a 1,000,000 line web application that can write to the same database as the big COBOL application. Manual testing of the COBOL system takes three weeks. The web team targets having a potentially shippable application each sprint. They end each sprint pretty sure that they haven’t messed up the COBOL system because they are conscientious developers. However, they don’t know that they haven’t messed up the COBOL system. And they do know that eventually the odds will catch up with them and given enough sprints they will eventually mess up the COBOL system somehow. So they run a release sprint every few sprints before the website goes live. They do the manual system testing that would be prohibitively expensive to do in each sprint at that time. Of course we’d all love the COBOL system testing to be an automated single-button click that could be done in each sprint but since that’s unrealistic in their situation right now it’s done during the release sprint.
If you use a release sprint be sure that the length of the release sprint is independent of the number of sprints that precede it. To continue with the team above, if they ran five regular sprints before deciding to ship the web application, they need three weeks for a release sprint. If they ran twenty regular sprints before deciding to ship, they should also need just one three-week release sprint. To say otherwise would be to plan for work (such as bugs) to slop over from one sprint to the next.
Tags: agile release planning, sprints
Fort Short cycle requests using Software WISE packaging engineering project efforts, what do you recomend in term of Agile methods, processes adn tools (e.g. Daily Meetings, Product Backlog, and Spring releases….
These are short cycles, where the requests are to package three type of Media : ORACLE Universal installers, Microsoft MSI, and Customer installs. The expectation is to have them done in 1-week including rollouts (SMS or citrix,. We realize the rollout itself will not occour for packages processed using WISE and SMS if the media size is over 100 meg as they need to be scheduled based on lcoation and number of usrs concurrently conducting the SMS installation (Pull software apckaged), specially if they require installation in Europe, Asia and North America.
I am using daily meeting to disucss current work load, priorities, Project Critical path deliverables, roadblocks, and and obtain status.
From the Scrum and Agile processes and tools, what can be leveraged to create a 1-week packaing model inclusive of the rollout/deployment ?
Thanks
Francisco
This is very interesting for us; we’re trying to do something like this by having a development schedule of sprints and a deployment (release) schedule of sprints. I’ve recently been discussing this with the developers by invoking the concept of two rooms in with a corridor separating them; the room on the left is for the development team and the room on the right is for the deployment team. When the development team finsih a sprint they pass code across the corridor to the deployment team in what ever state its in and nothing ever comes back in the other direction. At this point we define three catogories of “potentially deployable code”;
* a deployable candidate – one that passed its customer review with flying colours
* a buggy candidate – one that finished customer revie with a few bugs
* and a change candidate – one that threw up changes during customer review that preclude deployment.
Which ever state its in the deployment team have a different task ahead. Of course we often end up with a buggy change candidate.
Feedback indicates that the concept is helpful but because in reality we have limited resources some of which must belong to both teams and only one room (lol) it is really only a frame of reference for us.
Also we find that the concept is easier to manage were we have the freedom to take a “completion of artefacts” approach and deploy as soon as we have something worth deploying – but much more challenging when we have commercial dates demanding minimum and target value deployments. The biggest challenge is with buggy candidates because there is a critical path to get them fixed in limited time – with change candidates there is the flexibility to re-prioritise the backlog.
Can you offer any comments on how to cope with very buggy release candidates? And, do you think our reference against this concept is valid?
Cheers
Dave
Hi Dave–
I think I’ve got a pretty easy answer (although not necessarily an easy solution) for you. If your team wrote higher quality code and hit the Deployable Candidate goal every (or nearly every) sprint, wouldn’t these problems go away? If you hardly ever had a bug found once the team was ready to deploy you wouldn’t have to worry about the interruption caused by fixing those bugs in the following sprint. All other similar problems go away. In my mind then I would focus exclusively on addressing this issue. Learning how to produce high-quality code within a sprint/iteration is a different skill that teams have to learn. A team that can’t do it yet isn’t a bad team; it’s just something they need to learn on the way to being agile. Anything else you do will make the current situation more tolerable, which means the team will be less likely to undertake the hard effort of solving the root cause.
Regards,
Mike
We are also having a similar problem and I agree with Mike that we should get to the point where each iteration is releasable instead of having to reserve some time to fix and converge things at the end.
On a related topic however, what’s the best way to do QA? The QA topic seems to be missing in most if not all papers and presentation in this site. We are finding ourselves completing features too late in the iteration and therefore putting QA against the wall. Should we reduce the scope of each iteration, or is there any other approach to a steady state development/testing flow.?
Mike,
Thanks – great advice.
Buggy code is a major frustration for us which we know we need to tackle, but is feels like its one of those issues that lies at the core while we struggle to peel off the outer layers.
I have been posing the question raised by Domingo M for some time; is it simply that the team is taking on too much in each iteration? Difficult to tell, because I guess information about the team’s velocity for creating buggy code isn’t particulalry useful data.
Cheers
Dave
Domingo: QA may be missing from my site because it’s not my specialty, therefore I don’t do conference presentations on it. QA is an integral part of agile, however, and I can certainly offer advice on the subject. First, you are right to be striving to achieve a steady flow through your process. The best way to do that is to have small product backlog items being pulled into the sprint/iteration. To take an extreme example, suppose a team is doing a two-week iteration and they complete 100 product backlog items (features/user stories) during that period. They will undoubtedly have a smooth flow from programmers to testers. Contrast that with a team that only finishes 3 items in the 2 weeks. They’re flow will not be as smooth–they can perhaps get close but it will take a lot of effort and commitment. So, first bit of advice: perhaps use smaller backlog items.
Second, never load up an iteration with all large items. Sometimes you will need to bring in an item that is large and that won’t be easily delivered to QA until late in the iteration. In those cases, supplement that big product backlog item not with another big one but with some small ones. Start working in the iteration with part of the team on the big one and others on the team on the small one (assuming that makes sense). In that way a series of small ones will get handed over to QA while the big one is still being worked on. This will help smooth the flow to QA.
Another thing you can do is realize that the product backlog item (user story / feature /etc.) is *not* the unit of transfer between programmers and testers. Start the iteration by having the programmer and tester involved in an item discuss a good way to split up the work. Perhaps they spent the first few days getting the new feature to work with positive numbers (through a math algorithm). When both are done it is added to a nightly build, both as code and an automated tests. They both then go on to work on support for non-positive numbers and two days later add that to the nightly build, and so on.
A final bit of advice: Along the lines of the Scrum burndown chart, put up a chart that says “product backlog items finished” and graph it every day. Show the number of items done (all the way through testing). It will be 0 the first couple of days. On a good agile team it will then start going straight toward wherever it ends (that is, moving at a consistent rate). On a team with the issues you describe it will stay flat and then move straight up the last two days. That’s a bad size and usually false–there’s no way nothing was truly done for 8 days and then all the work got finished the last two days. Perhaps all the work was given to QA on those last two days but it’s not tested and potentially shippable.
Regards,
Mike
Dave–
You’re absolutely right. The team’s velocity for producing buggy code is actually a meaningless number because our goal isn’t buggy code. The effort to product buggy code is related to the effort to product bug-free code somehow but we don’t know the relationship. Is it 1:1 or 3:1 or 1:3? Impossible to know. Yes, I suggest in the situation you’ve described you should have the team take on less. A rule of thumb I use is to cut the amount of work the team commits to (in hours on a sprint backlog) by 2x the number of hours of estimated work remaining. So if a team commits to 400 hours and claims to “finish” but then plans 50 hours of work into the next sprint to “clean up” or “improve” the work of the last sprint, I will only let that team commit to 300 (=400 – 50×2) hours. The logic is that we missed by 50 and that we probably cut another 50 hours of unseen as of yet quality corners AND I don’t want to miss the goal twice in a row for fear of it becoming a habit.
Regards,
Mike
Domingo:
I have found that a great place to start is to ensure that your acceptance criteria is clearly defined before development starts. That is a great launching pad to get tests automated early in the sprint which leads to getting rid of that end of sprint QA bottleneck that so many teams experience.
Regarding QA, you may want to look at how you are approaching your testing as well. If you are doing all manual testing, you are going to have a tough time getting to true agile bliss
Mike – great post. I was recently on a project that introduced “hardening periods” and had a tough time getting my head around what was appropriate for this and what wasn’t. This was very helpful!! Thank you.
Thanks, Megan. I’m glad you found the post useful. Thank you for contributing to the dialogue with your advice to Domingo. I completely agree with you.
On a CSM training I did with Michael Vizdos, Mike taught to avoid turning sprints into short waterfalls, when they analyze, design, code, test in stages. As a sprint becomes a short waterfall due to possible lack of time some of the staged activities are not performed at all leaving the undone tasks to the next sprints. When this happens with testing a release sprint will definitely be needed and this appears to be a sign of a smell.
Some teams tend to have a between-sprint time used to fix bugs from the last sprints and hence they end up with something like: 2 week sprint – 1 week bugfix/retest cycles. Seems also to be an evidence of a smell had it happen to you.
Hi Mike – I see that this is an old blog, however, I hope you are still picking up the comments.
I have come from a heavy SDLC background, mainly implementing ERP systems such as Siebel and SAP. Generally in these sort of projects after build is complete, the project enters System Test, System Integration Test and then UAT. If the release is into a live environment, Regression Test will be added to that list.
With clients moving from waterfall to SCRUM, how do you explain to your clients how SCRUM will address these phases of testing?
Adam–
To the extent possible, those activities will be performed during each sprint. (This is a good reason why automated tests are important.) This is the whole idea behind the iterative and incremental nature of a sprint. Sometimes some of this work is done in a release sprint if the cost of iterating over it (e.g., a large, manual regression test) is too high.
Along the same lines, I’ve heard of sprints that are devoted to analysis and ones devoted to testing. Are those truly “allowed” by Scrum? To me, it sounds like someone trying to claim that they are doing Scrum but still really doing waterfall. If that’s the case, wouldn’t stabilization sprints and release sprints fit in the same category? Rather than having individual devoted release sprints, should additional time be added to each sprint to allow for the code release? Thanks!
Hi Aaron–
The difference between an analysis sprint and a release sprint is that an analysis sprint is activity specific. We want to avoid *activity-specific sprints*. A release sprint can still involve the work of a cross-functional team but is focused on work that is too expensive to do every iteration. For example, the Windows 8 team at Microsoft could not possibly want to test every single application they can get their hands on (e.g., a 1999 version of dBase, a copy of Word 1995) even though they may have an eventual goal of supporting all those things. I’d expect them to have a reasonable subset that they test with every sprint under the premise that “if these work, just about everything works.” But every few months, and certainly nearing release, I’d expect them to pull out additional old stuff like that to test as part of a compatibility test.
Mike,
I realize this is an old blog but it seems to age like good wine
I just saw the next release plan for our product and was horrified to see four “feature” sprints and no less than six “hardening” sprints, with the first two doing actual coding, and the last four doing deployment, staging, UAT, beta and fixing defects arising from those.
Is this even sane? How does the team maintain sustainable pace with feature freeze 40% along the way? How do they focus on quality if they know, up front, that they have more time to “finish it up” than to do it right in the first place? How do you get fast feedback if it takes 50% longer to get software to the customer than it takes you to actually write it?
The main (and quite valid) concern Engineering has is a huge inventory of bugs, in the thousands. It seems they want to do features first, then spend some time fixing bugs. Wouldn’t it make more sense to devote some time on each sprint (50% of the velocity even) to improving the code base — refactoring, fixing bugs, whatever the team thinks fits best? Make the release six sprints long, too. Yes, you get half the velocity for 6 sprints vs. the full velocity for 4 sprints, but I am willing to bet the improved code quality, better packing of large and small items, and sustained pace would make up for it. Once the six sprints are up, tag it, branch it, put it on staging and start a new release as long as the staging/UAT/whatever cycle of this one is. This way, you have one release done with better code quality run in staging, and another one incorporating all the fixes + new features, in the time frame originally envisioned for just the first one.
Does this make sense? Did I just ask for my last check when I suggested it (rather vocally, too) to management? How do you see this release plan playing out in real life? Thanks!
Hi Peter–
I’d have to say that four feature sprints followed by six hardening ones does sound a bit crazy. But I don’t think they are really doing six hardening sprints. A hardening sprint isn’t for fixing known bugs. I hear this team saying “We have two work queues–one of new features and one of old bugs. We’re going to work for the first 40% of the time on the feature queue and then 60% of the time on the bug queue.”
I’m not sure I’d do things in that sequence. But, in the situation you are describing, I can see a possibility that I would want to add new functionality first. After all, the app is buggy as can be already. There’s a lot of uncertainty in the new functionality and the team will be in trouble if it isn’t added on time. So they want to add it first. They probably also think that it won’t matter much if they fix 438 or 461 bugs during the latter 60% of the project. So, there’s some logic to the sequence in which they want to pull work. I just wouldn’t call those hardening sprints.
Good luck!
Mike,
Thanks for the prompt response! So what you’re saying is, as long as there is a clear commitment that what’s done is really done, and that by the end of Sprint #4 we’re indeed done with the features committed, there will be no temptation to brush anything under the rug. Personally, I trust the team to do that and can work with the Scrummaster to help them keep this promise (they are still rather new to Scrum but very motivated).
Here’s an interesting analogy. A cab driver once told me that, on average, by the fourth hour of his shift, he’d just about paid for lease, tax, gas, and car wash. Anything else he would make from then on, he would take home, so if he got a particularly good fare, he could come home early. I argued with him all the way home that while lease was a capital expense, tax and gas were operational costs and he couldn’t just be “done” with them while production was going. He, on the other hand, knew his average income per shift so statistically, tax and gas were quite predictable, too. We never came to an agreement but obviously he made a point with me if I still recall the conversation after five years
My concern remains with bugs being stored in a separate system (don’t get me started…) with no time tracking capabilities, so the directive right now is to just “close as much as you can”. However, even under these limitations it should be possible to track a separate velocity in #bugs/sprint, and use that for estimation and planning. We close several dozen per Sprint so the law of large numbers should apply. Either that, or I manually pull them in the Agile system and the team estimates them in points. As you might imagine, being a Product Owner this overhead doesn’t make me all too happy. We’ve already agreed to track defects on current deliverables in the Agile system as part of the story they belong to. Hopefully when we clear enough of the bug backlog we’ll be able to get rid of the second system, and the necessity for 60% “hardening” sprints.
Thanks again! Hope I can make it to Oslo
Hi Peter–It’s more than just a “clear commitment” because I wouldn’t want a team to clearly commit to coding for four sprints and then testing after that. To me the key was that the bugs you have are really the same as new feature requests except you/the team have found it helpful to categorize them as feature or bug. Each sprint the team pulls some amount of each type into a sprint and truly finishes each—whether a new feature or a bugfix, the item should be coded, tested, ready for use.
It would be great to see you in Oslo. I’m there in September and am always excited to travel to Norway.
On a separate note, how have you (anyone) implemented a separate testing sprint? We are constantly running out of testing time in our 4 week development sprint. And the tester is losing morale as they are squeezed for more to do in less time. Lately, we decided to implement a separate sprint for testing. But how the development and testing sprint will work is yet to be planned. I am looking for ideas and successful implementations else where. Thanks.
Jude-
Activity-specific sprints are a bad idea. There’s a section on why in Succeeding with Agile.