As you probably know I’m a big fan of writing the product backlog in the form of user stories and of writing user stories in the form, “As a , I so that .” An example being, “As a frequent flyer, I really want to be able to connect to the internet while flying so that I can update my blog while traveling rather than having to save this as a text file and updating my blog later.” (Can you guess where I am while writing this?)

What I’ve found makes a user story in this format very easy to work with in a spreadsheet is to take the boilerplate parts and put them into column headings. So we’ll have column headings like “As a” and “I” and “so that”. The meat of each story is then clearly visible in each row. Additional columns can be added for things like a unique identifier, notes, status and such. In this example, I’ve also included a column for the theme or grouping of which the story is a part. You can see this in the screen capture below. You can click the image for a larger view.

In order to clean up the product backlog, I want to introduce you to a new Scrum artifact–the Long-Term Product Backlog. The Long-Term Product Backlog is maintained by the product owner and is usually round, black and sits next to or under the product owner’s desk. Left unattended, a product backlog can become large and hard to work with. If your backlog has reached this point, take some time to do some spring cleaning—review the backlog and delete / throw away user stories that you can finally admit you’re never going to get to.

Some product owners or teams feel this is an admission of failure. It’s not. In most cases it’s an admission of success–we’re throwing away feature ideas that we once thought important because since writing them down we’ve discovered even more important features. So celebrate the fact that you’ve got newer, bigger, better feature ideas and delete the less valuable ones from your product backlog.

If permanently deleting such ideas is too big of a step, perhaps you really do want to introduce a new artifact into your process. Create a spreadsheet and paste the user stories there for safe keeping. Or print a report from your product backlog tool and file the report somewhere safe just in case. In my experience, though, you won’t miss them. And just like the bright new view through the windows of the Conderosa, you’ll be able to see the rest of your product backlog much more clearly.

But any dependencies that remain can raise questions about how to estimate the individual product backlog items. One common agile project management question is what to do with work that could be performed as part of either of two backlog items but that only needs to be done once. As an example, consider these two product backlog items, which are written as user stories:

• As a user, I can add an invoice to the system so that it gets paid.
• As a user, I can delete an invoice.

These two user stories share some work. Supposing this is a very simple system, whichever one of these stories is done first will also involve the team much of the database design for invoices. Some database design work (a cascading delete between Invoice and Line_Items, for example) will be done only when a specific one of the stories is developed. But, some work (deciding that a table called Invoice exists at all, perhaps) will be done as part of whichever story is implemented first. Assuming that this work is non-trivial, the estimates given to these stories will be influenced by which will be implemented first.

In deciding how to estimate these and how to handle the common work between them, I suggest there are two guiding principles we can rely on:

• We should assume work will be done in its natural order.
• The sum of all estimates on the product backlog should represent the total size of the project (as understood at that time, of course).

Let’s see how these two principles lead to the answer about how to estimate our two user stories about adding and deleting invoices. Although a good agile team should be able to implement these stories in either order, it is most likely that the product owner will ask the team to add invoices first. After all, you can’t delete an invoice until it has been added. This is what I mean by the natural order.

Although it is natural to add invoices to a database before deleting them, it is not hard to imagine a need to do these in the opposite order. Suppose our team is writing a new system to interface with an existing invoice database. That existing system is full of old invoices from the 1940s and the first thing the product owner wants is the ability to delete old invoices.

When a product wants stories done in the natural order or when it seems likely the product owner will want them done in that order, my advice is that the team should estimate with that assumption in mind and not bother noting it on the story cards. Documenting all logical assumptions about the natural order gets tedious. To document all such assumptions, our add an invoice story would also need to be documented with “assumes login story is done first,” and so on.

But when a team knows or suspects they will be asked to work out of the natural order, they should document the assumption. If the team assumes that adding an invoice will be quicker to implement because deleting is already done, they should add appropriate notes to these story cards.

Teams are, of course, sometimes wrong in their assumptions or are surprised by product owners who change their minds. These situations are resolved by simple discussion along the lines of saying “We assumed you’d want these stories in a different order. So, a few points [or ideal days] of work move off this story and onto that story.”

It is even possible that working outside the natural order can sometimes increase the total size of the work. For example, to do the delete invoices user story before the add invoices story, our team might need to write a short script to preload the database with dummy data just so we can be sure delete is working. Usually when this happens the change in project size is very small and washes out in the noise of other estimating and planning errors. (In other words, it’s rarely worse than someone being out sick for a day.)

Why not get around this problem of potentially moving points from one story to another by assuming that each story is done first, which would be a worst case for its estimate? This is where our second principle comes in: The sum of the estimates should add up to the total size of work being considered. If the total work is 8 and that’s what we’d put on a single add+delete user story then our two user stories need a total of 8. To put a higher number on them overstates the size of the project. This would lead to incorrect projections of the project completion date based on velocity.

Teams that are new to agile project management often do not go far enough in eliminating these hand-offs. They often make the assumption that the programmers should finish programming a product backlog item before handing it off to the testers. This results in lengthy delays at the start of a sprint, when the testers are waiting for a first product backlog item to be handed to them. On a Scrum project, the unit of transfer between disciplines should be smaller than an individual product backlog item. That is, although there will always be some hand-offs (not everyone can be working on everything all the time), the amount of work being transferred from one person to the next should generally be as small as possible.

To facilitate these small transfers, Scrum teams learn to work by doing a little of everything all the time. Rather than an analysis phase (done without the programmer and tester) followed by a programming phase followed by a testing phase, a little of each of those activities is happening at all times.

Even with an emphasis on minimal hand-offs, there will be some product backlog items that will require a week or more of programming time before the programmer can give something even beginning to be testable to a tester. That’s OK. Not everything can be split as small as we might like. To avoid a flurry of testing activity toward the end-of-sprint, avoid bringing a bunch of these complex items into the same sprint. Instead of planning a sprint with, for example, three very large items that cannot be partially implemented, bring one or two large items into the sprint along with two or three smaller items. Some of the programmers can work on the large items, handing them over to testers whenever possible. The remaining programmers can work on the smaller items, ensuring a somewhat smoother flow of work to testers early in the sprint.

While your goal in each sprint should be to do a little bit of everything all the time, some backlog items are complex by nature and will not be completed until near or on the last day of a sprint. To avoid having multiple product backlog items wrapping up the last few days of the sprint, mix the size of the product backlog items you choose to bring into each sprint.

• Detailed Appropriately. User stories on the product backlog that will be done soon need to be sufficiently well understood that they can be completed in the coming sprint. Stories that will not be developed for awhile should be described with less detail.
• Estimated. The product backlog is more than a list of all work to be done; it is also a useful planning tool. Because items further down the backlog are not as well understood (yet), the estimates associated with them will be less precise than estimates given items at the top.
• Emergent. A product backlog is not static. It will change over time. As more is learned, user stories on the product backlog will be added, removed, or reprioritized.
• Prioritized. The product backlog should be sorted with the most valuable items at the top and the least valuable at the bottom. By always working in priority order, the team is able to maximize the value of the product or system being developed.

Product backlogs are discussed in much more detail in Succeeding with Agile.

This means that on a Scrum project, design is both intentional and emergent. The design emerges because there is no up-front design phase (even though there are design activities during all sprints). Design is intentional because product backlog items are deliberately chosen with an eye toward pushing the design in different directions at different times.

As an example of how the product backlog items can be sequenced to influence the architecture of the system, consider a workflow system I worked on. The system supported a fund-raising company that produced specialized T-shirts and similar products. School-age children would go door to door selling these items. The sales revenue would be split between the company and the organization the kids represented, such as a school, sports team, or other group. For each sale, the kid would complete a form and send it to the company, where it was scanned, sent through an optical character recognition (OCR) process, and converted into an order. To keep shipping costs down, orders from the same organization were batched together and sent back to the organization, after which the kids would hand-deliver the items.

Our software handled the entire process—from when the paper was received by the company until the shipment went out the door. Kids have notoriously bad writing and are bad spellers, so our system had to do more than just scan forms and prepare packing lists. There were various levels of validation depending upon how accurately we thought each order form had been read. Some forms were routed to human data-entry clerks who were presented the scanned form on one side of the screen, the system’s interpretation on the right, and an additional space to make corrections.

Because thousands of shirts were processed on the busiest days, this process needed to be as automated as possible. I worked with the product owner, Steve, to write the product backlog. After that I met with the development team to discuss which areas of the system were the highest risk or we were the most uncertain about how to develop. We decided that our first sprint would focus on getting a high-quality document to run through the system from end to end. It would be scanned, go through OCR, and generate a packing list. We would bypass optional steps such as deskewing crooked pages, despeckling pages, and so on but would prove that the workflow could be completed from start to finish. This wasn’t highly valuable but it was something that needed to be done, and it let the developers test out the general architecture. After we accomplished this, we had a basic database in place and could move documents from state to state, triggering the correct workflow steps.

Next the developers asked the product owner if they could work on the part of the system that would display a scanned document to a human who would be able to override the scanned and interpreted values. This was chosen as the second architectural goal of the project for three reasons:

• It was a manual step, making it different from the workflow steps handled already.
• Getting the user interface right was critical. With the volume of documents flowing through this system, saving seconds was important. We wanted to get early feedback from users to allow time to iterate on usability.
• After this feature was added, users could start processing shirt orders.

The project continued in this way for a few months and was ultimately tremendously successful, meeting all of the prerelease targets for reliability and throughput. A key to the success was that the product owner and technical personnel worked together to sequence the work. The closest the team got to a design phase was the first afternoon in the conference room when we identified risky areas and dark corners and decided which one we wanted to tackle first. From there the design emerged sprint by sprint, yet was intentionally guided by which product backlog items were selected to illuminate the dark corners and risks of the project.

Succeeding with Agile goes into much more detail on agile design, including how the roles of architect and user experience designer change with Scrum, the concept of emergent design, and how teams work together and with the product owner to deliver increments of functionality that guide the design of the final product.

But what about a bug found today in something the team wrote a month or a year ago? And what about the existing bug database that a team new to agile is almost sure to bring with them?

The ideal situation is to put the bugs right onto the product backlog. To see why, consider the situation from a user’s perspective: Do you think a user cares whether something is considered a New Feature or a Bug? No. The user just wants the system to behave in some way different from how it behaves today. They don’t care what it’s called.

But notice I called this the ideal situation. It isn’t always practical–and sometimes for reasons out of the team’s control or influence. Bug databases have a way of becoming embedded into organizations. The tech support group uses them as a primary way of communicating with the developers. The marketing group uses the bug database in a similar way. This makes it quit the bug database cold turkey.

In these cases, the most common solution teams use is to have two backlogs

• a product backlog of features
• a bug backlog

This approach is a bit harder on the team and the product owner but allows an agile team to work more easily with existing processes in the organization.

Sometimes when we make such accommodations to the overall organization, the accommodation can damage or destroy the agile adoption. Allowing everyone to work on five concurrent projects is an example of a crippling accommodation. Accommodating the organization’s use of a bug database is not crippling. It’s just a bit more work.

The product owner takes on most of the additional work by having to prioritize two separate work queues and then present them to the team in a more or less consolidated manner (“My top priorities are the first three items on the product backlog, then bugs #12403 and 12415, then these next two items on the product backlog…”). This isn’t too onerous as the items would need to be prioritized even if all were in one backlog.

So: Ideally bugs belong on the product backlog just like any feature request. But, that would often necessitate a significant change for the rest of the organization so two backlogs are used.

• Big Visible Charts. Alistair Cockburn coined the term “Big Visible Charts” to describe the charts that agile teams like to hang on their walls. One of the most common of these is the sprint burndown chart, showing the number of hours remaining as of each day of the current sprint. Charts like these provide a strong visual reminder of the current state of the project. What is shown on these charts will get the attention of team members so display charts showing the most important information for that sprint. Ron Jeffries suggests considering big visible charts showing the number of passing customer acceptance tests, the pass/fail status of tests by day, sprint and release burndown charts, number of new stories introduced to the product backlog per sprint, and more.
• Additional feedback devices. In addition to big, visible charts, it is common for an agile team to use additional visual feedback devices in their workspace. One of the most common is a lava lamp that is turned on whenever the automated build is broken. I’ve also worked with teams that use flashing red traffic lights to indicate exceptional conditions such as an issue on a production server. Also popular are ambient orbs and Nabaztag rabbits, which are wireless programmable devices that can also be configured to change colors, speak messages, or wiggle their ears as a team desires. Devices like these make a workspace more dynamic, unobtrusively bringing into it information the team may find helpful.
• Everyone on your team. Each person on the team should ideally be able to see each other person on the team. This absolutely includes the ScrumMaster and ideally includes the product owner. I do understand, however, that product owners often have responsibilities to other groups outside the development team and so may sit near them instead. Still, in an ideal world the product owner would be visible to everyone in the team workspace.
• The sprint backlog. One of the best ways to ensure that everything necessary is completed in the sprint is to make the sprint backlog visible. The best way to do that is by displaying the sprint backlog on a wall, ideally in the form of a task board A task board is usually oriented in rows and columns with each row containing a particular user story and one index card or sticky note for each task involved in that story. Task cards are organized in columns, minimally including “To Do” “In Process,” and “Done.” In this way, team members are able to see work progressing across the task board during the sprint and all work to be done is visible at all times.
• The product backlog. One problem with running an endless series of sprints is that each can feel disconnected or isolated from the whole of a planned released or related set of new capabilities. A good way to reduce the impact of this problem is by displaying the product backlog somewhere clearly visible. This can be as simple as keeping the shoebox full of user stories written on index cards on a table in the middle of the team’s space. Even better, tack the index cards with those upcoming user stories on a wall where all can see them. This allows team members to see how the user stories they are working on in the current sprint relate to others that are coming soon.
• At least one big white board. Every team needs at least one big whiteboard. Locating this in the team’s common workspace encourages spontaneous meetings. One developer may start using the board to think through a problem; others may notice and offer to help.
• Someplace quiet and private. As important as open communication is, there are times when someone needs some peace and quiet. Sometimes this is for something as simple as a private phone call. Other times it can be to think through a particularly challenging problem without being interrupted.
• Food and drink. It’s always a good idea to have food and drink available. These don’t need to be fancy, and they don’t even need to be provided by the organization. I’ve worked with plenty of teams that buy a small under-desk refrigerator and share the expense of buying water bottles or soda for it. Other teams buy a coffee machine, depending on team preferences. Some teams rotate bringing in snacks, both healthful and not.
• A window. Windows are often a scarce commodity and are doled out to an organization’s favored employees. One of the nice things about an open workspace is that windows are shared. Even if the view is only of our parking lot and can only be seen across three messy desks, at least I can see the window and some natural light.

It’s unlikely that every one of these will be visible from your workspace, but the more of them visible, the better. Let me know what else you think should be visible from within the ideal agile workspace.

First, let’s clarify what people mean when they refer to a “release backlog.” A release backlog is a subset of the product backlog that is planned to be delivered in the coming release, typically a three- to six-month horizon. A release backlog would presumably contain the same type of items as on a product backlog. My preference for this, of course, would be user stories. So, at the start of some release planning horizon, a product owner with help from the team and other stakeholders would example the product backlog, select some amount of high priority work and move those items to the release backlog. Later, in sprint (iteration) planning, the team would examine the release product and select some amount of work to do. (Notice how this already goes against existing agile literature that says the team selects top priority items for the sprint from the product backlog?)

I am opposed to the use of a release backlog for a couple of reasons:

First, in all projects that are not being done as fixed-scope contracts (and, arguably, even then) we don’t know exactly what user stories or features will be delivered in a release. To say that there is a release backlog may not imply a guarantee that all features on it are delivered. However, it does create additional work as items are moved off back and forth between the product and release backlogs. This will happen as the team’s velocity changes (even by small amounts) from sprint to sprint. If the release backlog is to show what a team will deliver in a release, it should contain an amount of work equal to the number of story points (or ideal days) times the number of remaining sprints. If you have an average velocity of 20 and 6 sprints left, the release backlog should contain 120 points worth of work. Suppose the team finishes 24 points of work in a sprint. The backlog is now down to 96 points (120-24) but should contain 100 (5 sprints left x an average velocity of 20). Things get even more difficult if that good velocity of 24 increases the team’s average velocity to 21. In that case the release backlog should contain 5×21=105 points of work; we need to move 9 points of work from the product backlog to the sprint backlog. If there’s a release backlog, this moving of items back and forth from product backlog to release backlog will happen every sprint.

Second, introducing a release backlog creates additional confusion. We’ve already overloaded the word “backlog” with “product backlog” and “sprint backlog.” Why confuse things further with a third type of backlog unless there is an immensely compelling reason to do so?

Third, introducing the concept of a Release Backlog actually makes it harder for product owners to do one of the most important things they should be doing–looking at the features or user stories that just miss the cut of being in a release. When I look at a product backlog and count down the number of story points that I anticipate will be in the release, the most interesting thing to me is not necessarily the set of user stories that will make it into the release. I’m often just as interested in the next few user stories below that cut-off point. That is, I want to know what user stories won’t quite make it into the release. After all, we say that one of the benefits of agile (on some, not all, projects) is the ability to decide later if we should release a few sprints early (if a lot has been done), on the planned date, or perhaps a sprint or two late if we want to add more functionality before a release. This is made more difficult with a Release Backlog because a product owner now needs to examine both a Product Backlog and a Release Backlog to make these decisions.

So, what do I propose instead?

I advocate that all teams track their velocities. This leads to a graph like this:

This graph shows a team calculating three average velocities:

1. Long term average, defined as the mean of the last eight sprints
2. Worst case average, defined as the mean of the worst three chosen among the last eight sprints
3. Best case average, defined as the mean of the best three chosen among the last eight sprints

You can choose your own ways of calculating long-term, best-case, and worst-case average velocities. These are just the ones I like. We use these average velocities as shown in this figure:

In this figure, the product backlog is shown on the left as a stack of index cards. We use the three average velocities multiplied by the number of remaining sprints to draw arrows pointing into the product backlog. The range created by these arrows indicates the likely amount of work finished by the planned date. Our release plan is the work defined by the arrows. Rather than a Release Backlog as a new work product or artifact, my equivalent of a release backlog is the product backlog and these arrows pointing into it.

As you can imagine, predicting velocity as a range is much more likely to be accurate than using a single point estimate (“Our velocity is 31.”) as has been implied by everyone I’ve heard talk about a Release Backlog. Additionally, looking at figures like these should make it apparent that the amount of work expected to be delivered in a release will change from sprint to sprint. Pointing arrows into a product backlog is much easier and more helpful than creating a new work product, the Release Backlog.

In my view, the purpose of the iteration planning is for the team to arrive at a commitment to some set of functionality that they feel reasonably confident they can complete in the coming iteration. The purpose is not to identify tasks. The purpose is not to estimate the number of hours for each of those tasks. The purpose is to figure out how many and which product backlog items they can commit to delivering in the coming iteration.

If a team can do this without discussing tasks and hours, so be it. If a team can walk into the iteration planning meeting and one team member announces, “Sssh, I’ve receiving divine inspiration……… We can do the first three items plus the sixth on the product backlog,” I would call the meeting over and we’d have our commitment. I’d be impressed and I’d be skeptical, but we’d be done with the meeting.

Since I haven’t seen a team do this yet, I strongly recommend that an iteration planning meeting involve identifying tasks, estimating the effort of each, and then using that information to arrive at a set of product backlog items the team can commit to. I think this is the best way for a team to arrive at that reasonable commitment that I believe is the purpose of this meeting. Is it the only way? No. I’ve also worked with teams who decide that they will split work up such that all tasks are “about half a day”. They then can skip explicitly estimating the hours for each task because, in a way they’ve already done it.

But, until I learn how to receive divine inspiration at the start of my iteration planning meetings, I’m sticking with identifying tasks as hours as a good way to figure out how much to commit to.