First, the terms don’t matter that much. These are not terms with important specific meanings like “pointer” to a programmer or “collateralized debt obligation” to whomever it is that’s important. Story, epic and theme are merely terms we use to help simplify some discussions teams have. The terms do have standard meanings that date back to some of the earliest Extreme Programming (XP) teams. And it’s nice to use terms in industry-standard ways. But, if these terms didn’t exist, you’d make up your own.

So let’s see what each means.

A user story is simply something a user wants. Stories are more than just text written on an index card but for our purposes here, just think of user story as a bit of text saying something like “Paginate the monthly sales report” or “Change tax calculations on invoices.” Many teams have learned the benefits of writing user stories in the form of “As a I so that .” But it is not necessary that a user story be written that way. The advantages of that format are covered elsewhere on this site.

An epic is a large user story. There’s no magic threshold at which we call a particular story an epic. It just means “big user story.” I like to think of this in relation to movies. If I tell you a particular movie was an “action-adventure movie” that tells you something about the movie. There’s probably some car chases, probably some shooting, and so on. It tells you this even though there is no universal definition that we’ve agreed to follow and that an action-adventure movie must contain at least 3 car chases, at least 45 bullets must be shot, and …

So, “epic” is just a label we apply to a large story. Calling a story an epic can sometimes convey additional meaning. Suppose you ask me if I had time yesterday to write the user stories about the monthly reporting part of the system. “Yes,” I reply, “but they are mostly epics.” That tells you that while I did write them, I didn’t get the chance to break most of them down into stories that are probably small enough to implement directly.

Finally, “theme” is a collection of stories. We could put a rubber band around that group of stories I wrote about monthly reporting and we’d call that a “theme.” Sometimes it’s helpful to think about a group of stories so we have a term for that. Sticking with the movie analogy above, in my DVD rack I have filed the James Bond movies together. They are a theme or grouping.

Hopefully you’ve found this short explanation helpful. I look forward to your thoughts.

As you probably know I’m a big fan of writing the product backlog in the form of user stories and of writing user stories in the form, “As a , I so that .” An example being, “As a frequent flyer, I really want to be able to connect to the internet while flying so that I can update my blog while traveling rather than having to save this as a text file and updating my blog later.” (Can you guess where I am while writing this?)

What I’ve found makes a user story in this format very easy to work with in a spreadsheet is to take the boilerplate parts and put them into column headings. So we’ll have column headings like “As a” and “I” and “so that”. The meat of each story is then clearly visible in each row. Additional columns can be added for things like a unique identifier, notes, status and such. In this example, I’ve also included a column for the theme or grouping of which the story is a part. You can see this in the screen capture below. You can click the image for a larger view.

In order to clean up the product backlog, I want to introduce you to a new Scrum artifact–the Long-Term Product Backlog. The Long-Term Product Backlog is maintained by the product owner and is usually round, black and sits next to or under the product owner’s desk. Left unattended, a product backlog can become large and hard to work with. If your backlog has reached this point, take some time to do some spring cleaning—review the backlog and delete / throw away user stories that you can finally admit you’re never going to get to.

Some product owners or teams feel this is an admission of failure. It’s not. In most cases it’s an admission of success–we’re throwing away feature ideas that we once thought important because since writing them down we’ve discovered even more important features. So celebrate the fact that you’ve got newer, bigger, better feature ideas and delete the less valuable ones from your product backlog.

If permanently deleting such ideas is too big of a step, perhaps you really do want to introduce a new artifact into your process. Create a spreadsheet and paste the user stories there for safe keeping. Or print a report from your product backlog tool and file the report somewhere safe just in case. In my experience, though, you won’t miss them. And just like the bright new view through the windows of the Conderosa, you’ll be able to see the rest of your product backlog much more clearly.

• Detailed Appropriately. User stories on the product backlog that will be done soon need to be sufficiently well understood that they can be completed in the coming sprint. Stories that will not be developed for awhile should be described with less detail.
• Estimated. The product backlog is more than a list of all work to be done; it is also a useful planning tool. Because items further down the backlog are not as well understood (yet), the estimates associated with them will be less precise than estimates given items at the top.
• Emergent. A product backlog is not static. It will change over time. As more is learned, user stories on the product backlog will be added, removed, or reprioritized.
• Prioritized. The product backlog should be sorted with the most valuable items at the top and the least valuable at the bottom. By always working in priority order, the team is able to maximize the value of the product or system being developed.

Product backlogs are discussed in much more detail in Succeeding with Agile.

But what about a bug found today in something the team wrote a month or a year ago? And what about the existing bug database that a team new to agile is almost sure to bring with them?

The ideal situation is to put the bugs right onto the product backlog. To see why, consider the situation from a user’s perspective: Do you think a user cares whether something is considered a New Feature or a Bug? No. The user just wants the system to behave in some way different from how it behaves today. They don’t care what it’s called.

But notice I called this the ideal situation. It isn’t always practical–and sometimes for reasons out of the team’s control or influence. Bug databases have a way of becoming embedded into organizations. The tech support group uses them as a primary way of communicating with the developers. The marketing group uses the bug database in a similar way. This makes it quit the bug database cold turkey.

In these cases, the most common solution teams use is to have two backlogs

• a product backlog of features
• a bug backlog

This approach is a bit harder on the team and the product owner but allows an agile team to work more easily with existing processes in the organization.

Sometimes when we make such accommodations to the overall organization, the accommodation can damage or destroy the agile adoption. Allowing everyone to work on five concurrent projects is an example of a crippling accommodation. Accommodating the organization’s use of a bug database is not crippling. It’s just a bit more work.

The product owner takes on most of the additional work by having to prioritize two separate work queues and then present them to the team in a more or less consolidated manner (“My top priorities are the first three items on the product backlog, then bugs #12403 and 12415, then these next two items on the product backlog…”). This isn’t too onerous as the items would need to be prioritized even if all were in one backlog.

So: Ideally bugs belong on the product backlog just like any feature request. But, that would often necessitate a significant change for the rest of the organization so two backlogs are used.

My first answer is always: Try to get the one person to move to where the rest of the team is. I don’t expect to see any moving trucks roll out when I ask this, but I have to ask. I figured if I keep asking, some team somewhere will eventually have the person move. Having one remote is a cost that must be borne by the full team. For the right person, it’s easily worth it. But sometimes, the person who is remote has not special skills, knowledge or experience to justify the added hassle.

So, assuming that we’ve tried the “move here” solution and it didn’t work, what can a team do that likes the physical task board but that has one remote member?

My recommendation is to continue to use the physical task board–it is simply too beneficial to the collocated team members to give it up in favor of a product backlog tool, especially if team members are already used to it and like it. How the team conveys information on the task board to the remote team member depends on what that person does.

Sometimes the remote person works remotely because they have a very specialized skill that couldn’t be filled in the office where the rest of the team is. This would be the case, for example, if our remote person is an expert in Windows internals and is writing boot-time code in C++. It would also be the case if our remote person has twenty years of experience in the domain and with some of our really old code. In these cases, the remote person usually (not always) works for a day or two relatively alone. The remote person in this case could identify during the daily Scrum what he or she will work on and not need further interaction with the task board until selecting the next task. Here, the remote person doesn’t really interact with the task board at all and interacts only with the team. Not ideal as I’d like the person to see the tasks, but this can work in some situations.

What is more common is for the ScrumMaster to take on the responsibility of updating an electronic version that mirrors the physical task board. A shared spreadsheet is normally sufficient for this, but some teams opt for a more specialized tool, which is fine. Many tools offer 5-10 user free licenses and since the tool is only needed by the ScrumMaster and one remote employees, the free license is adequate.

Normally the ScrumMaster updates the electronic task board once a day, usually right after the daily scrum meeting. Of course, reading the physical task board and updating the electronic one can be quite time-consuming because the ScrumMaster has to look at each task in both places to see if that item needs to be updated.

One good way of minimizing the time the ScrumMaster spends doing this is to mark the cards on the physical task board. The mark indicates “I’ve updated this task. Please update it in the online task board.” I like to use Post-It flags for this. As team members do their daily scrum, they stick one of these flags (of any color) on the card. If the estimate changes, if the task is done, if a new task is added, those cards are tagged with a flag. When the meeting is over, the ScrumMaster can very easily see which items need to be updated. The ScrumMaster removes the flags once the online task board is updated. This approach also works in situations where the ScrumMaster updates the board more than once a day. Any time someone changes the board, flag the task.

Other teams do something similar using color-coded dots. Anything touched on Monday is blue, Tuesday is red, and so on. Two problems occur though: First, the dot packs usually come with four colors to a pack so you have to buy a fifth color. Second, it’s a hassle to make sure we have the right color on hand.

First, let’s clarify what people mean when they refer to a “release backlog.” A release backlog is a subset of the product backlog that is planned to be delivered in the coming release, typically a three- to six-month horizon. A release backlog would presumably contain the same type of items as on a product backlog. My preference for this, of course, would be user stories. So, at the start of some release planning horizon, a product owner with help from the team and other stakeholders would example the product backlog, select some amount of high priority work and move those items to the release backlog. Later, in sprint (iteration) planning, the team would examine the release product and select some amount of work to do. (Notice how this already goes against existing agile literature that says the team selects top priority items for the sprint from the product backlog?)

I am opposed to the use of a release backlog for a couple of reasons:

First, in all projects that are not being done as fixed-scope contracts (and, arguably, even then) we don’t know exactly what user stories or features will be delivered in a release. To say that there is a release backlog may not imply a guarantee that all features on it are delivered. However, it does create additional work as items are moved off back and forth between the product and release backlogs. This will happen as the team’s velocity changes (even by small amounts) from sprint to sprint. If the release backlog is to show what a team will deliver in a release, it should contain an amount of work equal to the number of story points (or ideal days) times the number of remaining sprints. If you have an average velocity of 20 and 6 sprints left, the release backlog should contain 120 points worth of work. Suppose the team finishes 24 points of work in a sprint. The backlog is now down to 96 points (120-24) but should contain 100 (5 sprints left x an average velocity of 20). Things get even more difficult if that good velocity of 24 increases the team’s average velocity to 21. In that case the release backlog should contain 5×21=105 points of work; we need to move 9 points of work from the product backlog to the sprint backlog. If there’s a release backlog, this moving of items back and forth from product backlog to release backlog will happen every sprint.

Second, introducing a release backlog creates additional confusion. We’ve already overloaded the word “backlog” with “product backlog” and “sprint backlog.” Why confuse things further with a third type of backlog unless there is an immensely compelling reason to do so?

Third, introducing the concept of a Release Backlog actually makes it harder for product owners to do one of the most important things they should be doing–looking at the features or user stories that just miss the cut of being in a release. When I look at a product backlog and count down the number of story points that I anticipate will be in the release, the most interesting thing to me is not necessarily the set of user stories that will make it into the release. I’m often just as interested in the next few user stories below that cut-off point. That is, I want to know what user stories won’t quite make it into the release. After all, we say that one of the benefits of agile (on some, not all, projects) is the ability to decide later if we should release a few sprints early (if a lot has been done), on the planned date, or perhaps a sprint or two late if we want to add more functionality before a release. This is made more difficult with a Release Backlog because a product owner now needs to examine both a Product Backlog and a Release Backlog to make these decisions.

So, what do I propose instead?

I advocate that all teams track their velocities. This leads to a graph like this:

This graph shows a team calculating three average velocities:

1. Long term average, defined as the mean of the last eight sprints
2. Worst case average, defined as the mean of the worst three chosen among the last eight sprints
3. Best case average, defined as the mean of the best three chosen among the last eight sprints

You can choose your own ways of calculating long-term, best-case, and worst-case average velocities. These are just the ones I like. We use these average velocities as shown in this figure:

In this figure, the product backlog is shown on the left as a stack of index cards. We use the three average velocities multiplied by the number of remaining sprints to draw arrows pointing into the product backlog. The range created by these arrows indicates the likely amount of work finished by the planned date. Our release plan is the work defined by the arrows. Rather than a Release Backlog as a new work product or artifact, my equivalent of a release backlog is the product backlog and these arrows pointing into it.

As you can imagine, predicting velocity as a range is much more likely to be accurate than using a single point estimate (“Our velocity is 31.”) as has been implied by everyone I’ve heard talk about a Release Backlog. Additionally, looking at figures like these should make it apparent that the amount of work expected to be delivered in a release will change from sprint to sprint. Pointing arrows into a product backlog is much easier and more helpful than creating a new work product, the Release Backlog.

Thinking back into the dark ages, I can remember when I first read about “non-functional requirements.” The term threw me for a loop. “If it’s non-functional, why do I care about it?” I’m sure the author of that book clarified this for me a page later, but the term has always seemed an odd one to me. I prefer to think of non-functional requirements as “constraints” we put on the system. When a product owner says “this system must perform adequately with 100,000 concurrent users” the product owner is putting a constraint on the development team. The product owner is effectively saying, “Develop this software any way you’d like as long as you achieve 100,000 concurrent users.” Each constraint we put on a system narrows the design choices a bit; calling these “constraints” rather than “non-functional requirements” helps us remember this.

Fortunately constraints / non-functional requirements can be easily handled as user stories. Here are a couple of examples:

• As a customer, I want to be able to run your product on all versions of Windows from Windows 95 on.
• As the CTO, I want the system to use our existing orders database rather than create a new one sot that we don’t have one more database to maintain.
• As a user, I want the site to be available 99.999% of the time I try to access it so that I don’t get frustrated and find another site to use.
• As someone who speaks a Latin-based language, I might want to run your software someday.
• As a user, I want the driving directions to be the best 90% of the time and reasonable 99% of the time.

As you can see from these I was quite easily able to stick with the “As a <type of user>, I want <some goal>, so that <some reason>” template that I prefer for most user stories. I do this for a couple of reasons but want to comment further on only one of them here. Consider the example of the CTO constraining the team to use the existing orders database. (This was the real situation; the team was considering a second orders database that would be sychronized at night. The CTO overheard this and said “no!”) If we wrote this requirement as simply “Must use existing orders database” it would be entirely possible that a few months down the road we wouldn’t remember why we should be constrained in that way. We’d ask the product owner if she cared if we used a secondary database, and she’d say she had no objections. And we’d make a mistake of using one. Embedding the person who wants something can be very useful.

But, you should be careful not to get obsessed with that template. It’s a thinking tool only. Trying to put a constraint into this template is a good exercise as it helps make sure you understand who wants what and why. If you end up with a confusingly worded statement, drop the template. If you can’t find a way to word the constraint, just write the constraint in whatever way feels natural

These days, though, there are many truly large projects being done with agile methodologies. In preparing for this posting I started counting on my fingers the number of 100+ person projects I’m aware of. I quickly ran out of fingers. I’ve been involved in a couple of 500+ person projects and am aware of three projects that each have over 1,000 people. We are truly at the point of doing large scale agile.

Unfortunately, the charts and tools we use for such large projects have not entirely caught up with us. For example, the time-tested Scrum burndown chart is absolutely wonderful but is really limited to showing only one thing: a single team’s rate of progress through their product backlog. This month I want to write about visualizing large product backlogs using a technique I’ve been advocating with my clients for three years but will be writing about here for the first time.

Suppose you have a very large product backlog–such as one with lots of themes (groups of related user stories) or being worked on by multiple agile teams. The best way to visualize this product backlog is with a treemap. Treemaps were invented in the 1990s by Dr. Ben Shneiderman as a way of visualizing hierarchical (that is, tree-structured) data.

The following is a simple treemap of a two-story house:

In this treemap, each rectangle is sized to represent the relative area of the room. From it you can see that the master bedroom is about twice the size of either kid’s room and a little larger than the downstairs family room. The combined green area of the downstairs rooms is slightly larger than the area of blue upstairs rooms. From this very simple treemap you can get a feel of certain aspects of this house.

To visualize a product backlog with a treemap we need to conceptualize it as hierarchical data. We can do this a couple of different ways. For example,

Rental Car Theme

• As a traveler, I can rent a car
• As a traveler, I can get collision insurance on a car rental policy.
• As a traveler, I can request a baby car seat be inside my car.

Airplane Theme

• As a traveler, I can request an aisle seat.
• As a frequent flyer, I can request an upgrade to first class.

Or we could create a product backlog as a tree with levels for

• Team
• Product Backlog Items Being Worked On By That Team

The following figure shows a product backlog as a treemap. There are five themes in the treemap. (Theme 4 is in the top left; Theme 1 is in the bottom right. You can click on it to enlarge it but be sure t come back.) Each theme is made up of a number of individual user stories. Story 4-28 (indicating the 28th story of theme 4) is in the top left. I’ve used this “theme-dash-story” notation for simplicity only for this example. I wouldn’t do that on a real project, of course.

The size of each story in the preceding figure represents the number of story points that the story was assigned when estimated. Colors can be used on a treemap to represent an additional attribute of the data. On agile projects I’ve used colors to represent whether a user story has been developed or not. One color coding we used was:

1. Done
2. Started but not done
3. Not started but not planned to have been started yet
4. Not started but it should have been started by now
5. Blocked

I’ve also used color to indicate which team would work on a user story / product backlog item or whether the item was ready for development or not. Treemaps are very flexible in this regard. Check out the link to the stock market as a treemap at the end to see a great example of using color.

A good treemap is interactive–you can mouse over, click on regions of it, and so on. For example, in the treemap above you’ll notice that some of the user stories of Theme 4 are so small you can’t read them. Clicking on a part of Theme 4 should zoom the treemap in so it displays only Theme 4, meaning more room is available and more detail can be shown as below:

Sometimes zooming isn’t enough. A good treemap implementation will also display additional detail when mousing over part of it as shown in this example:

Treemaps are an excellent way of visualizing large product backlogs. To date, I’ve made use of various open source or relatively inexpensive general treemapping tools to create these on my projects or for my clients. Hopefully now some of the leading agile tool vendors begin to incorporate this type of visualization technique into their products. I would love to be able to visualize a large product backlog and interact with directly from within some of those tools. Until then, check out some of the links below for useful ways to create treemaps. The ones in this posting were created with IBM’s free Many Eyes tool.

Useful links:

1. The stock market as a treemap
2. IBM’s Free Many Eyes tool
3. A good interactive example of a treemap
4. Read news headlines as a treemap
5. A simple JavaScript implementation you could add to your project homepage

At a recent conference, someone asked me why. Because I get that question fairly often I want to give three reasons why here:

Reason 1
Something significant and I’m tempted to say magical happens when requirements are put in the first person. Obviously by saying “As a such-and-such, I want….” you can see how the person’s mind goes instantly to imagining he or she is a such-and-such. As for the magic, Paul McCartney was interviewed and asked about why the Beatles songs were so amazingly popular. One of his responses was that their songs were among the first to use a lot of pronouns. Think about it: She Loves You, I Wanna Hold Your Hand, I Saw Her Standing There, I Am The Walrus, Baby You Can Drive My Car, etc. His point was that these helped people more closely identify with the songs. I tried briefly to find a source for this interview tonight and the closest I found was this. The information in that reference fits my recollection of hearing McCartney says this during a radio interview in 1973 or 74 that I assume was recorded when the Beatles were together.

Reason Two
Having a structure to the stories actually helps the product owner prioritize. If the product backlog is a jumble of things like Fix exception handing, Let users make reservations, Users want to see photos, Show room size options, and so on, the product owner has to work harder to understand what the feature is, who benefits from it, and what the value of it is.

Reason Three
I’ve heard an argument that writing stories with this template actually suppresses the information content of the story because there is so much boilerplate in the text. If you find that true then correct it in how you present the story. I’ve seen backlogs in Word that present the boilerplate in grayed text with the unique parts in black. I’ve created product backlogs in Excel that use column headings to filter out the common text. Look and you’ll see what I mean:

Seriously take a look at the image above before continuing….I’ll wait……Please, really look at it before reading further…..

OK, here’s how I bet you read the spreadsheet. I bet that as you read most of the rows you added in the “As a…”, “I want….” and “so that….” text, possibly even by looking at the heading as you read across each row. I’ve experimented by asking people and this is what most people do. If that text is unnecessary, why do we mentally mouth the words to ourselves or even glance at the heading while reading the row? Perhaps it’s not so non-essential after all.

For now, and probably a long time to come, I’ll be sticking with the “As a <type of user>, I want <some goal> so that <some reason>” template for these and other reasons.