Mike Cohn’s Blog - Succeeding With Agile™

In my user stories book and in all my training and conference sessions on user stories I advocate writing user stories in the form of “As a <type of user>, I want <some goal> so that <some reason>.” While I consider the so-that clause optional, I really like this template.

At a recent conference, someone asked me why. Because I get that question fairly often I want to give three reasons why here:

Reason 1
Something significant and I’m tempted to say magical happens when requirements are put in the first person. Obviously by saying “As a such-and-such, I want….” you can see how the person’s mind goes instantly to imagining he or she is a such-and-such. As for the magic, Paul McCartney was interviewed and asked about why the Beatles songs were so amazingly popular. One of his responses was that their songs were among the first to use a lot of pronouns. Think about it: She Loves You, I Wanna Hold Your Hand, I Saw Her Standing There, I Am The Walrus, Baby You Can Drive My Car, etc. His point was that these helped people more closely identify with the songs. I tried briefly to find a source for this interview tonight and the closest I found was this. The information in that reference fits my recollection of hearing McCartney says this during a radio interview in 1973 or 74 that I assume was recording when the Beatles were together.

Reason Two
Having a structure to the stories actually helps the product owner prioritize. If the product backlog is a jumble of things like Fix exception handing, Let users make reservations, Users want to see photos, Show room size options, and so on, the product owner has to work harder to understand what the feature is, who benefits from it, and what the value of it is.

Reason Three
I’ve heard an argument that writing stories with this template actually suppresses the information content of the story because there is so much boilerplate in the text. If you find that true then correct it in how you present the story. I’ve seen backlogs in Word that present the boilerplate in grayed text with the unique parts in black. I’ve created product backlogs in Excel that use column headings to filter out the common text. Look and you’ll see what I mean:

Seriously take a look at the image above before continuing….I’ll wait……Please, really look at it before reading further…..

OK, here’s how I bet you read the spreadsheet. I bet that as you read most of the rows you added in the “As a…”, “I want….” and “so that….” text, possibly even by looking at the heading as you read across each row. I’ve experimented by asking people and this is what most people do. If that text is unnecessary, why do we mentally mouth the words to ourselves or even glance at the heading while reading the row? Perhaps it’s not so non-essential after all.

For now, and probably a long time to come, I’ll be sticking with the “As a <type of user>, I want <some goal> so that <some reason>” template for these and other reasons.

In the discussion that ensued from another post here, Brian Bailey asked a great set of questions. The questions were big enough that I’ve moved them here. I’m going to intersperse Brian’s questions and comments with my responses. Brian started with:

If the product owner is onsite and part of the same company, what role should she have in prioritizing tasks within a sprint? To clarify, I assume that a 30-day sprint does not mean that new features that are completed are held until the end of the sprint. If a feature is finished and ready to move to production after the first week, it can be released, correct?

If that’s the case, should a PO be allowed to prioritize the stories and tasks the team committed to (”Please start with this critical item”) or should the team have full control over the order things are worked on?

First, the concept of prioritizing tasks within a sprint is one that doesn’t make sense. When a team plans a sprint they make a commitment to complete the user stories they select from the product backlog. It’s not a come-hell-or-high-water commitment, but the team is expected to do its best to complete the work they select. And averaging things out over the course of many sprints, the team should generally meet their commitment.

The idea of prioritizing items within a commitment doesn’t make sense. Back in 1988 I got married and made a commitment to my wife, Laura. I promised to love, honor, and cherish her. (We mutually agreed to drop “obey.”) I didn’t promise to love her all the time, honor her if there was more time and then cherish her as a stretch goal.

But what about Brian’s statement that, “I assume that a 30-day sprint does not mean that new features that are completed are held until the end of the sprint.” Well, yes and no. This depends upon what was agreed to during sprint planning. The default assumption should be that the team has the right to rip the system apart on day one of the sprint and does not have to have it put back together until the last day. (For argument’s sake, I’m ignoring that this would be a bad idea for many reasons.) If the product owner needs an interim deliverable during the sprint, that should be discussed and planned during the sprint planning meeting. There will almost certainly be some overhead in doing this and the team needs to account for that. Even better, though, I’d suggest trying a shorter sprint length if this is common.

Back to Brian:

And of course, the dangerous follow-up that almost seems inevitable - could the PO then decide or shift priorities (not adding or replacing tasks, though) throughout the sprint?

I’m confident this is a bad idea and could lead to micro-management instead of team-driven development. But I can also see how the PO might have a good reason for something to be moved to the front of the line.

You’re correct. This is a bad idea. One of the challenges with 30-day sprints is that it is a long time to make the business go without changing its collective mind. If this questions are legitimate concerns then a shorter sprint length is a likely solution.

I was recently emailed a question asking whether the sprint planning meeting should start with time allocated for putting story point estimates on any user stories that have not yet been estimated.

No, I don’t think this is a good idea. Keep in mind that we put estimates on product backlog items (which I recommend be user stories) so that:

• the product backlog can be prioritized. It is impossible to fully prioritize a set of items without knowing at least their relative cost.
• we can make high-level forecasts about how much will be done by when
• we can make tradeoff decisions between scope and schedule

We can achieve these goals with approximate, relative estimates such as given by story points. For example, if I decide to buy a new car this weekend it is sufficient for me to know that I can get a Toyota or Honda for around $30,000 and that a Ferrari goes for closer to $800,000. I do not need to know a more precise cost of the Honda ($31,850) before knowing it should be on my short list of cars to evaluate while the Ferrari should not.

Sprint planning meetings typically go into deeper detail than is appropriate for product backlog item estimating (whether in story points or ideal days). Since we become accustomed during sprint planning to breaking user stories into tasks and considering those tasks in more detail, there is a chance this will carry over into any story point estimating done during the same meeting.

So, when should story point estimating happen? I’ll describe the ideal case, which you can easily adjust for the real-world intrusions on your project. Projects typically start in either of two ways:

1. A reasonably fully stocked product backlog is written before the first sprint begins and all items are estimated before the first sprint planning meeting. [If you do this, be careful not to write all user stories with too much detail. Each product backlog item you write represents an investment. User stories should therefore be elaborated just-in-time and in just-enough detail that they can be turned into functionality in one sprint.]
2. We know we’ve got to do the project so we dive in. During the first sprint or two, all the user stories are written and estimated just like above.

On an ongoing basis, once per sprint I recommend that the ScrumMaster tell the team something like, “Hey, we’ve had five new user stories come in this sprint and we need to estimate them. Everyone plan on hanging around for a bit after tomorrow’s daily scrum meeting and we’ll play Planning Poker to estimate the new items.” Doing it right after the daily scrum helps cut down on the number of interruptions in total. I usually aim for having that meeting about two days before the end of the sprint. That way the product owner will have estimates on them so she can prioritize prior to the start of the sprint.

I was asked recently how to go about writing user stories for a back-end financial system. This is an interesting example and is a question I get asked a lot, so I thought I should answer it here. This example brings up a couple of key interesting challenges:

• While there may be users of the system they are not often not direct users (i.e., with hands on the keyboard waiting for something to happen)
• The functionality is often larger than will fit in one iteration

So that we can deal with these challenges, let’s make up some more context for the example. Suppose our financial system takes in a lot of daily data and produces flat files that will be sent to banks and other partners at the end of the day. Some of the files are simple formats. Other files are in more involved formats with multiple record types within the file and possibly with multiple lines for some of the transactions. This would be a fairly standard batch-processing type application.

I want to write user stories in the template I recommend in the book: As a , I want so that . So, let’s deal with these challenges in order and first try to figure out who the user will be in our stories? Given the context provided above the user, is probably a bank or business partner. This will let us write stories like “As a bank, I want…” It’s entirely possible that we will want to get more specific and sometimes write stories for more specific users:

• As a commercial bank, I want….
• As a savings & loan, I want…
• As the Bank of America, I want… (assuming we have some specific business partnership that provides BofA with unique functionality)

Now, what is it these banks want?

At a high level we can start out with some epics, perhaps like these:

• As a bank, I want to receive a file showing all checks to be cleared so that I can debit and credit the right accounts.
• As a bank, I want to receive a correctly formatted 5300 file so that I can adjust balances as appropriate.
• As a bank, I want to receive a file showing the status of all resubmitted transactions so that I can update the database with new information.

(Note that I am liberally making up specific business rules or details. Presumably someone writing the real user stories could add real and relevant detail instead.)

First, notice it is OK to humanize the bank with “As a bank, I want…” Programmers do this all the time in conversation. “OK, suppose I’m the bank and you send me a 5300 file with a bad record. In that case, I’ll…”

As I said at the start of this post, one of the challenges in an application like this is that some of these user stories we just made up could be large. Suppose our mythical file format 5300 includes a header, three different record types, and a footer. We could break up the 5300 file user story above into things like:

• As a bank, I want to receive a 5300 file with the correct header and footer so that I can process the file correctly.
• As a bank, I want to receive a 5300 file with correctly formatted prearranged payment records so that I can process those correctly.
• As a bank, I want to receive a 5300 file with correctly formatted deposit entry records so that I can process them.
• As a bank, I want to receive a 5300 file with correctly formatted addendum records so that I can process them.

If those were still too big, I would find ways to split them smaller based on the specific formatting of the records. Suppose our deposit entry record type sometimes fits on one line but can also span multiple lines. In that case I could write these user stories:

• As a bank, I want to receive a 5300 file with correctly formatted single-line deposit entry records so that I can process them.
• As a bank, I want to receive a 5300 file with correctly formatted deposit entry records that include continuation lines so that I can process them.

Is the system “potentially shippable” after each of these user stories is delivered? Yes, it kind of is. While I’ll admit that we’re unlikely to find a bank that will take a 5300 file with only a header and a footer, it’s possible. Especially if we think of the situation in which the bank is simultaneously writing their own software to process the 5300 files we send them. Imagine meeting with the bank before the start of the next iteration and perhaps agreeing, “OK, in this iteration we’ll create some blank 5300 files with just headers and footers. But that will let us make sure that the communication problems are worked out, that you can read the files we write, and so on….” These stories also work well because they will likely fit with how the programmers will want to build the system.

Nothing very profound in this comment but something I wanted to post because I’ve been thinking about it all day…

I started reading a new book called Made to Stick by Chip and Dan Heath. It’s about why some ideas are “sticky” and spread while other ideas die out. They start with the classic urban legend of the guy who is drugged and wakes up in a bathtub full of ice and discovers his kidney has been stolen. That’s a sticky idea. Many of us have heard it, and we remember it after hearing it. Sticky ideas and stories aren’t necessarily bad; they are just things that change our attention at just the right time and that are so intriguing/compelling/interesting that they spread like wildfire.

This got me thinking about sticky stories from my past and how perhaps stickiness has changed given the rapid communication of today, including and especially the Internet. One real story from my life is from back in January 1981. I was in college and living in the dorm. Ronald Reagan was newly sworn-in as President. A common joke at the time was that Reagan would start a nuclear war and blow up the whole world. The previous summer the US had boycotted the Olympics because of the Russian invasion of Afghanistan. It was a bit of frightening time.

That’s why I remember how scary it was one Sunday morning when our whole dorm woke to rumors that the US and the USSR had declared war on one another. We couldn’t confirm this, though. Not a single person in our dorm had a TV. None of the radio stations were saying anything about it. This was before Al Gore had invented the Internet. This rumor of US-USSR war was *sticky.* It spread around the whole dorm building within minutes. People were waking others, pounding on their doors saying, “Wake up, we’ve declared war on Russia!” Still, we couldn’t get any real confirmation of this.

I finally called my parents who lived two timezones away and were still asleep. I sheepishly asked, “Mom, did we declare war on Russia this morning?” She thought I was crazy so I explained. While I did that she turned on her television and confirmed nothing was being reported. That was good enough for us. All of us college students went back to sleep.

Two thoughts strike me form this: First, Communication has changed that much in 27 years. It was possible back then to be unable to verify something like a war. If that happened today I’d browse the web on my phone and know in seconds. Second, that was a very first-hand example of how quickly a sticky idea could spread.

It seems that agile might be a sticky idea.

I recently read an article about how fire ants have evolved such that some ant colonies now have two queens. This has helped fire ants spread and more densely populate certain regions of the United States. If a queen who insists on being the sole monarch is born into a densely populated area, she is likely out of luck. But, a queen fire ant with the genetic adaptation that allows her to share the colony will thrive in a densely ant-populated area.

This got me to wondering whether product owners evolve. The answer is, yes, of course they do.

Just as the fire ants have evolved in ways that allow them to better survive in their environments, I learned recently that product owners do the same. I was speaking with someone who works at one of my clients that I hadn’t visited for a few months. He told me that his product owners have suddenly stopped emphasizing high quality work and bug fixes.

They remain nominally agile–they work in short iterations, have ScrumMasters, pair program, have thousands of automated tests, dabble in test-driven development, and so on. But, product owners no longer treat quality as important.

They now push teams teams to go faster, faster, faster even if that means leaving bugs and sloppy code behind. It took my friend at this client a bit of thinking to figure out what was going on. But what he discovered was frightening.

One of the product owners came completely clean and explained it to him: In their organization there are many projects that compete for resources. When a project is funded it is approved for a certain number of sprints with a certain team or teams. Once those sprints are over the project can request more funding, but it’s likely that another project has already been lined up behind this one.

Being a very customer-foucsed company, this organization has instituted a policy common to many organizations: “Customer-reported quality issues are to fixed immediately.”

Pause and reflect on that for a moment. Suppose you are  a product owner. Your project is approved for a fixed number of sprints. You’d have really preferred more time for your project, but you understand why you only got what you were given. But, if there are quality issues found after your product goes live they will be treated as high priority items and fixed immediately. So, as a sneaky, devious product owner the best way to get more time for your product is to cut quality, cram in half working features and then get them fixed “for free” after you’re out of sprints.

What my friend had found is that his product owners had evolved in adaptation to their environment, just as the fire ants had evolved in adaptation to theirs. His unfortunate situation won’t change until some other environmental factor comes along that changes the motivation and incentives of the product owners.

Using story points or ideal days to measure productivity is a bad idea because it will lead the team to gradually inflate the meaning of a point–when trying to decide between calling something “two points” or “three points” it is clear they will round up if they are being evaluated on productivity as measured by the number of story points (or ideal days) finished per iteration.

My view is that points can be used as the best way to estimate and assess progress that we’ve ever had or they can be used as another weapon with which to hit the team. There are plenty of weapons with which you can hit your team. We don’t need to ruin points by using them that way as well.

Some teams have measured productivity with things like the number of backlog items delivered or the % of backlog items completed vs. planned into a sprint. Teams will alter their behavior on those as well though so they can be gamed and misleading. These metrics can be useful but only as part of a suite of metrics collected at the end of each iteration.

If we rethink the question of “how do we measure productivity” we might get a better answer. Suppose you own a sandwich shop and want to measure the productivity of the sandwich maker in the back. He responds to our metric by making as many sandwiches as he can–regardless of whether anyone ordered them! At the end of the day there will be 200 extra sandwiches to throw away. A better measure of him might be how quickly he makes any sandwich. So we’d measure the time from when the customer placed the order until the sandwich is put on a tray. Or for a more complete metric we may want to measure the time from when he receives an order until he is ready to receive the next order as this captures any cleanup or restart time.

So, one measure we may want to include in our suite of metrics could be the responsiveness of the development organization. This would be measured in the same way as in the sandwich shop. Datestamp each product backlog item and track the time from when something enters the product backlog until it either (a) comes out of an iteration or (b) is delivered into the hands of customers. Choosing between (a) and (b) will largely be a matter of how often you ship software. Option (b) is a better measure of rapid delivery of customer value but is impractical in some cases. It would be a bit of a useless measure for the Microsoft Vista team, for example.

As described in Agile Estimating and Planning, I’m a huge fan of using story points for estimating the product backlog. However, I also recommend estimating the sprint backlog in hours rather than in points. Why this seeming contradiction?

I’ve previously blogged on the reasons why I recommend using different estimation units (points and hours) for the different backlogs. But I’m often asked this related question I want to address here:

I’m curious why you aren’t using story points to do your sprint planning.  I thought that the point of measuring story point velocity was partly to determine how much we can take on (or commit to) in a sprint.  Do you only use story points for longer-term planning (e.g. release planning)?

I don’t use story points for sprint planning because story points are a useful long-term measure. They are not useful in the short-term. It would be appropriate for a team to say “We have an average velocity of 20 story points and we have 6 sprints left; therefore we will finish about 120 points in those six sprints.” It would be inappropriate for a team to say, “We have an average velocity of 20 story points so we will finish in the next sprint.” It doesn’t work that way.

Suppose a basketball team is in the middle of their season. They’ve scored an average of 98 points per game through the 41 games thus far. It would be appropriate for them to say “We will probably average 98 points per game the rest of the season.” But they should not say before any one game, “Our average is 98 therefore we will score 98 tonight.”

This is why I say velocity is a useful long-term predictor but is not a useful short-term predictor.

Velocity will bounce around from sprint to sprint. That’s why I want teams to plan their sprints by looking at the product backlog, selecting the one most important thing they could do, breaking that product backlog item / user story into tasks and estimating the tasks, asking themselves if they can commit to delivering the product backlog item, and then repeating until they are full. No discussion of story points. No discussion of velocity. It’s just about commitment and we decide how much we can commit to by breaking product backlog items into tasks and estimating each. This is called commitment-driven sprint planning.

When a team finishes planning a sprint in this way it is indeed likely that the number of story points they have unknowingly committed to should be close to their long-term average but it will vary some. It will also be true that a team will commit to approximately the same number of hours from one sprint to the next. I use the term capacity to refer to this number of hours because velocity is reserved for referring to measuring the amount of work planned or completed as given in the units used to estimate the product backlog (which I recommend be done using story points).

I like to use Planning Poker to estimate the user stories on an agile team’s product backlog. In this approach individual estimators hold up cards showing their estimates. If estimators disagree they discuss why, ask questions of their product owner (who should be present), and repeat until they come to consensus. Team members often ask me whether they really need to come to consensus or  whether they can just take the mean of the individual estimates.
The problem with averaging is that it is too easy–rather than have the fierce discussion that is one of the huge benefits of playing Planning Poker teams fall into a trap of playing one or two rounds and then just averaging. An obvious dysfunction is that one estimator may play the 100 card not because he thinks it will take that long but because he thinks 20 is the right number and other estimators are thinking 8 and 13. For this reason and others, if a team truly feels compelled to average, they should take the median (middle value) rather than the mean (sum of estimates divided by number of estimates).

A lot of dark corners are enlightened through the discussion; teams lose out on that when they average. So while I want teams to come to agreement, I don’t care how heartfelt the agreement is. If we agree on 13 some of us may really believe that’s the right number. Others may think 8 is right but that 13 is “close enough.” Still others may think we’ve discussed the item too long and even though it should be a 20 will give in and call it a 13 just to be done with it.

So, rather than average if the team is an impasse I suggest going another round. If still stuck, someone should suggest a reasonable number and see if everyone can “support it” rather than “think it’s the absolutely perfect number.”

Should a team that is estimating in story points ever re-estimate? This is a question I’m commonly asked and would like to address here.

Most people have a natural feeling that re-estimating is somehow wrong but they can’t quite say way. I’ll encourage those individuals to stick to their hunches, and hopefully I can provide of the reasoning that supports your natural inclination that most re-estimating is wrong. Philosophers talk about two types of knowledge. The first is a priori knowledge, which is knowledge before you experience something. Let’s call this knowledge-before-the-fact. This is the type of knowledge we have when we estimate something. Before I estimating development of the new search screen I think it’s about 8 story points, because it seems to be about the same total effort as some other 8 point story. The other type of knowledge is called a posteriori knowledge by the philosophers. This is knowledge after the fact.

When we estimate it is important that we not mix knowledge-before-the-fact with knowledge-after-the-fact. Suppose you are looking at a Scrum product backlog that has just been estimated with none of the work started. Each of those estimates was given before-the-fact (a priori). Now suppose you are looking at the same project a few months later. You’ve got a list of completed work, some of the items on that list still show their original, before-the-fact estimates but some have been re-estimated with after-the-fact estimates. The product backlog is similarly mixed: mostly the initial, before-the-fact estimates but some estimates that have been revised after-the-fact because of what was learned by developing previous user stories off the backlog.

Having both before-the-fact and after-the-fact estimates on your product backlog and list of finished work can cause a lot of confusion for the project. When all estimates are given in before-the-fact numbers we can reason about them and compare them. Suppose the team is estimating a new item and want to say its equivalent to 20 story points because it’s similar to another item that has been estimated at 20 story points. That logic makes sense if the original item has not been re-estimated. If the old item was given an estimate of 10 before the fact and re-estimated to 20 after the fact then it is harder to know if the new item should get a 10 or a 20. With the re-estimation having occurred we’re in the position of saying “Before I start this one I think it’s a 20 because the other one felt like a 20 after I did it.” That’s weaker than “Before I do either of these they seem the same size.”

So, does this mean you should never re-estimated? Absolutely not. There are times when you want to re-estimated. Generally re-estimating is useful when you completely blew it on the original estimate and can see that the mistake was a rare occurrence. (That is, if every estimate is systematically off by half I wouldn’t re-estimate.) Second, you should re-estimate when there has been a change in relative size. For example, the team has discovered that learning AJAX will be about half as hard as they thought. We’d want to fix that because the new knowledge tells us that our relative estimates are off-kilter for the AJAX-heavy stories.