Resampling is based on the idea that things we’ll observe in the future will be similar to the things we’ve observed in the past. In the examples below we’re saying the velocities a project team will see in the future will be similar to ones that occurred in the past. Resampling works by imagining we’ve put all old sample velocities into a bag. If we have past velocities of 18, 17, 18, 19, 22, and 20 imagine each of those written on separate slips of paper and dropped into a bag. Note we’ll have two slips of paper with “18? because our Agile or Scrum team here had that velocity twice.

To predict future velocity we reach into the bag and pull out one piece of paper. What’s written on it is our prediction of velocity in the first sprint. To predict the team’s velocity in the second sprint, we reach into the bag and pull out another slip of paper. But, before we do that, it’s important we put the first slip of paper back into the bag. This is called “resampling with replacement.” We want to replace because for any given sprint the team is equally likely to get any of their past velocities.

Suppose we’re trying to predict how much work a given team can complete in a coming ten-sprint period. We would resample (with replacement) ten times. Each time we’d note the number we pulled from the bag. After we pull the tenth slip, we would sum the ten values we pulled and that is one possible result for this team over the coming ten-sprint period.

It’s quite possible we could get lucky and pull the highest-valued slip (22) each of ten times. Or we could pull the worst-case slip, 17, each of the ten times. But these are unlikely. If we could in some way in the real world run the project hundreds or thousands of times, we would occasionally see the team repeat their highest (or lowest) velocity every sprint but it wouldn’t happen very often.

Since we can’t run the project hundreds or thousands of times in the real world, we want to simulate doing so on a computer. We can then learn a great deal from the results. For example, suppose we are ready to start a project that will have ten sprints. It would be helpful to know things like:

• what is the average amount of work completed over those ten sprints?
• what percentage of the time does the team finish more than say 200 points of work?

It’s actually quite straightforward to answer these questions using simulation and resampling. Let’s see how it’s done. You can follow along with this velocity resampling spreadsheet. In the figure below (from that spreadsheet), cells B3 through B28 show historical velocity.

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
Because our hypothetical project here involves ten sprints, cells D3 through E12 show the sprint numbers (1-10) and a resampled velocity for each.

Selecting a resampled velocity is simply a matter of randomly selecting any velocity from the B3:B28 range (our 26 historical velocities). This is done with the formula:
=SMALL($B$3:$B$28,INT(COUNT($B$3:$B$28)*RAND())+1)
which generates a random number between 1 and 26 and then uses the SMALL method to select that item from the list of values. (Note: SMALL selects the nth smallest item from the list of values; we could have used LARGE instead. The goal is just to randomly select a value from the list of velocity in B3:B28.)

Because we’re using the RAND() function in E3:E12, any time you change any cell in the spreadsheet, the values in E3:E12 change. This is a desirable side effect of using RAND().

E3:E12 simulates one run of ten sprints. We’d like, though, to simulate 100, 200 or even 1000 runs of the project (each with ten sprints in it). To do this is slightly tricky because we’re going to use something a lot of people aren’t familiar with in Excel: a Data Table. In our spreadsheet, the Data Table is in G3 through H202, a portion of which is shown below.

The G column shows a sprint number, the H column shows a sum of ten velocities, representing one ten-sprint project in our case. In the example, in the figure at left you can see that the first simulation of the project yielded a total of 230 points done in the ten sprints of the project. In the next row (cell H4, but labeled ’2′ in column G of the spreadsheet), the team got a much higher simulated velocity, 264. In the spreadsheet, I repeated this 200 times but you can do more or less as you prefer.

For brief instructions on how to create a data table, see the end of this post. For more complete instructions, see the Excel documentation.

Armed with 200 simulations of the ten sprints of the project (or ideally even more), we can now answer the question we started with, which is, How much can this team finish in ten sprints? Cells E17 and E18 of the spreadsheet show the average total work finished from the 200 simulations and the standard deviation around that work.

In this case the resampled average is 240 points (in ten sprints) with a standard deviation of 12. This means our single best guess (50/50) of how much the team can complete is 240 points. Knowing that 95% of the time the value will be within two standard deviations we know that there is a 95% chance of finishing between 240 +/- (2*12), which is 216 to 264 points.

If my boss wants a guarantee, I might say, “We can pretty much guarantee 216.” Technically I know the math doesn’t support the guarantee. There’s about a 2.5% chance we fall below that. However, humans are involved and just about any good team I’ve been on would be happy to kick in some extra effort sometime over the ten sprints to finish the 216 we committed to rather finishing with 210 if that 2.5% chance does occur.

Another interesting issue we can address with this type of simulation is the boss who says, “I need you to get 250 points done in the next 10 sprints.” You can see how likely this is to occur by scanning down the resampled values (the H column) and seeing how often the value there equals or exceeds the value the boss, client or customer wants. The spreadsheet is set to do this automatically as shown in this figure:

Type in the total number of points desired in L20, the spreadsheet will see how many times that number or higher occurred in the simulations (L21) and then report it as a percentage (L22). In this example, if the boss wants 250 points in 10 sprints, we can reply that while the team will try to achieve that, historical data shows that there is only a 20% chance of that occurring.

Hopefully you’ve found these examples of working with resampling to simulate projects helpful. There are many more things that can be done using this technique. I’ll provide additional examples in future posts.

You can download the velocity resampling spreadsheet used in this examples.

Note on Creating a Table Table

To create a Data Table, in cell H2 (the cell above where you want to put the simulation results), enter the formula you want to calculate as a result of each simulation. Because we’re interested in the sum of their ten sprints worth of velocity, I’ve entered:
=SUM($E$3:$E$12)

Next, if you’re interested fill the sprint cells (the G column) with numbers from 1 to 200. Then highlight cells G2:through 202 (assuming you want to do 200 simulations as I’ve done here). Notice you’re highlighting starting one row above our simulations through the last row of simulations. Now create the data table. In my version of Excel (Mac 2011), I do this by selecting Data | What If | Data Tables. You’ll get a little dialog box asking for the row or column input cell. Move the cursor to Column Input cell and select G2. Close the dialog and you’ll see the 200 rows fill with simulation results.

In these situations a common question is whether the team should assign some number of story points to the work of fixing these legacy bugs.

If the team does not assign a story point value to this work, velocity will show the amount of “forward progress” the team is making each sprint. This has the advantage of making visible that the team is going more slowly through the work than it could if these legacy bugs had been fixed when originally found.

On the other hand, if the team does assign points to the bug-fixing effort, velocity comes to represent the team’s true capacity to accomplish work.

My usual recommendation is to assign points to the bug fixing. This really achieves the best of both worlds. We are able to see how much work the team is really able to accomplish but also able to look at the historical data and see how much went into the bug-fixing story each sprint. Knowing this can be helpful to a team and its product owner. For example, imagine a situation in which the product owner is considering suspending bug fixing for the next six sprints in order to add a valuable different feature into a release. If we know that the team’s full historical average velocity was 25 but that 5 points went to bug fixing each sprint, we know that suspending bug fixing for the next six sprints will result in 30 (6*5) more points of new functionality.

First, let’s clarify what people mean when they refer to a “release backlog.” A release backlog is a subset of the product backlog that is planned to be delivered in the coming release, typically a three- to six-month horizon. A release backlog would presumably contain the same type of items as on a product backlog. My preference for this, of course, would be user stories. So, at the start of some release planning horizon, a product owner with help from the team and other stakeholders would example the product backlog, select some amount of high priority work and move those items to the release backlog. Later, in sprint (iteration) planning, the team would examine the release product and select some amount of work to do. (Notice how this already goes against existing agile literature that says the team selects top priority items for the sprint from the product backlog?)

I am opposed to the use of a release backlog for a couple of reasons:

First, in all projects that are not being done as fixed-scope contracts (and, arguably, even then) we don’t know exactly what user stories or features will be delivered in a release. To say that there is a release backlog may not imply a guarantee that all features on it are delivered. However, it does create additional work as items are moved off back and forth between the product and release backlogs. This will happen as the team’s velocity changes (even by small amounts) from sprint to sprint. If the release backlog is to show what a team will deliver in a release, it should contain an amount of work equal to the number of story points (or ideal days) times the number of remaining sprints. If you have an average velocity of 20 and 6 sprints left, the release backlog should contain 120 points worth of work. Suppose the team finishes 24 points of work in a sprint. The backlog is now down to 96 points (120-24) but should contain 100 (5 sprints left x an average velocity of 20). Things get even more difficult if that good velocity of 24 increases the team’s average velocity to 21. In that case the release backlog should contain 5×21=105 points of work; we need to move 9 points of work from the product backlog to the sprint backlog. If there’s a release backlog, this moving of items back and forth from product backlog to release backlog will happen every sprint.

Second, introducing a release backlog creates additional confusion. We’ve already overloaded the word “backlog” with “product backlog” and “sprint backlog.” Why confuse things further with a third type of backlog unless there is an immensely compelling reason to do so?

Third, introducing the concept of a Release Backlog actually makes it harder for product owners to do one of the most important things they should be doing–looking at the features or user stories that just miss the cut of being in a release. When I look at a product backlog and count down the number of story points that I anticipate will be in the release, the most interesting thing to me is not necessarily the set of user stories that will make it into the release. I’m often just as interested in the next few user stories below that cut-off point. That is, I want to know what user stories won’t quite make it into the release. After all, we say that one of the benefits of agile (on some, not all, projects) is the ability to decide later if we should release a few sprints early (if a lot has been done), on the planned date, or perhaps a sprint or two late if we want to add more functionality before a release. This is made more difficult with a Release Backlog because a product owner now needs to examine both a Product Backlog and a Release Backlog to make these decisions.

So, what do I propose instead?

I advocate that all teams track their velocities. This leads to a graph like this:

This graph shows a team calculating three average velocities:

1. Long term average, defined as the mean of the last eight sprints
2. Worst case average, defined as the mean of the worst three chosen among the last eight sprints
3. Best case average, defined as the mean of the best three chosen among the last eight sprints

You can choose your own ways of calculating long-term, best-case, and worst-case average velocities. These are just the ones I like. We use these average velocities as shown in this figure:

In this figure, the product backlog is shown on the left as a stack of index cards. We use the three average velocities multiplied by the number of remaining sprints to draw arrows pointing into the product backlog. The range created by these arrows indicates the likely amount of work finished by the planned date. Our release plan is the work defined by the arrows. Rather than a Release Backlog as a new work product or artifact, my equivalent of a release backlog is the product backlog and these arrows pointing into it.

As you can imagine, predicting velocity as a range is much more likely to be accurate than using a single point estimate (“Our velocity is 31.”) as has been implied by everyone I’ve heard talk about a Release Backlog. Additionally, looking at figures like these should make it apparent that the amount of work expected to be delivered in a release will change from sprint to sprint. Pointing arrows into a product backlog is much easier and more helpful than creating a new work product, the Release Backlog.

But what do we do when we are managing an agile project where team membership or size changes frequently?

To answer this question most effectively, you should collect data on how teams of different sizes have performed over time in your organization. When I was a VP of Development at a couple of agile organizations, I used to collect data on velocity and team size changes in a simple spreadsheet similar to this:

The first column represented the size of the team before a change occurred. The next column represented the new size of the team (up or down). The third column was what I considered to be a reasonable “long term average” velocity for the team at its initial size. Because team’s could change frequently (by a person or two) I settled on using the median value of the team’s last five iterations. The tradeoff in using a longer measure (median of 15 iterations perhaps) is that you’d have fewer observations. The next columns represent the actual velocities of teams over the next three iterations. Notice that for the last team values are not shown for the last two iterations. This is usually because the team size changed again. If you have a significant number of teams, the rows in this type of spreadsheet will accumulate quite quickly.

In some of the organizations where I used this approach we did not have a standardized definition of story point (or we’re using ideal days, which were not as normalized as you might think). So all analysis was done on a percentage basis. What I wanted to know was “What is the average impact of adding a person to a seven-person team?” I would have loved the answer to be something like “Velocity goes up 15%.” Unfortunately, it wasn’t that straightforward because velocity often dipped for a couple of iterations before going up. By tracking data I found that usually by the third iteration a team had settled in on a new velocity, which is why my spreadsheet above only tracks through Iteration+3. By all means, track more and see what you find. (But keep in mind that the data will get sparse as team sizes will change again.)

Another tab in my spreadsheet, expressed all the data in percentage terms. The first two rows

(Example: Iteration +1 in the first row is -20% based on the team dropping its velocity from 25 to 20.)

I then simply averaged these percentages for each team size change to get results like:

This allowed me to answer all sorts of questions, including:

• What will this team’s velocity be if we add two people?
• How soon could we get this project if we added a person to each team?
• If I want all those projects done by the end of the year, how many people would we need to add?
• What would be the impact of not approving the new employees in the budget?
• What would be the impact of a 15% layoff?

There are of course many flaws with this approach. Adding Susan to the project is very different than “adding an unknown person” to the project. Still, if I have the data on averages across the board in our organization I can make assumptions about specifically adding Susan (if I want, there can be many more risks in doing this). Notice that the approach does not attempt to take into consideration who it was that was added or even what skillset the person had. You could collect such data if you wanted. As anal about collecting all sorts of data like this as I am when I have access to it, I knew though that collecting that type of data would have made this just hard enough that I wouldn’t have done it regularly.

I did collect a few other bits of data that I left out of the initial table (so as to have more horizontal room for the data of real interest). For example, I collected data such as iteration length and the name of the team’s ScrumMaster (the latter was in case I had questions a few weeks later).

The approach described here was just simple enough that I could get empirical evidence of the impact of team size changes. This was invaluable when discussing headcount changes with product owners and the CEO.

The need for a common baseline to story points usually arises from the reasonable desire to know how big the entire project is. To know that, we must know the size of the work to be done by each team. Unfortunately, along with this goal comes the ability to compare teams based on their velocities. Since many managers are constantly looking for ways to compare team and individual performance it is not surprising that they begin to make such velocity comparisons. Almost all such comparisons are disruptive to performance of the combined, overall group or department.

A chart such as the one that follows can show a lot of interesting information.

However, this chart can be very dangerous because of how teams will assume the data is being interpreted. Shown a chart like this a common team response will be to feel that they need to faster than the other teams. Achieving this additional speed may come from working in a more focused manner (a good thing), but it may come instead from sacrificing quality, leaving important refactorings undone, or a variety of other not-so-good manners.

Some teams may respond to the pressure for their abstract measure of velocity to increase by gradually inflating the number of story points assigned to a story. This can happen in subtle and not particularly nefarious ways that can accumulate into large problems. Consider, for example, a team that is arguing over whether a particular story should be estimated at 5 or 8 points. If the team is under pressure (real or just perceived) to increase velocity they will be more likely to assign the 8. The next story the team considers is slightly larger. They compare it to the newly assigned 8 and decide to give it a 13. Without pressure to improve velocity, this same team may have given the first item a 5 and the second (slightly larger still) item an 8. In this one scenario the team has inflated their points from 5+8=13 to 8+13=21, or more than 50%. Story point inflation such as this tends to happen very quickly if it happens at all.

Consider what happened in the next few iterations for the four teams shown in the previous figure.

Not surprisingly, someone in the Project Management Office distributed the chart showing the similarities over the first three iterations. Two of the teams reacted by instantly inflating their story points. After seeing that, the yellow team followed suit. The green team is either extremely virtuous or they haven’t noticed the charts yet.

So, should you establish a common baseline? Yes, if there are advantages to doing so on your project. If you do, however, you need to make sure you go out of your way to create safety around that baseline for the teams. Stress that this isn’t being done as a way to compare teams and that you (and your bosses know) that there are many factors that influence velocity, not just “how good” a team.

The best way I’ve found to do this is to bring a broad group of individuals representing various teams together and have them estimate a dozen or so product backlog items (ideally in the form of user stories in my opinion). Not each estimator needs to understand every item but most people should understand most items. The items being estimated do not need to be new items; some could be from a project finished recently that many estimators remember or worked on. Some items could be artificial; perhaps the team is asked to estimate, “a typical transaction activity report.” If that meant something to most estimators, it would be a good candidate item.

I’ve done with this 46 people in a large conference room–44 estimators plus me and a coach from my client who wanted to watch so he could moderate such a meeting the next time one would be needed. The 44 estimators represented 22 teams; two estimators per team were in the meeting. If you’ve seen or used the Mountain Goat planning poker cards, you’ll have noticed that they feature a very large number in the middle (plus the number in a smaller font in the corners). We could have done something cute like put eight little goats on the eight card. We put the very large number there deliberately, though: We wanted it to be visible across a potentially large conference room.

You can probably imagine how difficult it might be to gain consensus among 46 people playing planning poker. While it will not take proportionately longer to derive estimates, it does take quite awhile with that many people. I think it took us about two hours to estimate twelve items.

But when that meeting was over, each pair of estimators went back to their teams with twelve estimates. Those estimates could then be used as the basis for estimating future work. As each team estimated new product backlog items they would do so by comparing them to the initial 12 plus any estimates that had been produced since (by them or any other team).

I’ll blog next about when it may or may not be a good idea to establish such a common baseline.