Resampling is based on the idea that things we’ll observe in the future will be similar to the things we’ve observed in the past. In the examples below we’re saying the velocities a project team will see in the future will be similar to ones that occurred in the past. Resampling works by imagining we’ve put all old sample velocities into a bag. If we have past velocities of 18, 17, 18, 19, 22, and 20 imagine each of those written on separate slips of paper and dropped into a bag. Note we’ll have two slips of paper with “18? because our Agile or Scrum team here had that velocity twice.

To predict future velocity we reach into the bag and pull out one piece of paper. What’s written on it is our prediction of velocity in the first sprint. To predict the team’s velocity in the second sprint, we reach into the bag and pull out another slip of paper. But, before we do that, it’s important we put the first slip of paper back into the bag. This is called “resampling with replacement.” We want to replace because for any given sprint the team is equally likely to get any of their past velocities.

Suppose we’re trying to predict how much work a given team can complete in a coming ten-sprint period. We would resample (with replacement) ten times. Each time we’d note the number we pulled from the bag. After we pull the tenth slip, we would sum the ten values we pulled and that is one possible result for this team over the coming ten-sprint period.

It’s quite possible we could get lucky and pull the highest-valued slip (22) each of ten times. Or we could pull the worst-case slip, 17, each of the ten times. But these are unlikely. If we could in some way in the real world run the project hundreds or thousands of times, we would occasionally see the team repeat their highest (or lowest) velocity every sprint but it wouldn’t happen very often.

Since we can’t run the project hundreds or thousands of times in the real world, we want to simulate doing so on a computer. We can then learn a great deal from the results. For example, suppose we are ready to start a project that will have ten sprints. It would be helpful to know things like:

• what is the average amount of work completed over those ten sprints?
• what percentage of the time does the team finish more than say 200 points of work?

It’s actually quite straightforward to answer these questions using simulation and resampling. Let’s see how it’s done. You can follow along with this velocity resampling spreadsheet. In the figure below (from that spreadsheet), cells B3 through B28 show historical velocity.

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
Because our hypothetical project here involves ten sprints, cells D3 through E12 show the sprint numbers (1-10) and a resampled velocity for each.

Selecting a resampled velocity is simply a matter of randomly selecting any velocity from the B3:B28 range (our 26 historical velocities). This is done with the formula:
=SMALL($B$3:$B$28,INT(COUNT($B$3:$B$28)*RAND())+1)
which generates a random number between 1 and 26 and then uses the SMALL method to select that item from the list of values. (Note: SMALL selects the nth smallest item from the list of values; we could have used LARGE instead. The goal is just to randomly select a value from the list of velocity in B3:B28.)

Because we’re using the RAND() function in E3:E12, any time you change any cell in the spreadsheet, the values in E3:E12 change. This is a desirable side effect of using RAND().

E3:E12 simulates one run of ten sprints. We’d like, though, to simulate 100, 200 or even 1000 runs of the project (each with ten sprints in it). To do this is slightly tricky because we’re going to use something a lot of people aren’t familiar with in Excel: a Data Table. In our spreadsheet, the Data Table is in G3 through H202, a portion of which is shown below.

The G column shows a sprint number, the H column shows a sum of ten velocities, representing one ten-sprint project in our case. In the example, in the figure at left you can see that the first simulation of the project yielded a total of 230 points done in the ten sprints of the project. In the next row (cell H4, but labeled ’2′ in column G of the spreadsheet), the team got a much higher simulated velocity, 264. In the spreadsheet, I repeated this 200 times but you can do more or less as you prefer.

For brief instructions on how to create a data table, see the end of this post. For more complete instructions, see the Excel documentation.

Armed with 200 simulations of the ten sprints of the project (or ideally even more), we can now answer the question we started with, which is, How much can this team finish in ten sprints? Cells E17 and E18 of the spreadsheet show the average total work finished from the 200 simulations and the standard deviation around that work.

In this case the resampled average is 240 points (in ten sprints) with a standard deviation of 12. This means our single best guess (50/50) of how much the team can complete is 240 points. Knowing that 95% of the time the value will be within two standard deviations we know that there is a 95% chance of finishing between 240 +/- (2*12), which is 216 to 264 points.

If my boss wants a guarantee, I might say, “We can pretty much guarantee 216.” Technically I know the math doesn’t support the guarantee. There’s about a 2.5% chance we fall below that. However, humans are involved and just about any good team I’ve been on would be happy to kick in some extra effort sometime over the ten sprints to finish the 216 we committed to rather finishing with 210 if that 2.5% chance does occur.

Another interesting issue we can address with this type of simulation is the boss who says, “I need you to get 250 points done in the next 10 sprints.” You can see how likely this is to occur by scanning down the resampled values (the H column) and seeing how often the value there equals or exceeds the value the boss, client or customer wants. The spreadsheet is set to do this automatically as shown in this figure:

Type in the total number of points desired in L20, the spreadsheet will see how many times that number or higher occurred in the simulations (L21) and then report it as a percentage (L22). In this example, if the boss wants 250 points in 10 sprints, we can reply that while the team will try to achieve that, historical data shows that there is only a 20% chance of that occurring.

Hopefully you’ve found these examples of working with resampling to simulate projects helpful. There are many more things that can be done using this technique. I’ll provide additional examples in future posts.

You can download the velocity resampling spreadsheet used in this examples.

Note on Creating a Table Table

To create a Data Table, in cell H2 (the cell above where you want to put the simulation results), enter the formula you want to calculate as a result of each simulation. Because we’re interested in the sum of their ten sprints worth of velocity, I’ve entered:
=SUM($E$3:$E$12)

Next, if you’re interested fill the sprint cells (the G column) with numbers from 1 to 200. Then highlight cells G2:through 202 (assuming you want to do 200 simulations as I’ve done here). Notice you’re highlighting starting one row above our simulations through the last row of simulations. Now create the data table. In my version of Excel (Mac 2011), I do this by selecting Data | What If | Data Tables. You’ll get a little dialog box asking for the row or column input cell. Move the cursor to Column Input cell and select G2. Close the dialog and you’ll see the 200 rows fill with simulation results.

Each column of data and pair of bars shows data from stories of the size given in the Points row. The first set of data is for one-point stories, the second set of data is for two-point stories and so on. Looking at the one-point data, we see that the median number of hours to complete a one-point story (at this company) was 21 hours. The shortest (From) took 6 hours and the longest (To) took 36 hours. The shortest and longest are shown in the green and blue bars above the table. The median is graphed as the red line.

Let’s look at the two-point stories. We see there a median effort of 52 hours and a range from 31 to 73. If we assume the 1-point stories were perfectly estimated, we would expect 2 point stories to have a median of 42 instead of the 52 we see here. Or perhaps the 2-point stories were done perfectly and the 52 is right. In that case, the median for one-point stories should have been 26. Most likely, neither is perfect and the “perfect” estimates are somewhere between. Either way, the team has drifted a little bit because the median of the 2-pointers does not equal twice the median of the 1-pointers.

But let’s look at the three-point stories. Three-point stories should presumably be triple the median of the 1-point stories. The median should be 63 and we see that it is 64. Wow. Very close.

The five-point stories should presumably have a median of 5×21=105 and they come in at 100. The eight-point stories should presumably have a a median of 8×21=166 but they only come in at 111. But, hold on, we have to make a bit of an adjustment in this thinking for the 5- and 8-point stories. If you use these numbers the way I recommend, you know to think of them as buckets. An 8 is a bucket that holds all stories from six through eight points in size. That is, if you have a story you think is a six, you can’t fit it into the five-point bucket so it goes into the eight-point bucket.

This means that our five-point stories are really fours and fives. If we have the same number of each, the average size of a story in the five-point bucket is really 4.5 points. Multiplying 4.5 points x 21 hours (length of the 1-point story) give 94.5, off by about 5% from our measured median of 101 hours. Not so bad.

Since the 8-point bucket holds stories of size 6, 7, and 8, we can assume the average story given an 8 is really worth 7 points. And 7 points times 21 hours = 147 hours. The 8-point stories should have a median of 147. They don’t, they have a median observed value of 111. So this team has definitely drifted by the time they reach their 8-point stories.

(Let me add a sidenote here that rather than simply assuming the 1-point stories were perfect and using their 21 for all this mulitplication, a better approach would be to do linear regression analysis, which can be done with Excel. You can then look at the r-squared value to see how well the values fit. But, I didn’t want to go through all that math here and with data like this, we can, I believe, see that things work out pretty well up to 8.)

I encourage you to think about collecting data like this at your company. You need to be careful though. Because you’ll be collecting actual effort expended on each user story, it’s possible that team members feel more than the normal amount of pressure to finish within any estimates they give. They may then respond by padding their estimates. This defeats the whole purpose. So, show a graph like the one above to the team and be clear that having this data can help them. For example, the team above could learn that they put 8s on stories that should perhaps have been 5s. (Looking at the data, they could also learn that they estimated correctly but that they really had more sixes in the eight bucket.)

Most teams who do this will find that they are good through about 8, as in this example. With some awareness of data like this and some additional practice and calibration, just about any team can get good across a 1-13 range. Beyond that is tough and numbers above 13 (and possibly starting with 13) should be used with caution or only for answering rough, long-term questions like, “Is this project a couple of months or are we talking about a year to do it?”

If you collect data like this for your teams, I would appreciate it if you would share it with me. You can just email it to mike@mountaingoatsoftware.com rather than posting it here. I’ve been working on some analysis of data like this and the more teams I have, the better.

I’ve recently decided there’s a better way to convey the points we’ve been trying to make with the iron triangle–we use The One-Handed Clock of Project Goals.

To use the one-handed clock, position Scope, Schedule and Budget where twelve o’clock, four o’clock, and eight o’clock would be on a clock. It doesn’t matter which is positioned where but I put them as shown in the figure. Quality is again assumable fixed and not needed on the clock.

Next, ask the product owner or key stakeholder to point the one hand where it best indicates the project’s goals. The one hand can be aimed, for example, directly at Schedule. This would indicate that Schedule is the most important goal and Scope and Budget both take a back seat to it.

Or perhaps the one hand is pointed between Scope and Schedule showing a mix of importance between them.

To see how the One-Handed Clock of Project Goals works, take a moment to think about it. You can position the hand anywhere between any two of three goals but one goal is always left out. In the terms of the iron triangle, that would be the side left flexible.

There are a lot of things I like about this new way of visualizing the relative importance of Scope, Schedule and Budget. I’ll mention two here:

• The One-Handed Clock allows stakeholders or product owners to convey a position more precisely than saying something like “I pick Schedule and Budget.” The ability to point the arrow precisely rather than only directly at an item is essential.
• The One-Handed Clock is a useful visual metaphor that can be hung in a team room. The iron triangle doesn’t really work for that as it’s hard to convey which sides were selected other than by darkening them, which doens’t show much.

Try this out and let me know what you think. The teams and product owners I’ve introduced it to so far have found it to be a very helpful agile project management tool. I suspect you will as well. Also, let me know what else you use a one-handed clock for as it’s a useful visualization for any three competing factors.

Suppose for some reason you have tracked how long every one-story-point story took to develop for a given team. If you graphed that data you would have something that would look like this:

This shows that some stories took more time than others and some stories took less time, but overall the amount of time spent on your one-point stories takes on the shape of the familiar normal distribution.

Now suppose you had also tracked the amount of time spent on two-point user stories. Graphing that data as well, we would see something like this:

Number of hours to develop various one- and two-point stories

If the one-point stories are centered around a mean of x, ideally the two-point stories will be centered around a mean of 2x. This will never be exactly the case, of course, but a team that does a good job of estimating will be sufficiently close for reliable plans to be made from their estimates.

What these two figures show us is that is the relationship between points and hours is a distribution. One point equals a distribution with a mean of x and some standard deviation. The same is true, of course, for two-point stories, and so on…

By the way, notice that I’ve drawn the distributions of one- and two-point stories as having overlapping tails. It should be totally realistic that the biggest story that a team put “one story point” on might turn out to take more time than the smallest story they put a two on. After all, no team can estimate with perfect insight, especially at the story point level. So, while the tails of the one- and two-point distributions will overlap, it would be extraordinarily unlikely that the tails of, say, the one- and thirteen-point distributions will overlap.

But what do we do when we are managing an agile project where team membership or size changes frequently?

To answer this question most effectively, you should collect data on how teams of different sizes have performed over time in your organization. When I was a VP of Development at a couple of agile organizations, I used to collect data on velocity and team size changes in a simple spreadsheet similar to this:

The first column represented the size of the team before a change occurred. The next column represented the new size of the team (up or down). The third column was what I considered to be a reasonable “long term average” velocity for the team at its initial size. Because team’s could change frequently (by a person or two) I settled on using the median value of the team’s last five iterations. The tradeoff in using a longer measure (median of 15 iterations perhaps) is that you’d have fewer observations. The next columns represent the actual velocities of teams over the next three iterations. Notice that for the last team values are not shown for the last two iterations. This is usually because the team size changed again. If you have a significant number of teams, the rows in this type of spreadsheet will accumulate quite quickly.

In some of the organizations where I used this approach we did not have a standardized definition of story point (or we’re using ideal days, which were not as normalized as you might think). So all analysis was done on a percentage basis. What I wanted to know was “What is the average impact of adding a person to a seven-person team?” I would have loved the answer to be something like “Velocity goes up 15%.” Unfortunately, it wasn’t that straightforward because velocity often dipped for a couple of iterations before going up. By tracking data I found that usually by the third iteration a team had settled in on a new velocity, which is why my spreadsheet above only tracks through Iteration+3. By all means, track more and see what you find. (But keep in mind that the data will get sparse as team sizes will change again.)

Another tab in my spreadsheet, expressed all the data in percentage terms. The first two rows

(Example: Iteration +1 in the first row is -20% based on the team dropping its velocity from 25 to 20.)

I then simply averaged these percentages for each team size change to get results like:

This allowed me to answer all sorts of questions, including:

• What will this team’s velocity be if we add two people?
• How soon could we get this project if we added a person to each team?
• If I want all those projects done by the end of the year, how many people would we need to add?
• What would be the impact of not approving the new employees in the budget?
• What would be the impact of a 15% layoff?

There are of course many flaws with this approach. Adding Susan to the project is very different than “adding an unknown person” to the project. Still, if I have the data on averages across the board in our organization I can make assumptions about specifically adding Susan (if I want, there can be many more risks in doing this). Notice that the approach does not attempt to take into consideration who it was that was added or even what skillset the person had. You could collect such data if you wanted. As anal about collecting all sorts of data like this as I am when I have access to it, I knew though that collecting that type of data would have made this just hard enough that I wouldn’t have done it regularly.

I did collect a few other bits of data that I left out of the initial table (so as to have more horizontal room for the data of real interest). For example, I collected data such as iteration length and the name of the team’s ScrumMaster (the latter was in case I had questions a few weeks later).

The approach described here was just simple enough that I could get empirical evidence of the impact of team size changes. This was invaluable when discussing headcount changes with product owners and the CEO.

The need for a common baseline to story points usually arises from the reasonable desire to know how big the entire project is. To know that, we must know the size of the work to be done by each team. Unfortunately, along with this goal comes the ability to compare teams based on their velocities. Since many managers are constantly looking for ways to compare team and individual performance it is not surprising that they begin to make such velocity comparisons. Almost all such comparisons are disruptive to performance of the combined, overall group or department.

A chart such as the one that follows can show a lot of interesting information.

However, this chart can be very dangerous because of how teams will assume the data is being interpreted. Shown a chart like this a common team response will be to feel that they need to faster than the other teams. Achieving this additional speed may come from working in a more focused manner (a good thing), but it may come instead from sacrificing quality, leaving important refactorings undone, or a variety of other not-so-good manners.

Some teams may respond to the pressure for their abstract measure of velocity to increase by gradually inflating the number of story points assigned to a story. This can happen in subtle and not particularly nefarious ways that can accumulate into large problems. Consider, for example, a team that is arguing over whether a particular story should be estimated at 5 or 8 points. If the team is under pressure (real or just perceived) to increase velocity they will be more likely to assign the 8. The next story the team considers is slightly larger. They compare it to the newly assigned 8 and decide to give it a 13. Without pressure to improve velocity, this same team may have given the first item a 5 and the second (slightly larger still) item an 8. In this one scenario the team has inflated their points from 5+8=13 to 8+13=21, or more than 50%. Story point inflation such as this tends to happen very quickly if it happens at all.

Consider what happened in the next few iterations for the four teams shown in the previous figure.

Not surprisingly, someone in the Project Management Office distributed the chart showing the similarities over the first three iterations. Two of the teams reacted by instantly inflating their story points. After seeing that, the yellow team followed suit. The green team is either extremely virtuous or they haven’t noticed the charts yet.

So, should you establish a common baseline? Yes, if there are advantages to doing so on your project. If you do, however, you need to make sure you go out of your way to create safety around that baseline for the teams. Stress that this isn’t being done as a way to compare teams and that you (and your bosses know) that there are many factors that influence velocity, not just “how good” a team.

The classic Scrum release burndown chart is good at showing whether a team will finish “on time” as can be seen in the following example burndown chart:

A release burndown chart such as this one shows sprints on the horizontal axis and can show story points or ideal days on the vertical. It is updated once per sprint to show the team’s net progress that sprint. A team’s net progress is the amount of work they finished net of any changes in scope. So a team that completes 30 points of work but that has 10 points added to their product backlog will show net progress of 20.

But while a traditional release burndown chart excels at showing whether a team is on pace to finishing on time, it is not very good at showing what will be included in that “on time” delivery. To see this, imagine two teams that each start with 200 story points of work. The first team finishes twenty points of work for each of ten sprints. The second team is incompetent and rather than completing twenty points each sprint they drop twenty points of scope each sprint. The two burndown charts will be identical–perfect lines descending over ten sprints from 200 to 0.

At one level this is OK: the burndown chart shows whether a team will be finished (or by when the will be finished). The simplicity of the standard release burndown chart has much in its favor.

It isn’t hard, though, to extend a release burndown chart to also show what will be in the product by the final sprint. Look at the next figure, which is a hypothetical example of an eCommerce product.

In this figure, you can see the burndown is tracked in the normal way through the end of the current sprint, the seventh. The company desires to release this product after the fourteenth two-week sprint. The right side of the burndown chart shows the team’s product backlog with the highest priority theme (“Returns”) at the top. This top block represents some “must-have” user stories related to returning purchased items. Below that is a theme for gift wrapping purchased items, followed by some “nice to have” aspects of returning items. At the bottom right is the Coupons theme.

Extending out from the team’s current position at the end of the seventh sprint are four lines. These lines represent the following:

1. The team’s current position, drawn as a horizontal line from the current burndown position over to the product backlog. This tells us what is in the product so far. We can see that the mandatory return user stories and the gift wrap user stories are finished and that the team is partially into the nice-to-have return user stories.
2. A black, dashed line showing the team’s most likely finish. This is the first of three trend lines meant to show the likely range of work the team might deliver. To draw a team’s most likely finish use the team’s long-term average velocity. You can define “long-term average velocity” in whatever way you want but my preference is to use the average velocity of the last 8-12 sprints. Pick the number of historical sprints that is most suitable for your team based on how long the sprints are and how long the team stays together.
3. A pessimistic forecast of the amount of functionality that may be delivered. I recommend forecasting this based on a team’s worst-case but likely velocity. Calculate this by averaging the worst three or so velocities chosen among the same 8-12 iterations you looked back at to determine the team’s long-term average velocity.
4. An optimistic forecast of the amount of functionality that may be delivered. Calculate this in the same was as in the pessimistic case but use the three (or so) best velocities of the team.

The figures in this blog are static images I’ve cut from a presentation. If you apply this technique for your team, the backlog items on the right should be clickable, allowing users to drill down into a product backlog theme to see specifically which items (typically user stories) make up the backlog.

By producing a single chart that shows both a team’s rate of progress (its burndown) and the product backlog, we have a single visualization that shows both when a team is likely to finish and what features will be in the product by that time. This makes is easier for product owners to make scope vs. schedule tradeoff decisions.

Check back in a few weeks when I’ll show an even more powerful technique for visualizing large product backlogs.

My view is that points can be used as the best way to estimate and assess progress that we’ve ever had or they can be used as another weapon with which to hit the team. There are plenty of weapons with which you can hit your team. We don’t need to ruin points by using them that way as well.

Some teams have measured productivity with things like the number of backlog items delivered or the % of backlog items completed vs. planned into a sprint. Teams will alter their behavior on those as well though so they can be gamed and misleading. These metrics can be useful but only as part of a suite of metrics collected at the end of each iteration.

If we rethink the question of “how do we measure productivity” we might get a better answer. Suppose you own a sandwich shop and want to measure the productivity of the sandwich maker in the back. He responds to our metric by making as many sandwiches as he can–regardless of whether anyone ordered them! At the end of the day there will be 200 extra sandwiches to throw away. A better measure of him might be how quickly he makes any sandwich. So we’d measure the time from when the customer placed the order until the sandwich is put on a tray. Or for a more complete metric we may want to measure the time from when he receives an order until he is ready to receive the next order as this captures any cleanup or restart time.

So, one measure we may want to include in our suite of metrics could be the responsiveness of the development organization. This would be measured in the same way as in the sandwich shop. Datestamp each product backlog item and track the time from when something enters the product backlog until it either (a) comes out of an iteration or (b) is delivered into the hands of customers. Choosing between (a) and (b) will largely be a matter of how often you ship software. Option (b) is a better measure of rapid delivery of customer value but is impractical in some cases. It would be a bit of a useless measure for the Microsoft Vista team, for example.