Archive for the ‘Transitioning to Agile’ Category

New Article in Agile Journal on “Comparative Agility”

Wednesday, January 13th, 2010

Agile Journal has published an article of mine called, “Determining How Agile You Are Comparatively.” It is about the Comparative Agility project. If you haven’t looked at this before, please do. It’s an effort to collect data on how agile various companies are so that they can compare themselves (anonymously).

The Agile Journal article includes a sidebar by Laurie Williams and Kenny Rubin, my partners on this project. Laurie is currently heading up an effort to refine the questions that form the survey. We could use your help.

Share a Waterfallacy; Win a Book

Monday, January 11th, 2010

It seems like time for a new contest with the winner getting a free copy of Succeeding with Agile, my new book.

In Succeeding with Agile, I describe a waterfallacy as “a mistaken belief or idea about agile or Scrum created from working too long on waterfall projects.” And I give some examples, including these:

  • Scrum teams don’t plan, so we’re unable to make commitments to customers.
  • Scrum requires everyone to be a generalist.
  • Our team is spread around the world, and Scrum requires face-to-face communication.
  • Scrum is OK for simple websites, but our system is too complicated.

To enter the post, add a comment to this post telling us about one waterfallacy you’ve encountered and how you overcame that waterfallacy or convinced someone that it was a waterfallacy instead of the truth.

Let’s run this contest until midnight Mountain time on next Monday, 18 January. I’ll then announce winners on Tuesday, 19 January. I’ll pick two winners–the entry that I personally like the best plus one that will be randomly selected.

Good luck and let’s enjoying putting some waterfallacies to rest!

Reduce Manual Test Technical Debt

Monday, December 21st, 2009

When a team that has relied mostly or entirely on manual testing decides to adopt Scrum, it will quickly discover how hard it is to run short sprints when there’s a lot of manual testing to be done each sprint. It will also realize that unless it does something drastic, the technical debt will continue to accumulate. Teams in this situation can follow a three-step process to extricate themselves from at least the worst of these problems:

  1. Stop the bleeding.
  2. Stay current.
  3. Catch up.

The first priority of a team with technical debt in the form of an over-reliance on manual testing is to stop the bleeding, stop things from getting worse. The best tourniquet is to find ways to automate some of what is being tested manually. To mix metaphors, teams should find the low-hanging fruit: tests that will be easy to automate but save a lot of manual effort. Brian Marick, a leading authority on testing and coauthor of the Agile Manifesto, has told me that “the real low-hanging fruit is often not automating some test execution but automating other testing tasks, like populating databases or automatic navigation to the page where you’ll start manual testing. You’re not reducing the number of manual tests, but you’re reducing the total time it takes to run them.”

After the bleeding has been stopped, the situation will no longer be getting worse from sprint to sprint. Manual tests still will be added each sprint but the team is finding enough low-hanging fruit each sprint to offset the time needed to run the new manual tests. At this point, it is time to move to step two: learning to stay current. During this phase, the team focuses on learning how to write and automate tests for whatever new features are added during the sprint. While doing this, no more debt is accumulating, so the situation isn’t getting any worse, but it’s not yet getting any better either. Learning to add automated tests in the same sprint as the feature will be a new skill for the team. It won’t be as hard to learn as the initial skills were during the first phase but will require new discipline.

Eventually the team enters the final phase, which is when it catches up on additional outstanding testing debt. I generally tell teams that I coach that I don’t care how quickly the outstanding testing debt is brought down as long as it is indeed coming down. Obviously, I would prefer the debt to come down as fast as possible. But, by stating it this way I am emphasizing that what I am most concerned with are the first two steps: stop the bleeding and stay current.

For more on testing with an agile process like Scrum, pick up a copy of Succeeding with Agile.

The Forgotten Layer of the Test Automation Pyramid

Thursday, December 17th, 2009

Even before the ascendancy of agile methodologies like Scrum, we knew we should automate our tests. But we didn’t. Automated tests were considered expensive to write and were often written months, or in some cases years, after a feature had been programmed. One reason teams found it difficult to write tests sooner was because they were automating at the wrong level. An effective test automation strategy calls for automating tests at three different levels, as shown in the figure below, which depicts the test automation pyramid.

Test Automation Pyramid

At the base of the test automation pyramid is unit testing. Unit testing should be the foundation of a solid test automation strategy and as such represents the largest part of the pyramid. Automated unit tests are wonderful because they give specific data to a programmer—there is a bug and it’s on line 47. Programmers have learned that the bug may really be on line 51 or 42, but it’s much nicer to have an automated unit test narrow it down than it is to have a tester say, “There’s a bug in how you’re retrieving member records from the database,” which might represent 1,000 or more lines of code. Also, because unit tests are usually written in the same language as the system, programmers are often most comfortable writing them.

Let’s skip for a moment the middle of the test automation pyramid and jump right to the top: the user interface level. Automated user interface testing is placed at the top of the test automation pyramid because we want to do as little of it as possible.

Suppose we wish to test a very simple calculator that allows a user to enter two integers, click either a multiply or divide button, and then see the result of that operation. To test this through the user interface, we would script a series of tests to drive the user interface, type the appropriate values into the fields, press the multiply or divide button, and then compare expected and actual values. Testing in this manner would certainly work but would be brittle, expensive, and time consuming.

Additionally, testing an application this way is partially redundant—think about how many times a suite of tests like this will test the user interface. Each test case will invoke the code that connects the multiply or divide button to the code in the guts of the application that does the math. Each test case will also test the code that displays results. And so on. Testing through the user interface like this is expensive and should be minimized. Although there are many test cases that need to be invoked, not all need to be run through the user interface.

And this is where the service layer of the test automation pyramid comes in. Although I refer to the middle layer of the test automation pyramid as the service layer, I am not restricting us to using only a service-oriented architecture. All applications are made up of various services. In the way I’m using it, a service is something the application does in response to some input or set of inputs. Our example calculator involves two services: multiply and divide.

Service-level testing is about testing the services of an application separately from its user interface. So instead of running a dozen or so multiplication test cases through the calculator’s user interface, we instead perform those tests at the service level.

Where many organizations have gone wrong in their test automation efforts over the years has been in ignoring this whole middle layer of service testing. Although automated unit testing is wonderful, it can cover only so much of an application’s testing needs. Without service-level testing to fill the gap between unit and user interface testing, all other testing ends up being performed through the user interface, resulting in tests that are expensive to run, expensive to write, and brittle.

For more on Scrum and agile testing, pick up a copy of Succeeding with Agile.

Four Types of Resistors When Adopting Agile

Wednesday, December 2nd, 2009

People resist changing to Scrum for many different reasons. Some may resist because they are comfortable with their current work and colleagues. It has taken years to get to their current levels in the organization, to be on this team, to work for that manager, or to know exactly how to do their jobs each day. Others may resist changing to Scrum because of a fear of the unknown. “Better the devil you know than the devil you don’t” is their mantra. Still others may resist due to a genuine dislike or distrust of the Scrum approach. They may be convinced that building complex products iteratively without significant up-front design will lead to disaster.

Just as there are many reasons why some people will resist Scrum, there are many ways someone might resist. One person may resist with well-reasoned logic and fierce arguments. Another may resist by quietly sabotaging the change effort. Still another may resist by quietly ignoring the change, working the old way as much as possible, and waiting for the next change du jour to come along and sweep Scrum away.

Each act of resistance carries with it information about how people feel about adopting Scrum. As a change agent or leader in the organization, your goal should be to understand the root cause of an individual’s resistance, learn from it, and then help the person overcome it. There are many techniques you can use for doing this. But unless the technique is carefully chosen, it is unlikely to have the desired effect. To help select the right technique, I find it useful to think about how and why someone is resisting. We can group the reasons why someone is resisting Scrum into two general categories:

  • They like the status quo.
  • They don’t like Scrum.

Reasons for resistance fall into the first category if they are actually a defense of the current approach. This type of resistance to changing to Scrum would likely result no matter what type of change was being contemplated. Reasons fall into the second category if they are arguments against the specific implications of beginning to work in an agile manner.

Categorizing how individuals resist is even simpler: Is the resistance active or passive? Active resistance occurs when someone takes a specific action intended to impede or derail the transition to Scrum. Passive resistance occurs when someone fails to take a specific action, usually after saying he will.

The figure below combines the whys and hows of resistance into four general types of resistance, each with a name descriptive of the person who resists in the way indicated by the labels on the axes.

four types of resistors

A skeptic is someone who does not agree with the principles or practices of Scrum but who only passively resists the transition. Skeptics are the ones who politely argue against Scrum, forget to attend the daily scrum a little too often, and so on. I am referring here to individuals who are truly trying to stop the transition, not people with the healthy attitude of “this sounds different from anything I’ve done before but I’m intrigued. Let’s give it a try and see if it works.”

Shown above the skeptics are the saboteurs. Like skeptics, saboteurs resist the transition more from a dislike of Scrum than support for whatever software development process exists currently. Unlike a skeptic, a saboteur provides active resistance by trying to undermine the transition effort, perhaps by continuing to write lengthy up-front design documents, and so on.

On the left side of the figure are those who resist because they like the status quo. They are comfortable with their current activities, prestige, and coworkers. In principle, these individuals may not be opposed to Scrum; they are, however, opposed to any change that puts their current situation at risk. Those who like the status quo and who actively resist changing from it are known as diehards. They often attempt to prevent the transition by rallying others to their cause.

The bottom left of the figure shows the followers, who like the status quo and resist changing from it passively. Followers are usually not enraged by the prospect of change, so they do little more than hope it passes like a fad. They need to be shown that Scrum has become the new status quo.

When introducing a complex change into a large organization, resistance will be inevitable. What isn’t inevitable is the reaction of an organization’s leaders to that resistance. In a 1969 Harvard Business Review article, Paul Lawrence describes an appropriate response to resistance.

When resistance does appear, it should not be thought of as something to be overcome. Instead, it can best be thought of as a useful red flag—a signal that something is going wrong….Therefore, when resistance appears, it is time to listen carefully to find out what the trouble is. What is needed is not a long harangue on the logics of the new recommendations but a careful exploration of the difficulty.

While its important to confront resistance, be careful not to create an atmosphere of “us” against “them.” The real goal is to create a feeling that the transition to Scrum is inevitable and that, as the Borg of Star Trek taught us, “resistance is futile.” The need to foster this atmosphere does not give you carte blanche to ignore the feelings and reactions of employees or to steamroll Scrum into an organization. Instead, to paraphrase Nigel Nicholson in a 2003 article for the Harvard Business Review, when an employee resists, an effective leader looks at the employee not as a problem to be solved but as a person to be understood.

For more help with overcoming obstacles to your agile transition, see Chapter 6 of Succeeding with Agile.

Setting and Managing Expectations

Monday, November 30th, 2009

In 1994 I managed a team that delivered a project that any outsider or any project team member would have considered a success. The product represented a great leap forward for the company. It included far more features than the product that was being replaced, was built using new state-of-the-art technologies with which the company had no prior experience, and included the development of three data centers that went on to provide 99.99999% uptime over the next six years. However, the project was almost considered a failure.

The project was to be delivered into multiple call centers with more than 300 nurses on the phones. It was to replace a quirky but familiar system that the company was rapidly outgrowing. The nurses’ expectations of what the new system would deliver were sky high. In monthly sprint reviews with the nurses, I was routinely shocked by what they’d come to expect, some of which wasn’t even technically feasible.

With about three months left on the year-long project, I realized my focus had to change. From then on, I spent almost all of my time on expectations management. I met with nurses in each of the call centers and described exactly what would and would not be in the delivered system. I toned down their expectations about the system’s impact on world peace, global warming, and personal weight loss. Without this effort, the product would have been perceived as a failure.

Since that project, I have been acutely aware of the importance of expectations management to the overall success of any project. Setting and managing expectations is perhaps even more important at the start of a major shift such as adopting Scrum. In initiating a transition to Scrum, I find it helpful to set and manage expectations about four things:

  • How quickly teams will improve
  • How long it will take to gain additional predictability from the team’s new way of working
  • How there will almost always come a time when turning back looks easier than sticking with it
  • The level of involvement in the transition that will be necessary from various stakeholders and organization leaders

By properly setting expectations you can avoid the problem of having an otherwise successful transition or project sunk by unrealistic expectations.

More details about setting and managing expectations can be found in Chapter 5 of Succeeding with Agile.

How Do You Get from Here to Agile? Iterate.

Thursday, November 19th, 2009

Historically, when an organization needed to change, it undertook a “change program.” The change was designed, had an identifiable beginning and ending, and was imposed from above. This worked well in an era when change was necessary only once every few years. But in today’s fast-paced, ever changing environment, it makes more sense to create agile organizations, ready to adapt to whatever comes their way.

But how do you manage the effort of moving from the point you are today—whether that’s just starting to adopt Scrum or fine-tuning your use of Scrum—to a place where you can readily react and respond to the vagaries of the market? By following an iterative transition process. Making small changes on a continual basis is a logical way to adopt a development process that is itself iterative. Doing so will be much more likely to result in a successful and sustainable transition. This is why I believe that the effort of adopting Scrum is best managed using Scrum itself. With its iterative nature, fixed timeboxes, and emphasis on teamwork and action, it seems best suited to manage the enormous project of becoming and then growing agile with Scrum.

Just as Scrum development projects use product backlogs, you should use an improvement backlog to track the effort of adopting Scrum in your organization. An improvement backlog lists everything that the organization could do better in its use of Scrum. If you’re just starting with Scrum, your improvement backlog will emphasize creating awareness and desire. If the transition is already well underway, your improvement backlog may contain more items around developing the ability to do Scrum well, to promote successes, or to transfer it to other groups. A small department or single-project transition may involve a single improvement backlog. But when Scrum is being adopted across a large site, department, or organization, the transition effort becomes large enough that multiple improvement backlogs are used, each of which is created by a community of individuals who are passionate about improving the organization in a particular way.

Many corporate improvement initiatives fail because plans are not made specific and actionable. Using Scrum to manage the effort of becoming agile allows you to divide the work into stories and tasks with concrete deliverables and definite end dates. At the end of every iteration, the organization will have improved in measurable and visible ways. Eventually, you will have successfully transformed the organization into one that not only is agile but also that seeks to become more agile each day.

Additional advice on using Scrum to manage your transition effort is provided in Chapter 4, “Iterating Toward Agility” of Succeeding with Agile.

Four Attributes of the Ideal Pilot Project

Monday, November 16th, 2009

Selecting the right project as a pilot can be challenging. Jeff Honious, vice president in charge of innovation at Reed Elsevier, led his company’s transition to Scrum. He and colleague Jonathan Clark wrote of their struggle to select the right pilot.

Finding the right project was the most critical and challenging task. We needed a meaty project that people would not dismiss as being a special case, yet we did not want a project to fill every possible challenge—too much was riding on its success.

Not every project is equally suited to be your first. The ideal pilot project sits at the confluence of project size, project duration, project importance, and the engagement of the business sponsor, as shown in the figure below. You may find it impossible to identify the “perfect” pilot project. That’s OK. Consider the projects you do have and make appropriate trade-offs between the four factors presented in the figure shown below. It is far better to pick a project that is close enough and get started than it is to delay six or more months waiting for the perfect pilot to present itself.

Four attributes of ideal pilot project

Duration.
If you select a project that is too short, skeptics will claim that Scrum works only on short projects. At the same time, if you select a project that is too long, you risk not being able to claim success until the project is over. Many traditionally managed projects claim to be on track 9 months in to a 12-month schedule, yet in the end are over budget and late, so a Scrum project proclaiming the same may not be very convincing.
What I find best is to select a project whose length is near the middle of what is normal for an organization. Ideally and frequently this is around three or four months. This gives a team plenty of time to start getting good at working within sprints, to enjoy it, and to see the benefits to them and to the product. A three- or four-month project is also usually sufficient for claiming that Scrum will lead to similar success on longer projects.

Size.
Select a project that can be started with one team whose members are all collocated, if at all possible. Start with one team, even if the pilot project will grow to include more teams. Try to select a pilot project that will not grow to more than five or so teams, even if such projects will be common in your organization. Not only is coordinating work among that many Scrum teams more than you want to bite off initially, but you also probably wouldn’t have time to grow from one team to more than five anyway if you also looking for a project that can be completed in three or four months.

Importance.

It can be tempting to select a low-importance, low-risk project. If things go badly, not much will be lost. And people may not even notice a failure on a low-importance project. Don’t give in to this temptation. Instead, pick an important project. An unimportant project will not get the necessary attention from the rest of the organization. Additionally, some of the things required of a team transitioning to Scrum are difficult; if the project isn’t important, people may not do all that is required of them. Early agilist and inventor of the Adaptive Software Development process Jim Highsmith gave advice about this in Agile Software Development Ecosystems.

Don’t start with an initial ‘learning project’ that is of marginal importance. Start on a project that is absolutely critical to your company; otherwise it will be too difficult to implement all the hard things Scrum will ask of you.


Business sponsor engagement.

Adopting Scrum requires changes on the business side of the development equation, not just the technical side. Having someone on the business side who has the time and inclination to work with the team is critical. An engaged business sponsor can help the team if it needs to push against entrenched business processes, departments, or individuals. Similarly, there is no one more useful in promoting the success of the project afterward than a sponsor who got what was expected. One sponsor commenting to another that a recent project tried Scrum and delivered more than past projects did will do wonders in getting other sponsors to ask their teams to also try the new approach.

And Finally: People.
I’m not forgetting the importance of the people involved to the success of a pilot. The people involved are, of course, the most important factor in the success of any project. The advice provided in this blog post assumes that you have already chosen the appropriate team and are now seeking to match them to the best possible pilot. Additional advice on selecting the individuals who will form pilot teams is given in the “Selecting a Pilot Team” in Chapter 5, “Your First Projects” of Succeeding with Agile.

There Is No End State When Transitioning to Agile

Monday, November 9th, 2009

None of the agile processes as described by their originators is perfect for your organization. Any may be a good starting point, but you will need to tailor the process to more precisely fit the unique circumstances of your organization, individuals, and industry. As Alistair Cockburn once told me, “Having a chance to change or personalize a process to fit themselves seems to be a critical success factor for a team to adopt a process. It’s the act of creation that seems to bind teams to ‘their own’ process.”

You may have a clear vision of what “doing Scrum” or “doing XP” means to you, and you may get others to buy into exactly that same vision, but where the organization ends up is likely to be somewhat different. In fact, to even refer to end states in an agile transition is incorrect; there can be no end state in a process that calls for continuous improvement.

This creates a problem for an organization that wants to transition to agile through a traditional change approach that relies on gap analysis and then on closing the identified gaps. If we cannot anticipate the end state of an agile transition, we cannot identify all of the gaps between there and the current state. So, a gap analysis–driven change approach will not work.

The closest we can come is to identify gaps between where we are now and an improved, intermediate state. After identifying these smaller gaps, though, we are still left with the problem of how to close them. It is difficult (and often impossible) to predict exactly how people will respond to the many small changes that will be needed on the way to becoming agile. Teamwork expert Christopher Avery views organizations as living systems.

We can never direct a living system, only disturb it and wait to see the response…. We can’t know all the forces shaping an organization we wish to change, so all we can do is provoke the system in some way by experimenting with a force we think might have some impact, then watch to see what happens.

So, a transition to agile cannot be a process that “articulates and defines the entire change process required to bridge the gap between ‘as is’ and ‘to be’ and creates tactical plans,” as I read in a traditional change management book recently. Creating such a plan would require leaping two impossible hurdles: first, knowing exactly where we’ll want to end up; and second, knowing exactly the steps to get there. Because we cannot overcome these impossibilities, the best we can do is adopt a “provoke and observe” approach in which we try something, see if it moves us closer to an intermediate, improved state, and if so do more of it.

These pokings and proddings of the organization are not random. They are carefully selected based on experience, wisdom, and intuition to drive a successful transition to agile.

For more information on this topic, see Succeeding with Agile: Software Development using Scrum.

Presentation on the Benefits of Agile

Wednesday, October 28th, 2009

Chapter 1 of Succeeding with Agile summarizes of the published research on the benefits of adopting an agile process. I created a presentation based on that research.

The presentation is available in Keynote and PowerPoint format and is released under a Creative Commons license with the hope that others will use and extend it.