• We have five product owners. What do we do?
• We drop work from every sprint. How do we get out of that habit?
• We can’t ever get things “potentially shippable” at the end of a sprint. How can we?
• We spend forever in planning meetings. How can we spend less time doing that?
• etc…

So, what problems have you encountered or heard of teams encountering?

Thanks for your help.

That leads me to my new year’s resolution for 2012 and one that I hope a lot of others will make with me: make recommendations not rules.

There are very few hard-and-fast rules to agile software development. I’d put things like:

• work in iterations of no more than a month long
• by the end of each iteration be “done” with something to some pre-agreed upon definition of done and solicit feedback from your key stakeholders on it
• at the start of an iteration, get together and figure out what you’re doing to do during the iteration
• at the end of the iteration, reflect on how well you did during the iteration
• talk a lot during the iteration

Beyond that, it’s much more about recommendations. And there are plenty of things we’ve learned in the nearly 20 years that some agile processes have been around in even informal forms. For example, I recommend teams use user stories as their approach to requirements. I recommend teams use story points for estimating. I recommend that the team pick a day other than Mondays for starting their iterations. I recommend the Szechuan Chicken at Spice China. But, none of these things is required for success with agile. Each may help a team be better, and I have reasons I recommend each. But, these things are not required.

So, my resolution for 2012: Make recommendations not rules. I’d kind of like to make it a rule that you join me in this resolution, but I’ve just resolved not to make such rules.

Well, the organization probably does need some changes because if it were operating optimally, we wouldn’t be looking at changing its approach to software development. This idea that a methodology is better because it can be introduced without changing an organization seems a little too much like a sales effort. One of the best things about most people in the Scrum community is that we openly say that changing will be hard. In fact, in my Scrum classes I warn attendees of what I call the Scrum Promise: If you adopt Scrum, there will be a day you come into the office nearly in tears over how hard the change can be. This is because Scrum doesn’t solve problems, it uncovers them and puts them in our face. Then, through hard work we address them.

I’m reminded of visiting my father years ago after he had quadruple bypass heart surgery. His doctor had warned my dad that he needed to eat less fat. My dad diligently went on a low-fat diet but when he returned a few months later to the doctor he had surprisingly gained weight.

The mystery was solved shortly later when I visited my dad. He had a pantry full of fat-free muffins and cakes, which he would eat by the box. He was also enjoying Hershey’s chocolate syrup straight from the bottle. When I asked him about it, he just said, “What’s wrong? It’s fat free?” And indeed it was.

Still, I suspect my dad’s surgeon was looking for a bit more dramatic of a change in my dad’s eating habits. My dad did eventually change his eating habits and it’s been 20 years since his bypass surgery and his doctor tells him his heart is doing well.

For an organization to have similarly good long-term results, it, too, will likely need to make some hard changes. Trying to become agile without hard change is too much like drinking chocolate syrup from the bottle and expecting to lose weight.

We want to use agile when we are doing something that is new, or at least new to the team building it. If it’s something the team has done before over and over then the team probably doesn’t need an agile approach. To my mind, this is where some of the manufacturing analogies come in. If we are building the same car day after day, we learn pretty quickly all the nuances of building that car. We don’t need an agile approach because the novelty of the situation is low.

Novelty alone does not mean we should use an agile process. I went to my favorite Chinese restaurant for lunch today. I ordered an entree “triple extra spicy and with jalapenos.” It was probably the first time they cooked this particular dish that way and so it was a novel or unique order. The cook prepared it wonderfully though and because I could see into the kitchen I’m sure they didn’t need a daily standup or even TDD to make my lunch. (I might have noticed a kanban back there, however. So, in addition to novelty, the project needs a certain amount of complexity.

One final element I believe is required in making a project appropriate for agile is urgency. The timeboxes and iterations of an agile approach are devised to keep the intensity and focus going on a project. If there’s no urgency to the project, those are unneeded.

So let’s see how these three factors–urgency, complexity, and novelty–mix on various projects, starting of course with software projects. There couldn’t be a better fit. Software projects are notoriously complex. Each software project is largely a new endeavor. And in today’s world, there is almost always a sense of urgency.

But let’s look at one other situation where we commonly here about Scrum (in particular) being applied: getting married. At least a couple of times a year I hear about a couple who planned their wedding using Scrum. There is always a wedding backlog–buy cake, pick photographer, send invitations, pick dress, etc. How does planning a wedding do against the three factors I’m proposing. Sense of urgency? Check. There’s always a deadline and it’s usually pretty fixed. Complexity? Well, it’s not like a software project but it does have its own complexities often enhanced by non-functional requirements such as a fixed budget, who sits next to whom, the type of food to be served, the need to let Cousin Ira’s band perform at the reception, etc. Novelty? Yep. Most people don’t get married enough times with large celebrations that planning the event becomes second hand.

So, agile is most appropriate on any urgent project with significant complexity and novelty–and that includes software development and weddings. It does raise the question though of whether a couple’s first kiss at the end of the ceremony is a product backlog item or part of the done criteria for the overall product.

The Agile Journal article includes a sidebar by Laurie Williams and Kenny Rubin, my partners on this project. Laurie is currently heading up an effort to refine the questions that form the survey. We could use your help.

In Succeeding with Agile, I describe a waterfallacy as “a mistaken belief or idea about agile or Scrum created from working too long on waterfall projects.” And I give some examples, including these:

• Scrum teams don’t plan, so we’re unable to make commitments to customers.
• Scrum requires everyone to be a generalist.
• Our team is spread around the world, and Scrum requires face-to-face communication.
• Scrum is OK for simple websites, but our system is too complicated.

To enter the post, add a comment to this post telling us about one waterfallacy you’ve encountered and how you overcame that waterfallacy or convinced someone that it was a waterfallacy instead of the truth.

Let’s run this contest until midnight Mountain time on next Monday, 18 January. I’ll then announce winners on Tuesday, 19 January. I’ll pick two winners–the entry that I personally like the best plus one that will be randomly selected.

Good luck and let’s enjoying putting some waterfallacies to rest!

1. Stop the bleeding.
2. Stay current.
3. Catch up.

The first priority of a team with technical debt in the form of an over-reliance on manual testing is to stop the bleeding, stop things from getting worse. The best tourniquet is to find ways to automate some of what is being tested manually. To mix metaphors, teams should find the low-hanging fruit: tests that will be easy to automate but save a lot of manual effort. Brian Marick, a leading authority on testing and coauthor of the Agile Manifesto, has told me that “the real low-hanging fruit is often not automating some test execution but automating other testing tasks, like populating databases or automatic navigation to the page where you’ll start manual testing. You’re not reducing the number of manual tests, but you’re reducing the total time it takes to run them.”

After the bleeding has been stopped, the situation will no longer be getting worse from sprint to sprint. Manual tests still will be added each sprint but the team is finding enough low-hanging fruit each sprint to offset the time needed to run the new manual tests. At this point, it is time to move to step two: learning to stay current. During this phase, the team focuses on learning how to write and automate tests for whatever new features are added during the sprint. While doing this, no more debt is accumulating, so the situation isn’t getting any worse, but it’s not yet getting any better either. Learning to add automated tests in the same sprint as the feature will be a new skill for the team. It won’t be as hard to learn as the initial skills were during the first phase but will require new discipline.

Eventually the team enters the final phase, which is when it catches up on additional outstanding testing debt. I generally tell teams that I coach that I don’t care how quickly the outstanding testing debt is brought down as long as it is indeed coming down. Obviously, I would prefer the debt to come down as fast as possible. But, by stating it this way I am emphasizing that what I am most concerned with are the first two steps: stop the bleeding and stay current.

For more on testing with an agile process like Scrum, pick up a copy of Succeeding with Agile.

At the base of the test automation pyramid is unit testing. Unit testing should be the foundation of a solid test automation strategy and as such represents the largest part of the pyramid. Automated unit tests are wonderful because they give specific data to a programmer—there is a bug and it’s on line 47. Programmers have learned that the bug may really be on line 51 or 42, but it’s much nicer to have an automated unit test narrow it down than it is to have a tester say, “There’s a bug in how you’re retrieving member records from the database,” which might represent 1,000 or more lines of code. Also, because unit tests are usually written in the same language as the system, programmers are often most comfortable writing them.

Let’s skip for a moment the middle of the test automation pyramid and jump right to the top: the user interface level. Automated user interface testing is placed at the top of the test automation pyramid because we want to do as little of it as possible.

Suppose we wish to test a very simple calculator that allows a user to enter two integers, click either a multiply or divide button, and then see the result of that operation. To test this through the user interface, we would script a series of tests to drive the user interface, type the appropriate values into the fields, press the multiply or divide button, and then compare expected and actual values. Testing in this manner would certainly work but would be brittle, expensive, and time consuming.

Additionally, testing an application this way is partially redundant—think about how many times a suite of tests like this will test the user interface. Each test case will invoke the code that connects the multiply or divide button to the code in the guts of the application that does the math. Each test case will also test the code that displays results. And so on. Testing through the user interface like this is expensive and should be minimized. Although there are many test cases that need to be invoked, not all need to be run through the user interface.

And this is where the service layer of the test automation pyramid comes in. Although I refer to the middle layer of the test automation pyramid as the service layer, I am not restricting us to using only a service-oriented architecture. All applications are made up of various services. In the way I’m using it, a service is something the application does in response to some input or set of inputs. Our example calculator involves two services: multiply and divide.

Service-level testing is about testing the services of an application separately from its user interface. So instead of running a dozen or so multiplication test cases through the calculator’s user interface, we instead perform those tests at the service level.

Where many organizations have gone wrong in their test automation efforts over the years has been in ignoring this whole middle layer of service testing. Although automated unit testing is wonderful, it can cover only so much of an application’s testing needs. Without service-level testing to fill the gap between unit and user interface testing, all other testing ends up being performed through the user interface, resulting in tests that are expensive to run, expensive to write, and brittle.

For more on Scrum and agile testing, pick up a copy of Succeeding with Agile.

Just as there are many reasons why some people will resist Scrum, there are many ways someone might resist. One person may resist with well-reasoned logic and fierce arguments. Another may resist by quietly sabotaging the change effort. Still another may resist by quietly ignoring the change, working the old way as much as possible, and waiting for the next change du jour to come along and sweep Scrum away.

Each act of resistance carries with it information about how people feel about adopting Scrum. As a change agent or leader in the organization, your goal should be to understand the root cause of an individual’s resistance, learn from it, and then help the person overcome it. There are many techniques you can use for doing this. But unless the technique is carefully chosen, it is unlikely to have the desired effect. To help select the right technique, I find it useful to think about how and why someone is resisting. We can group the reasons why someone is resisting Scrum into two general categories:

• They like the status quo.
• They don’t like Scrum.

Reasons for resistance fall into the first category if they are actually a defense of the current approach. This type of resistance to changing to Scrum would likely result no matter what type of change was being contemplated. Reasons fall into the second category if they are arguments against the specific implications of beginning to work in an agile manner.

Categorizing how individuals resist is even simpler: Is the resistance active or passive? Active resistance occurs when someone takes a specific action intended to impede or derail the transition to Scrum. Passive resistance occurs when someone fails to take a specific action, usually after saying he will.

The figure below combines the whys and hows of resistance into four general types of resistance, each with a name descriptive of the person who resists in the way indicated by the labels on the axes.

A skeptic is someone who does not agree with the principles or practices of Scrum but who only passively resists the transition. Skeptics are the ones who politely argue against Scrum, forget to attend the daily scrum a little too often, and so on. I am referring here to individuals who are truly trying to stop the transition, not people with the healthy attitude of “this sounds different from anything I’ve done before but I’m intrigued. Let’s give it a try and see if it works.”

Shown above the skeptics are the saboteurs. Like skeptics, saboteurs resist the transition more from a dislike of Scrum than support for whatever software development process exists currently. Unlike a skeptic, a saboteur provides active resistance by trying to undermine the transition effort, perhaps by continuing to write lengthy up-front design documents, and so on.

On the left side of the figure are those who resist because they like the status quo. They are comfortable with their current activities, prestige, and coworkers. In principle, these individuals may not be opposed to Scrum; they are, however, opposed to any change that puts their current situation at risk. Those who like the status quo and who actively resist changing from it are known as diehards. They often attempt to prevent the transition by rallying others to their cause.

The bottom left of the figure shows the followers, who like the status quo and resist changing from it passively. Followers are usually not enraged by the prospect of change, so they do little more than hope it passes like a fad. They need to be shown that Scrum has become the new status quo.

When introducing a complex change into a large organization, resistance will be inevitable. What isn’t inevitable is the reaction of an organization’s leaders to that resistance. In a 1969 Harvard Business Review article, Paul Lawrence describes an appropriate response to resistance.

When resistance does appear, it should not be thought of as something to be overcome. Instead, it can best be thought of as a useful red flag—a signal that something is going wrong….Therefore, when resistance appears, it is time to listen carefully to find out what the trouble is. What is needed is not a long harangue on the logics of the new recommendations but a careful exploration of the difficulty.

While its important to confront resistance, be careful not to create an atmosphere of “us” against “them.” The real goal is to create a feeling that the transition to Scrum is inevitable and that, as the Borg of Star Trek taught us, “resistance is futile.” The need to foster this atmosphere does not give you carte blanche to ignore the feelings and reactions of employees or to steamroll Scrum into an organization. Instead, to paraphrase Nigel Nicholson in a 2003 article for the Harvard Business Review, when an employee resists, an effective leader looks at the employee not as a problem to be solved but as a person to be understood.

For more help with overcoming obstacles to your agile transition, see Chapter 6 of Succeeding with Agile.

The project was to be delivered into multiple call centers with more than 300 nurses on the phones. It was to replace a quirky but familiar system that the company was rapidly outgrowing. The nurses’ expectations of what the new system would deliver were sky high. In monthly sprint reviews with the nurses, I was routinely shocked by what they’d come to expect, some of which wasn’t even technically feasible.

With about three months left on the year-long project, I realized my focus had to change. From then on, I spent almost all of my time on expectations management. I met with nurses in each of the call centers and described exactly what would and would not be in the delivered system. I toned down their expectations about the system’s impact on world peace, global warming, and personal weight loss. Without this effort, the product would have been perceived as a failure.

Since that project, I have been acutely aware of the importance of expectations management to the overall success of any project. Setting and managing expectations is perhaps even more important at the start of a major shift such as adopting an agile project management approach like Scrum. In initiating a transition to Scrum, I find it helpful to set and manage expectations about four things:

• How quickly teams will improve
• How long it will take to gain additional predictability from the team’s new way of working
• How there will almost always come a time when turning back looks easier than sticking with it
• The level of involvement in the transition that will be necessary from various stakeholders and organization leaders

By properly setting expectations you can avoid the problem of having an otherwise successful transition or project sunk by unrealistic expectations.

More details about setting and managing expectations can be found in Chapter 5 of Succeeding with Agile.