Giving birth to Agility and parenting it to pre-pubescence was a miraculous feat. The thought leaders who brought us the manifesto and subsequent culture shift deserve our thanks for seeing the need and creating the right message at the right time. Thanks to these revolutionaries Agile is not a footnote, but the most promising path forward to improving our profession.

One focus of the last 10 years of agile discussion has been, “There are better ways to develop software.” While the state of spaghetti code in the universe is still a big problem, progress has been made in this area. It is no longer radical to think in terms of test-first, pattern-based, or *-driven. Many teams are just acting more professionally, and that’s a wonderful thing.

The other primary focus of the last 10 years has been teaching technologists to actually speak human. You know, with emotion and stuff. And there’s great news; as a profession, we developers aren’t as jerky now as we were 10 years ago. See? It’s working.

The Agile Consensus

I get to see many implementations of Agile Software Development. Not surprisingly, most teams out there aren’t living the dream, but are actively trying to improve. Indeed, recent studies and surveys have noted that projects using Agile methods now outnumber plan-driven, or waterfall, projects.

Sam Guckenheimer calls this change the Agile Consensus in his recent book with Neno Loje (disclosure: I was a technical reviewer on this book). The idea behind the Agile Consensus is simply this: Agile won. Creating software with plan-driven techniques obviously falls short of the advantages of developing with a focus on humanity and exploiting shorter feedback cycles.

Bottom line: Agility has crossed the chasm and is no longer only the domain of developers. The enterprise wants big-A Agility.

Not Crazy Anymore

Whether or not a given practice is crazy or “edgy” depends on who you talk to. Organizational acceptance of a given practice is typically rooted in the values that practice supports, and the practices here seem to have reached a level of general acceptance in our industry. Mostly.

Daily Team Meeting

Whether you call it the Daily Scrum, the Daily Standup, or simply a daily team meeting, the practice of a quick, informal team meeting held at the same time and place each day has really caught on. The reason for this is simple; when done well, the daily team meeting helps teams be more productive and improves situational awareness in complex environments. And regardless of what kind of software development you do, odds are it is fairly complex.

The daily team meeting is one of the most misunderstood and poorly executed practices mentioned in this article, but I digress.

Bottom Line: Daily Team Meetings are commonplace and useful. They aren’t considered crazy by most teams anymore.

Sprints

Sprints, or iterations, are simply short periods of dedicated time within which teams will deliver working software. These time boxes in which teams agree to deliver something have changed the way we think about long, death march projects. Many teams know that progressing toward the broader goal of releasing software is often best managed by delivering working software all the way through a delivery pipeline, with increasing amounts of functionality each time.

Businesses leaders often love Sprints because Sprints are an obvious way to manage risk. More on this later.

Bottom line: While there are other ways to organize work, doing it in small batches within a Sprint of 30 days or less is a model that has proven itself time and again.

Test as We Develop

Lean thinking encourages “testing at the point of work”, which is a broad concept we’ve apply to software development to derive TDD, BDD, ATDD, and other forms of simply proving that software works as we create it. We’ve learned along the way that we don’t even need to think of this as a verification process. Test-First practices have proven to not only help developers build the “right” software, but to build the right software well.

Not every developer has drunk this Koolaid, but most understand the basic value behind Test-First as either a design tool or a verification tool. Those of us who really drank deeply see Test-First development as the de facto way to write code. Sure, I’ll bang out a quick shell script without an accompanying automated test harness, but that’s a simple fit-for-purpose decision.

Bottom line: When making software that needs to be work right and be crafted well, many developers see Test-First practices as indispensable.

Deliver Frequently

Delivering working software frequently allows development teams to actually deliver something, and that’s half the battle. Frequent delivery of working software enables the most valuable feedback loop in software development. The conversation around this feedback loop is simple and sounds like this:

“Here’s what we made. Here is how it works. What do you think?”

And then comes the tricky part – Actually listening to the response. This helps teams build the right thing next. I loved watching eBay a few years back as it changed its color scheme and skin layout gradually over a few months. They design a destination look and feel and migrated to it with frequent delivery of small changes in their UI. This allowed the product to evolve, rather than re-release. This is how software delivery is evolving and the most extreme form is called “Continuous Delivery.”

Delivering frequently may sound a lot like “Sprints”, but there is more going on here. Sprints are simply a forcing function for frequent delivery. There are other ways to pull it off. Frequent delivery all the way to customer feedback can change the way a company engages its customers and plans its strategic moves.

Bottom line: Frequent delivery is craved by executives for the business advantages offered and by technical teams because it makes actually shipping ubiquitous.

Still Crazy

While some Agile practices have crossed into “just plain old good ideas,” many are still seen as edgy, or extreme. Despite evidence that these practices offer real value and better alternatives to traditional thinking, the old ways of looking at the world are just so ingrained that these practices provide fodder to skeptics.

Pairing

No technical practice has drawn more fire than Pair Programming. Hard data has begun to emerge about the practice of pairing, and all that data shows (to varying degrees) how pairing creates higher quality and simply better software. A paper

There are also a ton of human advantages, like increased learning, knowledge sharing, and removing single points of failure within a team.

Why then has formal pairing been relegated to the domain of roman sandal wearing hippie agilistas? Most development team leaders or managers simply see pairing as an investment of two people doing what one could accomplish. I won’t try and convince you otherwise in this article, but I will mention this:

Barry Bohm has made a very distinguished career of studying software development. In Balancing Agility and Discipline, he asserts that 60% of all software defects in production could have been caught with a peer review. Pair programming is continuous peer review. You do the math.

Finally, most developers treasure their alone-time with the code. Sharing the way I approach problems or write code can feel like a job interview every day. That kind of scrutiny can feel very uncomfortable unless I am in an environment of absolute trust. That ties the success of this technical practice to the culture of the team and company.

Bottom line: Pair programming is still seen as eXtreme, and the transparency it forces can terrify many developers.

Funding Alternatives

Companies spend a lot of time and energy developing golden plans for the next year. Strategic planning is a dependable activity of middle-management in those months counting down to the end of the current fiscal year.

We know good and well that we can’t predict the evolution of a software project beyond a few months in most thriving businesses. Change just happens. Why then do we persist in thinking Big Funding Up Front is any different than Big Design Up Front? Some are making inroads with models of T&M funding, fixed cost, adjustable scope, and other techniques like incremental funding. However, for the most part we remain stuck in annual funding models because business Agility, the real promise of Agile, remains elusive.

Bottom line: Software development projects are still funded when we know the least about how we’ll be spending that money.

Strategic Iteration

While Sprints, or iterations, are very popular on the operations side of the house, few companies see them as the strategic advantage they really are. Sprints are loved by the business because they reduce risk, but actually refining the scope, plans, and functionality based on an iterative feedback model is a foreign idea. Iterative delivery provides a regular cadence that can be interpreted as “milestones” by traditionally trained most project managers.

The innovation companies could have with regular Sprints is lost because of the aforementioned Big Up Front Funding that causes Sprints to be seen as a tool of operations.

Bottom line: The potential value of iterative incremental teams is being wasted by a determination to fund fixed-scope projects up front.

The Next Challenge

Professionalism

The profession of software development is reflecting upon itself right now and the question of what it means to be a software professional is coming to a head. The craftsmanship movement has a genuine toehold with many introspective developers; Universities are actively looking beyond computer science programs to fill the supply void of industry; and my mom thinks she’s “writing code” when her excel macro runs without error.

Writing solid code is now table stakes for being a software professional. The expectations we have of true professionals are becoming appropriately greater. As technology matures and abstractions go higher, the productivity of development teams should be through the roof. Yet, it isn’t necessarily the case and hiring organizations are desperate for some way to assess prospective developers en masse.

One desperate attempt at identifying professionals is the ridiculous history of Scrum certification. Certification teases with the allure of simply trusting a credential. Unfortunately, this isn’t working for any known certification yet, university, private, or otherwise.

Bottom line: Professionalism in software is finally being demanded by those creating it, and by those asking for it.

Maturity

Just like the allure of hiring a professional, the temptation of the perfect development process is just too tempting for the ignorant to ignore. The success of simple frameworks like Scrum and Kanban provide just enough structure to get things done, without providing prescribing specific practices. That scares plan-driven organizations that value control over creativity.

To get real traction with Agile methods means getting not just permission, but support; and support means money. Before bureaucracies spend money, they want assurances and guarantees. False ones will do; look at how well RUP and MSF sold.

Providing any compelling story for change requires supporting data. The willingness of good leaders to instigate and support true change will start with the end in mind. While the end state of an Agile transition can’t be predicted, case studies and measurements of established Agile teams are the catalysts for getting Agile transitions started.

The demand for reassurance will drive development of tools like assessments, maturity models, and formal adoption programs. As older and more established industries explore Agile, these tools will be in heavy demand by those wanting to make data-driven decisions.

Bottom line: Agility is moving into more mature organizations and Agile itself will need more accessories of maturity.

The way our sprint reviews work, everyone in the company gets to rotate between teams who show their work in 15 minute segments. Picture a science fair with parents moving between booths and spending 15 minutes at each one. This has been a hugely successful format because teams get to have more intimate conversations with people seeing their work. Discussions are more interactive because there is a smaller group watching the demo and folks are more likely to speak up.

From the presenter’s point of view, this is showing the same thing 4 times in a row. This is okay, but gets a bit tedious. Worse, because each group brings up individual issues there may be things about the feature that get skipped in a given 15 minute segment.

What I Did

I recorded a feature walk through on Camtasia and played it during sprint review. Four times. This left 3 minutes for discussion, but with the transition times, it pretty much took up the entire 15 minutes.

The most common feedback was, “Cool idea. It was too long.”

Noted. And, I think the idea of doing a complete walk through as a recording has merit because it sort of forces that all areas I want to show will get shown.

It is important to realize that I made the recording a mere hour before Sprint Review. Thus, the software was real and did do what I showed.

What I Will Do Next Time

• Limit the Camtasia video to 5 minutes and leave plenty of time for discussion.
• Be more animated. No monotone voice.

Your Thoughts? Am I a big, fat cheater?

Jeffery weighed in that he prefers low tech note cards on the cork board because of the high tactile nature of using them.

I pointed out that our teams have huge electronic dashboards for use during their daily Scrum meetings, and the dynamic nature of the Scrum board I saw from Conchango would work similarly to a a cork board experience.

“Yeah,” someone interjected, “and you could get a touch screen to touch the work item cards. Touch it, move it, hold it and be mad at it, whatever.”

“That’d be sweet,” I thought.

Jeffrey responded, “You just told me how to spend a whole lot of money so the Scrum master doesn’t have to do 15 minutes of data entry per day and write out the cards.”

Pause.

Pause.

Damn it, he’s right.

The geek in me really wanted to like the bigger, more expensive, technical solution to this problem. You know what, though? The Scrum Master is there to facilitate the team’s success and to be the gate to management.

I will freely admit right here and now that I have been slacking. I was recently working as Scrum Master for a team back home and I taught the team how to use TFS to manage their own work items. I also asked them to update their time remaining on SBLIs before our daily Scrum. You know what? That was wrong of me. That was flat out laziness. They even had a cork board in their work space.

I resolve right here and now that when I get home I will make sure the team has their Scrum board. I will make sure the team doesn’t have to deal with time tracking. I will cover the administration of the team; that’s my job as Scrum Master.

The team needs frictionless tools. They’ll get them. It’s important.

If you are thinking I am declaring project management or cost accounting tools unnecessary, you are dead wrong. The transparency into the team and the overall planning of features and work items is hugely important. I just know it is the job of the Scrum Master to deal with it, not the team’s.

Frankly, I am willing to bet this kind of data management model will result in more accurate and detailed reporting anyway.

Lesson learned. Thanks, Jeff.

Colin Bird, CTO for Conchango, showed the 2.1 version of the process template today. It solved almost 100% of the problems I had been struggling with in my customized versions of the template.

1. Support for different teams on every work item type. This is HUGE because it allows for all teams to pull work from a single backlog.
2. Burn down reports for sprint, product, and release, that all split against teams. This is the biggy. If you haven’t tried working with reporting in TFS, you don’t know how HUGE this is. Awesome.
3. Code churn reporting
4. Value flow diagrams
5. Elimination of the Sprint data element in a work item in favor of using a release model based on iterations. Thank you.
6. Bugs as work items that appear on the product backlog

Honestly, there is very little left to do! I can only think if a few things I will be adding to this already existing functionality.

• Epics and Themes as work items and reports to support them. Even this will get better in Rosario with hierarchical work items.
• Tagging

That’s about it.

Colin also showed us an awesome new product that Conchango will be charging for and I would happily pay for. It is a WPF application that is a Scrum board working right off of TFS directly. There are other attempts at solving this same problem in the open source world, but this one is fully baked and soon to be available. It looks very usable. I can’t wait to see this thing in my shop. It will help the daily standup so much, and the overhead of making out physical cards will be a thing of the past.

I took a picture of it with my cell phone, but I don’t think I was supposed to do that. Oh, what the hell. Here it is.

You can actually drag the cards around and they change state and work remaining. If Colin weren’t English I would have hugged him. And then tickled him.

How does a Project Manager or Product Owner coordinate work across multiple Scrum teams?

There is a prescribed set of practices and techniques we can bring to bear to make scaling Scrum easier and more formulaic to answer this question.

Don’t Get Caught up on “Product”

The more I work with Scrum across the enterprise, the more I loath the word “product” in the Scrum vernacular. The Product Backlog is an essential part of Scrum and other Agile techniques. Although we use the term product, any significant internal system, strategic initiative, or product needs a roadmap and backlog. Since the word product elicits particular pre-conceived ideas , it is important to find a word that can be used in its place. Some people suggest Epic as a reasonable term and I have found great success with it.

Some possible Epics include:

• An internal resume tracking system
• A sales initiative to capture a particular share of a given market
• Moving to another platform for your Internet Data Center
• A MOSS 2007 rollout
• An internal CRM
• A new version of a flagship product
• A major services contract with a long term commitment

Get it? Any really big deal that will be around for a long time and generate a big piles of code, work, or revenue can be an Epic. Now that we can see what our Epics are, what do we do with them?

Start With a Roadmap

An Epic Roadmap is a clear, concise, and ubiquitously available document that provides a guidance for general features or accomplishments over the course of time. A good roadmap is essential to scaling Scrum, providing long to medium term direction, and communicating with stakeholders and clients.

Here is a fictitious Epic Roadmap for a year in the life of MOSS at a major organization.

And Then Create a Backlog that Supports the Roadmap

The collection of Epic Backlog Items can be written in sets, each set directly supporting one of the significant areas of work in the roadmap. For example, if we look at just 2008 Q1 of the above roadmap, there will be some work around introducing the My Sites feature of MOSS to the company. A backlog of work for this feature rollout may look similar to this:

• Configure sustainable authentication and roles based security model for My Sites features.
• Train the company on the My Sites feature.
• Disable personal blog sites capability in Share Point.

Obviously, some of these backlog items are fine grained and some will need to be further decomposed into smaller items later. For example, in a larger organizations, there may be many training sessions for many different teams.

Summary

Although Epics are needed to manage large systems or initiatives, they are really just a large scale backlog item. Epics still foster a very one dimensional, albeit hierarchical, representation of our backlog. This role that Epics play is important because without it, there would be not good way to talk about big systems, or to fund big initiatives.

Slicing through Epics with Themes is a subject we will explore in a later article.

Our organization is fairly new to Agile/SCRUM. We have always struggled to create the master backlog for this project. Along with adapting to a new process, it has been especially challenging for our product owners to consider the 60,000+ customers in which they proxy for.

The project is 16 iterations in (2 week), we are making great progress and are trying to finalize the vision of our first release. Over the past 3 weeks our product owners & scrum masters took over a conference room and flushed out the remaining backlog items for our first delivery. As you can imagine it was a very tedious task, but I for one am extremely excited about the results. No it is not perfect, Yes there will be changes, but its great to visualize what we are trying to do in its entirety.

This last week the teams were brought in to do estimations and tweak stories (combine/split) to complete the backlog. Here are a few photos from the last 15 minutes on Friday as we were finishing up.

Chickens!

Here is our awesome Scrum Master John P in action

Windows are great with sticky cards

We use planning poker (fibonacci seq) for estimations, here is my sweet deck. You also have to take note of the orange with the painted face in the coffee cup. He is our bad mojo orange, so anytime you are getting frustrated there is a big scene and you have to touch the orange to remove your evil thoughts. Though by the end of the week it was kind of getting gross.

And this is what happens when your test engineers are trapped in a room with markers for a long period of time

And of course my favorite card, inspired by “doodle the developer” (Aka Steve J). It works nicely in place of the ‘Infinity’ card.

The backlog items are organized in priority order, determined by the Product Owner (PO). The PO can use any criteria to prioritize the backlog items. Priorities may be driven by risk, return on investment, or client demand. No matter what technique is used, this prioritization is crucial to the use and effectiveness of the backlog.

This all works quite well in a  situation of a single team creating a single product. It even works well when we scale up a bit and have one team working on several products. It just becomes necessary for the two product owners to strike a bargain on how to share team capacity.

It turns out that scaling to even bigger scenarios with an Enterprise Backlog and many teams working from it presents some new challenges. In this model you manage all work in the enterprise in a single backlog. This immediately draws into question what the "real" backlog is. Backlog managers may be interested in a view of the backlog that shows the prioritized list of items for a particular system, a team, a product, a release, or some other grouping.It becomes quickly apparent there are many ways to "see" the backlog.

Which Backlog View Wins?

Obviously, a PO is primarily interested in the view that shows the backlog for a particular product. For a release manager, coordinating the overall release of a product suite, the view of items to be included across many products in a big release is the perfect view. If you are a project manager, interested in a theme of work that will affect many products (think the Smart Art feature in Office 2007), you are looking for the theme view. We can think of this problem as dimensions in a multidimensional cube if it helps you BI types.

With all these views of the backlog going on, and each one of theme prioritized from 1-n, which one is the actual backlog?

• Theme View
• System View
• Release Cycle View
• Scrum Team View
• Product View

The easy answer is, "They are all important, depending on who you are and what you are interested in." It is true, though, that two of these view are closer to reality than all the others. While most of these views represent what we hope will get done, 2 of them are just a bit closer to what will get done, or has been done.

The Scrum Team View

This view aggregates the work items into a collection that actually represents what the team will be doing. This is the backlog the team will use in Sprint planning as they plan an iteration of work. If you are a theme owner and your work items aren’t showing up in the Scrum Team View, you’re in trouble.

The Release Cycle View

This view (and an associated burndown chart) helps us see the reality of features that are scheduled for release and those that have been completed. This view represents the absolute reality of what we can tell the clients will be in the next release.

Just Get Them

The real truth is that all of the views really are important. The real challenge is in deriving the views in the first place. If you are trying to work with an Enterprise Backlog and haven’t got a good model for segmenting it, you may soon find it unwieldy. Find the right view to help you interact with the requirements and make sure your backlog items provide the data needed to see it. Just remember the 2 views that live closer to where the rubber meets the road.

Here are links to sufficiently ambiguous definitions from some trusted sources, all saying effectively the same thing.

http://www.scrumalliance.org/articles/39-glossary-of-scrum-terms#1122

http://www.mountaingoatsoftware.com/product_owner

More informative is this course description from the Ken Schwaber for his Certified Product Owner Course.

http://www.controlchaos.com/certification/cspo.php

and this course description from Mike Cohn:

http://www.mountaingoatsoftware.com/product_owner_training

There is another question commonly asked in this discussion, though. “Who writes the requirements?”

The idea behind Agile product development is that requirements DO exist, typically in the form of a backlog. The next point is that they are expected to change. In this regard, the Product Owner has the responsibility to continuously and actively manage the requirements. It is easily seen that the only way for a Product Owner to effectively do this is through intimate familiarity with the requirements. While the Product Owner may not have been the person to initially place an item on the Product Backlog, they are accountable to the team for maturing that requirement into an executable state.

Therefore, who is responsible for requirements? Clearly, the Product Owner.

Lastly, if you are still having trouble identifying the Product Owner for a given system, product, project, or initiative, remember this one thing:

"The Product Owner is the one person in an organization responsible for P&L (Profit and Loss) of the work." — Jeff Sutherland

Lean / Agile

• Net Objectives
• EMS Strategies

Scrum

• Ken Schwaber
• Mountain Goat Software (Mike Cohn)
• Danube

Software Development Practices

• Object Mentor
• Net Objectives

See the contradiction?

These folks (and I have met many over the years) are quick to point out the flaws, inconsistencies, and downright disfunction of their current waterfall process.

“We get the requirements from the Business Analysts,” they say, “and they are never right.”

“We build an deliver to QA and move on to the next project,” they say, “and it always comes back with defects.”

“There is no such thing as defect-free software,” they say, “an in their world they are right.”

“It hurts when I hit my finger with the hammer,” they say while placing their index finger atop the next nail.

These same folks will dismiss Agile and iterative development practices in the next breath with words like “impractical”, “unrealistic”, or “chaotic”. Is this fear of change? Is it threatening to have the system you hate so much be threatened by an up-and-coming idea? This seems to me a bit like hating the king and leaving him in power for fear of who might will replace him.

This is one of the reasons I find it terribly important to discuss not just the benefits of all things Agile like Scrum and TDD, but to also introduce the history and practices of failed engineering projects that caused these things to emerge. This provides a pathway to enlightenment for the most diehard anti-Agilista. Once we can establish the history, theory, demonstrated success, and techniques, engineering minded folks are more able to see these Agile practices as real tools for their toolbox.

Another useful technique is to start these folks off with engineering process first, not methodology changes. Almost everyone can agree to the demonstrated value of TDD. Once a surly developer sees TDD as a small, iterative feedback loop, we can start asking why the next level of the feedback loop isnt’t happening with automated builds. Then Continuous Integration. Then iterative delivery to a product owner. Then Agile because by now they are already there without knowing it.

Leading with process in a top down implementation can be ineffective when selling to grizzled engineers. Let’s help people stop hitting their fingers before we try to get them to build something beautiful.