Now that the term “Agile” is sufficiently compromised as to be near meaningless, Agile Software Development is old enough to stand on its own, make its own business case, and demonstrate its value. But it still isn’t a mature adult. Agile Software Development is a hormonally unbalanced pre-teen with ugly spots, occasional outbursts of irrational anger, and the promising potential of smart-assed intelligence.
Giving birth to Agility and parenting it to pre-pubescence was a miraculous feat. The thought leaders who brought us the manifesto and subsequent culture shift deserve our thanks for seeing the need and creating the right message at the right time. Thanks to these revolutionaries Agile is not a footnote, but the most promising path forward to improving our profession.
One focus of the last 10 years of agile discussion has been, “There are better ways to develop software.” While the state of spaghetti code in the universe is still a big problem, progress has been made in this area. It is no longer radical to think in terms of test-first, pattern-based, or *-driven. Many teams are just acting more professionally, and that’s a wonderful thing.
The other primary focus of the last 10 years has been teaching technologists to actually speak human. You know, with emotion and stuff. And there’s great news; as a profession, we developers aren’t as jerky now as we were 10 years ago. See? It’s working.
The Agile Consensus
I get to see many implementations of Agile Software Development. Not surprisingly, most teams out there aren’t living the dream, but are actively trying to improve. Indeed, recent studies and surveys have noted that projects using Agile methods now outnumber plan-driven, or waterfall, projects.
Sam Guckenheimer calls this change the Agile Consensus in his recent book with Neno Loje (disclosure: I was a technical reviewer on this book). The idea behind the Agile Consensus is simply this: Agile won. Creating software with plan-driven techniques obviously falls short of the advantages of developing with a focus on humanity and exploiting shorter feedback cycles.
Bottom line: Agility has crossed the chasm and is no longer only the domain of developers. The enterprise wants big-A Agility.
Not Crazy Anymore
Whether or not a given practice is crazy or “edgy” depends on who you talk to. Organizational acceptance of a given practice is typically rooted in the values that practice supports, and the practices here seem to have reached a level of general acceptance in our industry. Mostly.
Daily Team Meeting
Whether you call it the Daily Scrum, the Daily Standup, or simply a daily team meeting, the practice of a quick, informal team meeting held at the same time and place each day has really caught on. The reason for this is simple; when done well, the daily team meeting helps teams be more productive and improves situational awareness in complex environments. And regardless of what kind of software development you do, odds are it is fairly complex.
The daily team meeting is one of the most misunderstood and poorly executed practices mentioned in this article, but I digress.
Bottom Line: Daily Team Meetings are commonplace and useful. They aren’t considered crazy by most teams anymore.
Sprints
Sprints, or iterations, are simply short periods of dedicated time within which teams will deliver working software. These time boxes in which teams agree to deliver something have changed the way we think about long, death march projects. Many teams know that progressing toward the broader goal of releasing software is often best managed by delivering working software all the way through a delivery pipeline, with increasing amounts of functionality each time.
Businesses leaders often love Sprints because Sprints are an obvious way to manage risk. More on this later.
Bottom line: While there are other ways to organize work, doing it in small batches within a Sprint of 30 days or less is a model that has proven itself time and again.
Test as We Develop
Lean thinking encourages “testing at the point of work”, which is a broad concept we’ve apply to software development to derive TDD, BDD, ATDD, and other forms of simply proving that software works as we create it. We’ve learned along the way that we don’t even need to think of this as a verification process. Test-First practices have proven to not only help developers build the “right” software, but to build the right software well.
Not every developer has drunk this Koolaid, but most understand the basic value behind Test-First as either a design tool or a verification tool. Those of us who really drank deeply see Test-First development as the de facto way to write code. Sure, I’ll bang out a quick shell script without an accompanying automated test harness, but that’s a simple fit-for-purpose decision.
Bottom line: When making software that needs to be work right and be crafted well, many developers see Test-First practices as indispensable.
Deliver Frequently
Delivering working software frequently allows development teams to actually deliver something, and that’s half the battle. Frequent delivery of working software enables the most valuable feedback loop in software development. The conversation around this feedback loop is simple and sounds like this:
“Here’s what we made. Here is how it works. What do you think?”
And then comes the tricky part – Actually listening to the response. This helps teams build the right thing next. I loved watching eBay a few years back as it changed its color scheme and skin layout gradually over a few months. They design a destination look and feel and migrated to it with frequent delivery of small changes in their UI. This allowed the product to evolve, rather than re-release. This is how software delivery is evolving and the most extreme form is called “Continuous Delivery.”
Delivering frequently may sound a lot like “Sprints”, but there is more going on here. Sprints are simply a forcing function for frequent delivery. There are other ways to pull it off. Frequent delivery all the way to customer feedback can change the way a company engages its customers and plans its strategic moves.
Bottom line: Frequent delivery is craved by executives for the business advantages offered and by technical teams because it makes actually shipping ubiquitous.
Still Crazy
While some Agile practices have crossed into “just plain old good ideas,” many are still seen as edgy, or extreme. Despite evidence that these practices offer real value and better alternatives to traditional thinking, the old ways of looking at the world are just so ingrained that these practices provide fodder to skeptics.
Pairing
No technical practice has drawn more fire than Pair Programming. Hard data has begun to emerge about the practice of pairing, and all that data shows (to varying degrees) how pairing creates higher quality and simply better software. A paper
There are also a ton of human advantages, like increased learning, knowledge sharing, and removing single points of failure within a team.
Why then has formal pairing been relegated to the domain of roman sandal wearing hippie agilistas? Most development team leaders or managers simply see pairing as an investment of two people doing what one could accomplish. I won’t try and convince you otherwise in this article, but I will mention this:
Barry Bohm has made a very distinguished career of studying software development. In Balancing Agility and Discipline, he asserts that 60% of all software defects in production could have been caught with a peer review. Pair programming is continuous peer review. You do the math.
Finally, most developers treasure their alone-time with the code. Sharing the way I approach problems or write code can feel like a job interview every day. That kind of scrutiny can feel very uncomfortable unless I am in an environment of absolute trust. That ties the success of this technical practice to the culture of the team and company.
Bottom line: Pair programming is still seen as eXtreme, and the transparency it forces can terrify many developers.
Funding Alternatives
Companies spend a lot of time and energy developing golden plans for the next year. Strategic planning is a dependable activity of middle-management in those months counting down to the end of the current fiscal year.
We know good and well that we can’t predict the evolution of a software project beyond a few months in most thriving businesses. Change just happens. Why then do we persist in thinking Big Funding Up Front is any different than Big Design Up Front? Some are making inroads with models of T&M funding, fixed cost, adjustable scope, and other techniques like incremental funding. However, for the most part we remain stuck in annual funding models because business Agility, the real promise of Agile, remains elusive.
Bottom line: Software development projects are still funded when we know the least about how we’ll be spending that money.
Strategic Iteration
While Sprints, or iterations, are very popular on the operations side of the house, few companies see them as the strategic advantage they really are. Sprints are loved by the business because they reduce risk, but actually refining the scope, plans, and functionality based on an iterative feedback model is a foreign idea. Iterative delivery provides a regular cadence that can be interpreted as “milestones” by traditionally trained most project managers.
The innovation companies could have with regular Sprints is lost because of the aforementioned Big Up Front Funding that causes Sprints to be seen as a tool of operations.
Bottom line: The potential value of iterative incremental teams is being wasted by a determination to fund fixed-scope projects up front.
The Next Challenge
Professionalism
The profession of software development is reflecting upon itself right now and the question of what it means to be a software professional is coming to a head. The craftsmanship movement has a genuine toehold with many introspective developers; Universities are actively looking beyond computer science programs to fill the supply void of industry; and my mom thinks she’s “writing code” when her excel macro runs without error.
Writing solid code is now table stakes for being a software professional. The expectations we have of true professionals are becoming appropriately greater. As technology matures and abstractions go higher, the productivity of development teams should be through the roof. Yet, it isn’t necessarily the case and hiring organizations are desperate for some way to assess prospective developers en masse.
One desperate attempt at identifying professionals is the ridiculous history of Scrum certification. Certification teases with the allure of simply trusting a credential. Unfortunately, this isn’t working for any known certification yet, university, private, or otherwise.
Bottom line: Professionalism in software is finally being demanded by those creating it, and by those asking for it.
Maturity
Just like the allure of hiring a professional, the temptation of the perfect development process is just too tempting for the ignorant to ignore. The success of simple frameworks like Scrum and Kanban provide just enough structure to get things done, without providing prescribing specific practices. That scares plan-driven organizations that value control over creativity.
To get real traction with Agile methods means getting not just permission, but support; and support means money. Before bureaucracies spend money, they want assurances and guarantees. False ones will do; look at how well RUP and MSF sold.
Providing any compelling story for change requires supporting data. The willingness of good leaders to instigate and support true change will start with the end in mind. While the end state of an Agile transition can’t be predicted, case studies and measurements of established Agile teams are the catalysts for getting Agile transitions started.
The demand for reassurance will drive development of tools like assessments, maturity models, and formal adoption programs. As older and more established industries explore Agile, these tools will be in heavy demand by those wanting to make data-driven decisions.
Bottom line: Agility is moving into more mature organizations and Agile itself will need more accessories of maturity.
Keep fiting the good fight! I’ve recently found your article on the SCC, however, it appears that the domain name is expired. Is this initative still on the move or in remission? I am an independent developer (and software craftsman in the making) and I was wondering if there is some sort of mentoring program, through the CSS or otherwise, for individuals/small businesses dedicated to the craftsmanship movement.
Grea
Great article David! By far the biggest challenge in Agile software development at the moment is the enormous cultural shift required to move to incremental deliveries. This may well prove to be insurmountable in the long run but hopefully business will see the light. In times of “austerity” this may be the perfect storm for Agile to really take off.
(apologies for the partial comment below – itchy trigger finger…)
Great article 😀
I like the “Next Challenge” part a lot. Agile, as a framework/methodology (or whatever you think it is) lacks both professionalism and maturity, and that’s why it’s probably looked upon with disdain from project managers. That needs to be fixed!
The SCC was successful for the immediate members. So much so that 1 year into it, no one had any time to dedicte to the org because everyone had so much work. And that was during the slump!
Too bad, though it is, there was just not enough momentum to keep it moving forward, so it passed away.
It was one of my biggest learning moments. Even good ideas sometimes aren’t going to fly and that’s okay. All I could do was try something else, so I did.
Thanks!
A catastrophic success! Sounds like a good problem to have although, it definately seems to have closed the door to an almost eutopian developer environment!
Can you recomend any big brother/mentor type organizations (or people) that are willing to help a small startup shop follow in your footprints?
I’m the owner of a (very) small programming shop. There isn’t a good deal of professionalism in my area and trying to advocate for the software craftsman movement. I’ve recently started up a web developer user group and I’m constantly learning on my own (blogs, books, podcasts, etc.) However, the lack of a preexisting community often leads to grasping straws.
I’m beginning to implement the ‘good things’ one piece at a time. I am currently working on introducing continuous integration and BDD. I guess what I’m reaching for is someone take us under their wing and give us the benefit of their expertise; I’m talking about a pay-it-forward type of arrangement, of course. Any recommendations or advice?
Thanks!
Great article. I am missing one thing though: Retrospectives. Or continually learning from your mistakes and continually looking at how you can improve your process.
It’s working, but we have such along way to go.
On a recent Developer recruitment drive it was frightening to see how much “just get it done” mentality exists out there among Developers. Not saying they are to blame, but still we need to work much harder to show “the business” that we have a way working that we must follow. Do you see many lawyers taking short-cuts?
I would strongly advise you reach out and find a local coach for your organization. I am offering my services without charge to a local small startup firm. I would reach out at local user groups and such and offer your shop up as a laboratory for agile adoption allowing the coach to use you as a reference and whitepaper source, etc.
I find that:
a. Retros are the most powerful Kaizen tool in a team’s toolbox.
b. Most teams don’t use them.
c. Of the teams that do use retros, most don’t use them well at all.
Bottom line: We still have a long way to go with retros.
David, thank you for sharing your observations. Your assessment is very useful to me, as we begin the 2nd year of “trying to improve”. Reading your assessment, it becomes crystal clear that the hard part is really not technical but psychological. As I see it, if business leaders want change badly enough to understand and support it, implementing TDD and changing the way we plan and budget would fall into place. Minds and behaviors can change through vision, courage, and commitment. Speaking as someone who sees the many advantages of Agile, I ask myself every day what I can do to provide leadership and help others embrace the vision. Again, thanks for reminding me that others keep struggling just as we are.
We’re always trying to improve ourselves and learning from previous mistakes. But I agree, still a long way to go.