Team Safari – Beating the Central Services Team Problem


The Central Services Problem

In large enterprises, a customer-facing product development team often has dependencies on other teams for frameworks, underlying systems, and (very often) centralized platform APIs. The most common scenario in large enterprises is a central Platform Services team that provides APIs to product teams for access to underlying data and service workflows.

The point of this post is not to critique that model, but instead to accept this model exists and work with it as best we can. These central Platform Service APIs are a product in themselves, with customer-facing product teams as customers. 

Delivering a full slice of functionality of a given feature is the only true example of delivering value, but this can be challenging when team dependencies like this exist, which they often when dealing with scale. This central services model can produce frustrated customer-facing application development teams because they may be waiting on a Platform Services API team to provide those services or endpoints they depend on. So, what does the product team do? They can:

  1. Mock out the Platform Services API based on proposed contract (which may change)
  2. Put in an abstraction layer over the Platform APIs that would be unnecessary should the services be directly available, because it’s easier to mock and doesn’t predict service contracts. Then mock that, adding a layer of premature abstraction to your design.
  3. Send the Product Owner over to fight for positioning in the Platform Service team’s backlog. (This should probably happen anyway, but let’s assume that fails because the product team needs aren’t at the top of the list).
  4. Submit a Pull Request to the Platform Service team’s repository to provide the service capabilities the product team needs.

Ding, ding, ding. That last one is the right answer. The product-facing team submits a PR to the Platform Services API repository. What a great solution! The ultimate prescription for success here is to use an internal open-source workflow model and submit code to the Platform Services team to accept into their codebase.

Then the PR is rejected for a myriad of reasons that boil down to the PR-issuing team not understanding the conventions and policies of developing code for the Platform Services team APIs.

A Team Safari

Consider a Team Safari (we call these Agile Safaris at GoDaddy).

A Team Safari is when one or more members of a team leave their home team to join with another team for a Sprint or two to solve a problem and learn more about that team’s software systems.

Why would a we do this? After all, it decreases the capacity of the team left behind for the time the team-member is away. Further, the receiving team capacity is eaten up teaching the new developer how everything works.

The short answer is everyone benefits from high-quality PR being accepted. If the product-facing team can issue PRs that get accepted to the Platform Services team, everyone benefits. The Platform Services team grows its API services footprint. The customer-facing team gets the service it needs.

A Successful Safari Recipe

When a Team Safari is used, there are some ground rules to help things go much better than for a developer to just move a desk over and announce, “Here I am! Safari me!” In addition to core platform capabilities, the visiting engineer will greatly benefit from gaining a working knowledge of all aspect of the visited team practices, patterns, underlying architectures, etc.

First and foremost, ensure the Engineer on Safari is there to solve a specific problem, or learn a specific skill. This should be summarized in a clear and concise “purpose statement” everyone involved understands.

If the following are approached methodically, once the engineer returns to the home team, he or she will later be able to issue successful PRs to the team visited. The following knowledge will help ensure those PRs are accepted the first time.

  1. Standards, conventions, and architectural guidelines of the Platform Services APIs public interfaces.
  2. Standards, conventions, and architectural guidelines of underlying systems capabilities including data access, logging, security, etc.
  3. The underlying language (probably a prerequisite to the Team Safari anyway), documentation, and coding standards.
  4. Full understanding of the CICD team’s CICD pipeline, to ensure appropriate unit and integration tests are included with the PR and pass appropriately.
  5. The specific test-first or other developer testing practices used by the visited team.
  6. The greater architecture (if applicable) of the visited system, framework, or technology if appropriate. Not just solving for the stated purpose, but doing so in the visited team’s context.
  7. Any teams and system the Platform Services team depends on, and how those dependencies relate to each other. 

Paying Back on the Investment

Not only does the Team Safari inform the developer who went on Safari, but that knowledge eventually gets transferred back into the home, or product-facing team then the developer returns.

And now, we don’t have to wait for the Platform Service team to work our Product Backlogs to the top of the queue. The newly informed team has someone who can create real PRs that are accepted and deliver on the promise of working software each Sprint.

As stated, the visited team now has the added capability the accepted PRs provide. Everybody wins.

Do you and your teams use Team Safaris or something like them? Let e know in the comments.

Leftover Work in Scrum

I recently received an email from someone who had taken one of my Scrum courses on Pluralsight. He asked:

  1. For the next Sprint, how is this unfinished User Story/Documentation task handled and is the User Story re-estimated for its remainder to arrive at Story Points?
  2. If the User Story is not considered for Sprint 1 Velocity given that it is not complete, and if its considered entirely in the second sprint when complete, this will skew the Velocity. As organizations love to measure Velocity and would like this to be fairly consistent, it does not help. Wouldn’t it help to have the story points prorated for Velocity discussions? If Documentation would have taken 2 % (we can use task estimations in hours to find out the relative contribution), we can consider 98% of the story points for velocity. What do you think?


This was my reply to this poor soul who is clearly being beaten down by a manager or someone else who doesn’t understand what this whole velocity thing is all about.

I do not advocate for re-estimating the work. Just leave it on the board and let it finish, taking the Story Points in the next Sprint. It will all come out in the wash. Just try to limit this by learning to plan better to your team’s capacity.
Trying to arrive at a consistent velocity is a dysfunction. Take the averages and don’t work about Sprints being consistent. What happens when people go on vacation, become sick, etc.? Velocity is a historical record and lagging indicator. It is a poor predictor as a leading indicator because of the reality of life.
I hope this makes sense. In short: Don’t let the process beat you to death. Let it flow and don’t worry so much about perfection 🙂

Mac – Install Xcode for git?!

It is so lame that I have to install Xcode to get git to work on the command line. I did’t have this problem last time I used git, but I just a few days ago upgraded to Sierra and I received this message while trying to clone a repository from GitHub.

git clone error: invalid active developer path (/Library/Developer/CommandLineTools), missing xcrun at: /Library/Developer/CommandLineTools/usr/bin/xcrun

Aaaand, lame. I don’t think I’ve ever read a really long license agreement via a terminal window, btw. That was a thing to behold.

That said, once I installed the tools, Xcode reqs, everything was peachy and worked fine.

Here’s to doing that in the shell:

xcode-select --install

Or if you already have Xcode

xcode-select --reset

Yep, I’m in Mac all the way now. Instead of griping about MSFT tools, I get to gripe about other tools 🙂 Stay tuned.

Starting the New Year with Less

I would like to make a more concerted effort to post a bit more. Not necessarily huge, rambling tomes of articles, but short and meaningful drips of things I learn, experience, and encounter along my professional path.

So, first things first! I am starting the new year with an old technology purge. This is actually taking longer than I’d thought because of all the unidentifiable cords, power adapters, and other detritus that’s piled up in my office over the years. The three things I did manage to accomplish yesterday were to:

  1. Admit my old Windows Home Server adds no value and put it on the “take to e-cycling” shelf in the garage. I pulled all photos from it, out them up in my personal OneDrive, and pulled the plug.
  2. Gather together all my old cords, mice with wires, some mice without wires, 4G thumb drives, and other do-dads. Those still need to be sorted so I can put most of that pile on the “take to e-cycling” pile, too.
  3. Pull all my old books I will no longer receive value in owning from my shelves. Put them in a box and was pretty stunned at how much it weighs. I guess I’ll donate these to Goodwill or something, but below is a picture of my first real book purge.

Book Purge

I hope this is a good start to the year.

What can you purge from your life that would make things feel lighter for you?

Attributes of a great development team member

My boss’s boss recently asked me, “If an Agile leader were to outline the top characteristic of their best developers, what are they? “

The response was relatively quick, and rather “off the cuff”. After re-reading it, I like my answer!

  1. Seeks to help others with any aspect of product development, regardless of her individual specialty
  2. Independently pursues continuous learning and shares new knowledge with the team
  3. Encourages and enacts technical practices to improve code and product quality
  4. Collaboratively and respectfully engages during all Scrum events
  5. Pro-actively seeks and provides feedback on code and the product being created
  6. Encourages, coaches, and embodies team-level agility
  7. Respectfully collaborates with other teams to ensure valuable features are DONE within each Sprint
  8. Embraces and actively supports organizational-level agility
  9. Enacts and coaches product instrumentation enabling Build > Measure > Learn

What’s missing from this list?

What could be improved in the original list?

nodemon leaving Node.js process running

nodemon is a great tool for use when writing Node.js applications. It is a simple little tool that simply watches for changes in the file system of your project and restarts the Node.js application when a file changes.

Of course, there are more options, but that’s the essential idea. Unfortunately, once you launch nodemon it is off and running and the only way to stop it is to brek out of the command shell with CTRL+C.

Doing this can leave a running instance of your Node.js applciation running as a background process. HUGELY annoying, especially when your app is a long-running application, like a web application.

Fortunately, nodemon subscribes to a couple of Node events and we can tap into those if we wrap nodemon in a JavaScript task runner like Gulp.

Below is my simple gulpfile.js which starts nodemon and has it watch for certain events that will be emitted when shutting down the nodemon process. This catches the exiting nodemon process and kills the Node.js process it spawned.

Is Agile still applicable?

I just read this interview article with Bob Martin and I have a few thoughts about it.

Uncle Bob is extremely code-focused and always has been in his career. This isn’t a criticism, far from it, but one must take that into account when reading his opinions on such matters. He is heavily involved in the Software Craftsmanship movement, which he sees as a follow-on to the Agile Manifesto itself.

Like any business practice, Agile has evolved and matured in directions the original 17 signatories never imagined . Others people beyond the original signatories have done a lot to further the principles into techniques like Scrum, Kanban, DevOps, etc. This is to be expected, imo, as something matures. Even Lean has evolved far beyond Demming’s orginal models.

Though the work of others in the field, we’ve learned that the original 4 values and 12 principles are applicable outside the context of software development. The primary direction it has evolved is toward frequent delivery with applied empiricism to use data in guiding decisions. We are beginning to do this quite well as an industry, imo. At GoDaddy, we are even using Agile practices to manage the work of our training team and that has been quite successful.

I would offer that his views are, of course, quite opinionated and one would likely hear different opinions from the other 16 original signatories. I would even call his views on Agile slightly dated.

In summary, Agile principles can indeed be applied to business practices within the right contexts and we’ve seen through evidence at other organizations (like Intuit, GE Medical, and others) that Agile practices can indeed scale. It’s not easy to do, of course, because scaling ANYTHING is difficult and more error prone than small scale efforts.

I respect and appreciate Bob Martin’s opinions, but in reading this particular interview I think they should be tempered with the evidence we have that shows Agile’s success beyond its original intent.

Memo on O-Ring and Software Erosion

One of the most fascinating documents I’ve read to date is the memo from Roger Boisjoly on O-Ring Erosion. The original target audience for this memo he’d written were the management folks of Morton Thiokol back in 1985, about six months before the Challenger disaster.

What I find so striking about this whole story is its resemblance to the field of software engineering. We software developers can relate to this story all too well. I’ve personally been in this situation more than once. Heck, some of us are in similar kinds of situations all the time.

Roger Boisjoly attempted to halt the launch of the Space Shuttle, unfortunately to no avail. In his particular case, a failure of the Space Shuttle would cost the lives of the astronauts on board. But what about the software that we are crafting? In most cases, a failure of the software would not result in the loss of human life. It would cost the company a particular amount of money. Worst case scenario, its reputation would be flushed down the drain alongside the money.

But what should we do when we encounter reliability issues in the software that we’re working on? Well, first of all, we make an assessment about the severity and the impact of these problems and fix them accordingly. But what if it takes us a couple of days, a week, two weeks or even a few months to fix things? Well, then we usually turn to the management staff, explaining in our best non-technical terms what’s going on and ask them for the time (aka budget) we need. But here comes the tricky part. What if they don’t want to listen? What if they don’t take us serious? What if they don’t trust us that our findings are correct? What if they simply ignore us, wondering why we still have the energy to even come talk to them after or during this dead march that we’re in? What if they do listen to our plea, wholeheartedly agree with everything we say, walk us back to the hallway and then simply don’t give us anything except more business requirements and feature requests?

But now on to the million-dollar question. What do we software developers do when we don’t get the time to do the right thing? Well, we could try to fix things in our spare time, sacrificing the precious time we have with our family and friends or perhaps even our sleep for the next couple of days, weeks, months, perhaps even years? I don’t think that this is very sustainable. Or we could simply ignore management, stop adding new features and business capabilities and start fixing things on our own behalf, like some kind of software development team mutiny? I guess this will end up with one or more developers getting fired. Or we could start a campaign on social media about how software developers are being wronged at this or that particular company. Would anyone even care? And this will probably end up in some layoffs as well.

So what should we as professional software developers do to make things right in these kinds of scenarios? Maybe the people of the software craftsmanship movement could draft a script with concrete actions that guide software developers when they find themselves in the same situation as Roger Boisjoly?

I can only wonder what would happen to our industry if only 10% of all software developers would follow the actions described in such a manifesto. Wouldn’t that be useful advice?