Sunday, August 2, 2009

Releasing, Branches, and Agile Teams

When Brad Appleton and I wrote the SCM Patterns book we discussed two branching patterns related to releasing software: Release Line and Release Prep Codeline. The first, is a pattern that allows teams to easily support released code. The second is a pattern to help you "stabilize" a codeline before a release while allowing parallel work for the next release to proceed. Too often teams start Release Prep Codelines at the wrong time or for the wrong reasons, without understanding the costs . Given the rate at which changes happen on agile codelines, the costs of creating a branch can be large if do it for the wrong reasons.

If an ideal agile team had "shippable" software at the end of every iteration, a Release-Prep codeline isn't necessary. You just label your code at the end of the final sprint, and if you need to provide a fix that can't be done from the mainline code, you start a release branch from that label.

Many teams can't ship at the end of every sprint, creating a Release Prep Codeline (branch) is a useful: it avoid some poor alternatives, like having a code freeze. The branch can receive small changes to fix problems, and the mainline can add new features, refactor, and integrate the fixes from the release-prep branch.

As the time between when the branch is created and the project is released grows, the cost of merging changes between the branch and the Mainline increases because the source code diverges. This decreases the velocity of the team and can make the time to release grow more.
A long interval between branching and release often happens for reasons like:
  • Quality issues. There are a lot of problems, so going from "feature complete" to "shippable" takes longer than expected.
  • "Code Freeze" happens before "Feature Freeze." Not explicitly, but after the branch is created you identify new "must-have" features. This gets worse as the time between branch and ship increases.
So what is a team to do? Here are some suggestions:
  • Be agile and prioritize: If the release is the most important task, do that work on the mainline, and have everyone work on it. Don't branch until you are ready to ship.
  • Add automated tests early. Try to be "ready to ship at the end of the sprint," so you can avoid the costs of branching.
  • Don't branch until you really are feature complete, and use the Release-Prep Branch only for a constrained set of fixes.
If you really need to start future work before the current release is ready to ship consider either:
  • Doing all work on the main line and isolate "future work" by architectural techniques. (Make new features plugins, for example).
  • Keeping the work that is highest priority on the main line, and create a Task Branch for the future activity. Those on the task branch are responsible for merging main line code into their branch, and when the release is done, the task branch get copied to the main line.
My tendency is to want to keep the highest priority work on the main line as long as possible. Usually the code you are about to release meets this criterion.

Technically, these are simple technical approaches to implement. Like all SCM issue, there a are many non-technical issues to address. Product owners need to understand the costs of "working in parallel" and an agile team is responsible for making sure that the product owners know these costs so that they can make the correct decisions about when to start a parallel work effort.

How has your team addressed pre-release and release branches? If you read this and have an opinion, please comment!

3 comments:

Michael Hüttermann said...

Very good post. My favorite strategy is to avoid branches altogether. ;-) .. and create them if needed ex post. This works pretty good if you have an optimal Agile flow. If a bug is detected it is becoming part of the next release. Critical bugs can be detected early enough through CI.

Mario Moreira said...

When it is a new product, mainline is the way to go. The challenge becomes when once a release is placed in production and there is a need to support the next release and maintenance/bug fix work. Then the question becomes how module people want and whether a separate "next" release branch or patch branch is needed (or both). Althought keeping branching to a minimum is always recommended. However, you indicate the challenges that people have of getting testing up to speed enough to be able to get the code release ready amongst others. When an organization adds stages after a product is release-ready (convergence, beta, etc.) on mainline or the project branch, then there may be a need for a pre-release branch, although why they wouldn't just use the mainline or project branch is questionable applying labels or timestamps as needed.

Constantine Vasilyev said...

Steve, good post. I agree that branching should be avoided, whenever possible. The necessity to branch, and complications related to code maintenance, in my opinion more often than not have to do with the level of competence of the software engineers - not with the process or methodology adopted on the project. The idea of Agile development is great. However, like any methodology or tool, it only works if used by competent professionals. No process, methodology, or framework can help if architects and programmers are not capable of good design.

In my opinion, the best strategy to ensure a successful agile development process is to

1) hire only competent software engineers capable of truly modular design;
2) keep designs modular and pluggable so that pieces that are work-in-progress may be easily stubbed and later swapped for (well tested individually) actual implementations;
3) keep the system in an intellectually manageable and stable state at any given moment, at any iteration. It certainly doesn't have to be "shippable" after each iteration - if not all features are complete. But the Intellectual Manageability of software at any stage of development is an absolute necessity. (Edsger Dijkstra first suggested this in his "Humble Programmer" Turing Award acceptance speech in 1972, but very few people seem to have listened.)

If the software system at any given moment is represented by a set of intellectually manageable independent/swappable modules/components with minimal couplings (which is always possible but requires a true software engineer to design), maintenance (including the necessity for branching) becomes a trivial issue. Unfortunately, only a small percentage of software architects and programmers today are capable of designing software in such a way.

Site Reliability Engineering; The Book and The Practices

Site Reliability Engineering It’s difficult to walk into a software development organization without hearing about the discipline of Site ...