Skip to main content


Showing posts from 2009

Continuous Integration of Python Code with Unit Tests and Maven

My main development language is Java, but I also some work in Python for deployment and related tools. Being a big fan of unit testing I write unit tests in Python using PyUnit. Being a big fan of Maven and Continuous Integration, I really want the  Python unit tests to run as part of the build. I wanted to have a solution that met the following criteria:
Used commonly available pluginsKeep the maven structure of test and src files in the appropriate directories.Have the tests run in the test phase and fail the build when the tests fail.
The simplest approach I came up with to do this was to use the Exec Maven Plugin by adding the following configuration to your (python) project's POM.

<plugin> <groupId>org.codehaus.mojo</groupId> <artifactId>exec-maven-plugin</artifactId> <executions> <execution> <configuration> <executable>python</executable> <workingDirectory>src/test/python</workingDirect…

Versions in Maven and Source

I'm a big fan of Maven, a build (and project) management tool. When you are working with Maven, each artifact  that you develop (jar, or war file for example) has a version that's distinct from version in your SCM system.  The Maven Book has a good discussion about how versions are managed, but some there are often questions on projects about how to use Maven versions when you also have the SCM version. "SCM Version 6453" gives more information that "Artifact version 1.1" for example. yet we have 2 version numbers to manage. Here's one approach that works well for simple projects.

I'm assuming that you know the basics of how to specify dependencies in Maven. If you need a quick intro to Maven, see the Maven Book.

If you are developing a project that has artifacts that external clients will incorporate with Maven, you need to change the artifact version with each release, as you  specify the artifact version in the dependency element in your pom. If …

Uncertainty and Agile Requirements

The key value of Agile methods is to help you to manage uncertainty. By being incremental and iterative, you manage risks by not investing a lot of effort in specifying things that may "wrong." At the start of each iteration you can look at what you have and decide that it's the right thing, in which case you can build on it, or the wrong thing, in which case you can try something else. Since you've only invested a small amount of effort relative to the specification you do in a waterfall process, you've wasted less effort and, in the end, money if you are wrong.

This approach of small stories with only some details works really well in many cases. An agile team runs into trouble when the project team confuses "uncertainty" with "vagueness." To be successful, an agile team needs to work off of a backlog that has stories that are precise enough that the team can iterate effectively with the stakeholders at the end of each iteration, and can dev…

Silver Bullets and Simple Solutions

Recently I was reading the Boston Globe and saw a letter to the editor about problems with H1N1 vaccine distribution and lamenting that "We’re eight months into a pandemic, and it seems that all the government can do is tell us to wash our hands!" While I understand the writer's frustration about the availability of vaccines, and the technology used to produce them, I was  struck by the writer's attitude that a simple solution couldn't possibly be effective.  I'm not a medical professional, but from what I've read, hand washing, while mundane sounding, is effective in preventing the spread of disease. Since I  am a software development professional, I also thought that the attitude that the more exotic solution  is always better, is common in software development as well.

When people ask me about improving their release management process to be more agile,  they are disappointed when I don't focus on the latest SCM tool, but rather talk about approach…

Planning Time and Sprint Duration

While having lunch with a friend of mine he mentioned that his team had frequently changing priorities and how the team tried having short (1 week) sprints to be able to adapt to business changes. He discussed how the team felt like the overhead of planning for a 1 week sprint was too high, so the team decided to abandon a sprint model. This conversation reminded me that this kind of question comes up a lot, especially with teams transitioning  to agile.

It makes a lot of sense to tune your sprint length to the rate at which requirements change and the rate at which the team can deliver functionality. Adding work as you go makes it difficult to make commitments and to measure progress, and new "high-priority" work can disrupt flow. If your sprints are 4 weeks long, then there is a greater temptation to add work mid-stream. If a sprint is 1 week long, then it's easier for a Product Owner to be comfortable slotting work into the next sprint.

A sprint isn't just the tim…

Doing Less to Get Things Done

Have you ever been in a situation where someone walks into the room and announces that they just got off the phone with a customer you need to add some functionality, described in very specific terms. As described the feature could take a lot of work, so you bounce around some ideas about how to do what the customer asked for. Along the way you realize that maybe, perhaps, there is another way that you can add a similar feature that meets the needs at much lower cost. But no one asked the customer what problem they wanted to solve. So what do you do now?

Some options are:
Saying, sorry, we don't really know what the requirement is, so come back when you have more to say.Spend the next couple of hours discussing how to implement all of the options you think of, and planning how to get them done in detailList some options for what the customer might really mean, then delegate someone to fine out more, using your options as a basis for conversation.Option 1 sounds appealing, but doe…

Fail, To Succeed

I was listening to a commentary on NPR about a pre-school graduation which mentioned a comment from an education expert Leon Botstein that "we should be rewarding: Curiosity. Creativity. Taking risks. Taking the subjects that you're afraid you might fail. Working hard in those subjects, even if you do fail. We should reward children when they show joy in learning."

This led me to thinking about a reason that some teams struggle with being agile. Agile teams are good at making corrections based on feedback. For this to work you need to be willing to honestly evaluate your progress against a plan, and be willing to revise the plan (and how you work) based on this feedback. This is a hard thing to do if you're used to the idea that any feedback other than "you're doing OK" is bad. (I have more to say about this in a contribution to the 97 Things Every Programmer Should Know project.)

Agile methods help you create an environment where it's safer to try t…

The 2009 Software Test and Performance Conference

Last week I gave a class on SCM for Agile Teams at the 2009 Software Test and Performance Conference in Cambridge, MA. The conference had a focus on Agile software development. Good SCM is essential to agile (and any) software development, though it's an oft ignored topic, so I applaud the organizers for considering the topic worth a session. And I thank them for inviting me to give the class.

At the risk of making sweeping generalizations, I like testers, and I tend to find that the way good testers think is very well aligned with the way that I think. Maybe this is part of the reason that I enjoy agile software development so much: testing and automation are very closely tied to the development process. One of the messages in my talk on SCM for Agile Teams is that testing is an essential part of the configuration management process. If you have good automated testing, you have to worry less about branching for isolation, and you save the overhead that (unnecessary) branching ad…

Information, Interaction, and Scrum

A Scrum team that has dependencies on another group often struggles with how to integrate the other group's deliverables into their sprints. Since the other group hasn't signed up for the sprint commitment, and also has commitments to other teams, the Scrum team has the problem of how to commit to a delivery when they depend on someone outside of the team.

One way to manage dependencies is to list them as roadblocks. However, while that's a good start, it's not the whole answer. A few years ago I was working in the IT department of a large (not-a-software) company. My team was building a tool to archive a class of instant messages. We used Scrum, and consistently followed "the three questions" : what I did yesterday, what I plan to do today, roadblocks. A couple of items relating to the operations group were on the roadblock board for quite a while. While we had a designated contact in that group, he had many other commitments, and our requirements kept gett…

Continuous Integration as Metaphor for Agile Methods

Build and SCM practices are essential for agile teams to deliver working software on a periodic basis. Continuous integration is an essential practice for agile teams, as described in the XP book: Extreme Programming Explained: Embrace Change, and Paul Duvall's excellent book: Continuous Integration: Improving Software Quality and Reducing Risk.

For those not familiar with the continuous integration (CI), it's a practice where a team has an automated build that monitors the source code repository for changes and builds and tests the software periodically. CI allows the team to monitor the state of the project and detect mistakes and integration errors quickly, minimizing the risk of someone on the team being stuck because they checked out broken code, and thus enabling the team to deliver quickly.

Continuous integration is also a good metaphor for agile software development not just because it's about providing feedback and enabling change, but because of the way that CI t…

Enabling Change

I'm starting to gather my thoughts for the October issues of CM Crossroads which has a theme of "Overcoming Resistance to Change." Some of my favorite books on the topic of enabling change are:
Fearless Change: Patterns for Introducing New Ideas
by Linda Rising and Mary Lynn MannsBecoming a Technical Leader: An Organic Problem-Solving Approach by Jerry Weinberg
Quality Software Management: Anticipating Change (and other books in the QSM Series) b Jerry Weinberg
While I can't possibly cover all of the ground that these books do, I can share some observations I've made while trying to help teams to do things differently, such as adopting Scrum or developing a more agile release management approach.

A common reason people resist change is because they follow "rules" that they don't understand no longer apply. When these "rules" are embedded in the culture of an organization the challend to change is greater, since its often more pardonable to f…

IDEs and Builds Scripts (September Better Software)

I wrote an article for the September/October issue of Better Software Magazine: IDEs and Build Scripts in Harmony where I discuss how to use build tools, and IDEs while minimizing duplicate work and getting the benefits of both tools.

As usual, the editors did a great job selecting metaphorically appropriate art work, and there is lots of other good content in the issue, including an interesting commentary by Lee Copeland on testing, Scrum, and "testing sprints."

I won't repeat anything I say in the article (since I already said it), but I want to add some philosophical comments. The question of working in environments that use IDEs for development and integration builds that are driven by other tools is one that I care about because it involves some of the issues that often cause a lot of angst and discussion on development teams:
Standardization and the effects of tools on individual and team productivity, andRepetition: having the same information in two places.Many of …

97 Things Every Programmer Should Know

This past week 97 Things Every Programmer Should Know was made available to the public. This project was driven by Kevlin Henney, who I know from the early Pattern Languages of Programming conferences, . The list of contributors contains many familiar names, and I'm honored to be among them.

My contributions are about the agility, deployment, and their intersection. They include:
Own (and Refactor) the BuildDeploy Early and OftenUnderstand the Principles behind Practices, and Acknowledge (and Learn From) Failures
This project has contributions from many really interesting and insightful people. Maybe you already know everything on the lists, but you might get a fresh perspective by reading the contributions. You are very likely to start thinking.

On a related note, this past week, there was a conversation in my office between one of my colleagues who is a parent of a toddler, and one who isn't about why one works so hard to sound excited when talking to children about tasks.


Streamed Lines at 11

As I was starting to prepare for a class I'm giving at the Software Test and Performance conference in Cambridge MA this October I looked over a paper Brad Appleton and I wrote in 1998 on branching patterns: Streamed Lines, and I started to think about the path from this paper to the SCM Patterns book. Streamed Lines describes a number of branching patterns and when it's appropriate to use each one. From time to time people still tell me or Brad that this is one of the better overviews of branching strategies they have seen.

The paper grew out of material we gathered in a workshop at ChiliPLoP 1998 in Wickenberg AZ. We organized the set of index cards, categorized by color-coded stickers into the paper we prepared for workshopping at PLoP 98 (which was the PLoP conference I was program chair for). Many steps later, with encouragement from John Vlissides, we submitted a book proposal and started working on Software Configuration Management Patterns. The SCM Patterns book says a …

Sprint Review as Agile Enabler

An agile process such as Scrum is built on a number of both project management and engineering practices. The engineering practices support the project management practices and the project practices guide engineering decisions. While it takes more than the presence (or absence) of any one practice to cause your agile project to succeed or fail, some practices can drive your process in a powerful way. Sprint reviews are one practice that, when done with the right attitude, can help teams develop and maintain a good project rhythm.

An iteration process has mechanisms in place to help steer teams. Of all of the practices that support iteration, regular sprint reviews help teams and product owners get the feedback that they need to improve their performance. Reviews are also one practice that I've noticed that teams slack off on, especially when they are first starting out.

A common rationale for not having an iteration review are that there isn't enough to show. When the team is …

Releasing, Branches, and Agile Teams

When Brad Appleton and I wrote the SCM Patterns book we discussed two branching patterns related to releasing software: Release Line and Release Prep Codeline. The first, is a pattern that allows teams to easily support released code. The second is a pattern to help you "stabilize" a codeline before a release while allowing parallel work for the next release to proceed. Too often teams start Release Prep Codelines at the wrong time or for the wrong reasons, without understanding the costs . Given the rate at which changes happen on agile codelines, the costs of creating a branch can be large if do it for the wrong reasons.

If an ideal agile team had "shippable" software at the end of every iteration, a Release-Prep codeline isn't necessary. You just label your code at the end of the final sprint, and if you need to provide a fix that can't be done from the mainline code, you start a release branch from that label.

Many teams can't ship at the end of eve…

Yours, Mine, and Ours- Ownership on Agile Teams

During a panel discussion on people issues on agile teams I participated in during a meeting of the New England Agile Bazaar the issue of how to address the problem of people taking credit for work someone else did arose. (The short answer is that you can solve the problem through a combination of agile tracking methods to make progress visible, and some people management techniques.) This entry isn't about that question.

The issue of credit in agile teams reminded me of a recurring puzzlement: when someone on a team says something like "we used Jim's component...." what does that mean? Some of the choices are:
I want to give Jim Credit for doing something useful.I want to make it clear who's responsible if it doesn't work.Jim is the only one who knows how it works, and the only one who can make changes.
There may be other choices, but this type of dynamic comes up a lot and I wonder what it has to do with the tension between the collective code ownership value…

Finding and Evaluating Options

Of all the rules, techniques, and heuristics I've tried for making design (and other) decisions, the "Rule of Three" keeps surfacing as one of the simplest and most effective to use.

There are two variations of the Rule of Three. The first, from Secrets of Consulting: A Guide to Giving and Getting Advice Successfully by Jerry Weinberg is about evaluating options:

If you can't think of three things that might go wrong with your plans, then there's something wrong with your thinking.
This sounds rather negative, but if you think for a second, it makes sense, and it's really about being constructive. Design decisions are about evaluating trade-offs. You want to pick the decision that has the problems that you can live with. When we've been struggling with a problem and come up with a plausible option, we tend to want it to work. If you don't think about what might go wrong you're making it harder for a good idea to succeed. The worst that can happen…

Begin with the End in Mind: Deploy Early

In many projects I've worked on the task of developing a deployment process is deferred until close to the release date. This is a bad idea. Deferring deployment-related decisions adds risk to a project as deployment to a target environment exposes development time assumptions and operational requirements that may need to be addressed by the engineering team before an application is released.

I suggest that a project develop (and use) a process to deploy the software to the target platform in the first iteration. Starting on deployment early gets you closer to the agile goal of shippable code at the end of each iteration.

Deployment is important, and something for which there should be a story in your process early on because:
Deployment is the first thing a customer sees, but it's only an enabler, it doesn't directly deliver value, so you'd like for the process to be efficient. Incremental improvement of the process is a great way to make it work well.Design decisions ca…

STPCon 2009 in Boston

I'll be giving a class about software configuration management for agile teams at the Software Test and Performance Conference in Cambridge, MA on Friday Oct 23. The conference is Oct 19-23, 2009.

The abstract for my talk is:

Version management, build, and release practices are essential elements of any effective development environment, especially for agile teams, which rely on feedback and maintaining high quality during rapid change. Many agile teams are puzzled about how to apply good software configuration management (SCM) practices in an agile environment. This session will provide an overview of SCM concepts and explain the patterns and practices that teams need to maintain an agile SCM environment. You’ll learn how agile testing practices and continuous integration change how teams use SCM, and how to set up the essentials of an agile SCM environment.

Testing Cost Benefit Analysis

I'm probably one of the first people to advocate writing tests, even for seemingly obvious cases. (See for example, Really Dumb Tests.) There are some cases where I suggest that testing might best be skipped. There are cases where tests may not only have little value but can also add unnecessary cost to change. It's important to honor the principles of agile development and not let the "rule" of a test for everything get in the way of the the "goal" of effective testing for higher productivity and quality.

While writing a unit test can increase the cost of a change (since you're writing the code and the test), but the cost is relatively low because of good frameworks, and the benefits outweigh the costs:
The unit test documents how to use the code that you're writing,The test provides a quicker feedback cycle while developing functionality that, say, running the application, and The test ensures that changes that break the functionality will be fo…

3 Books every engineer should read

There are many excellent books to read if you write software (Software Configuration Management Patterns: Effective Teamwork, Practical Integration) among them :) ). Given that each of us have limited time, its difficult to read everything one might want to, so we tend to focus on reading material that addresses our immediate problems. Having said that, there are 3 books by Jerry Weinberg that have incredibly valuable advice that I use frequently, and I'd like to recommend them to anyone who's serious about software engineering work:
Are Your Lights On?: How to Figure Out What the Problem Really IsBecoming a Technical Leader: An Organic Problem-Solving ApproachSecrets of Consulting: A Guide to Giving and Getting Advice SuccessfullyThese books are more about working effectively with people to solve technical problems than actual technology problems. This is an area many people neglect; tools and technologies come and go, and you need to maintain your skill level, but knowing how…

Feeback, Writers Workshop Style

I was fortunate enough to participate in the first few Pattern Languages of Programs conferences, where I learned quite a bit about technology, writing, problem solving, and giving and receiving feedback. What made the biggest impression on me was the process the patterns workshops used.

Before I started to write patterns, I was used to getting feedback in a way that focused on what was wrong or missing in my presentation. The patterns community approach is to evaluate papers through a shepherding process that ends in a writers workshop.

A patterns writers workshop has the following parts:
The author of a paper reads a selection. This gives the other participants a chance to understand something about how what the author thinks is most important about the paper. After this reading the author becomes a passive participant in the session, only speaking if someone makes comment that isn't clear. The author does not defend his writing.Someone in the workshop summarizes the paper. This al…

The Value of Agile Infrastructure

Engineering practices such as Continuous Integration, Refactoring, and Unit Testing are key to the success of agile projects, and you need to develop and maintain some infrastructure maintain these engineering practices. Some teams struggle with how to fit work that supports agile infrastructure into an agile planning process.

One approach is to create "infrastructure" or "technical" stories for tasks that are not directly tied to product features. In these stories the stakeholder may be a developer. For example, a user story for setting up a continuous integration server can go something like:

As a developer I can ensure that all the code integrates without errors so that I can make steady progress.
While there not all work on a project leads directly to a user-visible feature, thinking of infrastructure stories differently than other stories has a number of risks, and can compromise the relationship between the development team and other stakeholders. Agile met…

Pitching Agile and the Elevator Test

Some time ago I read Crossing the Chasm by Geoffrey Moore, which is about marketing high tech products to mainstream customers. The part of the book that stuck with me, and the one that has the most pages marked in my copy was the section on positioning a product, in particular the section on the elevator test, which I've found to be a useful framework for understanding the value of many things, not just high tech products and companies.

A successful 2 sentence statement of the value of product should pass the elevator test . While the book is focused on making a claim for getting the interest of investors, being able to cast any idea you are trying to sell into this format is a great exercise, as it forces you to be focused and to really understand the value to your audience of the idea you are describing. Once you've got someone interested you can then go into the nuances.

In an elevator pitch you should be able to describe:
Your target audienceThe market alternativeThe new …

Really Dumb Tests

When I mention the idea of automated unit testing (or developer testing as J.B. Rainsberger referred to it in his book JUnit Recipies) people either love it, or more likely are put off because all the tests that they want to write are too hard, and the tests that they can write seem too simple to be of much value. With the exception of tests that "test the framework" that you are using, I don't think that one should worry prematurely about tests that are too simple to be useful, as I've seem people (myself included) spin their wheels when there was a coding error that could have been caught by one of these "simple" tests.

A common example is the Java programmer who accidentally overrides hashcode() and equal() in such a way that 2 items which are equal do not have the same hashcode. This causes mysterious behavior when you add items to a collection and try to find them later (and you don't get a HashCodeNotImplementedCorrectly" exception.) True, y…