White Paper: CM: THE NEXT GENERATION of Agile CM

The two biggest gains of an Agile development process, in my opinion, are the continuous integration, which when properly administered leads to greater product quality, and the interactive customer feedback loop, which helps to get the requirements right before development is over. These are huge advantages. There are many agile design processes. But how do you extend the agile advantages to the CM function?

A tool-centric view of Agile CM will not lead to Agile CM. Why? Because, for the most part, with noteable exceptions, tool makers worry about their own little world, and when all of the little worlds that make up the ALM function are put together, they fall short. If you want Agile CM, start off with a good, agile process. If your CM tools can't support the process, look around for more flexible ones.

Achieving an Agile CM Process
What is an Agile CM process? If you said it's one that doesn't get in the way, you're only partially correct. CM adds a critical capability to a project. This capability is even more critical in an Agile environment. CM helps the agile process keep rolling. But it also provides the information required to make decisions with good judgement and timing. An Agile process is about good communication. Your CM tools should be your Central Communication Point. All project-specific information should reside there. I should have to go to people for judgement and for skills, but not for information.

An Agile CM Process is one that supports all team members, making them more productive and simplifying their management tasks. If your developers need to perform tedious labelling, merging or check-in tasks, odds are that the quality of these tasks will degrade over time. If on the other hand your developers add information to the CM vault as a side effect of their work or in order to realize productivity improvements, the opposite with hold true. Automation is a key for Agile CM, but automation that pervades the process, not just one that does a build. Tell me when I violate a dependency. Let me know when a new branch is required. Let me see my prioritized to-do list. Give me a single click to review my change, or any other change for that matter. Eliminate the need for me to label things. Make my life easier while helping me to make everyone else's job easier.

If you're looking for a quick fix to your current CM process to make it Agile, forget it. If you want an Agile process, you'll have to remove the road blocks all the way through. Just as the weakest link in a chain determines the strength of the chain, all parts of the process have to be agile to attain an Agile process. If you can't easily roll back a change, you'll hit road blocks. If you can't automate your build procedure, you'll introduce costly, time-consuming problems.

If you want to make your CM process Agile, go through it and identify how you can reduce end-user complexity, tool misfits, areas of risk, and so forth. If a developer asks "Why can't I just...?", don't answer right away. Instead, ask yourself "Should the developer be able to just...?" If the answer is yes, then look at what's preventing him/her from doing so, and fix the process.

Agile CM Checklist
Below is a checklist of process items you should consider to make your process Agile.

Easy Traceability. Traceability is much more important for agile processes than for traditional ones. Agile decision makers meet often and need to have the necessary information at their finger tips to make good decisions. If you can't trace easily from changes to features and from features to testing, this will not only slow down your process, but will necessarily make the traceability data less useful, less used, and hence lower quality. Your processes simply cannot be sufficiently agile if your data is difficult to get to or to relate to other data.

High Quality Data - Picking up data from context. High quality data is a result of having good forms with well-defined set of options. But more than that, data has to be populated from context to minimize user data errors and to simplify and speed up user interaction. If a user is working on a particular development stream of a particular product, that information should automatically be populated in any changes, problems, features, etc. that the user adds. If a new change is started by clicking on a problem to fix it, the problem should automatically be traced to the change.

Labels Can Mean Trouble - If you need to manually label things, you've introduced unnecessary administration into your process. You should be able to completely define and organize changes, baselines, build records, branches, etc. without having to create labels. Creating a label is a short cut for an inadequate process/tool, or a workaround for an overloaded capability. For example, if I check out a file against a change, the branching (if necessary) should look after itself based on the change. No need to apply labels. A common mistake of many CM tools is to overuse branching - for promotion, for parallel development streams, for release identification, etc. Because of this, a branch does not imply what it could and so labels are introduced to discriminate the overloading of the branching concept.

Reduce dependence on version codes at the user interface. Developers, and even CM managers should rarely have to look at version codes. If they do, then the tool is not doing a good enough job in specifying the context and user interaction, or the tool makes specifying the context too much of a chore. Version codes are fine for looking back at what was done in history. But the user interface of a modern CM tool should be change focused, not version focused.

Easy rollback of changes. Any agile development shop is going to have to pull changes that upset the stability of the product. This should be as easy as rolling back the status of the change, if there are no higher promoted versions in the development stream. If there are, the tool should let you know and leave you with the decision as to whether to roll back later changes as well, or to yank the change from the stream. Both of these options must also be easy to do.

Change-based CM - One step check-in, one step promotion, one step delta, one-step propagate/yank. If you have to type in the reasons for each file revision you check out, or check in each file separately, do delta reports on a file-by-file basis, etc., you're process is a lot more tedious than it should be. This will increase the reluctance to make and deal with changes and will subtlety affect your process and your product quality.

Code in Repository. One of the keys to agile development is knowing what's happening. The CM repository needs to be a communication tool. If you can't put you're code change in the repository when you're done your "unit" testing, only you know it's state, and others have to work from an older state of the corresponding files. This complicates everyone's work, forcing additional parallel changes and merge/test cycles. Make sure your process will allow developers to put code into the repository when it's ready, not when someone else is.

Object-oriented queries - Testcases missing, failed requirements. Communication is as much about getting information as it is about providing it. If you can't go to an object and find out the things you need to about it, your interface needs improvement. There's no room for poorly integrated management information in an agile environment. If you need to know if a requirement passed it's tests in a build, you need to go to the requirement and ask that question. Similarly if you have a requirements sub-tree and you want to know if there are requirements not covered by testcases, you need to go to the sub-tree and ask that question. If you need to find the build in which a problem was fixed, you need to go to the problem and ask that question. If the tools don't let you do that, you'll have to learn some other more awkward way of getting that information.

Interactive CRBs, weekly/daily meeting support. Your CM/ALM tools need to be able to support interactive CRBs, and weekly or daily team huddles. It must provide up to the minute information that can be easily summarized, and zoomed into for detail. If you need to have an expert in the room to do this, that's fine, as long as the expert can do it in real time for you to run your meeting. Better yet, your tools should be able to provide a list of objects that form the focus of your meetings, especially for CRBs. And tracking meetings and action items from the meeting would be an additional benefit.

Priority-based feature/problem management. Agile development is primarily feature/problem driven, rather than schedule driven. Perhaps you pick features/problems from a pool, or perhaps they're assigned to you. Your tools must provide you with the capability to easily prioritize features and problems and to look at the features/problems which are candidates for or designated for the next iteration. These must be priority sorted so that the view reflects what's most important. Ideally, when your users log into the CM/ALM environment, or even their IDE environments, they can see a list of problems/features assigned to them or to their group pool, at a click of a button. Similarly they should be able to see their changes in progress.

Instant queries/reports. This is important for meetings, but it's also important for communication. If queries and reports are less than instantaneous, they will fall into disuse to some degree. The longer it takes, the less it will be used. The less it is used, the poorer the communication of information. And this extends to operations on the query results, such as navigating traceability links and zooming in for more information. Metrics must be visible outside the tool, but when viewed inside the tool, it must be possible to look at them in more detail when necessary to help understand why a metric has fallen out of bounds. If there are significant wait times to get these answers, your process improvement efforts will suffer and as a result, your agility.

Easy rebase/workspace synchronization. If it's a chore to rebase your workspace to reflect the latest changes, don't expect it to be done often. The result will be development done against older environments, leading to less integration testing with other new features. As well there will be additional workload on your developers when the time comes to check in their new development.

Easily Identify Changes for Each Iteration. An agile process is generally characterized by short iterative development cycles, or at least "release" cycles, where we use the word release loosely here. When testing identifies problems with a new build, it's important to be able to rapidly compare it to the previous one so that the list of changes that went into the new iterations can be used as a starting point to identify the source of the problems. It's not good enough to identify a list of files that have changed. Often in an iteration, a dozen or two changes may involve several dozen file changes. If you can easily scan a list of 15 changes and pick out two or three potential problem sources, you'll streamline your process of correcting the iteration.

Branchless Promotions. If you are managing promotions using branches, a promotion will mean a merge operation is necessary. This is a pessimistic algorithm that assumes all changes are promoted in random order. For the most part, changes will flow through the system in the same order as their original creation (i.e. completion). It is the exception that requires rearranging the order of changes. Yet promotion-based branching incurs the merge penalty always. Time to rethink this strategy to eliminate extra promotion branches so that merging occurs only when required. Another bad side effect of a branch-based promotion model is that it is a big deal to introduce a new promotion level, so there is a lot of reluctance, even if it is necessary. In agile CM, this leads to not improving the process in the name of keeping things agile. Promotion should occur through promotion of changes, not through merging into a new branch. The tool should be able to look at the change state and determine the appropriate context view and/or baseline/configuration based on these states.

Automated Build for SI and Designers This is a big topic and is usually a central topic for Agile CM. Designers need to be able to do their own builds and testing. And all building must be automated. It's no good if the designer built it one way and it worked, but the System Integration team can't get it to build. If your nightly builds are not automated yet, make that a priority task. Do what you need so that your builds will run automatically every night. Perhaps one that has all the "checked in" changes, and another more stable one that has all the "selected/approved" changes. When this is done, make sure that the developers can use the same build automation capabilities that the nightly builds use. That will allow them to test what is going to be built and integrated before your nightly build. That will mean fewer errors in system builds.

Branch per stream. In an agile environment, it is important to be able to rapidly iterate, yet to work towards releases which are longer term than an iteration. The traditional main trunk model works well for a single release, but when subsequent releases appear, the support structure is compromised. As well, the strategy for working on future releases is typically complicated to the point that only developers themselves are aware of future release work until the future release is promoted to the main trunk. Instead, it is recommended that a main branch exist per release stream. That way, work may proceed on any stream at any time, allowing support, advanced work or normal development without having to change the process or artificially schedule dates for the main branch reflecting a new release. If you really need "leapfrog" check-in capability, whereby parallel teams are working on the same release but using overlapping iterations, consider two parallel streams per release, which eventually will merge into one.

What Else Can You Do
There are many things you can do beyond the above checklist.

First of all, look at the tools you are currently using. Make sure the architecture of those tools will take you well into the next generation. Odds are that if your company is successful with it's current project, the results of that project will be expanded upon for many years to come. You don't want your tools failing you along the way.

Look at how you handle parallel checkouts for short term parallelism. Many of the successful projects, including our own, rarely need to do parallel checkouts. Or if so, they are done in an informal fashion (i.e. get the code make the changes, reconcile on checkin). Why? Well, for us, several factors come into play.

Fortunately our tool allows short term parallel checkouts without the need for branching. The change record contains the same information as provided by a branch, and the CM+ tool allows us the option of recording or not recording parallel checkouts. At check-in time, we are informed if there have been any changes since the checkout operation and, if so, a reconcile operation for the change being checked in is initiated.

There's no real value to having a couple of extra branches to show the history of the past three days work. As long as the two parallel changes make it into the build and the change deltas clearly show the changes independently, I'm happy. In fact, often you'll find branching for short term parallel changes only confuses the branch history. Some tools even have ways of purging such branches to clean up the history. All this adds to the complexity. So why is branching done for short term parallel changes? To provide visibility of the changes. Your CM tool should provide this visibility without forcing you to complicate your branching architecture.

Tools must have a net payback for all users so as to encourage their continued usage. If your users currently have to collect all of the files and do a delta report on them and then email that report around, that few extra minutes of work will add up over the course of several hundred changes. Your tool should allow collaborative reviews from a single change identifier using a single click or two. If merging a change from one stream into another (we call this Change Propagation) is difficult because the user first has to go through a long process of figuring out which files need to be merged and what the merge ancestors are for each merge, what will happen? Any process requiring a merge will be strongly discouraged. If users have to create configuration view specifications to ensure that they can see the appropriate versions of their files, there will be loss of productivity and manual errors causing frustration. It all comes back to ensuring that the tools you select will support your agile processes.

Agile CM Requires Good Software Architecture
If you think that you can ignore your software architecture and still have an Agile CM process, think again. I've seen countless architectures where a handful of files are bottlenecks to the whole development process. The same file needs to be changed by virtually everyone. A single file is stretching the limits of your compiler or version control tool. Introducing product variations is causing nightmares in the software. Here are three key things that all projects can do to simplify their lives.

Minimize contention on files. Files have high levels of contention because they were designed to have high levels of contention. If I have to put a change number in a single file every time I make a change, I'll expect a lot of contention. If instead, all I have to do is call a routine with my change number, the contention disappears. Same functionality, no contention. Most files which have a high contention on them can be dealt with in this manner. Whether it's a change number, a message code, or even a function. Replace the one-file policy with a distributed, less contentious one. You'll also find that your version numbers for the file stay down to earth rather than growing astronomically, and your CM tool may like that.

File ownership for quality. Did you ever work in a car wash? How much attention did you give to each car? Now when you're cleaning you're 1972 Mustang do you give it the same level of attention or more? Code is something created. The creator has pride in his/her creation. Don't destroy that pride by making it open season on everyone's code. File ownership does not mean that the owner has to make all the changes - just that the owner has the responsibility and authority for the design of that code on an ongoing basis. The owner reviews all changes at a minimum. The CM tool can help support and enforce file ownership, ensuring that the otherwise "open season" Agile process does not degrade code quality.

Run-time Configuration. In the haste to get the product out the door, data is hard-coded into source code files. This may make changing the data more difficult, but the real crunch comes when product variants are produced by having variant files with different values of the hard-coded data. This can be a CM nightmare if there are many variants along different axes. Work from the outset at a strategy to make data run-time configurable. This does not involve more code (usually less in fact), but does require an architectural decision up front to allow data to be run-time configurable. The result is that code is written with that assumption. Allocations are done as required rather than up front, thus making use of changes to the configuration parameters. Variables that can be changed through the user interface are used rather than hard-coded "#define" statements. Slightly different algorithms are used to support some dynamically changing values at system start-up, or even throughout the life of the application, rather than assuming static values. The amount of flexibility not only reduces the CM load, but speeds up the design as the decision on what the hard-coded values should be can be deferred to the product team.

Agile is Just Good Design and Common Sense
Now go through all of these factors and tell me which ones you would not want to take advantage of if you weren't looking at creating an Agile CM Process. Are you surprised? Agile Design and Agile CM are just good common sense practices with some terminology for wrappers. So if you're not looking for an Agile CM Process, I might recommend that you read this article and the other related articles anyway. It just makes sense.

Joe Farah is the President and CEO of Neuma Technology . Prior to co-founding Neuma in 1990 and directing the development of CM+, Joe was Director of Software Architecture and Technology at Mitel, and in the 1970s a Development Manager at Nortel (Bell-Northern Research) where he developed the Program Library System (PLS) still heavily in use by Nortel's largest projects. A software developer since the late 1960s, Joe holds a B.A.Sc. degree in Engineering Science from the University of Toronto. You can contact Joe by email at farah@neuma.com

More white papers:

Find out more ...

Neuma White Paper:

CM: THE NEXT GENERATION of Agile CM