More white papers:
Find out more ...
Neuma White Paper:
CM: THE NEXT GENERATION - Building: Change Manangement Comes First
Building is considered to be one of the fundamentals of Configuration
Management, even though some might argue that Building isn't really
CM. The reason it is fundamental is that the build/verification cycle
provides proof of reproducibility if done correctly. It forces CM to
be done correctly so that only the objects from the CM repository are
used to reproduce the build. When formal build processes are correct,
they need no information that resides outside of a CM repository. And
when properly done, the build record is created prior to the build
(i.e. a build notice) rather than as a result of the build - that
record being used to drive the build process. A integral Change
Management capability is an essential component of such a build
Types of Builds
We can classify builds
into three distinct categories: Release, Integration and Developer.
They might be loosely categorized as:
- Release - Existing Build Record (i.e. Snapshot) Used (as authorized)
- Integration - Repository and Change Status Centric (typically nightly)
- Developer - Workspace Centric (repeatedly as required)
release build is one that is done for a formal or candidate release of
the build outside of development. This may be for formal Verification,
for Field Trial, Beta Trial, or Production. It differs from the other
two builds in that there already exists a snapshot of what is to be
built in the CM repository. You specify which snapshot you wish to
build and then, using only data from your CM repository, your build
tools/process produce the required outputs.
An integration build is similar to a release build except for a few distinctions.
of all, part of the integration build task is to assemble the correct
objects into a prospective snapshot which will be used for the build
Secondly, there is generally some latitude for
modifying the prospective snapshot prior to freezing it. This might
occur, for example, if the build attempt caused compile errors, or
perhaps if the built product would not successfully initialize during
basic sanity testing. Generally, urgent changes are produced to "fix
the build". Different shops have different rules for such latitude
within a build cycle.
Some will say: leave the initial snapshot
and always create a new snapshot for any corrections, especially if
there is not good traceability of what changes were required to fix
it. I don't generally agree with this as it may cause confusion when a
developer comes in the next day and sees four new snapshots, even if
they are properly marked with an appropriate status in the repository.
And it may make it more difficult to pinpoint a snapshot in the past.
My opinion is that a snapshot is frozen when the build team puts its
seal of "sanity tested" on it. This does not mean that all sanity
tests were passed, but instead, that it is more beneficial to "release"
this build to the development environment then to stick with the
previous one. And that will be a judgement call at times.
third distinction is that integration builds are generally done on a
specific timeline. In fact, in many shops, they are automatically
performed in the "middle of the night", that is, nightly builds. Does
such a term has a meaning in your global development shop? Some of
Neuma's customers do builds in the "middle of the night" at each
separate development site. In this way, they may actually be doing
multiple integration builds a day. Others focus each site on a
particular development stream and so different streams of development
are integrated at each of the different sites.
fourth distinction might be to say that integration builds might be
either incremental or full build operations, whereas the same is not
true of a release build. There is some wiggle room here. I might
recommend that, at least a couple times a month, full integration
builds are performed. This will help to verify the continued
correctness of the full build process. Most shops tend to do full
builds on a nightly basis. This is dependent on the capabilities
provided by the CM tools, as well as on the size of the operation. If
the CM tool knows what was built the previous night and knows the
impact of all changes going into the current build, there is not a
necessity to do a full rebuild. Similarly, if your build process has a
reliable way of ensuring, for example, that a "make" operation will
always do "the right thing", then a full build is not required every
night. On the other hand, for Release builds, there is always a need
to verify full reproducibility so that you know that at a point in the
future, if necessary, a full build may be generated without problems.
A developer build is different from release and integration builds in various ways:
The developer may choose to build in an isolated workspace, or to
continually re-sync (aka. re-base) active changes with the latest (or
perhaps another) integration environment, or perhaps to use a mixed
strategy: isolate until the changes are in a stable known state and
- The developer does not generally create a
snapshot of each build, especially if the edit/build/test cycle is only
a few minutes long. Instead, the developer will generally save (aka.
shelf) the source changes at a stable known state prior to continuing
to work on the change. It is recommended, however, that a complex
feature be broken down into a set of logical changes that can be
independently checked-in along a strategy that sees the riskiest
portions of the feature implementation absorbed into the integration
cycle as early on as possible (i.e. not close to release dates).
A developer build is performed ad hoc and typically a lot more
frequently than release or integration builds. It is not uncommon,
especially with today's IDEs, to see a developer perform dozens of
builds in a day on the way to producing a new feature.
developer build is most often done as an incremental build. Changes
are made and only what is necessary is re-compiled and built. Your
build environment should rarely require a developer to do a full
build. However, for smaller products (e.g. < 100 KLOC), a full
build can be done in under a minute these days. So this distinction is
only important in larger projects. Often the IDE is used to decide
between incremental and full builds.
- Developer builds are not
released in any way shape or form. Only the developer is waiting for
it. (At least this is recommended practice.) If another developer
requires the changes of one developer, the correct approach would be to
either check-in, or save (i.e. shelf) the change at the state that it
will be used by the other developer.
There are, no doubt, other distinctions that can be made, but we need to get on with the process.
Automation of Integration Builds
process of automating builds has a lot less to do with the build
process than with the change process in your shop. However many shops
ignore this and automate the builds regardless of the change process.
When this happens, the automated build becomes king and change
management, as well as configuration management, fall slave to the
build process. It is important that a proper change and configuration
management process be used from which build automation will flow out
rather than vice versa.
For example, consider a shop that says:
"We're going to use a basic CM tool and automate nightly builds by
pulling out the latest checked-in code, including 'make' files, and
perform the build operation. We'll then make the resulting build
environment available for anyone who needs to make/test changes the
next day." The process is easy to implement: pull out the latest code
(into the right places) and invoke the top-level make file (which will
invoke whatever other ones are required). It's simplicity means that
the process will be implemented accurately, so that's good.
there are numerous other implications and omissions in this statement,
and some have far-reaching effects. For example, a developer will not
check in any code that is not fully tested for fear of "breaking the
build". Good testing is good. Leaving code checked out longer than
necessary is not so good. In fact, a rapid add-on to the above process
is: "You need approval for checking in any code changes that are not
required to fix urgent bugs". In the context of the process, this is
reasonable and even good. But overall, code is going to be checked out
for even longer periods. When code is checked out longer than
necessary, more parallel changes are required. When the developer does
not have full control over when they are checked in, even more parallel
changes are required. So starts a complex maze of branching and
merging and a Configuration Management and labeling strategy begins to
grow for management of such branches and merges.
implication of the above is that the make files checked in are going to
be used for the build operation. That's not bad in itself, but does it
force a serialization of changes to the make files, and implicitly the
corresponding code changes, that is other than optimal. In other words,
the structural changes of the product are tied to the code changes of
the make file(s). One frequent response is to segment the make file
into many little pieces. Now anyone who has worked with make files
knows that they can be complex beasts. So now this complexity, along
with all of the maintenance of the complexity, is distributed and/or
replicated into many files.
You could probably identify some
other reprocussions from the above nightly build policy. For example,
what happens when a build breaks? What happens if the server is down
prior to the build and required changes don't make it in? How does a
change control board figure into this? What about parallel stream
development so that work on the next release can be started? These
questions all hinge on the CM Process and Tools used to implement your
Change Management Process.
CM Requirements for Build Automation
keep Change Management working properly we need to take a different
perspective. To effectively run an automated build and effective
integration shop there are a number of requirements that must be met by
the CM tool and process. Let look at some of these:
(1) Change Packages (aka: Updates):
Ability to treat code changes as logical changes, rather than as file
changes. Let's face it, a designer will fix a problem or implement a
feature. The files being changed, as well as the problem(s) or
feature(s) being addressed form a logical unit of change. The change
must move through the system as a whole, from check-in through all
levels of promotion. File-based CM is an old concept and if your tools
push you this way, it's time to look around.
Ability to snapshot a build record and to easily reproduce the build
from the snapshot. A build record must be something that drives the
build process rather than a record of a build being performed.
Reproducibility is the goal. The snapshot can be in the form of an
enumeration of file revisions (including tool/process components), as
is done for a typical baseline, or perhaps in terms of an existing
baseline plus a set of change packages relative to that baseline.
Ability to check-in code without forcing it into the next build(s) -
Completed code that's out on a developer's disk is a liability. It's
prone to disk crashes; it's prone to becoming out of date; it's likely
to go through fewer verification cycles. Completed code belongs in a
repository where it can be used by others, simply to review, or as a
basis for future changes, without the need to have a parallel change
branch. A developer should be able to put a source change into the
repository even before dependent changes are checked in. The developer
then pushes the change to the "ready" state after confirming it's ready
to go into an integration build.
(4) Pull: Ability to
differentiate between check-in authorization and build acceptance
[similar to (1)]. Often, especially at the beginning of a release
development effort, any code that has been marked ready is
automatically picked up by the nightly integration build. Although
this may be sometimes sufficient, it is not sufficient as the release
approaches and changes are more selectively pulled into the build for
integration. The developer pushes, the integration team pulls.
Ability to roll-back changes that "broke the build" in an easy manner.
A change breaks the build. The best solution is to put in a new change
to fix the broken part of the change. This provides full traceability
of the problem. However, sometimes, it's too complex to fix and the
best solution becomes "roll-back" of the change so that it is not part
of the build. Ideally this is a simple state change on the build. If
changes are already stacked on top of this change, it may require
"yanking" changes. More complex CM tools/processes use parallel
streams for promotion (as opposed to state changes). In such cases,
rolling back a change may entail additional steps, some of which may
even be required in more than one of the parallel promotion streams.
Ability to adjust the snapshot (ie continue to (re)define it) by adding
or remvoing changes, until it is frozen [extension of roll-back]. More
generally than a roll-back, it must be possible to arbitrarily modify a
build record (i.e. notice) so that additional fixes may be added to it
or errors in definition of the build record may be corrected. This
must be easy to do so that the build can be successfully completed for
the team's use. It should only be done on a change-by-change basis -
never on a file basis. A change must always be in or out, never
(7) Dependencies: Ability to keep the build
definitions in a consistent state for the build operations. As changes
are added or swapped out of a build record definition, the CM tool must
notify you of violations in dependent changes. For example, if a
second change has already been built on the one being pulled, the tool
must notify you of this and allow you to take approprate action to
ensure a consistent definition results.
(8 ) Structure:
Ability to associate structure changes with a change. It's no good
pushing and pulling changes if the structural changes that go with the
change are not also pushed/pulled. For example, if a change moves a
data file from one directory to another, or deletes an existing file,
these structural changes must be packaged with the source code changes
so that everything moves together through the system, whether forward
(9) Promotion Levels: Ability to support
multiple levels of integration, especially in larger projects.
Promotion to integration is a single promotion step. In larger
projects, there are several levels of integration, especially in
task-based development environments where a major task might be
sub-divided into smaller tasks which are then integrated prior to
integrating the larger task into the "mainline". Larger projects will
have more promotion levels. I am often reluctant to use parallel
streams for promotion because there is a natural reluctance to running
through an additional merge cycle when a new promotion level is added.
Instead, I prefer state changes with a process which tends to support a
higher percentage of changes moving through the promotion levels in the
same order as they are checked-in to the repository.
Ability to retrieve files for a build into an integration workspace.
Given the definition of a build record, it must always be possible to
easily retrieve the files from the repository so that they can be
readily and reliably built. This includes support for both incremental
and full workspace retrievals. Along side this is the ability to
easily identify differences between any snapshot and the contents of a
(11) Stream: Ability to identify changes and
do builds for multiple development streams of a product. A stream is a
separate codeline targetting a different release of a product. For
example, release 1 and release 2 would generally have separate
development codelines, with release 2 slightly, or not so slightly,
overlapping release 1. A stream is basically a codeline that
culminates in a new release. The CM process and tools must support
this function. Many processes will use a single "mainline" which
switches at some point from release 1 to release 2. I do not recommend
this. It introduces a number of arbitrary decision points and more
complexity into the development process. For example, the process for
making a change to release 2 changes at least twice in the single
mainline approach: once when the switch of the mainline to release 2 is
completed, and again when the switch for a subsequent release is
(12) Product: Ability to identify changes and
do builds for multiple, possibly dependent, products. CM tools need to
support and clearly identify multiple products. Dependencies between
the products need to be identified, both for build purposes and for
(13) Compare: Ability to compare any to
builds to identify the difference in terms of changes (as opposed to
lines of code). For a successful build operation, it must be possible
to easily compare any two builds. For example, the build manager
should be able to take the new build definition and identify the
changes that are being added to the previous definition. It's not so
much the code changes that are important as the logical changes - i.e.
what is being done. It's the logical changes that will be pushed and
pulled in correcting an errant build. It's also necessary to be able
to identify problems fixed, features added, etc. so that integration
testing can first focus on the correctness of these features prior to
regression testing of pre-existing features. Ideally, your CM tool
will let you point to any two builds and compare them in such a manner.
a pretty long list of requirements. Are they comprehensive? I don't
think so, but it's a good start. Design your build processes around a
solid change management capability and you're well ahead of the game.
Start from an inadequate base and you'll be playing catch-up forever.
That was OK when we were trying to figure out what CM is. But now that
we know, and now that we have a much more solid understanding of Change
Change Management: Requests and Changes
Management has two key components and Build Management must fully
support these. On the one side, there's the Change Request; on the
other there are Changes (i.e. change packages). Builds are defined in
terms of Changes applied to a previous build or baseline definition.
This eliminates the tedium of managing file versions and labels. Each
Change needs to be targeted to a specific product and to a specific
stream of the product. The Change should exist prior to the first
checkout operation. The promotion level of a change, typically
indicated by its "state", is used to determine which builds it will be
part of. Promotion is always done on a Change basis. And Changes must
indicate their dependencies on one another, either implicitly through
file history, or explicitly through pre-requisite data. The build
process must use the dependency information to alert the build team,
the CM manager, or even the developer, whenever a promotion operation
leaves dependents behind.
Change Requests are quite different
from Changes and should not be used as a Change container. A Change
Request might be implemented in several Changes across several
releases. Or multiple Change Requests might be addressed by a Change.
The originator, or requestor, of a Change Request communicates the
request to the product team, which ensures that it has a proper
understanding of the request. Once this handoff occurs and the product
team accepts the Change Request, it is treated as a Requirement. The
originator will want to know when the request will be/has been
implemented and when it will be delivered. The originator is
responsible for closing the request - i.e. indicating that the
implementation meets the criteria of the request.
Builds as the Key Reference Points
where does the Build come into play? It must be possible to take any
build and identify which requests of a particular originator (or group
of originators) have been met by that build. On the flip side, that
leaves the set that is outstanding. If your CM traceability cannot
give you this information, you need to improve your data and
processes. Ideally, this information should be available at the click
of a button (or at least a simple report request panel).
this loop is closed, your builds are more than just a set of
executables. They become the key reference points for communicating
with your customers:
- What release is the customer running?
- What requests are still outstanding for that customer?
- What requests are addressed by the next release that weren't previously addressed?
- What problems have been fixed in the new release that were not fixed in the customer's current release?
above questions are answered by looking at Release Builds. But builds
are equally important for your internal customers, especially your
development and verification teams. And you can start to identify a
whole range of questions that can be answered using Integration Builds
as the handles of the queries.
- What features/problems should be tested that weren't available in the previous verification cycle?
- What changes went into the nightly build?
- What structural changes (ie. new/moved/removed files/directories) have occured since my last "rebase" operation
- What run-time data configuration files need to be retrieved to update my test bed.
Build is a key component of your CM data. It's the one that makes it
easy to work with all of the traceability data you've accumulated, as
long as your CM tool supports these sort of questions and comparisons
through simple menu functions. Pay attention to your Change Management
process and your CM data and Build automation will follow.
Joe Farah is the President and CEO of Neuma Technology . Prior to co-founding Neuma in 1990 and directing the development of CM+, Joe was Director of Software Architecture and Technology at Mitel, and in the 1970s a Development Manager at Nortel (Bell-Northern Research) where he developed the Program Library System (PLS) still heavily in use by Nortel's largest projects. A software developer since the late 1960s, Joe holds a B.A.Sc. degree in Engineering Science from the University of Toronto. You can contact Joe by email at firstname.lastname@example.org