White Paper: CM: THE NEXT GENERATION - Don't Let Our Mistakes Hold Us Back

Mistake #1: We assume that the most expensive, or most prevalent solution is the best
Why is this not the case? Is Windows the best? It may be getting there, but the Mac has put it to shame over the last quarter century, even though Mac's market share remains low. (By the way, I'm not, and never have been, a Mac user.) What about rockets? A little company (less than 1000) named "SpaceX" is raising eyebrows as it promises to lower the cost to space by an order of magnitude, with it's first successful launch last year and it's heavier lift vehicles targeted for this year.

CM is a complex business. The market leaders got there by carving out a niche and growing market share on it. Maybe its freeware (CVS), COTS packaging (PVCS), virtual file system (ClearCase). Whatever the reasons, these have captured market share. But don't make the mistake of assuming the market leading tools are the best. Look around and you'll see more able, more reliable and less costly, and less risky, solutions. Just as for CVS, SVN goes one better, the same likely holds for any solution that is not delivering.

Branching Strategy is Key
On another occasion I was exposed to a project while they were deciding on where to go from their current CM solution. They needed to work in a multiple site environment and they were taking the time to make sure that their revised branching strategy was going to work. They had spent a couple of months on it and had a strategy in the review process, but just weren't yet sure how well it would work. They did know that they would have to spend some time training the staff in the strategy and implement some quality controls to ensure that the strategy was being followed properly. In the mean time, another company about the same size, needing to work in a multple site environment, had a very simple release-based branching strategy with exclusive checkouts. It took them virtually no time to train staff and they used the tool to enforce the strategy.

Mistake #2: We develop ever more complex branching strategies to deal with ever more complex CM.
Why were these two situations so different? More than likely, the reason is one of process and one of matching technology to the process. The CM world invented branching almost 40 years ago. It allowed them to work on the next release in parallel with supporting the current one. It was and is a great concept and a valuable tool. However, processes evolved that said:

It's good to see process evolving. But when the technology doesn't evolve as well, guess what happens. Well two things happened. The first was the invention of the "MAIN" branch. The second was the overloading of the branching concept with a myriad set of reasons and the addition of labelling technology to help to sort out the mess caused by this overloading.

I know there are two camps out there: MAIN branch, and main-per-release branch. I hope there won't always be. Why is there a MAIN branch camp? Because we sculpt our tools to support it. However, however well we sculpt, a MAIN branch solution will always be burdened with the following problems:

You can probably add to this list. In a branch-per-release strategy, each release has its own branch. You always do release 2 specific work in the release 2 branch, and release 3 specific work in the release 3 branch. Now if your tools support the branch-per-release concept, they will help you. For example, they will let you know when you make a release 2 change, that it may have to be applied to release 3 or later streams as well. They will give you the option of automatically inheriting changes to an earlier release to the next and later releases, letting you establish this as a default policy which you need to override when you don't want it, or having you choose on a case-by-case basis. If your tool has a release stream context for development, it can automatically tell you when you need to branch for reasons of parallel release development, rather than you having to instruct your staff on how to determine when to branch.

The second thing I mentioned was the overloading of the branching concept. This is because the CM technology did not evolve to support the evolving process. Older technology may require that you branch code in order support promotion levels. It may require that you branch code so that you can label it with the feature, the change, the developer, etc. It may require that you create branches and/or add labels to identify baselines and builds. It may require that you branch code to support short-term (i.e. change duration) parallel changes. On top of all of this, there's a lot of labeling and merging that results - and unfortunately, that's where the CM technology tends to evolve - to make these tasks easier and less resource intensive, rather than eliminating most of them in the first place.

Now consider this. If you were looking at new tools, would you look first at how the tools simplify the entire branching architecture, or would it be more important to you that the tools provide advanced multi-way merging, have multiple ways of viewing branching, or have enhanced labeling mechanisms and a means to catagorize and navigate labels?

If that's not enough to consider, assume your product has been bought out by a new company, or that your development contract does not get renewed and the source code is being returned to the owner, or that one site is being shut down and all of the development code is being consolidated at a single site (without the staff from the other site). How long will it take the owner to unravel all of the branches and labels that exist in the development environment - even if the CM tool kept perfect track of them all? The answer here should be, at most, a few hours, not a few weeks or months or even years. If you think that's impossible, all it means is that it's time for you to review CM technology again.

Branching strategy is key, it's true. But the minimal strategy that meets the requirements must take precedence over the minimal CM tool functionality mentality. CM tools need to deal with promotion without forcing every file to branch. The same goes for change packaging, for short term parallel changes, for change owner identification, for baseline and build identification. Design teams need to use the tools effectively, handling variants as a software engineering capability first, and only when that's in place, using CM to support it. Undisciplined branching will eventually contribute to the downfall of a project.

A Monolithic Monster
I remember the days of PCTE (portable common tool environment), of "backplane" technology, and of other efforts to get all of the best of breed management tools to work together in a cooperative manner - requirements, test management, change management, version control, document management, build and release management. Contrast that with the companies that are trying to build all of those things into one giant monolithic monster. No way - I'd rather imagine having the best of breed and them all working together. Twenty-something years later, we're still working at such a solution.

If I'm using tool A, B, and C, and you give me a way of helping them work together so that I have better overall management and traceability, that's great. That's fantastic. I don't have to create the glue myself and, even better, I don't have to maintain it. In these days of open source, especially, someone else is doing that for me.

Mistake #3: Common backplane schemes reduce effort and costs, but they don't come close to well designed monolithic tools
I've been doing software development since the late '60s and there's one thing I've noticed - it's generally better, less costly and takes less time to add-on your own mouse trap than to try to integrate with an existing one. There are exceptions, of course. I wouldn't want to build an OS onto my CM tool and then try to sell the CM tool with the OS. But a CM tool is layered on top of an OS. A problem tracking tool is not. Nor is a requirements tool, nor a release management tool. They're more like siblings. And I wouldn't say that a Problem Tracking company could relatively easily build in a CM component.

But CM is the heart of the ALM cycle - it deals with configurations and changes to the configurations. That's a complex process. It requires data repositories, process engines, advanced user interfaces, etc. If you use these "engines" as a common platform for all of the N ALM functions, you simplify administration by a factor of N; you don't have to figure out how N multiple site solutions will work together; you don't have to build message structures and concepts for interchanging data among N different tools; you don't have to build glue for N*N/2 tool integrations; you don't have to adjust N-1 pieces of glue every time 1 of the N pieces are upgraded. In fact, you probably won't need training on N tools. You can probably customize all N tools in fairly much the same way.

With a monolithic system, if you're building an ALM tool, you can put most of your energy into the common engines, which will benefit all functions, and with the resultant resource savings, you can spend more time on individual functions to tailor them more specifically to the requirements of the day. This is precisely what two Canadian companies (MKS and Neuma) have decided is the best approach. Look also at IBM, a company which is recognizing this and is making a valiant effort to take old technology and merge it into one of them monolithic "monsters". Bravo. The payback is way more than you can imagine if you have not yet been exposed tosuch systems.

In a monolithic system, you have all of the data at hand. It's in a single repository so traceability is easy and fast, and is easily supported by a common process engine. Hopefully, you have a common user interface that is customized by role rather than by CM function, making it more user friendly and easier to navigate. That in turn allows you to focus on higher-level roles, like CRBs, product managers, VPs and Directors, etc. It also allows you to extend your ALM tools to support live meetings, since all of the data you need for decision making is at hand, hopefully in a well designed format that allows you both management summaries and drill down to details and across traceability links.

I've been exposed to a number of monolithic systems, even some bad ones. And although I'd stay very far away from the bad ones, I would never start a project with a glued together solution, or even with just a small portion of the functions in one tool. I'd always start with the "monster".

We Have The Expertise
Nortel selected a CM tool for corporate use because it was the "best". Really it was because they had expertise on board that knew the product. They had a few successful products that were using the tool, each with its own CM customization team. They knew it needed a lot of customization, consulting, administration, etc. But they also argued that they had the best minds, were up for the task and that money wasn't the issue. They took on the tool as a challange, one that could beat out its own home-grown technology. They probably put somewhere between $100 million and a quarter billion dollars into customizing, administering and operating their CM system. Today, Nortel is under bankrupcy protection.

Mistake #4: We hold on to a system because of our familiarity with it, regardless of the cost
There are many places I've been where the choice for the CM technology was based on the experience of the CM manager making the decision. One tends to go with that with which one is familiar. One tends not to look so much at the cost. And why? Well if the solution requires lots of training for administration, customization and just plain usage, and that training takes lots of time and money, and I've already been through the training, why shouldn't I hasten the time to deployment and cut training costs by going with something I already know? Well, why not? The reason may be that there is technology out there that doesn't require nearly the same amount of training and has better functionality and operating costs as well. There are technical people out there that still like to use only command line interfaces because they know them well and can do anything with them - be it a CM tool or an OS (e.g. Unix). I don't have any problem with that. But I do when they then try to impose it on the entire team. I know of companies that are hurting but are burdened with large CM operating costs and I suspect that there will be more than one such company where such costs help prevent it from surviving the current economic crisis. When you look for technology, don't start from where you were 5 or 10 years ago - look at the market anew.

A similar situation arises in building your solution in-house. Nortel, did this in the mid-'70s, and had perhaps the best CM system of its time, running on an IBM mainframe. AT&T had similar success and even took their system commercial at one point. IBM as well. These were great systems and while Nortel, AT&T and IBM are able to finance such ventures, they are not their core competence. These companies did so in their day because there were no other solutions that compared favorably. They held on to them for long periods and built large teams around them, even though they started out as very small team efforts, even through initial maturity. But as the teams grew in size, the business case for them became less compelling. This is why Nortel started moving away, about a decade ago, from its in-house CM, though I'm sure much of it is still in use. They had the expertise, but they didn't manage the overall costs well. Less than $1 million to create a CM tool that saved them millions - then they grew the team so that it was too expensive to evovle. Rather than try to spin out the group into a separate company, or reduce it back down to its original size of a dozen people or less, they abandoned their own in-house tool and moved on to new technology. A good move perhaps, except guess what... Mistake #1 followed by Mistake #4.

Change Control, Not File Control
CM tools evolved in the late '60s and early '70s, with the advent of SCCS and other less known tools. These would allow you to store multiple versions of a file in a single file, delta compressed. This capability evolved and evolved - but, with some exceptions, remaining file based. A few organizations recognized the need for change-based CM, tracking changes, not to file, but which were implemented by revisions to a set of files. Even Nortel's home system realized this (hence it's longevity). But commercial vendors, with the exception of a couple, pushed the file-based agenda.

Mistake #5: Change-based CM, not File-based
It wasn't until the late '90s that most vendors conceeded that change packaging was essential and file-based CM just got you into trouble. So the scramble started to improve tools. Few did an admirable job. Most looked at adding a separate concept called a change package or task or similar so that files could be tied together into a change package, but required the use of additional tools or user interfaces. As we turned the century, some realized that there were actually some benefits to looking at things from the change perspective rather than from a file perspective.

Very few CM tools have grown up with or have moved to a change-centric focus. As a result, most developers still look at file-based tools and what improvements can be made from a file-based perspective. A change-centric CM tool makes life so much easier. It makes CM natural - not an extra process that developers must put up with. In fact, it greatly reduces their effort. Imagine merging a change from release 3 to release 2 by selecting the change and saying "merge please". This is so much different than identifying all of the files that changed to implement a feature, looking at the branching history of each and figuring out which files need to merge and then merging each one. Or imagine looking through a set of 20 changes for the smoking gun that broke the build, as opposed to finding and looking through the 73 files that were involved in those changes.

CM's rather slow evolution, at least compared to other technologies (OK, other than RDBMS systems), is largely due to the file-centric view that still dominates a number of tools. Perhaps some can be forgiven, if they evolved from a hardware support background. Hardware CM is primarily a part-based change tracking system to this day. That's because hardware has a greater affinity to parts. My mouse stops working... what part failed? I've got no power... is it the power wiring or the power supply? Not to say that hardware can't benefit from change packages - just that it can survive better without them, especially because of the more rigid ECR/ECO process built into most hardware development shops.

But software is functional - the menu button is not highlighted... maybe it's the specification, or are there certain conditions somewhere that have to be met, or is it a bug in the menu software, or did someone overwrite some memory, or am I out of resources, or... Software functions span the architecture. A bug doesn't point to a file like a flat tire points to a wheel. But if it used to work, it may point to a recent change. As well, software doesn't wear out like hardware, and isn't as environmentally particular. So the failure is not fatigue or operating conditions. Software changes usually invovle more than one file, possibly in totally disjoint parts of the system - e.g. the User Interface and the Database. Fixes can usually be done in multiple ways - we can check for the symptom and fail, we can find the root cause and fix it there, we can disable that functionality, or if that's in the stable code, we can code around using it, and so on.

Change-based CM has been embraced too late by the industy. And it really won't be much longer that file-based CM is tollerated, I hope.

What's a Few Mistakes
So these are some of the industry's mistakes, whether by tool vendors, process engineers or CM teams. What's the impact? Has this really slowed us down? Well, yes. In fact, YES. Whereas operating systems have moved from very complex tuneable systems that software engineers had to customize and babysit, to plug-in-and-play software to which anyone can make substantial customizations, CM is largely still a technical mess - and in many cases we continue to build capabilities to manage the mess.

CM is complex, but so are operating systems. Although more expansive, an OS is still not as complex as CM. Why? Because one deals with making well behaved processes conform to the software rules for the benefit of one or a few individuals, while the other deals with trying to get a team to work together. In my years I have contemplated, with more than one CM system, evolving the technology into an OS platform. Only one OS (VMS) have I ever considered evolving the technology into a CM system. I've done neither, but keep my options open on the former.

CM is complex and difficult. But the user roles must not be. Even though hundreds of users are working with tens of thousands of files, hundreds or thousands of requirements, thousands upon thousands of problem reports/defects, myriads of test cases, in multple development/support streams of dozens of products having several variants, it must not be complex and difficult for the user.

It's time to throw away the knives and stones and start using real technology. It's time to capture the data from the thousands of user actions each day and figure out what's going on, and even further, provide active feedback to the users so that rather than asking the CM tool questions, the CM tool is providing the answers before they're asked. You need to branch. Your workspace file is different than the one in your view. This change may have to be applied to other "future" streams. Here's all the information you need to know about why this line of code was changed. Your schedule is at risk because of x, y and z. Here's what your customer is using now and this is what the functional and fix differences are between that product and what you're releasing tomorrow.

Impressive. Pie-in-the-sky perhaps? Perhaps not. In fact, let's go one step further. We've got accountants that have variants of their spreadsheets as they work through their reconciliations. We've got lawyers that evovle their agreements as they work through negotiations. We've got sons and daughters who work through essays and theses. We've got physicists who repeat experiments, subtly different from the previous run, or many the same with different results. These have not yet even been exposed to CM, yet they are just as much in need as developers. Can we actually make CM tools that are easy enough for such a class of users? You betcha. We can. We don't. We'd rather evolve our band-aids to deal with our knives and stones rather than move on.

Surely I've only touched on a few of our mistakes. And surely new vendors sprout up with a great new concept that drives sales for their CM tools. But we've got to move away from the days of "I like this CPU instruction set" and "SRAM has a lot of advanatages over DRAM" to the days of "Here's a computer that your grandma can use." We haven't done this with CM, and until the industry starts looking seriously at the potential, it will be content with replacing the knife with a Swiss Army Knife, and the stones with rubies and gems. Until we start to admit to our mistakes and drive to the next generation of CM, we'll stagnate.

Ask yourself, are the processes and tools I'm using today for CM substantially different than the ones I was using 5 years ago, 10 years ago, 15 years ago? Or are they just packaged prettier, with add-ons to make some of the admin-intensive tasks more tollerable? I know of 2 or 3 companies who can (and many that will) say their tools are significantly better than 3 years ago. I know of many, many more who have just spruced up the package, or perhaps customized it for another sector. Nothing wrong with that, but that won't get us ahead. Let's hope the 2 or 3 can stick with it and bring about the change needed to drag the rest of the industry forward. That's exactly what I think is going to happen... and then the pace will quicken.

Joe Farah is the President and CEO of Neuma Technology . Prior to co-founding Neuma in 1990 and directing the development of CM+, Joe was Director of Software Architecture and Technology at Mitel, and in the 1970s a Development Manager at Nortel (Bell-Northern Research) where he developed the Program Library System (PLS) still heavily in use by Nortel's largest projects. A software developer since the late 1960s, Joe holds a B.A.Sc. degree in Engineering Science from the University of Toronto. You can contact Joe by email at farah@neuma.com

More white papers:

Find out more ...

Neuma White Paper:

CM: THE NEXT GENERATION - Don't Let Our Mistakes Hold Us Back