More white papers:
Find out more ...
|
|
Neuma White Paper:
CM: THE NEXT GENERATION of Test Case Management
Testing is a complex discipline. There are various approaches,
methodologies, strategies. So where is the connection with CM? As
with development, requirements specifications, and other aspects of
product development, the connection is on the management side. A
Software CM Audit (FCA - Functional Configuration Audit) is really
about demonstrating that you have test case coverage for your
requirements, and that the test cases have been successfully run
against the target build. As it's Christmas/Hanukkah season, I'll keep
my comments brief so that you have more time to celebrate.
Viva la difference!
Test
case management is similar to software management and to requirements
management, to some extent. But it's the differences that make test
case management interesting. Let's explore a few of these differences.
(1)
Just as requirements may be applicable to more than one product, so too
test cases may be common from one product to another, especially when
the same manufacturer is involved, or when an industry standard is
being addressed.
(2) Test case management also involves tracking
and management of test case results so that the FCA can be automated.
Unlike the test cases, the test case results are specific to a
particular product build.
(3) Whereas software tends to evolve,
changes to test cases tend more to be additive. That is, new test
cases are written to supplement the existing ones, much more frequently
than upgrades to existing test cases are made. This depends on the
granularity of a test case, however.
(4) Test cases are clearly
divided along automation lines. Automated test cases can be run
frequently, repeatedly and without human intervention. Presumably, the
results can be extracted automatically as well. Manual test cases chew
up resources. So the goal is always to move more and more of these test
cases to the automated side of the fence.
(5) Test cases require
specific conditions under which they must be run. Management of these
conditions, the test environment, are as much a part of the test case
as the operational test itself is.
(6) Depending upon
granularity, the number of test cases may be very large. So it is both
important that test case management has some degree of automated
identification, as well as hierarchical, and other organizational
capabilities. This includes option and variant tagging.
So what is the impact of these attributes on current and future CM systems?
Product Management
The
CM process/tools must incorporate product management. The CM
repositories must share test cases among products in a product family,
and beyond. It's not good enough to be able to export test cases from
one product repository to another. That leaves you managing multiple
copies of the same test suite, which is really no different than
copying a complete set of source code for each development stream.
Test Case Results
It
is not sufficient to track the test cases in the CM repository. The
data runs must also be tracked. It must be possible to associate a
test case run with a specific build and with a specific tester
verification session. We need to be able to asked who tested what,
when and what were the results. This is the proof we need to conduct
the Functional Configuration Audit, so we might as well automate that
audit while we're at it, or at least leave the audit to auditing the
CM/Verification tools and processes rather than having to wade through
the mounds of data.
Tracking test case results in the CM repository also allows a number of other capabilities and queries. For example:
- When did this test case last fail.
- Which requirements have never been tested?
- Which requirements are not yet verified as complete?
- What's the test history for a particular test case?
A
number of testing metrics are also easily generated from the
combination of test case run data and its traceability back to test
cases, builds, and testers.
Change Control of Test Cases
Change
to the repository of test cases is most often in the form of adding in
new test cases. This is especially the case if the granularity of a
test case is small. A good CM system helps to manage aggregates of
small test cases so that individual tests can be represented as
individual test cases while still allowing the management of less
granular clusters.
A fine granularity has the side effect for
test cases that changes to the test case repository are generally of
the form of adding in new test cases, rather than testing existing
ones. This obviously depends on the test specification complexity. But
generally, the user interface must be optimized for adding and
tagging/organizing test cases. This is not to say that full change and
version control will not be necessary. But often, the original test
case is as important as the revised and must be maintained as a
separate test case. Thus, there is less revisioning and more
management of test cases.
Automated Test Cases
Test
case automation is essential to quality. But CM tools can help by
allowing easy packaging of an automated test suite based on the build
and the features and options contained within it or targeted for
testing. The CM tools can also help to automate the collection of test
results, especially when it can inject the test case identification
into test suites. This is highly dependent on both the types of tests
and the test engine technology.
The Test Environment
The
CM tool can track not only the test case but the conditions for running
the test. More advanced systems should allow specific environments to
be defined that can be shared by multiple test cases. When test
environments and data setups can be tracked separate from the test case
itself, there is an economy of specification, and the resulting economy
of management effort for changes to the environment. At the same time,
the tools should allow a complete test case description to be presented
entirely as a single set of instructions. This is especially important
when addressing failed test cases individually. This occurs when the
failed test case is fed to the development team for diagnosis, starting
with reproduction of the test case that caused the failure.
Scalability
When
the granularity of the test cases is fine, the number of test cases can
easily grow into the tens of thousands. It is important that CM tools
be able to scale to these requirements - especially considering that it
is desireable to track multiple products in the same repository to
facilitate sharing of test cases and requirements across products.
Test Case Management - An Integrated Solution
A
next generation CM tool will consider test case management as part of a
full ALM solution, and not as an isolating test case management
function. Role management, generation of problem reports, coverage of
requirements, traceability to builds, and hence to the changes in each
build - these are all essential components of test case management. A
solution that is designed in the context of an end-to-end lifecycle
will yield real advantages.
The user interface is critical, but
not just from a verification team perspective. The developers need to
be able to navigate test results. Management needs to navigate the
data but also correlate it with changes, problem report arrival rates,
audit requirements, builds, etc. A combination of object-oriented
capabilities, management dashboards and traceability navigation aids
should be an important component of the UI. Object-oriented operations
should be applicable to hierarchical displays to facilitate reporting,
metrics, test case extraction, etc. on hierarchical subtrees.
Merry Christmas
I've
kept the column shorter and lighter than usual. Hopefully these few
ideas will help you both in your process design and tool acquisition.
But that can wait a couple of weeks. Have a Merry Christmas or Happy
Hanukkah and get some rest so that you can take on those important CM
projects with more enthusiasm in the New Year.
Joe Farah is the President and CEO of Neuma Technology . Prior to co-founding Neuma in 1990 and directing the development of CM+, Joe was Director of Software Architecture and Technology at Mitel, and in the 1970s a Development Manager at Nortel (Bell-Northern Research) where he developed the Program Library System (PLS) still heavily in use by Nortel's largest projects. A software developer since the late 1960s, Joe holds a B.A.Sc. degree in Engineering Science from the University of Toronto. You can contact Joe by email at farah@neuma.com
|
