time a new build is created it needs to be tested for basic functionality.
The point of such testing is to flag serious problems in the new build,
and make it easy for the project manager to ensure that the program
will pass the first round of testing. If the new build fails initial
verification, some companies prefer to withdraw it from the test process
and to save QA resources for other tasks. Other companies evaluate
the test result that tells testers what problem areas they need to
focus on, or what areas to avoid because they are not ready yet.
This first round of testing is time
consuming, boring and requires a significant amount of QA resources.
Moreover, the test result usually is available only the day after
the new build is created.
Automated Build Certification (ABC)
is a framework of scripts that automatically installs the new build
and runs some test cases to make sure the build is not broken and is
ready for the second round of tests. It starts immediately after
the new build is created. If builds are done at the end of the day,
then the QA team can utilize the overnight time to do some up-front
automated testing before accepting the new build for the next test
There are significant benefits to having
an automated build certification process that does some automated
testing before the QA team accepts the build for testing:
Part of the acceptability testing can be automated
and run overnight. QA no longer has to take a new build and
verify that it is acceptable for testing because they can rest assured
that if the build happens and the ABC runs successfully, the build
is stable and ready for testing.
Developers are alerted ASAP if there are any problems with
the build. Often the development team knows about build problems
simultaneously with the QA team and can start working on the discovered
problem even before QA reports bugs.
As a result of using ABC, company saves
significant time on internal procedures between build creating and
starting bug fixing.
How Automated Build Certification works
The ABC contains configuration file,
main script and some testing modules that are responsible for different
areas of application testing. The configuration file allows adding
and turning on/off testing modules and their parameter configuration.
In the configuration file the user can change the order of executing
test modules. The file also contains mail lists of dedicated people
who receive ABC reports by e-mail.
Each testing module (TM) is an independent
script or application that is called by the main script. It can
have its own configuration file and creates a separate log file.
ABC can use any SilkTest, WinRunner or other automated tool script
with some minor updates as testing module. One of the testing
modules is responsible for install/uninstall verification.
The main script drives test modules,
analyzes TM log files and sends e-mail notifications with ABC results.
It can also create general Web reports for the build under certification.
The main script along with the configuration file allows running
testing modules in any order any number of times on any specified
ABC Delivery Package
CrystalTest delivers the following
1. Main script with hooks to add new modules
2. Configuration file
3. Test module that is able to install/uninstall application under
test and verify the install result
4. ABC documentation
1. Web report
2. Test modules according to customer specifications.
3. Existing SilkTest/Winrunner scripts for acceptance/regression tests
or company internal tools and scripts integrated in ABC.