AutoTest
Automatic test case generation
AutoTest automatically generates passing Cantata unit tests to exercise all the C source code. It can be used on new or legacy projects, making it easy to:
Autotest Video
What is Cantata AutoTest?
Cantata AutoTest creates complete unit test scripts or adds test cases to existing Cantata scripts. It parses the C source code to determine all possible paths through the code as required by a structural code coverage metric target:
- 100% function Entry-points
- 100% Statements
- 100% Decisions
- 100% Unique Cause MC/DC
An algorithm creates test case vectors which exercise all required code paths, using the Cantata powerful white-box capabilities to set data, parameters and control function call interfaces.
The test vectors drive the code, and check the parameters passed between functions, values of accessible global data, order of calls and return values.
AutoTest generated cases, are editable in the same ways as user generated cases, and each test case has a description of what path through the code it was created to exercise, making them easy to maintain.
What does AutoTest help me do?
CONFIGURE AUTOMATIC TEST GENERATION
AutoTest generation options can be easily configured via the GUI or command line:
Code coverage required (Rule Set)
How function calls are simulated or intercepted
Whether to access global static variables
Isolation unit test or cluster integration test static functions
What data to check (i.e. global data, function call arguments or return values);
Whether global data is modified during function calls to force paths;
Generation max limits on time, paths, arrays etc.
Selecting a code coverage Rule Set will determine the depth (i.e. number of test cases needed to achieve the target). The style is determined by use of the AutoTest specific and standard Cantata Workspace code generation options.
Isolation unit tests or cluster integration tests, using black-box or white-box testing approaches, provide flexibility to fit with whatever initial testing thoroughness and test maintainability for code refactoring is required.
IDENTIFY CODE TESTABILITY ISSUES
Where AutoTest is unable to generate tests, or the tests are incomplete, the AutoTest Generation Report provides useful insight into any testability issues within the source code.
Warning messages are written to the AutoTest Generation Report, showing where and why it was not possible to generate test cases, indicating possible problems in the source code or the ability to dynamically test it. It can therefore be a valuable supplement to code review, even if AutoTests are not dynamically run.
GENERATE TESTS WITH FULL CODE COVERAGE
The Cantata AutoTest Generation Report identifies the expected code coverage which will be achieved by executing the generated tests. As AutoTest cases are initially generated to pass, the only failing checks will be where a Coverage Rule Set checks achieved coverage against a target.
Achieving full code coverage is of course only part of the testing objectives. AutoTest generated test cases can be reviewed and where suitable mapped to requirements (see Cantata Requirements Traceability capability for more details). In this case, AutoTest dramatically reduces the developer effort to select test case vectors to verify the requirements.
PLUG EDGE CASE GAPS IN COVERAGE
Where complex code is being tested, even detailed requirements, and thorough robustness tests may leave untested execution paths in the code, because the different combinations of input vectors required for full coverage are difficult to identify. For these ‘edge cases’, AutoTest can generate test cases to plug any gaps in code coverage if the code is not redundant.
AutoTest can be used on just selected functions to generate a focused set of test cases achieving 100% coverage. Cantata automatic code coverage optimisation can at a single click delete/disable any test cases which do not add to the overall code coverage achieved.
CREATE A BASELINE REGRESSION TEST SAFETY NET
Legacy code that relies on system testing is very expensive and inefficient to modify, but writing a comprehensive set of unit tests from scratch is simply not commercially viable. Cantata AutoTest automatically generates a suite of unit tests for all selected code. These tests can act as a baseline safety net against future regression errors as code changes. This reduces reliance on expensive and time consuming system tests as well as being more thorough and pin-pointing errors precisely.
Cantata AutoTest generates a set of Cantata Makefiles, these can be easily integrated into any open source or commercial CI tools to enable fully automated testing during development, at every integration.
UPGRADE FROM LEGACY TEST TOOLS
Even where other unit testing tools are already in use, there can be real advantages in upgrading to the modern powerful test facilities available in Cantata. This especially true where legacy tools are not well supported or are not qualified for safety standards tool certification needs.
Cantata AutoTest provides two highly automated routes for upgrading unit test tools:
For a code base with an accepted level of quality, existing tests can simply be replaced by an AutoTest generated set of Cantata unit tests.
For legacy unit tests written in C (e.g. xUnit tools), existing tests can be incorporated into Cantata test scripts, retaining investment in legacy test cases and then enhanced with additional Cantata capabilities including AutoTest.
How does AutoTest work?
AutoTest can be invoked using a Test Script Wizard, or it can generate additional test cases in an existing test script in the Test Script Manager view. For larger code-bases it may be more convenient to run AutoTest from the command line.
AutoTest parses the source code and uses Cantata options to generate:
- AutoTest Generation Report, which provides both summary and detailed information on the tests that have been generated.
- Cantata test scripts (or additional test cases to existing scripts)
- Cantata Makefiles to build the test executable
The generated tests are then built and run, producing the standard set of Cantata results and reports.
AUTOTEST GENERATION ALGORITHM
Cantata AutoTest creates complete Cantata unit test scripts from source code using an algorithm that automatically generates test case vectors that exercise code paths, check data, check parameters and record call orders. The code paths used to generate the test cases are determined by selecting the metric types (Entry-point, Statement, Decision or unique cause MC/DC) in a code coverage Rule Set.
Preferences are available to influence the AutoTest generation algorithm, including the required level of code coverage, the method of controlling calls, what types of data to check, how data may be modified to force execution paths, time limits, path limits, and many more. These preferences affect the style of the Cantata tests, and limit the test generation scope.
Tests can be more or less thorough, and can use white-box or black-box approaches. The decision on what style of unit tests to generate with Cantata AutoTest will depend on regression plan objectives and how code changes. For example, allowing flexibility for code refactoring of static functions without breaking black-box tests.
AUTOTEST WARNING AND ERROR MESSAGES
Where Cantata AutoTest generation results in incomplete tests or encounters suspect code quality, warning messages are written to the AutoTest Generation Report. These warning messages indicate possible problems in the source code or its dynamic testability, so they can be of significant value.
Cantata AutoTest will report:
- unreachable code, where it cannot generate test cases to exercise all parts of the software under test
- potential crash scenarios such as null pointers, an out-of-bounds array index, decisions using uninitialised variables, and division by zero
- type truncation, where type casting might result in data loss
- uninitialised data use, such as a field within a large structure
- non-constant static data within a function, as its value will be unknown at the start of a test case
- non-returning functions, where a function does not return control to the caller – although Cantata does have the capability to work around this for testing purposes.
- implicit function declarations, where a function is called before it has been declared
More information about Cantata Autotest
For more detailed information about Cantata Autotest please see: