AutoTest
What is Cantata AutoTest?
AutoTest is a powerful tool that automates the generation of Cantata unit tests for C and C++ source code, ensuring full code coverage and identifying testability issues. It can be used on both new and legacy projects, making it easy to configure automatic test generation, plug edge case gaps in coverage, and create a safety net of baseline regression tests. With AutoTest, you can easily upgrade from legacy test tools to Cantata.
Cantata AutoTest creates complete unit test scripts or adds test cases to existing Cantata scripts. It parses the C or C++ source code to determine all possible paths through the code as required by a structural code coverage metric target:
An algorithm generates test case vectors that exercise all required code paths, checking parameters passed between functions, values of accessible global data, order of calls, and return values.
Automatic test generation results are editable in the same way as user-generated cases, and each test case has a description of the path through the code it was created to exercise, making them easy to maintain.
More information about Cantata Autotest
For more detailed information about Cantata automatic test generation please see:
What does AutoTest help me do?
CONFIGURE AUTOMATIC TEST GENERATION
As an automated test case generation tool, Cantata AutoTest can be easily configured via the GUI or command line:
By selecting a code coverage Rule Set, you can determine the depth, which is the number of test cases required to achieve the target. The test style is determined by using AutoTest-specific and standard Cantata Workspace code generation options. Isolation unit tests or cluster integration tests, using black-box or white-box testing approaches, offer the flexibility to fit the initial testing thoroughness and test maintainability required for code refactoring.
IDENTIFY CODE TESTABILITY ISSUES
If AutoTest cannot generate tests or the tests are incomplete, the AutoTest Generation Report provides useful insight into any testability issues within the source code.
Warning messages are written to the report, showing where and why it was not possible to generate test cases, indicating possible problems in the source code or the ability to dynamically test it. Therefore, the report can be a valuable supplement to code review, even if AutoTests are not dynamically run.
GENERATE TESTS WITH FULL CODE COVERAGE
The Cantata AutoTest Generation Report identifies the expected code coverage that will be achieved by executing the generated tests. As AutoTest cases are initially generated to pass, the only failing checks will be where a Coverage Rule Set checks achieved coverage against a target.
Achieving full code coverage is of course only part of the testing objectives. Achieving full code coverage is, of course, only part of the testing objectives. AutoTest-generated test cases can be reviewed and, where suitable, mapped to requirements (see Cantata Requirements Traceability capability for more details). In this case, AutoTest dramatically reduces the developer’s effort to select test case vectors to verify the requirements.
PLUG EDGE CASE GAPS IN COVERAGE
When testing complex code, even detailed requirements and thorough robustness tests may leave untested execution paths in the code because the different combinations of input vectors required for full coverage are difficult to identify. For these “edge cases,” AutoTest, test case generation tool, can fill any gaps in code coverage if the code is not redundant.
Automatic test case generation can be used on just selected functions to generate a focused set of test cases achieving 100% coverage. Cantata’s automatic code coverage optimization can, with a single click, delete or disable any test cases that do not add to the overall code coverage achieved.
CREATE A BASELINE REGRESSION TEST SAFETY NET
Legacy code that relies on system testing is very expensive and inefficient to modify, but writing a comprehensive set of unit tests from scratch is simply not commercially viable. Cantata AutoTest automatically generates a suite of unit tests for all selected code. These tests can act as a baseline safety net against future regression errors as code changes. This reduces reliance on expensive and time-consuming system tests while being more thorough and pinpointing errors precisely.
Cantata AutoTest generates a set of Cantata Makefiles, which can be easily integrated into any open-source or commercial CI tools to enable fully automated testing during development at every integration.
UPGRADE FROM LEGACY TEST TOOLS
Even where other unit testing tools are already in use, there can be real advantages in upgrading to the modern, powerful test facilities available in Cantata. This is especially true where legacy tools are not well supported or is not qualified for safety standards and tool certification needs.
Cantata automatic test generation provides two highly automated routes for upgrading unit test tools to generate test cases automatically:
For a codebase with an acceptable level of quality, existing tests can simply be replaced by an AutoTest-generated set of Cantata unit tests.
For legacy unit tests written in C or C++ (e.g., xUnit tools), existing tests can be incorporated into Cantata test scripts, retaining investment in legacy test cases and then enhanced with additional Cantata capabilities including automatic test case generation.
How does AutoTest work?
Automatic test generation can be invoked using the Test Script Wizard, or it can generate additional test cases in an existing test script in the Test Script Manager view. For larger codebases, it may be more convenient to run AutoTest from the command line.
AutoTest parses the source code and uses Cantata options to generate:
The generated tests are then built and run, producing the standard set of Cantata results and reports.
AUTOTEST GENERATION ALGORITHM
Cantata AutoTest creates complete Cantata unit test scripts from source code using an algorithm that automatically generates test case vectors that exercise code paths, check data, check parameters, and record call orders. The code paths used to automatic test case generation are determined by selecting the metric types (entry-point, statement, decision, or unique cause MC/DC) in a code coverage Rule Set.
Preferences are available to influence the AutoTest generation algorithm, including the required level of code coverage, the method of controlling calls, what types of data to check, how data may be modified to force execution paths, time limits, path limits, and many more. These preferences affect the style of the Cantata tests and limit the test generation scope.
Tests can be more or less thorough and can use white-box or black-box approaches. The decision on what style of unit tests to generate with Cantata AutoTest will depend on the regression plan objectives and how the code changes. For example, allowing flexibility for code refactoring of static functions without breaking black-box tests.
AUTOTEST WARNING AND ERROR MESSAGES
When Cantata automatic test case generation tool results in incomplete tests or encounters suspect code quality, warning messages are written to the AutoTest Generation Report. These warning messages indicate possible problems in the source code or its dynamic testability, and can be of significant value.
Cantata automatic test generation will report:
