testing.rst

CMake Testing Guide

The following is a guide to the CMake test suite for developers. See documentation on CMake Development for more information.

See CMake Integration Testing for running integration testing builds.

See Tests/README.rst for the test suite layout in the source tree.

Running Tests in the Build Tree

After Building CMake, one may run the test suite in the build tree using ctest(1):

• With a single-configuration CMake generator, such as Ninja or Unix Makefiles, one may simply run ctest:

  $ ctest

• With a multi-configuration CMake generator, such as :generator:`Ninja Multi-Config`, :generator:`Visual Studio <Visual Studio Generators>`, or :generator:`Xcode`, one must tell ctest which configuration to test by passing the -C <config> option:

  $ ctest -C Debug

Some useful ctest(1) options include:

-N

List test names without running them.

-V

Show verbose output from each test.

-j <N>

Run to run up to N tests concurrently.

-R <regex>

Select tests for which the regular expression matches a substring of their name.

Cleaning Test Build Trees

Many CMake tests create their own test project build trees underneath the Tests/ directory at the top of the CMake build tree. These build trees are left behind after testing completes in order to facilitate manual investigation of results. Many of the tests do not clean their build trees if they are run again, with the exception of tests using the RunCMake infrastructure.

In order to clear test build trees, drive the test_clean custom target in the CMake build tree:

$ cmake --build . --target test_clean

This removes the Tests/ subdirectories created by individual tests so they will use a fresh directory next time they run.

Running Tests with a Different Generator

After Building CMake with one CMake generator, one may configure the test suite using a different generator in a separate build tree, without building CMake itself again, by defining CMake_TEST_EXTERNAL_CMAKE to be the absolute path to the bin directory containing the cmake, ctest, and cpack executables.

For example, after building CMake with the Ninja generator:

$ cmake -B build-ninja -G Ninja -DCMAKE_BUILD_TYPE=Debug
$ cmake --build build-ninja

one may configure a second build tree to drive tests with the Ninja Multi-Config generator:

$ cmake -B build-nmc-tests -G "Ninja Multi-Config" \
  -DCMake_TEST_EXTERNAL_CMAKE="$PWD/build-ninja/bin"
$ cmake --build build-nmc-tests --config Release

The second build tree does not build CMake itself, but does configure the test suite and build test binaries. One may then run tests normally:

$ cd build-nmc-tests
$ ctest -C Release

Note that the configuration with which one drives tests in the second build tree is independent of the configuration with which CMake was built in the first.

Measuring Coverage of CMake Code

To see which lines of CMake language code, such as the files in Modules/, the tests run, use the CMAKE coverage tool of ctest:

1. Configure the CMake build with -DCTEST_TEST_COVERAGE_TOOL=CMAKE using cmake version 4.4 or newer. A plain ctest run in the build tree is then configured to collect coverage data.

2. Run the tests with the ctest that the build produced, for example:

   $ bin/ctest -R '^RunCMake\.Example$'

   For each test, ctest writes an lcov tracefile named <test-name>.cmake-coverage.info into the directory of the build tree that holds the test, here Tests/RunCMake. The tracefile leaves out the files in the build tree, such as the build trees of the test cases.

3. Keep only the records of files in the CMake source tree. The tracefile can also name files outside both trees, such as the package configuration files that find_package() reads. With lcov, where <source> is the absolute path of the CMake source tree:

   $ lcov --extract Tests/RunCMake/RunCMake.Example.cmake-coverage.info \
       '<source>/*' -o example.info

4. Make an HTML report with genhtml from lcov:

   $ genhtml -o coverage example.info