Tests not being discovered by GCC and Clang Oldest builds in C++ Internal

[jpihl]

I just noticed that some of ours build jobs are not running the tests. Maybe there's some issue with older versions of ctest?
Notice that we also specify --no-tests=error, but the step doesn't fail even though no tests are found.

Run python3 waf --run_tests
Project version                          : 5c26ec7 
Using config file                        : /opt/actions-runner/.wurf_config 
Resolve path                             : ~/wurf_dependencies 
Load "asio" (load)                       : 4.0.1      (resolve_symlinks/asio => /opt/actions-runner/wurf_dependencies/asio-a17865/af54bc82e7) 
Load "asio-source" (load)                : asio-1-... (resolve_symlinks/asio-source => /opt/actions-runner/wurf_dependencies/asio-source-aa34f2/12e0ce9e05) 
Load "polka" (load)                      : 3.0.0      (resolve_symlinks/polka => /opt/actions-runner/wurf_dependencies/polka-13f413/8cd61545ec) 
Load "bourne" (load)                     : 11.0.0     (resolve_symlinks/bourne => /opt/actions-runner/wurf_dependencies/bourne-e56c59/cf4fed3ea1) 
Load "endian" (load)                     : 14.0.0     (resolve_symlinks/endian => /opt/actions-runner/wurf_dependencies/endian-10cf49/b3702d8d13) 
Load "rely" (load)                       : 33.0.0     (resolve_symlinks/rely => /opt/actions-runner/wurf_dependencies/rely-eddec1/d879907caa) 
Load "slide" (load)                      : 15.0.0     (resolve_symlinks/slide => /opt/actions-runner/wurf_dependencies/slide-6be11f/92c7e462e4) 
Load "fifi" (load)                       : 49.0.0     (resolve_symlinks/fifi => /opt/actions-runner/wurf_dependencies/fifi-f85dcd/163bdb1dc4) 
Load "platform" (load)                   : 6.0.0      (resolve_symlinks/platform => /opt/actions-runner/wurf_dependencies/platform-e774c1/7740600dfa) 
Load "cpuid" (load)                      : 10.0.0     (resolve_symlinks/cpuid => /opt/actions-runner/wurf_dependencies/cpuid-a4173a/e3747802bd) 
Load "poke" (load)                       : 16.0.2     (resolve_symlinks/poke => /opt/actions-runner/wurf_dependencies/poke-3d576e/97669413ea) 
Load "fmt" (load)                        : 5.0.0      (resolve_symlinks/fmt => /opt/actions-runner/wurf_dependencies/fmt-807bda/5d5eb0f166) 
Load "fmt-source" (load)                 : 11.0.2     (resolve_symlinks/fmt-source => /opt/actions-runner/wurf_dependencies/fmt-source-9df202/0c9fce2ffe) 
Load "abacus" (load)                     : 8.0.0      (resolve_symlinks/abacus => /opt/actions-runner/wurf_dependencies/abacus-f8880b/964e203558) 
Load "protobuf" (load)                   : 3.0.0      (resolve_symlinks/protobuf => /opt/actions-runner/wurf_dependencies/protobuf-9cf85f/a32b195628) 
Load "protobuf-source" (load)            : v31.0      (resolve_symlinks/protobuf-source => /opt/actions-runner/wurf_dependencies/protobuf-source-42561d/3d4adad5c4) 
Load "abseil" (load)                     : 1.0.2      (resolve_symlinks/abseil => /opt/actions-runner/wurf_dependencies/abseil-216eb1/d6a9562e97) 
Load "abseil-source" (load)              : 20250512.0 (resolve_symlinks/abseil-source => /opt/actions-runner/wurf_dependencies/abseil-source-d220b4/bc257a88f7) 
Load "source_location" (load)            : 2.0.0      (resolve_symlinks/source_location => /opt/actions-runner/wurf_dependencies/source_location-8f595d/c2dbc1e904) 
Load "source_location-source" (load)     : e384945    (resolve_symlinks/source_location-source => /opt/actions-runner/wurf_dependencies/source_location-source-6869c6/e3849453a3) 
Load "aybabtu" (load)                    : 5.0.0      (resolve_symlinks/aybabtu => /opt/actions-runner/wurf_dependencies/aybabtu-2c2b54/9a374094fd) 
Load "recycle" (load)                    : 8.0.0      (resolve_symlinks/recycle => /opt/actions-runner/wurf_dependencies/recycle-fa0f6e/a590e0e123) 
Load "ranbo" (load)                      : 3.0.0      (resolve_symlinks/ranbo => /opt/actions-runner/wurf_dependencies/ranbo-0f71e4/886dba79bd) 
Load "cli11-source" (load)               : f862849... (resolve_symlinks/cli11-source => /opt/actions-runner/wurf_dependencies/cli11-source-e67ced/f862849488) 
Load "tunnel" (load)                     : 19.0.1     (resolve_symlinks/tunnel => /opt/actions-runner/wurf_dependencies/tunnel-d16b01/e1d5709ec7) 
Load "kodo" (load)                       : 33.0.1     (resolve_symlinks/kodo => /opt/actions-runner/wurf_dependencies/kodo-f3a9b9/63db019740) 
Load "tiny-process-lib" (load)           : 2.1.0      (resolve_symlinks/tiny-process-lib => /opt/actions-runner/wurf_dependencies/tiny-process-lib-472bc6/23bc729ca2) 
Load "tiny-process-lib-source" (load)    : 6166ba5... (resolve_symlinks/tiny-process-lib-source => /opt/actions-runner/wurf_dependencies/tiny-process-lib-source-13cd31/6166ba5dce) 
Load "gtest" (load)                      : 6.0.2      (resolve_symlinks/gtest => /opt/actions-runner/wurf_dependencies/gtest-39e104/8e1f80ae86) 
Load "gtest-source" (load)               : v1.17.0    (resolve_symlinks/gtest-source => /opt/actions-runner/wurf_dependencies/gtest-source-ffb79b/52eb8108c5) 
Load "protolint-linux" (load)            : protoli... (resolve_symlinks/protolint-linux => /opt/actions-runner/wurf_dependencies/protolint-linux-752e44/extract-45bd85) 
Load "protolint-macos" (load)            : protoli... (resolve_symlinks/protolint-macos => /opt/actions-runner/wurf_dependencies/protolint-macos-748ae6/extract-7d14a3) 
Load "protolock-linux" (load)            : protolo... (resolve_symlinks/protolock-linux => /opt/actions-runner/wurf_dependencies/protolock-linux-4fabe8/extract-a7f0ff) 
Load "protolock-macos" (load)            : protolo... (resolve_symlinks/protolock-macos => /opt/actions-runner/wurf_dependencies/protolock-macos-a79d78/extract-59da85) 
Load "expected" (load)                   : 2.0.0      (resolve_symlinks/expected => /opt/actions-runner/wurf_dependencies/expected-bf0168/e84368a61a) 
Load "expected-source" (load)            : v1.1.0     (resolve_symlinks/expected-source => /opt/actions-runner/wurf_dependencies/expected-source-3b6b8b/292eff8bd8) 
Load "boringssl" (load)                  : 781a72b... (resolve_symlinks/boringssl => /opt/actions-runner/wurf_dependencies/boringssl-3aeb0c/781a72b2aa) 
Load "verify" (load)                     : 6.0.2      (resolve_symlinks/verify => /opt/actions-runner/wurf_dependencies/verify-652564/7f7c4b0d66) 
Load "libassert-source" (load)           : v2.1.5     (resolve_symlinks/libassert-source => /opt/actions-runner/wurf_dependencies/libassert-source-cdbe2c/6066782926) 
Load "zlib-source" (load)                : 51b7f2a... (resolve_symlinks/zlib-source => /opt/actions-runner/wurf_dependencies/zlib-source-696d19/51b7f2abda) 
Load "raft" (load)                       : 25.0.0     (resolve_symlinks/raft => /opt/actions-runner/wurf_dependencies/raft-3a0f2c/8127070442) 
Load "toolchains" (load)                 : 1.0.5      (resolve_symlinks/toolchains => /opt/actions-runner/wurf_dependencies/toolchains-e20223/d771e98c84) 
Waf: Entering directory `/opt/actions-runner/_work/plumr/plumr/build'
[0/2] /usr/bin/cmake -P /opt/actions-runner/_work/plumr/plumr/build/CMakeFiles/VerifyGlobs.cmake
ninja: no work to do.
UpdateCTestConfiguration  from :/opt/actions-runner/_work/plumr/plumr/DartConfiguration.tcl
UpdateCTestConfiguration  from :/opt/actions-runner/_work/plumr/plumr/DartConfiguration.tcl
Test project /opt/actions-runner/_work/plumr/plumr
Constructing a list of tests
Updating test list for fixtures
Added 0 tests to meet fixture requirements
Checking test dependency graph...
Checking test dependency graph end
No tests were found!!!
Running command: ['cmake', '--build', '/opt/actions-runner/_work/plumr/plumr/build', '--parallel', '7']
Running command: ['ctest', '-VV', '--no-tests=error', '-C', 'Debug', '--test-dir', '/opt/actions-runner/_work/plumr/plumr/build']
Waf: Leaving directory `/opt/actions-runner/_work/plumr/plumr/build'
'build' finished successfully (0.093s)

[jaxels20]

This is weird... Is this happening consistently? have you noticed any pattern?

[jpihl]

It seems to be consistent yes

[mortenvp]

This is an issue with older CMake versions. The option --no-tests was only added in CTest 3.26, and I think we are using an older version on the old Ubuntu runners. This probably only happens on those older runners. I noticed it when we first started moving fully to CMake.

https://cmake.org/cmake/help/latest/manual/ctest.1.html#cmdoption-ctest-no-tests

I considered making a fix in Waf by checking for the string No tests were found!!! in the output. At the time, I didn’t implement it since I assumed we would soon move to a newer CMake anyway.

But maybe we should fix it after all?

[jpihl]

I think that would be great. But it i think we also need to find a solution for detecting the tests with ctest (if possible).

[mortenvp]

But it i think we also need to find a solution for detecting the tests with ctest (if possible).

Not, 100% sure I know what you mean. But, if I understand correctly :) the issue is that ctest does not have any provisions for saying that no tests is a failure (until version that added the above flags). Before that no tests did not result in a non-zero exit code or anything similar. I think the only way to check if ctest finds any tests (with old ctest) is to check some of the generated files or parse the stdout/stderr and look for certain strings.

[jpihl]

What I mean is that this issue is twofold, and we need to address both problems:

1. Specifying --no-tests=error does not cause the ctest command to fail on older versions.
2. Older versions of ctest do not discover our unit tests in plumr.

[mortenvp]

Ah, yes only (1) can be fixed with Waf you are right 👍 Issue (2) would need some investigation.