Create a project with the following build def (on Windows, not tested on other platforms):

cmake_minimum_required(VERSION 3.27)
project(dirtest CXX)
enable_testing()
set(CMAKE_RUNTIME_OUTPUT_DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR}/bin)
add_executable(dirtest dirtest.cpp)
add_test(dirtest dirtest)
# This behaves identically to the line above:
# add_test(NAME dirtest COMMAND dirtest)

The dirtest.cpp file is just an exe that returns zero. If you build this and then run tests, CTest will error out claiming the executable can't be run.

If you remove the set command, and rebuild, CTest will succeed.

If you then add the line back in and rebuild, CTest succeeds because it will run the executable from the previous build step. If you delete it, CTest will fail again.

It seems that CTest does not take CMAKE_RUNTIME_OUTPUT_DIRECTORY into consideration when looking up executables even though CMake documentation says "If specifies an executable target created by add_executable(), it will automatically be replaced by the location of the executable created at build time."