Build, test and sanitizers

Key ideas

1. 01

   A test that has never failed might be unable to fail

   Watch every new test fail once, against missing or deliberately broken code, before you trust it to pass (F4.07). In the check project, a planted bug in a degrees-to-radians function is caught by a test whose expected value comes from elsewhere ($\sin 30^\circ = \tfrac12$), and sails past a test whose expected value was copied from the code.[1][2]

2. 02

   Expected values come from an independent route

   A hand calculation, a closed form, a known identity, a second method, a reference implementation: anything but the code under test. Module 06's inertia is checked this way, by hand against MuJoCo's own mass matrix (F6.04).

3. 03

   Compare doubles with intent

   EXPECT_DOUBLE_EQ allows 4 ULPs, a gap that scales with the numbers: at $10^9$ it is 4.8×10⁻⁷ in absolute terms, and near zero it rejects a difference as small as $10^{-20}$ (F4.04). For a computed result, choose a tolerance from the error you expect and use EXPECT_NEAR (F4.02).[3][4]

4. 04

   EXPECT reports and carries on; ASSERT returns

   Use ASSERT_* when the rest of the test is meaningless without the condition (a model that failed to load), and EXPECT_* for everything else, so one run reports every problem (F4.03). A failed ASSERT returns from the current function only.[5]

5. 05

   Sanitizers make silent bugs loud

   AddressSanitizer stops the program at a use-after-free or an out-of-bounds access; UBSan catches undefined behaviour such as signed overflow (F4.05). By default most UBSan checks print a report and carry on, exit code zero, so a test still passes; -fno-sanitize-recover makes them stop. The check project shows both outcomes.[1][6][7]

6. 06

   A sanitizer report is a diagnosis; switching it off is not a cure

   Turning a check off tells you which check fired. It does not change the program. In the check project, a real read past the end of a vector is reported, and with the check switched off the same program runs to the end, prints garbage and exits cleanly (F4.06).[1][8]

7. 07

   Know when your tests are discovered

   gtest_discover_tests lists the tests by running the test binary. With POST_BUILD that happens during the build, so a binary that cannot start breaks the build. With PRE_TEST it happens when ctest starts: the build succeeds, and ctest stops before running any test, healthy ones included (F4.08).[1][9]

8. 08

   Debug for correctness, Release for timing

   AddressSanitizer typically slows a program by about a factor of two, on top of an unoptimised build. Run the tests in the sanitized debug preset every day, and measure speed only in release.[6]

Formulas and rules

The same entries as the formula sheet. Open an entry's details for its symbols and the checks behind it.

1. Machine epsilon, double precision

   $$\varepsilon = 2^{-52} \approx 2.22\times10^{-16}$$
   F4.01✓ 1/1 checked[10][11]
   Symbols and checks

   Symbol	Meaning	Unit
   $ε$	gap between 1 and the next double	dimensionless

   • ✓ machine epsilon (numpy) vs 2^-52expected 2.22045×10⁻¹⁶ · got 2.22045×10⁻¹⁶ · absolute error 0

2. Floating-point comparison with a tolerance

   Choose rtol from the error you expect (round-off grows with the number of operations), and atol for values near zero.

   $$|a-b| \le \max\!\big(\text{atol},\ \text{rtol}\cdot\max(|a|,|b|)\big)$$
   F4.02[3][4]
   Symbols

   Symbol	Meaning	Unit
   $atol$	absolute tolerance	units of a
   $rtol$	relative tolerance	dimensionless

3. EXPECT carries on, ASSERT returns

   A failed ASSERT returns from the current function only, so code after it in that function does not run (including clean-up written there).

   ASSERT_NE(model, nullptr);       // fatal: no point continuing without it
   EXPECT_EQ(model->nv, 1);         // non-fatal: report, keep checking
   EXPECT_GT(model->opt.timestep, 0.0);

   F4.03✓ 2/2 checked[5]
   Symbols and checks

   • ✓ code after a failed EXPECT still runsobserved as stated
   • ✓ code after a failed ASSERT does not runobserved as stated

4. Comparing doubles in GoogleTest

   Four ULPs is between about 4×10⁻¹⁶ and 9×10⁻¹⁶ relative, so only round-off passes. Compared with exactly zero it passes nothing but the smallest subnormal numbers, so near zero use EXPECT_NEAR, with a bound from the error you expect (F4.02).

   EXPECT_DOUBLE_EQ(a, b);        // within 4 ULPs: scales with the numbers
   EXPECT_NEAR(a, b, abs_error);  // within an absolute bound you choose

   F4.04✓ 5/5 checked[3]
   Symbols and checks

   • ✓ EXPECT_DOUBLE_EQ accepts 4 ULPsobserved as stated
   • ✓ EXPECT_DOUBLE_EQ rejects 5 ULPsobserved as stated
   • ✓ 4 ULPs at 1e9 is 4.8e-7 in absolute termsexpected 4.76837×10⁻⁷ · got 4.768×10⁻⁷ · relative error 7.8×10⁻⁵the tolerance scales with the size of the numbers
   • ✓ EXPECT_DOUBLE_EQ(0, 1e-20) failsobserved as stated
   • ✓ EXPECT_NEAR(0, 1e-20, 1e-12) passesobserved as stated

5. The sanitizer flags, and why each one is there

   Use the same flags when linking. AddressSanitizer typically slows a program by about 2×. Without -fno-sanitize-recover, most UBSan checks print a report and carry on, and the exit code says nothing went wrong.

   -fsanitize=address,undefined      # memory errors; undefined behaviour
   -fno-sanitize-recover=undefined   # stop at the first UB, so tests fail
   -fno-omit-frame-pointer           # readable stack traces

   F4.05✓ 4/4 checked[6][7]
   Symbols and checks

   • ✓ use-after-free: ASan stops the programobserved as stated
   • ✓ signed overflow with -fno-sanitize-recover: stopsobserved as stated
   • ✓ same bug, default recovery: reported, but exit code 0observed as stateda test runner reading the exit code would call this a pass
   • ✓ control: the same program without the bug runs clean (exit code)expected 0 · got 0 · absolute error 0

6. Switching a check off silences real bugs too

   The documented false positive (the wiki knows of at least one kind) comes from mixing code built with and without AddressSanitizer. Even then, prefer building everything with it; failing that, confine the option to the affected binary and write down why.

   ASAN_OPTIONS=detect_container_overflow=0 ./container_overflow   # reports nothing, exits 0

   F4.06✓ 2/2 checked[8]
   Symbols and checks

   • ✓ real read past size(): reportedobserved as statedlibc++ std::vector
   • ✓ with detect_container_overflow=0: silent, exit 0observed as statedthe bug is still there; only the report is gone

7. A test must be able to fail

   Watch each new test fail once, against missing or deliberately broken code, before trusting its pass. Take expected values from an independent route, never from the code being tested.

   // copied from the code under test: agrees with it, bug and all
   EXPECT_NEAR(to_radians(30), 30 * pi / kHalfTurnDegrees, 1e-12);
   // from somewhere else: sin 30 degrees is one half
   EXPECT_NEAR(std::sin(to_radians(30)), 0.5, 1e-12);

   F4.07✓ 4/4 checked[1][2]
   Symbols and checks

   • ✓ as written: the test fails, on the limit and on the massobserved as stated
   • ✓ correct code: tests failingexpected 0 · got 0 · absolute error 0
   • ✓ bug planted: independent expected value failsobserved as stated
   • ✓ bug planted: expected value copied from the code still passesobserved as stated

8. When test discovery runs

   POST_BUILD (the default) lists the tests as part of the build, so a test binary that cannot start breaks the build. PRE_TEST lists them when ctest starts, where the binary's run-time dependencies are more likely to be found; a binary that cannot start then stops ctest before any test runs.

   gtest_discover_tests(arm_tests DISCOVERY_MODE PRE_TEST)

   F4.08✓ 3/3 checked[9]
   Symbols and checks

   • ✓ POST_BUILD: a test binary that cannot start breaks the buildobserved as stated
   • ✓ PRE_TEST: the build succeeds (exit code)expected 0 · got 0 · absolute error 0
   • ✓ PRE_TEST: ctest stops at discovery and runs no testsobserved as statedthe healthy test in the same project does not run either

Worked example: a test that can fail, and a sanitizer that is allowed to stop you

Two small experiments from the check project behind this page, built with the same preset layout as the arm repository. Everything in the tables was observed by tools/robotics/verify.py, not written by hand.

1. 01

   The function, and a planted bug

   A one-line conversion, built twice: as written, and with the half-turn constant changed to 360, the kind of slip that compiles cleanly.

   #ifdef MUTANT
   inline constexpr double kHalfTurnDegrees = 360.0;  // the bug
   #else
   inline constexpr double kHalfTurnDegrees = 180.0;
   #endif
   inline double to_radians(double deg) { return deg * std::numbers::pi / kHalfTurnDegrees; }

2. 02

   Two tests that look equally reasonable

   The first takes its expected value from the code's own formula and constant. The second takes it from somewhere else.

   TEST(ToRadians, CopiedFormula) {
     EXPECT_NEAR(to_radians(30.0), 30.0 * std::numbers::pi / kHalfTurnDegrees, 1e-12);
   }
   TEST(ToRadians, IndependentValue) {
     EXPECT_NEAR(std::sin(to_radians(30.0)), 0.5, 1e-12);
   }

3. 03

   Which test notices

   Both pass on correct code, so on correct code you cannot tell them apart. Only the planted bug shows which one is a test.[1]

   Test results, correct code and planted bug

   Test	Correct code	Bug planted
   CopiedFormula	passes	passes
   IndependentValue	passes	fails

4. 04

   What the exit code sees

   Signed integer overflow built two ways, a use-after-free, and a control with no bug. The first row uses -fsanitize=undefined alone; the others add the rest of the flags in F4.05. The exit code is what ctest and CI read.[1][7]

   What happens when the program runs

   Build	Report printed	Program stops	Exit code
   Overflow, UBSan default (recovers)	yes	no	0
   Overflow, -fno-sanitize-recover=undefined	yes	yes	non-zero
   Use-after-free, AddressSanitizer	yes	yes	non-zero
   No bug, same flags	no	no	0

5. 05

   Switching the check off

   A real read past the end of a std::vector (libc++, whose vector is annotated for AddressSanitizer), run with the container check on and off.[1][8]

   ./container_overflow                                          # reported, stops
   ASAN_OPTIONS=detect_container_overflow=0 ./container_overflow  # silent, exit 0

A green test suite means something only if each test has been seen to fail and every sanitizer report has been explained rather than silenced. Both are cheap to do once and expensive to discover later.

Best practice

• Write the test first and watch it fail for the right reason.

  A failure against missing or broken code proves the test can see the thing it claims to check. Then make it pass.[1][2]

• Take expected values from outside the code under test.

  An expected value derived from the implementation agrees with it, bugs included.[1]

• Test error conditions separately.

  LoadingAMissingFileThrows and LoadsThePendulumWithOneDegreeOfFreedom fail for different reasons, and their names say which.[2]

• Keep -fno-sanitize-recover=undefined.

  Without it, undefined behaviour is reported in the log and the test still passes, because the exit code is zero.[1][7]

• Never silence a sanitizer globally.

  If a report really is the documented false positive, scope the suppression to the one binary, with a comment that links the evidence. Anything wider hides real bugs with it.[8]

• Use ASSERT for preconditions, EXPECT for everything else.

  One run then reports every failing check, and a test never carries on with a null pointer.[5]

Questions

What does a test that fails first protect you from, that a test written after the code doesn't?

From a test that cannot fail: one that checks the wrong thing, never runs, or takes its expected value from the code it tests. Written after the code, a test tends to record what the code does rather than what it should do, and it passes on the first run, so you never learn whether it could fail. The check project's copied-formula test is exactly that: green on correct code, green on broken code.[1][2]

Why is switching off a sanitizer check fine as a diagnostic step, but not as the fix?

As a diagnostic, it tells you which check fired, and whether the failure goes away without it. As a fix, it changes nothing about the program: a real bug stays, unreported. The fix is to find the cause, which is either a bug in your code or a documented false positive, such as code built with AddressSanitizer mixed with code built without it. For a real bug, fix the code. For the false positive, build everything with the sanitizer or confine the suppression to the affected binary, and write down why.[1][8]

AddressSanitizer reports a container-overflow inside GoogleTest. Is it my bug?

It can be either. A prebuilt GoogleTest is not instrumented, and mixing instrumented and uninstrumented code that share std::vector is the documented source of false container-overflow reports. The arm repository does exactly this mix: its sanitized test binary links Homebrew's GoogleTest. Test the hypothesis instead of assuming it: build GoogleTest from source with the same sanitizer flags and see whether the report survives.[8]

Two doubles print identically but EXPECT_DOUBLE_EQ fails. Why?

Printing rounds; the comparison does not. They differ by more than 4 ULPs in their last bits. Decide how close they need to be for your purpose, and say so with EXPECT_NEAR.[3][4]

Why are my tests so much slower in the debug preset?

No optimisation, plus AddressSanitizer, which typically doubles run time on its own. That is the price of being told about memory errors; time things in release.[6]

Does ctest run my other tests if one test binary cannot start?

Not with test discovery at ctest time (PRE_TEST): discovery fails, and ctest stops before running anything. With discovery at build time (POST_BUILD), the build fails instead. Either way the failure is loud; it just arrives at a different step.[1][9]

Watch

• C++ Weekly, Episode 84: C++ Sanitizers (opens in a new tab)[12]

  Jason Turner · C++ Weekly (YouTube); announced on isocpp.org, 1 November 2017

  A short tour of the sanitizers in GCC and Clang and what each one catches.

  listed

• Sanitize your C++ code (opens in a new tab)[13]

  Kostya Serebryany · CppCon 2014; posted on isocpp.org

  The sanitizers explained by one of the people who built them: how they work, and how they are used on very large code bases.

  listed

• Back to Basics: Unit Testing in C++ (opens in a new tab)[2]

  Dave Steffen · CppCon 2024; posted on isocpp.org

  The principles behind unit-testing advice, including writing tests first, and where good practices pull against each other.

  listed

Read

• GoogleTest Primer (opens in a new tab)[5]

  Google

  The official introduction; assertions, test suites and fixtures in one sitting.

  read

• GoogleTest Assertions Reference (opens in a new tab)[3]

  Google, GoogleTest (docs/reference/assertions.md)

  Every assertion, with exactly what it compares.

  read

• AddressSanitizer (opens in a new tab)[6]

  Clang documentation, LLVM 18

  What AddressSanitizer detects, how to build with it, and its limits.

  read

• AddressSanitizerContainerOverflow (opens in a new tab)[8]

  google/sanitizers wiki, GitHub

  Container-overflow, and the false positive the wiki documents.

  read

Sources

Numbered in order of first use on this page. read: checked against the claim. listed: exists and covers the topic, wording not checked. derived: worked out here and checked numerically.

1. [1]

   Compiled checks for modules 02 to 04 (tools/robotics/cpp)

   A small CMake project that mirrors the arm repository, built with presets and run by verify.py; every C++, CMake and GoogleTest claim marked checked is observed there.

   derived
2. [2]

   Back to Basics: Unit Testing in C++ (opens in a new tab)

   Dave Steffen · CppCon 2024; posted on isocpp.org

   The principles behind common unit-testing advice, such as writing tests first and testing error conditions separately, and where they conflict.

   listed
3. [3]

   GoogleTest Assertions Reference (opens in a new tab)

   Google, GoogleTest (docs/reference/assertions.md)

   EXPECT_NEAR(val1, val2, abs_error); EXPECT_DOUBLE_EQ (within 4 ULPs); EXPECT_* continues after failure, ASSERT_* aborts the current function. Read from the GoogleTest repository's docs.

   read
4. [4]

   What Every Computer Scientist Should Know About Floating-Point Arithmetic (opens in a new tab)

   David Goldberg · ACM Computing Surveys, March 1991 (reprinted as Appendix D of the Oracle Numerical Computation Guide)

   Rounding error bounds per operation (at most ε relative for a correctly rounded multiply); why exact equality tests on computed doubles fail.

   read
5. [5]

   GoogleTest Primer (opens in a new tab)

   Google

   Assertions

   ASSERT_* failures are fatal and abort the current function; EXPECT_* failures are not, and are usually preferred. Read from the googletest repository (October 2026).

   read
6. [6]

   AddressSanitizer (opens in a new tab)

   Clang documentation, LLVM 18

   Introduction; Usage

   Detects out-of-bounds accesses, use-after-free, use-after-return and use-after-scope, double free; typical slowdown 2×; compile and link with -fsanitize=address, add -fno-omit-frame-pointer for better stack traces. Read from llvm-project at llvmorg-18.1.8.

   read
7. [7]

   UndefinedBehaviorSanitizer (opens in a new tab)

   Clang documentation, LLVM 18

   Usage; Available checks

   -fno-sanitize-recover makes a check print its report and exit instead of continuing; signed-integer-overflow is one of the checks. Read from llvm-project at llvmorg-18.1.8.

   read
8. [8]

   AddressSanitizerContainerOverflow (opens in a new tab)

   google/sanitizers wiki, GitHub

   Container-overflow is an access inside a vector's allocation but past its size; libc++ annotates std::vector; mixing instrumented and uninstrumented code can give false positives, worked around by building everything with ASan or by ASAN_OPTIONS=detect_container_overflow=0. Read from the wiki's git repository.

   read
9. [9]

   GoogleTest (CMake module) (opens in a new tab)

   Kitware, CMake documentation

   gtest_discover_tests: DISCOVERY_MODE (added in 3.18)

   POST_BUILD discovers tests at build time; PRE_TEST delays discovery until just before the tests run, where run-time dependencies are more likely to be found. Read from the documentation source in the CMake repository (master, October 2026).

   read
10. [10]

    IEEE Std 754-2019, IEEE Standard for Floating-Point Arithmetic (opens in a new tab)

    IEEE, 2019 (also ISO/IEC 60559:2020)

    binary64 (double) has a 52-bit fraction field, hence ε = 2⁻⁵².

    listed
11. [11]

    std::numeric_limits<T>::epsilon (opens in a new tab)

    cppreference.com

    Machine epsilon is the gap between 1.0 and the next representable value (DBL_EPSILON for double; 2⁻⁵² checked numerically).

    read
12. [12]

    C++ Weekly, Episode 84: C++ Sanitizers (opens in a new tab)

    Jason Turner · C++ Weekly (YouTube); announced on isocpp.org, 1 November 2017

    A short episode on the runtime sanitizers in GCC and Clang and what each detects.

    listed
13. [13]

    Sanitize your C++ code (opens in a new tab)

    Kostya Serebryany · CppCon 2014; posted on isocpp.org

    The sanitizers explained by one of their authors: AddressSanitizer, ThreadSanitizer, MemorySanitizer and UndefinedBehaviorSanitizer, how they work and how they are deployed.

    listed