Skip to content
Prathamesh Kasar
Work

Production case study

Tests that pass for the wrong reason

Growing a module's suite from 1,395 to 1,652 tests, and using mutation testing to find the tests that passed without proving anything.

Role
Backend owner
When
2025–2026
Where
bbSlate In-Person module, preparing for production release
Stack
  • xUnit
  • Mutation testing
  • ASP.NET Core 8

32 / 32API regression cases passed, no backend defects

Context

The module already had a large suite: 1,395 automated tests. A big number feels safe. The question I cared about was whether the tests would fail if the code were wrong.

Problem

A test can pass for the wrong reason. It may assert on something that is always true, set up data that skips the branch it claims to cover, or check a value the code never changes. Those tests count towards the total and protect nothing.

Investigation

Mutation testing asks the question directly. It makes small deliberate changes to the code, such as flipping a condition or changing a boundary, and runs the tests. If the tests still pass, nothing was really checking that line.

Root cause

The surviving mutants pointed at tests that ran the code without pinning down its behaviour. Coverage reports had counted those lines as covered.

Fix

I fixed the tests that still passed when the logic was wrong, and added new ones where behaviour had no real check, growing the suite from 1,395 to 1,652 tests.

Result

Full module regression passed 32 of 32 API test cases with no backend defects.

What I learned

Test count isn't test quality. Coverage tells you what ran; mutation testing tells you what was checked. I'd rather have fewer tests that fail for the right reason than more tests that only pass.