Mutation Testing at a Glance

At a Glance

One interesting thing I’ve worked on this month was conducting mutation testing and applying it to a real-world scenario. We had a situation where we’d just migrated our company services, rewriting them in Go for critical, write-intensive services (such as impression, tracking, and company search). We conducted mutation testing across 3 iterations, starting in the first week of July and wrapping up by mid-July. So there were a few valuable insights I got from this, hopefully you’ll find the same

1. What mutation testing actually does: it’s not real-time integration testing across services, but rather leans toward being a schema contract test. It takes a real request payload (sourced from production data), mutates it (using definitions from mutations.py), routes the mutation’s label through a particular assertion, and dispatches it to the matching function.

2. The particular assertion we talked about here consists of these utilities and functions:

• Assertion passes → mutation is accepted and complies with the API/schema contract. No violations
• Assertion is skipped → the function raises a KeyError, ValueError, or TypeError, meaning the parser rejects it
• Assertion raises and fails → the contract is violated, the test definitely fails

3. During v2 (as we called it in this cycle), we ran into several critical structural bugs while running the tests. We initially thought these were real bugs we’d caught during testing. So all of these turned out to be false positives

4. At a glance, some of the structural issues we had: an expired typo that masked 14 skipped tests, and an inverted company-published-date field

5. Once we addressed those structural bugs in v3, the test results went from 36 failed test cases to 49 failed test cases → indicating 13 real contract violations once we’d fixed the bugs in our mutation testing code.

6. One of the highest-impact contract violations was related to the salary range and published date fields. When we tried integrating this into an isolated dev environment and wiring it up with the mobile app build at the time, a clear issue resurfaced: on the mobile app, users visiting a company’s profile page would see nothing.

7. Some of the quality verdicts based on our mutation testing categories:

• ID consistency: 1 out of 3, a drift issue was detected on the company_id
• Mixed data types: 3 out of 5, boolean type passes as a subtype of int
• Expiry date logic: 1 out of 1, some records silently had an expiry date later than the created date

8. For us, it saves a lot of time and effort compared to having to perform full regression testing, even though we already have existing automation tests (either at the API level or unit tests). The reason is much simpler: we didn’t need to go back and forth investigating issues that might not have been caught or resurfaced using that method.

9. We need to be very strict and rigid when building the test assertions. For example: small subsets of the previous body, from before we migrated to the newest services, seemed to have drifted and changed, whether intentionally or not

Appendix

Here’s our mutation testing engine workflow (pretty straightforward):