killcheck

A Python mutation-testing harness that checks whether the tests an AI agent writes actually catch bugs, built for the micro1 Frontier Engineering Challenge 2026.

Category

Open Source

Industry

AI Evaluation

Role

Sole Developer

Stack

Python, pytest, mutation testing

Abstract grid of test cells with a few flagged survivors and a scan line
What I built

killcheck plants one mutant at each mutable site in a library. Mutants the existing tests miss become the agent's work queue, and a new test is kept only if it passes on the clean code and fails on its mutant. Exit codes are the ground truth, not the agent's own report.

Challenge and solution

An evaluation is only worth something if its numbers can't drift. I froze the measurement core, committed to the metric before running it, and logged every retraction. Across 455 mutants in 12 libraries, only 53 turned out to be reachable but unasserted.

Results

Public on GitHub with a full write-up on dev.to, including the corrections readers found after publication.