Testo

English

Русский

English

Русский

Appearance

...
Blog
Who Mutates the Mutator
2026-07-01Aleksei Gagarin

Who Mutates the Mutator? ​

Mutation testing works on a simple idea: the tool makes a small change to your code — a mutant — and checks whether your tests catch it. If a test fails, the mutant is "killed" and your checks are doing their job. If nothing fails, the mutant "survives" — and that means there's a hole somewhere.

All of this rests on one silent assumption: the observer never changes — only the object does. The framework stays the same; only the code beneath it breaks — and the difference is plain to see. But what if the framework tests itself? Then the observer becomes part of the observed system, and the assumption falls apart.

How it breaks in Testo ​

Run Infection over the Filter plugin — the one that selects tests by --path, --group, --filter, and so on — and at least one mutant will stubbornly "survive", even though it genuinely breaks the code.

The explanation turned out to be an elegant one. To check a mutant, Infection runs the tests that cover it — and it picks which ones through --filter, that is, through the very code that's mutated right now. From there the chain writes itself:

Mutate the filter → the filter breaks → Testo can't tell which test to run → no test runs at all → nothing to fail → Infection sees "success" → the mutant is marked as survived.

Note: this is not an equivalent mutant. An equivalent mutant doesn't change behavior for any input.

In a situation like this, you can't even be sure the killed mutants are being killed for the right reason.

As a fix, I could exclude Filter from mutation testing — and then every other plugin too, since they all share the problem. No, that's not a fix.

What if I ran the tests on a different framework? I'd still want to keep writing them in Testo, though.

Writing a shim to run Testo tests on PHPUnit is a dead end: PHPUnit isn't up to it. But what if I rewrote the tests from Testo to PHPUnit where possible, and ran Infection + PHPUnit on those?

That leaves "just" one thing: learning to rewrite tests from one framework to another without losing their meaning. For that there's Rector, which walks the AST and rewrites code according to rules.

Actually, the whole idea came from @samdark: I showed him the PEST-to-PHPUnit rule set, he suggested doing the same for Testo → PHPUnit, and I loved it.

You might think I'm playing for the wrong team, but let me remind you: Testo is on the developer's side. I often see people regret picking PEST and wishing they could go back to PHPUnit. The same could happen with Testo — so why not help them do exactly that?

That's how the testo/bridge-rector package was born: a set of conversion rules plus the harness to test them.

The Rector harness ​

Rector transforms code through rules. But every rule needs to be tested somehow. Rector has its own fixture format for this: *.php.inc files holding the "input" and the "expected output", plus RectorRunner\Testo\Bridge\Rector\Testing\Internal\RectorRunner to run them. Handy: one fixture — one self-contained "before → after" case.

Rector's native harness runs on PHPUnit through some scaffolding: for each rule you create a separate test class extending AbstractRectorTestCase\Rector\Testing\PHPUnit\AbstractRectorTestCase, and inside it always the same thing — a data provider of fixture paths and a cookie-cutter test method.

php
final class MarkTestIncompleteRectorTest extends AbstractRectorTestCase
{
    #[DataProvider('provideData')]
    public function test(string $filePath): void
    {
        $this->doTestFile($filePath);
    }

    public static function provideData(): Iterator
    {
        return self::yieldFilesFromDirectory(__DIR__ . '/Fixture');
    }
}

That's a lot of boilerplate, isn't it? Testo isn't that clunky, so I built myself a nicer harness: a single attribute on the rule itself, pointing at the fixtures folder.

php
#[TestRectorFixtures('MarkTestIncompleteRector')]
final class MarkTestIncompleteRector extends AbstractRector { /* … */ }

It works on the same principle as the Inline plugin: each #[TestRectorFixtures]#[TestRectorFixtures(string ...$paths)]Declares the fixtures that exercise a Rector rule. turns into a Data Provider, and each *.php.inc into a Data Set. The input and the expected output are piped into channels. No boilerplate classes, and all the detail right there in front of you.

Fixture in channels

If you write Rector rules and use Testo, this harness is available to you too — just add RectorTestingPlugin\Testo\Bridge\Rector\Testing\RectorTestingPlugin to testo.php.

The Rector rules ​

As long as conversion was "assert here, assert there", everything looked smooth. The real work started where the two frameworks diverge in semantics — and they diverge more often than you'd think.

For example:

  • $this in PHPUnit needs a class context. Static test methods have to be turned into regular ones.
  • You can't use a constructor or destructor in PHPUnit. Fine, they get converted into a #[Before]/#[After] hook. Not the same thing, but better than nothing.
  • Unfold the chain without shooting yourself in the foot. Assertions like Assert::array($log->all())->isList() expand into several lines in PHPUnit: assertIsArray($x) + assertIsList($x). But $log->all() mustn't be evaluated twice (what if there's a side effect?), so the subject is hoisted into a local variable. And its name has to be chosen so it doesn't clobber one already in the method — hence the $value, $value2, $value3… generator.
  • In Pest, a test is a call that takes a closure, not an explicit declaration — and full of magic and $this on top of that. You either turn them into functions or write yet another plugin; for now I've settled on functions.

Pest? Well, I just figured… since I was writing Testo → PHPUnit conversion rules anyway, why not make them bidirectional and toss Pest into the mix too? That's how three rule sets were born:

  • testo-to-phpunit
  • phpunit-to-testo
  • pest-to-testo

Feature parity ​

Not every feature is translatable — and that's fine:

Such cases aren't dropped silently — they get documented stub rules and an entry in TODO.md. In user tests, whatever can't be converted is marked skipped with a reason.

The mirror ​

Let's come back to the original problem — mutation testing by an outside observer.

Now a single command copies every test (except Self/Inline/Bench) into a temp folder and converts it to PHPUnit. The result is a set of mirror tests that then run against the original code — but this time under Infection + PHPUnit.

Like nesting dolls: Testo tests the Rector rules that rewrite Testo tests into PHPUnit, so that PHPUnit can test — and Infection can mutate — Testo's own code.

What came of it ​

The observer is finally on the outside.

  • The mirror builds and stays green: 864 tests, 0 errors or failures, ~37 skips (couldn't be converted), a single benign-risky.
  • The agent skill for PHPUnit → Testo conversion was extended with the Rector scripts.
  • Infection now has two fronts: Testo and PHPUnit.
  • Mutants die reliably.

It's too early to celebrate, though. The Self tests don't make it into the mirror, and they cover an order of magnitude more code than ordinary unit tests — so Infection still sees only a small fraction of the mutants. But at least it's something to work with now.