How to compare coding models with a bug you already fixed

Turn one resolved issue into a repeatable coding-agent trial using a failing test, a reviewable patch and the total cost of the session.

A bug your team fixed last week makes a good first model comparison. You already know how to reproduce it, what a successful fix looks like and which nearby behavior must keep working.

Consider a CSV importer that drops the last record when the file ends without a newline. Give each agent the same small repository and issue description. Keep the known fix out of its working copy. Before starting, confirm that your test fails on the original commit and passes on the corrected version.

Give each candidate the same starting point

Use separate clean checkouts of the original commit. Keep the client version, available tools, task instructions and time budget the same. Record the exact model and provider. Start a fresh session for each candidate so neither sees the other’s investigation.

Ask each agent to reproduce the defect, make a fix and run the relevant tests. When the session ends, run your own checks against the submitted patch. For the importer, check both forms of the file ending and a file with just one record. Then inspect whether the patch changed how blank rows or quoted fields are handled.

Decide what earns another trial

Measure What to record
Correctness Did the original failure disappear, and did the other cases still pass?
Review effort How long did a teammate need to understand and approve the patch?
Cost What did the entire session cost, including failed attempts?

If both patches pass, review time and cost can break the tie for this class of bug. Try a second task that exercises a different skill before choosing a default for the whole team. The model comparison catalog can help you pick candidates, and the API cost note shows how to account for retries.