A team choosing a model for support replies can learn more from ten resolved tickets than from a price list alone. Remove identifying details, keep the original questions and write down the facts a correct reply must include.
Compare the whole ticket
Run both candidates on the same questions with the same instructions. Have a reviewer mark each answer ready to send or in need of edits. Include the cost of every attempt, even when an answer is discarded.
For example, if one model costs $0.12 across ten tickets and produces eight acceptable replies, it costs $0.015 per accepted reply. If a second model costs $0.10 and produces five acceptable replies, the same measure is $0.02. These figures illustrate the calculation. Use the actual amounts from your own run when choosing a model.
Keep the result reproducible
Record the model ID, provider, date and prompt version alongside the score. A change in output length, a longer conversation or a different provider can move the result. Measure review time separately so a saving on tokens is visible alongside any extra editing.
You can find candidate pairs in TrustedRouter’s model comparison catalog and check how API billing works.