How to compare the cost of two model APIs on the same task

Use billed tokens, retries and accepted results to compare two AI models on one real workload.

A team choosing a model for support replies can learn more from ten resolved tickets than from a price list alone. Remove identifying details, keep the original questions and write down the facts a correct reply must include.

Compare the whole ticket

Run both candidates on the same questions with the same instructions. Have a reviewer mark each answer ready to send or in need of edits. Include the cost of every attempt, even when an answer is discarded.

For example, if one model costs $0.12 across ten tickets and produces eight acceptable replies, it costs $0.015 per accepted reply. If a second model costs $0.10 and produces five acceptable replies, the same measure is $0.02. These figures illustrate the calculation. Use the actual amounts from your own run when choosing a model.

Keep the result reproducible

Record the model ID, provider, date and prompt version alongside the score. A change in output length, a longer conversation or a different provider can move the result. Measure review time separately so a saving on tokens is visible alongside any extra editing.

You can find candidate pairs in TrustedRouter’s model comparison catalog and check how API billing works.