Token Price Isn't Everything: Pick AI Models Wisely

Organizations typically compare AI models by cost per token, but production workloads care about outcomes like resolved tickets or accurate summaries. A new open-source benchmarking harness tests OpenAI models on Amazon Bedrock against cost-efficient OpenAI API baselines, measuring cost per correct answer, agent turn efficiency, and professional deliverable quality. Results show capability tiers matter significantly: Sol solved 75% of AIME math problems versus Mini's 37%, while Luna delivered the lowest cost per correct answer even against cheaper-priced Nano. Following a July 30 price cut slashing Luna costs by 80%, Luna's recorded cost per correct AIME answer dropped to $0.0021 versus Mini's $0.0139, demonstrating that token efficiency and accuracy together determine real-world AI spending.