Sonnet 5 AI benchmark reveals mixed results for GTM use cases

The Gist

• Sonnet 5 costs $2/$10 per million input/output tokens through summer 2026
• In benchmarks, Sonnet 5 ranked near bottom for personal preference despite competitive pricing
• Claude Code enables repeatable benchmarks using frozen inputs and fixed rubrics

• Sonnet 5 is priced closer to previous Sonnet models than to Opus, but it doesn't automatically replace either one, requiring a quality argument for specific use cases.
• One-off vibe checks are not repeatable; a benchmark needs frozen inputs, a fixed rubric, and the same tasks for meaningful comparisons.
• Claude Code can generate benchmark ideas tailored to a person's actual work by reading old session history, a largely untapped resource.
• LLM-as-judge evals are too generous and miss visual flaws that humans catch immediately, like broken prototypes.
• The shift from 'agent prompter' to 'agent manager' is crucial for effective autonomous agent workflows, requiring better tooling and async management.
• AI's biggest unlocked opportunity is businesses built on heterogeneous data, like trading cards or vintage clothing, where LLMs handle inconsistency without extensive preprocessing.

• Build a custom benchmark with frozen inputs, a fixed rubric, and human scoring to evaluate AI models for your specific use cases.
• Use Claude Code or similar tools to generate tailored benchmark tasks from your historical work context.
• Track token costs per task to create feedback loops for improving agent specs and tooling.
• Purge skills files regularly to avoid contradictions and keep agent instructions concise.

• $2 per million input tokens, $10 per million output tokens (Sonnet 5 pricing through the end of summer.)
• 64 generations (Number of outputs Claire scored by hand across five models.)
• 15 million to 221 million tokens (Range of token costs per task in Alessio's agent setup.)
• 70/30 Claire-to-LLM weighted index (Weighting that reversed the LLM judges' rankings, favoring Sonnet 4.6.)