AI product teams are skipping critical eval steps, Shopify saw 68% cost savings

The Gist

• Shopify used evals to build an AI workflow 2.2x faster and 68% cheaper than frontier models
• Most teams skip critical eval steps and automate the wrong metrics
• Hamel and Shreya offer a free plugin for automated eval heavy lifting

• Most AI product teams skip the critical error discovery phase in evaluations, leading to measuring the wrong metrics.
• Error discovery is the eval equivalent of product discovery, revealing which AI failures are worth measuring.
• Shopify achieved 68% cost savings and 2.2x faster performance by using evals to guide AI workflow development.
• Automated agents are useful for finding obvious errors but less reliable for failures requiring product judgment.
• Evals help teams ship AI products faster by automatically checking if the product still works as intended.
• Investing in evals can compound advantages over time by turning production errors into additional test cases.

• Prioritize error discovery in AI product evaluations to identify critical failures worth measuring.
• Use a hybrid approach combining human judgment and automated agents for effective error discovery.
• Invest in evals to systematically improve AI products and compound advantages over time.
• Generate synthetic data for initial testing if real user traces are unavailable, but prioritize real user data.

• 68% (Cost savings achieved by Shopify using evals to guide AI workflow development.)
• 2.2x (Performance improvement in Shopify’s AI workflow builder compared to the frontier-model system it replaced.)
• 41% (Cost reduction achieved by Cursor through Auto Balance routing performance guided by evals.)
• 83% (Precision increase in Ramp’s on-device models for matching transactions in receipt photos after investing in evals.)