Research
How to Test AI Agents When No Benchmark Exists — Nathanael Weill, Air Canada|AI Engineer
The report in brief
Air Canada AI engineer Nathanael Weill says teams can evaluate AI agents without a public benchmark by creating test questions from known answers and checking results with code. For the airline’s supplier-contract agent, his team built 275 rule-based tests and used generated contract examples and checks that changes such as a new supplier name do not alter the flags. Evaluation grew from one test a week to dozens a day in an earlier project, but Weill says generated tests support development rather than decide whether a system is ready to launch. He warns that AI judges can be too unreliable to grade agents.
Read the original reporting