Skip to main content
AI CANADA PULSE
ReadDated · source-linked
The story archive

Research

How to Test AI Agents When No Benchmark Exists — Nathanael Weill, Air Canada|AI Engineer

The report in brief

Air Canada AI engineer Nathanael Weill says teams can evaluate AI agents without a public benchmark by creating test questions from known answers and checking results with code. For the airline’s supplier-contract agent, his team built 275 rule-based tests and used generated contract examples and checks that changes such as a new supplier name do not alter the flags. Evaluation grew from one test a week to dozens a day in an earlier project, but Weill says generated tests support development rather than decide whether a system is ready to launch. He warns that AI judges can be too unreliable to grade agents.

Read the original reporting