Evals Are What Let Me Flip the Flag

There’s a specific kind of dread that shows up right before you change one line of config in production. In my case the line was a feature flag that decides which provider actually runs our LLM calls - the ones that read real customer documents and make decisions about them. Migrating to the new provider was the easy part. Weeks of code, sure, but code you can review. The scary part was the moment I’d set that flag to true and real traffic would start flowing through a model I hadn’t watched behave on real inputs yet. Read more