All notes
Write your evals before your prompts
Why an evaluation set is the most valuable artefact in any LLM project, and how to start one in an afternoon.
The fastest way to improve an LLM feature is not a better prompt. It is a way to know whether the prompt you just wrote is better.
Start small
Fifty real examples with a clear definition of a good answer will tell you more than a week of vibes-based testing. Grow the set every time a user reports a bad answer.
Make it run on every change
An eval that runs only when someone remembers is an eval that doesn’t run. Put it in CI next to your unit tests.
Placeholder article. Replace with your own writing.