Four dimensions of evaluation
End-to-end vs component-based
End-to-end evals measure the final outcome: did the customer's issue get resolved? Component-based evals measure each step: did the extractor get the right order ID? Did the tool update the database? Did the email draft match the policy? Components tell you where to fix.
Objective vs subjective
Objective evals can be checked automatically: did the system extract the correct order number? (Compare input to extraction — a Python script can do this.) Subjective evals require judgement: was the email polite and helpful? (LLM judges with rubrics handle this.)
Quantitative vs qualitative
Quantitative: percentage of successful address updates, average response latency per component. Qualitative: error analysis — reading through interactions to find patterns like hallucinations, confused users, or tool failures. Start qualitative, then build quantitative metrics.
LLM traces are essential
An LLM trace records every step in the chain: what prompt was sent, what was returned, what tool was called, what the result was. Without traces, debugging an agentic system is nearly impossible. If you are evaluating an AI startup, ask them if they have traces.
If a customer support agent sends polite but factually wrong emails, which kind of evaluation would catch that?