How to use this checklist (and why it’s different)

This article follows an explicit logic so you can implement it as a playbook:

1. Personalization: identify your agent’s niche and failure modes.
2. Value prop: define what “better” means (speed, accuracy, safety, cost).
3. Their goal: translate to user and business outcomes.
4. Their value prop: align metrics to what your agent promises (resolution, bookings, shortlist speed, etc.).
5. Case study: see a numbers-and-timeline implementation.
6. Cliffhanger: the one lever most teams miss (calibration + thresholds).
7. CTA: operationalize in Evalvista.

It’s also intentionally not a rehash of generic “ranking vs scoring” discussions. You’ll get a concrete, step-by-step checklist with metric definitions, formulas, and decision rules.

Checklist Part 1: Define the evaluation unit (what exactly are you scoring?)

Before picking LLM evaluation metrics, lock down the unit of evaluation. This prevents teams from mixing apples (single response quality) with oranges (end-to-end workflow success).

1) Pick your unit: turn, step, or outcome

• Turn-level: one model response (e.g., “reply to user”).
• Step-level: one agent action (e.g., tool call, retrieval, classification, routing).
• Outcome-level: full workflow success (e.g., “resolved ticket,” “booked meeting,” “shortlisted candidate”).

2) Write your “definition of done” in one sentence

Examples (choose one that matches your niche):

• Recruiting intake agent: “Produces a same-day shortlist of 5 candidates that meet must-have criteria and includes evidence for each.”
• SaaS trial activation agent: “Guides users to complete the activation event within 24 hours and answers product questions accurately.”
• E-commerce support agent: “Resolves order issues in one conversation while following refund policy and reducing handle time.”
• Agency pipeline agent: “Qualifies inbound leads and books calls with ICP-fit prospects with correct routing.”

3) Identify your top 5 failure modes

Write these as testable statements. Common ones:

• Hallucinates policy details or product capabilities.
• Misses required fields (e.g., budget, timeline, role requirements).
• Calls tools with wrong parameters or in the wrong order.
• Violates compliance/safety constraints (PII, medical/legal advice).
• Over-answers instead of asking clarifying questions.

Checklist Part 2: Choose metrics by category (quality, safety, cost, speed)

Most teams over-index on “accuracy” and ignore the operational metrics that decide whether the agent is viable in production. Use this category checklist to build a balanced scorecard.

Quality metrics (does it solve the user’s problem correctly?)

• Task success rate (TSR): % of items where the agent achieves the defined outcome.
  • Formula: successes / total
  • Best for: outcome-level evaluation (booked calls, resolved tickets, completed activation)
• Instruction adherence: % of responses complying with required format/constraints (JSON schema, tone, policy).
• Groundedness / citation support: % of factual claims supported by retrieved sources (or internal KB IDs).
• Completeness: % of required fields captured (intake forms, lead qualification, incident triage).
• Answer correctness (graded): rubric-based score (e.g., 1–5) or binary label (correct/incorrect) for known-answer tasks.

Safety & compliance metrics (can it be trusted?)

• Policy violation rate: % of interactions with disallowed content or actions (PII leakage, prohibited advice).
• Refusal quality: when refusing, does it provide safe alternatives and remain helpful?
• Data handling compliance: % of tool calls that avoid restricted fields; % of logs properly redacted.
• Jailbreak susceptibility: success rate of adversarial prompts against your guardrails.

Tool-use & workflow metrics (does the agent execute reliably?)

• Tool call success rate: % of tool calls that execute without error.
• Tool call correctness: % of tool calls with correct parameters (IDs, dates, filters, query syntax).
• Recovery rate: when a tool fails, % of cases where the agent retries or falls back appropriately.
• Workflow completion rate: % of runs that reach terminal success state without human intervention.

Efficiency metrics (will this scale economically?)

• Latency (p50/p95): end-to-end response time and step latency.
• Token usage: input/output tokens per run; track by step to find bloated prompts.
• Cost per successful outcome: total cost / # successes (more useful than cost per call).
• Human escalation rate: % of runs requiring agent handoff; include reason codes.

Checklist rule: pick at least one metric from each category. If you can’t, your evaluation will be blind to a failure class.

Checklist Part 3: Decide how each metric is measured (judge, heuristic, or ground truth)

LLM evaluation metrics only become operational when you define the measurement method. Use this decision table to avoid inconsistent scoring.

• Ground truth comparison: best when you have labeled answers (classification, extraction, routing).
  • Metrics: accuracy, precision/recall/F1, exact match, field-level F1.
  • Watch-outs: label drift; ambiguous tasks need rubrics.
• Heuristic checks: best for format, schema, and deterministic constraints.
  • Metrics: JSON validity, required fields present, regex checks, citation count, tool call schema validity.
  • Watch-outs: heuristics can be gamed (citations that don’t support claims).
• LLM-as-judge: best for nuanced quality dimensions (helpfulness, reasoning quality, policy adherence with context).
  • Metrics: rubric scores, pairwise preference, refusal quality.
  • Watch-outs: judge bias, variance, and prompt sensitivity—must calibrate.

Checklist rule: for every judge-based metric, add at least one “anchor” metric that is deterministic (heuristic or ground truth). This reduces the risk of optimizing for judge quirks.

Checklist Part 4: Build a metric spec sheet (so results are repeatable)

Teams get stuck because “accuracy” means five different things across squads. Create a one-page spec for each metric.

1. Name: e.g., “Groundedness rate.”
2. Definition: what counts as success/failure.
3. Unit: turn/step/outcome.
4. Measurement method: ground truth / heuristic / judge.
5. Scoring: binary, 1–5 rubric, or continuous.
6. Aggregation: mean, median, pass@k, weighted score.
7. Threshold: ship gate (e.g., groundedness ≥ 0.92).
8. Slices: where to segment results (language, channel, customer tier, topic, tool availability).
9. Owner: who updates rubric, labels, and thresholds.

Concrete framework: use a 3-layer scorecard:

• Gating metrics: must-pass safety/compliance and schema validity.
• Core quality metrics: task success + correctness + groundedness.
• Business/ops metrics: latency, cost per success, escalation rate.

Checklist Part 5: Map metrics to your niche goal (8 templates)

Below are practical metric bundles aligned to common operator goals. Pick the closest template, then customize thresholds and slices.

1) Marketing agencies: TikTok ecom meetings playbook

• Outcome: booked call with qualified brand
• Core metrics: lead qualification completeness, ICP-fit precision, meeting booked rate
• Safety: brand-safe language rate
• Ops: speed-to-first-response, handoff rate to human setter

2) SaaS: activation + trial-to-paid automation

• Outcome: activation event completed + trial conversion lift
• Core metrics: answer correctness (product), next-best-action accuracy, task success rate
• Workflow: tool call correctness (CRM/product analytics), recovery rate
• Business: cost per activated user, conversion rate delta by cohort

3) E-commerce: UGC + cart recovery

• Outcome: recovered carts / UGC produced
• Core metrics: policy adherence (discount rules), personalization relevance score
• Safety: claims compliance (no false health claims)
• Business: revenue per conversation, unsubscribe/complaint rate

4) Agencies: pipeline fill and booked calls

• Outcome: booked calls with ICP
• Core metrics: routing accuracy, objection handling score (rubric), qualification completeness
• Ops: latency p95, follow-up persistence (touches before drop)

5) Recruiting: intake + scoring + same-day shortlist

• Outcome: shortlist delivered within SLA
• Core metrics: must-have criteria recall, evidence groundedness, ranking quality (NDCG@k)
• Safety: bias/fairness checks (disparate impact flags), PII handling compliance
• Ops: time-to-shortlist, escalation rate to recruiter

6) Professional services: DSO/admin reduction via automation

• Outcome: fewer touches per invoice / faster collections
• Core metrics: extraction accuracy (invoice fields), email correctness, policy adherence
• Business: touches per account, DSO delta, cost per resolved account

7) Real estate/local services: speed-to-lead routing

• Outcome: lead contacted and scheduled
• Core metrics: contact data capture rate, routing accuracy, scheduling success
• Ops: time-to-first-contact, after-hours coverage success

8) Creators/education: nurture → webinar → close

• Outcome: webinar attendance and offer conversion
• Core metrics: message relevance, objection handling, factual correctness
• Safety: claims compliance, spam policy adherence
• Business: show-up rate, conversion rate, refund/chargeback rate

Case study: implementing an LLM metrics checklist in 21 days (with numbers)

Scenario: A mid-market SaaS company shipped a trial assistant that answered product questions and guided setup. Users complained about confident wrong answers and slow responses. The team needed a repeatable evaluation system to improve quality without blowing up costs.

Baseline (Day 0)

• Dataset: 220 real trial conversations sampled from the last 30 days
• Primary outcome: activation event completed within 24 hours
• Baseline metrics:
  • Task success rate (activation within 24h): 32%
  • Answer correctness (rubric + spot ground truth): 3.1/5
  • Groundedness rate (claims supported by docs): 71%
  • Latency p95: 9.4s
  • Cost per successful activation: $4.80
  • Escalation rate to human support: 18%

Week 1 (Days 1–7): metric spec + gating

• Created spec sheets for: groundedness, correctness, tool call correctness, latency, cost per success.
• Added gating checks: JSON tool-call schema validity and a policy rule: “No feature claims without citation.”
• Built slices: new vs returning trials, top 10 question topics, and “docs coverage” buckets.

Week 2 (Days 8–14): fix the biggest metric drivers

• Improved retrieval prompts and enforced citation requirement for factual answers.
• Added a “clarify-first” rule when confidence is low (measured by judge rubric).
• Reduced prompt bloat by splitting system instructions into step-specific prompts.

Week 3 (Days 15–21): regression gates + thresholding

• Set ship thresholds: groundedness ≥ 0.88, correctness ≥ 3.8/5, latency p95 ≤ 6.5s.
• Added a “stop-ship” rule: any policy violation rate above 0.5%.
• Ran weekly eval on the same 220-conversation set plus 60 new holdout items.

Results (Day 21)

• Task success rate (activation within 24h): 32% → 46%
• Answer correctness: 3.1/5 → 4.0/5
• Groundedness rate: 71% → 90%
• Latency p95: 9.4s → 6.1s
• Cost per successful activation: $4.80 → $3.10
• Escalation rate: 18% → 11%

What made it work: they didn’t chase a single “overall score.” They used gating metrics to prevent unsafe/invalid behavior, core quality metrics to improve usefulness, and business metrics to ensure the system scaled.

The cliffhanger most teams miss: calibration, thresholds, and “metric gaming”

Once you publish metrics, the system will optimize for them—sometimes in undesirable ways. Prevent this with three safeguards:

• Calibration sets: keep a small, stable set (e.g., 50–100 items) that never changes. Use it to detect judge drift and prompt regressions.
• Multi-metric gates: don’t let one metric dominate. Example: require both groundedness and correctness to pass; citations alone are not enough.
• Adversarial slices: create “hard mode” slices (ambiguous queries, missing context, tool downtime) and track them separately.

If your team can’t explain why a metric moved, you don’t have a metric—you have a number.

FAQ: LLM evaluation metrics

What are the most important LLM evaluation metrics to start with?

Start with a balanced set: task success rate (outcome), groundedness or citation support (truthfulness), policy violation rate (safety), latency p95 (speed), and cost per successful outcome (economics).

Should we use LLM-as-judge or human evaluation?

Use both when possible: LLM-as-judge for scale and iteration speed, plus a small human-labeled calibration set to validate the judge and prevent drift. Add deterministic checks for schema and constraints.

How do we evaluate agents that use tools (RAG, APIs, workflows)?

Measure tool call success rate, tool call correctness (parameters), recovery rate after failures, and workflow completion rate. Pair these with groundedness and task success so you don’t “optimize the API calls” while harming user outcomes.

How do we connect LLM metrics to business impact?

Define a primary outcome (e.g., activation, booking, resolution). Track cost per successful outcome and segment by cohort. Then correlate quality metrics (correctness/groundedness) with outcome lift to find which improvements matter.

How big should an evaluation set be?

For early-stage iteration, 100–300 representative items is often enough to detect meaningful changes. Maintain a smaller fixed calibration set (50–100) and a rotating set for coverage of new behaviors.

CTA: Turn this checklist into a repeatable evaluation system

If you want these LLM evaluation metrics to run consistently—across prompt changes, model upgrades, and toolchain updates—put them into a repeatable agent evaluation framework.

Evalvista helps teams build test sets, define metric spec sheets, run automated evaluations (including judge + deterministic checks), slice results, and set ship gates so regressions don’t reach production.

Book a demo to operationalize this checklist for your agent, or explore Evalvista to start benchmarking your current stack.