How to evaluate a business AI agent before launch

Start with the user’s job

Choose tasks that reflect the intended workflow. A knowledge assistant should answer from the permitted sources. An assistant that updates records must also use the right identity and operation. A system that routes work should send the task to the appropriate destination with enough context.

Write acceptance criteria for each task before inspecting the output. They should describe a useful result, the facts that must remain correct and the conditions that require clarification or escalation. This helps reviewers judge behavior consistently without relying on whether a response sounds polished.

Include multi-turn changes

Real users refine their requests. They change a date, correct a name, reject an option or refer back to an earlier part of the conversation. Test whether the assistant uses the latest instruction and retains the relevant context without carrying forward discarded assumptions.

An illustrative CRM task might begin with one contact, then switch to another before approval. The final operation must use the corrected identity. A test that checks only the first answer would miss the more consequential failure later in the conversation.

Separate answer quality from action quality

Microsoft’s evaluation guidance includes functional behavior and human review against acceptance criteria. Apply that distinction by checking both the response and the underlying operation. A valid API call can still target the wrong record; a correct explanation can accompany a failed write.

Capture the evidence needed to investigate the task: relevant tool parameters, returned status and the final state where it can be verified. Keep sensitive information out of broad reports and retain only what the evaluation needs.

Reference: Microsoft: Agent evaluation checklist

Test missing evidence and unavailable tools

Ask a question the knowledge source cannot answer. Remove a required field from a test record. Make an integration temporarily unavailable in a controlled environment. The assistant should respond according to a defined failure policy rather than invent information or claim that the task succeeded.

For RAG systems, Microsoft’s design guidance treats evaluation as part of the solution lifecycle. Check retrieval separately from answer generation so that a missed source can be distinguished from a response that misuses a correctly retrieved passage.

• Unknown answer: acknowledge the gap and offer a useful next step.
• Ambiguous identity: clarify before acting.
• Failed operation: preserve uncertainty and recovery context.
• Changed instruction: use the latest authorized intent.

Reference: Microsoft: Designing and evaluating a RAG solution

Keep the evaluation reproducible

Record the application version, model configuration, prompts, tools and source collection used for the run. Save the actual outputs and reviewer decisions. A later improvement should be checked against the same conditions before concluding that quality changed.

Deterministic software checks and real-model evaluations answer different questions. Use ordinary tests for routing, validation and permissions, then run representative conversations through the actual configured system. Passing one category should not be presented as proof of the other.

Make release readiness a documented judgment

Summarize successful tasks, known failures and the controls that limit exposure. Identify who owns fixes and how the team can disable a problematic operation. After launch, review new failure patterns and add them to the evaluation set instead of treating the initial pass as permanent certification.

XT SCALE’s AI engineering work connects application behavior with business acceptance criteria. A launch review should leave the team with a clear account of what was tested, what worked and what still needs attention as usage expands.

Common questions

Can exact-match tests judge every AI answer?

They are useful for deterministic fields and rules, but open-ended responses also need task-specific review. Evaluate whether the answer meets the requirement, not whether it copies one preferred sentence.

Does a passing evaluation guarantee production performance?

No. It establishes evidence under the tested conditions. New requests, data changes and integration failures require continued monitoring and regression evaluation.