Shipping AI features safely: evals, guardrails and the boring parts

An AI demo takes an afternoon. A large language model, a clever prompt and a few hand-picked examples can impress any room. Then it meets real users, real data and edge cases nobody imagined, and the gap between demo and product becomes very clear.

The work that closes that gap is not glamorous. It is mostly testing, plumbing and careful limits. It is also what separates AI features people trust from the ones that are quietly switched off a month after launch.

Demos are easy, production is not

In a demo you choose the inputs. In production your users do, and they will paste in things you never expected: half-finished sentences, other languages, screenshots, hostile instructions, questions your data cannot answer.

A production AI feature has to answer three questions a demo never faces. How do we know it is good? What happens when it is wrong? And what does it cost when ten thousand people use it every day?

Define good before you write a prompt

Start with an evaluation set: a collection of realistic inputs paired with what a good response looks like. For a support assistant, that might be a few hundred real, anonymised tickets with the correct answers and the policies that apply.

Then decide how to score responses:

• Exact checks for things with a right answer, such as an order status or a refund amount.
• Rubric scoring for tone, completeness and policy compliance, often with a second model doing the grading and people spot-checking it.
• Must-never rules, such as never promising a refund above the limit, never revealing another customer’s data and never inventing a policy.

Run the full set on every change to prompts, models or retrieval. Without this, every “improvement” is a guess, and regressions reach users before you notice them.

Ground every answer

Most AI failures in business products are confident answers that are not true. The remedy is grounding: retrieve the relevant documents or records first, give them to the model, and require it to answer from them and cite them.

Good grounding also means teaching the system to say “I don’t know”. An assistant that admits a gap and hands over to a person is far more valuable than one that fills every gap with something plausible.

Put permissions around actions

Answering questions is one thing; taking actions is another. When an AI agent can update records, send emails or issue refunds, treat it like a new employee on day one:

1. Narrow tools. Give it specific, limited operations (“look up order”, “start return”) rather than broad access to an API or database.
2. Server-side limits. Enforce amounts, scopes and rate limits in code the model cannot talk its way around.
3. Approval steps. Route high-impact or unusual actions to a person, with the agent’s reasoning and the data it used attached.
4. Full audit logs. Record what the agent saw, decided and did, so every outcome can be explained later.

Watch cost and latency like any other budget

AI features have a running cost per use, and it can grow quickly. Set budgets early and design to them:

• Use the smallest model that passes your evaluation set, and reserve larger models for the hard cases.
• Cache responses and retrieved context where answers do not change often.
• Stream responses so users see progress, and set timeouts with sensible fallbacks.
• Track cost and latency per feature on the same dashboards as the rest of your product.

Keep a person where it counts

The goal is rarely full automation on day one. It is moving routine work to the AI and making people faster at the rest. Start with the AI drafting and a person approving. As evaluation results and real-world data build confidence, widen what the AI handles on its own, one category at a time.

A pre-launch checklist

Before an AI feature reaches customers, we make sure that:

• An evaluation set exists, runs on every change and passes agreed thresholds.
• Answers are grounded in retrieved sources, with citations where users can see them.
• The system says when it does not know, and hands over cleanly.
• Every action has narrow permissions, server-side limits and an audit trail.
• High-impact actions require human approval.
• Cost and latency budgets are set, monitored and alerted on.
• Users’ data is excluded from model training, and data stays where it must.

None of this makes for a flashy demo. All of it makes the difference between an AI feature that impresses once and one that keeps earning its place in the product.