Shipping AI features that survive production

Almost every AI feature I get called in to rescue has the same shape. Someone built a prompt, it worked beautifully in a demo, it went live, and then real users showed up with input nobody had tested. The model didn't get worse — the inputs did.

The distance between a demo and a production feature isn't model quality. It's everything around the model: what you feed it, what you accept back, and what happens on the bad day.

Write the evaluation before the prompt

The single highest-leverage habit is boring: collect 30–50 real inputs and write down what a correct output looks like for each, before you start tuning prompts. Not a benchmark — a spreadsheet. It takes an afternoon and it changes every decision after it.

Once that exists you can answer the questions that actually block a launch: did that prompt change help or just move the failures around? Is the cheaper model good enough here? Where exactly does this break?

Constrain the output, not just the input

Free-form text is the hardest thing to build on. When a feature has to drive UI or write to a database, ask for structured output and validate it before it touches anything downstream.

const Result = z.object({
  category: z.enum(["billing", "bug", "feature", "other"]),
  urgency: z.number().int().min(1).max(5),
  summary: z.string().max(280),
});

// Parse before you trust. A retry here is far cheaper
// than a malformed row that ships to a customer.
const parsed = Result.safeParse(JSON.parse(raw));
if (!parsed.success) return retryOnce(input);

A schema turns a whole class of vague failures — a stray sentence, a hallucinated field, a number as a word — into one explicit branch you can handle.

Design the failure state on purpose

Every AI feature needs an answer to "what does the user see when this is wrong or slow?" If you don't design it, you ship the default: a spinner that never resolves, or a confident wrong answer with no way out.

• Show the source. If the answer came from the user's own documents, link the passage — it turns a trust problem into a verification step.
• Make it correctable. A one-click edit beats a regeneration nobody asked for.
• Set a timeout and a fallback path, so a slow provider degrades instead of hanging.
• Log the input, the output and the user's reaction. That log is next month's evaluation set.

Put a number on the cost before launch

Token spend is easy to ignore in development and painful to discover in month two. Estimate cost per action, multiply by the volume you actually expect, and decide deliberately: cache aggressively, use a smaller model for the routine path, and reserve the expensive model for the cases that need it.

The short version

Evaluate with real inputs, constrain outputs with a schema, design the failure state, and know the cost. None of it is glamorous, and all of it is the difference between a feature that demos well and one that's still switched on six months later.