Production AI Architecture: What Changes Between Demo and Deployment

A demo needs a model call and a UI. A production system needs credential isolation, retrieval infrastructure, evaluation, observability, rate limiting, and a plan for what happens when the model provider has an outage — none of which show up in a five-minute walkthrough.

Credential isolation comes first architecturally: the browser talks to your own backend, and only your backend holds the API key and system prompt. This is a security requirement, not a nice-to-have — a client-side integration exposes both to anyone with dev tools open.

Observability is the part most often skipped under deadline pressure, and the one that matters most after launch. Logging what was retrieved, what was generated, and how users reacted is what lets a team actually improve accuracy over time instead of guessing at what's going wrong.

Rate limiting and cost controls matter for a different reason: LLM calls cost real money per request, and a single misbehaving client or bot can turn a reasonable budget into an unreasonable one within hours. This needs to be designed in, not added after the first surprising bill.

None of this is exotic engineering — it's the same discipline any production backend needs. The mistake is treating an AI feature as exempt from it because the interesting part is the model.