This is especially important because the main risks are rarely limited to the model itself. Production failures often come from unclear business ownership, poor data quality, weak access controls, missing evaluation criteria, insufficient capacity planning, or uncontrolled token and infrastructure costs.
1. What business outcome will AI improve?
Start with the user and the workflow—not the model.
Define:
- The user segment: employee, customer, analyst, support agent, developer, or operations team
- The workflow: what task is being performed today?
- The baseline: current time, cost, quality, error rate, or throughput
- The AI intervention: assist, recommend, draft, classify, retrieve, automate, or act
- The measurable expected outcome
Example: Instead of launching “an AI support assistant,” define: “Help Tier 1 support agents resolve password-reset and account-access tickets by retrieving approved knowledge and drafting a response. The target is to reduce average handling time without reducing quality or customer satisfaction.”
Metrics
| Metric | What it answers |
|---|---|
| Task-completion rate | Did users finish the intended task? |
| Average handling time | Did AI reduce time spent per task? |
| Automation rate | What percentage of eligible work was completed without manual execution? |
| First-contact resolution | Did AI improve resolution quality for support workflows? |
| Error or rework rate | Did AI create additional downstream work? |
| User adoption | Are users choosing the AI-enabled workflow? |
| Customer or employee satisfaction | Is the experience actually better? |
| Business KPI improvement | Did the use case improve revenue, retention, cost, risk, or productivity? |
A production go/no-go decision should include a target, not just a metric. For example: “At least 70% of eligible tickets receive an AI draft; agent acceptance exceeds 60%; average handling time falls by 15%; and no material decline occurs in customer satisfaction.”
2. Is AI the right solution?
Do not use an LLM because the problem sounds like an AI problem. Compare it with simpler alternatives:
- Search with improved metadata and permissions
- Rules or deterministic workflow automation
- A standard application feature
- Traditional machine learning classification
- A retrieval layer without open-ended generation
- Process redesign or better source documentation
AI is appropriate when ambiguity, unstructured information, natural-language interaction, summarization, extraction, reasoning support, personalization, or content generation creates a material advantage.
Example: If employees need to find a single policy document, well-indexed enterprise search may be more predictable and cheaper than a generative assistant. If they need to compare several policy documents and explain which one applies to a specific scenario, RAG with citations may be justified.
Metrics
| Metric | What it answers |
|---|---|
| Quality lift over baseline | Is the AI result better than search, rules, or the existing process? |
| Incremental value | What additional benefit does AI provide beyond the simpler option? |
| Adoption versus current workflow | Do users prefer and trust the AI-enabled workflow? |
| Automation rate | Does AI eliminate meaningful manual work? |
| Cost per successful task | Is the added value worth the operating cost? |
| Exception rate | How often does work require manual correction or fallback? |
| Time to resolution | Does AI improve end-to-end speed, not only response generation? |
A useful decision criterion is:
Incremental AI value>Additional cost+risk+operational complexityIf that cannot be demonstrated, the simpler solution is usually the better production choice.
3. Is the data ready, permitted, and governed?
Data readiness is not only about whether documents can be loaded into a vector database. It includes whether the data is accurate, current, legally permitted, appropriately classified, access-controlled, and suitable for the selected model provider and hosting environment.
Classify every source before it is made available to an AI workflow:
- Public
- Internal
- Confidential
- Personally identifiable information
- Financial, payment, health, legal, or regulated information
- Intellectual property, trade secrets, or source code
- Customer, employee, or third-party data
Under GDPR, personal data must be processed lawfully, fairly, and transparently; collected for specific purposes; limited to what is necessary; kept accurate and current; retained only as long as needed; and protected with appropriate security measures. GDPR also explicitly identifies measures such as pseudonymization and encryption as appropriate safeguards based on risk.[eur-lex.europa][eur-lex.europa]
For enterprise AI, answer these questions before production:
- Is the data allowed to leave the organization or approved cloud boundary?
- Will the provider retain prompts, responses, embeddings, or logs?
- Can the provider use submitted data for model training or service improvement?
- Is a data-processing agreement required?
- Does the selected cloud region meet data-residency obligations?
- Does the data contain PII, payment data, health information, source code, or confidential contracts?
- Do users retain the same permissions in the AI application that they had in the source system?
- Can data be deleted, corrected, or excluded when policy or regulation requires it?
- Who owns the source data and is accountable for keeping it current?
RAG access controls
RAG can improve answer quality by retrieving approved enterprise content, but it can also become an access-control bypass if implemented carelessly.
Example: An HR assistant may retrieve policy documents correctly but must not reveal compensation, performance, medical, or employee-relations information to a manager who lacks access in the original HR system.
Apply authorization at retrieval time—not only when content is indexed. The retrieval layer should filter documents and chunks using the user’s identity, role, department, geography, and source-system permissions.
NIST identifies privacy risks, sensitive-information leakage, and prompt injection as important generative-AI concerns. It also describes “confabulation”—confidently stated but false content—as a central risk that organizations must address through governance and measurement.[airc.nist]
Metrics
| Metric | What it answers |
|---|---|
| Data-source classification coverage | What percentage of connected sources have a documented classification? |
| Source-owner coverage | Does every source have an accountable business or data owner? |
| Freshness | How old is the retrieved data relative to the required update schedule? |
| Completeness | Are expected documents, fields, and metadata present? |
| Duplicate or stale-content rate | Is retrieval polluted by redundant or outdated content? |
| Sensitive-data detection rate | How much PII or restricted data is found before indexing or inference? |
| Access-control coverage | What percentage of sources enforce source-equivalent permissions? |
| Encryption coverage | Are data encrypted in transit and at rest? |
| Unauthorized retrieval attempts | Is the system blocking users from restricted information? |
| Data-leakage incidents | Has sensitive content appeared in outputs, logs, prompts, or telemetry? |
| Audit-log coverage | Can the organization trace who accessed, retrieved, or acted on data? |
| Data-subject request turnaround time | How quickly can the system support required access, correction, or deletion actions? |
Practical example: A knowledge assistant should fail closed. If the identity service or source authorization check is unavailable, it should not retrieve broad content “just in case.” It should return a controlled message or route the user to the original system.
4. How will quality, safety, and access be evaluated?
A few successful prompt examples are not an evaluation strategy. Before production, create a representative evaluation set based on real workflows, expected user questions, edge cases, adversarial inputs, and known failure modes.
For a support assistant, the test set might include:
- Common customer questions with verified answers
- Ambiguous questions that should trigger clarification
- Requests outside the product scope
- Outdated policy scenarios
- Conflicting source documents
- Prompts that attempt to override system instructions
- Questions that request restricted customer or employee data
- Requests that would trigger a tool, workflow, or external action
NIST’s Generative AI Profile identifies confabulation, privacy risks, and prompt injection among the risks that must be governed and measured. OWASP identifies prompt injection as a risk in which crafted inputs alter an LLM’s behavior in unintended ways, potentially leading to unauthorized access, data disclosure, or compromised decisions.[airc.nist][owasp][genai.owasp]
Evaluation layers
| Layer | What to validate | Example |
|---|---|---|
| Functional quality | Accuracy, completeness, relevance, format | Does the response correctly explain the approved refund policy? |
| Groundedness | Whether claims are supported by retrieved sources | Does every policy claim trace to an approved, current document? |
| Safety | Harmful, prohibited, or misleading output | Does the assistant refuse unsafe medical, legal, or financial advice where required? |
| Security | Prompt injection, data exfiltration, tool misuse | Can a malicious document instruct the agent to reveal secrets? |
| Authorization | User-level access enforcement | Can a manager retrieve only documents they can access directly? |
| Reliability | Expected behavior under failures | What happens if retrieval fails or the model provider times out? |
| Human oversight | Correct approval and escalation behavior | Does the agent request approval before changing a customer record? |
Metrics
| Metric | What it answers |
|---|---|
| Task-success rate | Did the AI produce an outcome that satisfies the business task? |
| Accuracy rate | How often is the output correct against verified ground truth? |
| Grounded-answer rate | How often are important claims supported by approved source material? |
| Citation correctness | Do citations point to the correct, relevant source? |
| Hallucination or confabulation rate | How often does the system state unsupported or false information? |
| User correction rate | How frequently do users edit or reject generated output? |
| Escalation rate | How often must a human take over? |
| Harmful-output rate | How often does the system produce disallowed or unsafe content? |
| Prompt-injection block rate | How often are malicious instructions detected and safely handled? |
| Data-leakage incidents | Did restricted information appear in output or telemetry? |
| Unauthorized tool-call attempts | Did the system attempt an action outside approved permissions? |
| Mean time to remediate | How quickly are identified quality or safety defects corrected? |
For high-impact workflows, use human approval before actions. A draft email, a recommended remediation step, and an autonomous payment change represent very different risk levels.
Example: An AI agent may prepare a change request for an infrastructure configuration. It should not apply the change until an authorized engineer reviews the diff, confirms the target environment, and approves execution.
5. Can it scale technically?
Scaling an AI application means more than selecting a model that responds well in a notebook. The production system includes identity, application APIs, orchestration, retrieval, vector search, model endpoints, tools, databases, queues, logging, monitoring, and fallbacks.
Capacity planning should estimate:
- Monthly and daily active users
- Requests per minute and concurrent sessions
- Peak periods and burst traffic
- Input and output tokens per request
- Context size and retrieved-document volume
- Tool calls per task
- Model rate limits and token-per-minute limits
- Latency requirements by workflow
- Vector-store index size, query rate, and retrieval latency
- Failure scenarios, retry behavior, and fallback paths
Google identifies model latency and token throughput as important generative-AI KPIs. For shared or multi-tenant agent systems, quota enforcement and context controls help prevent a single tenant or workflow from exhausting shared model capacity or creating unbounded token consumption.[cloud.google][docs.cloud.google]
Technical design considerations
- Synchronous versus asynchronous: A user-facing copilot may need a low-latency synchronous response. Document summarization or batch extraction can run asynchronously through a queue.
- Caching: Cache repeated retrieval results, embeddings, stable system context, and eligible responses. Do not cache personalized or sensitive responses without proper isolation.
- Rate limiting: Apply limits per user, tenant, API key, workload, and model to prevent abuse and protect shared capacity.
- Fallbacks: Define a degraded mode when the primary model, retrieval store, or external tool fails. This could mean switching to a smaller model, providing source search without generation, or asking the user to retry.
- Context controls: Use chunking, ranking, summarization, metadata filters, and maximum-token budgets. Passing entire documents or conversation histories by default increases both cost and error risk.
- Load testing: Test normal load, peak demand, burst traffic, dependency outages, and long-context requests before launch.
Metrics
| Metric | What it answers |
|---|---|
| Requests per minute | What steady and peak request volume must be supported? |
| Concurrent users | How much simultaneous demand exists? |
| Tokens per minute | Can model capacity support the expected workload? |
| p50, p95, and p99 latency | How quickly does the system respond for typical and worst-case users? |
| Availability | What proportion of valid requests are successfully served? |
| Error and timeout rate | How often does the system fail or exceed the response-time target? |
| Provider rate-limit failures | Is the model endpoint capacity sufficient? |
| Retrieval latency | Is search or vector retrieval delaying the user experience? |
| Cache-hit rate | How much repeated work is being avoided? |
| Queue depth and wait time | Are asynchronous workloads building up faster than they are processed? |
| Retry rate | Are dependency failures creating extra latency and cost? |
| Fallback success rate | Does degraded-mode behavior still enable users to complete work? |
| Peak-load task-success rate | Does quality remain acceptable under production demand? |
For reliability, define service-level indicators and objectives. For example, an SLI may be the proportion of valid requests served successfully or the proportion completed under a latency threshold.[static.googleusercontent]
Example SLO: “Over a rolling 30-day period, 99.5% of valid requests must receive a response, and 95% of interactive requests must complete in under 8 seconds.” The right threshold depends on the workflow; a real-time agent-assist use case has different expectations from overnight report summarization.
6. What will it cost to run?
AI costs are driven by more than the displayed token price of a model. Include:
- Input tokens from user prompts, system prompts, chat history, retrieved documents, and tool output
- Output tokens generated by the model
- Embedding creation and re-indexing
- Vector database storage and query operations
- Application compute, API gateways, queues, networking, and storage
- Observability, tracing, logging, and security tooling
- Retries, failed requests, and agent loops
- Model evaluation and human-review operations
- Reserved or provisioned throughput where applicable
For RAG applications, a model’s pricing commonly depends on input and output token volume, so excessive context directly affects cost. Production tooling can monitor token usage, quality, latency, and error rates together; Azure documents monitoring of prompt, completion, and total token usage alongside quality and operational measures.[docs.cloud.google][learn.microsoft]
Core cost calculations
Use the provider’s current pricing for the actual forecast; model pricing, quotas, and throughput options can change frequently.
Metrics
| Metric | What it answers |
|---|---|
| Input and output tokens per request | Where is token consumption coming from? |
| Average context size | Are prompts, history, and retrieved content larger than necessary? |
| Cost per request | What does a single invocation cost? |
| Cost per active user | How does cost change with adoption? |
| Cost per successful task | Is the AI workflow economically justified? |
| Cost by model | Is the selected model appropriate for the task? |
| Cost by feature or workflow | Which capabilities are consuming the budget? |
| Embedding and retrieval cost | Is RAG infrastructure contributing materially to spend? |
| Retry-related cost | Are errors or poor reliability increasing spend? |
| Cache savings | How much cost and latency does caching avoid? |
| Monthly spend versus budget | Is actual usage within the approved financial envelope? |
| Forecasted cost at peak demand | Can the system scale without unexpected budget overruns? |
Cost controls
- Set maximum input, output, and session token limits.
- Limit retrieved chunks and use relevance thresholds.
- Summarize long conversation history rather than sending the full transcript.
- Route simple tasks to smaller, lower-cost models.
- Cache stable system context and repeated retrieval results.
- Put ceilings on agent iterations and tool calls.
- Use asynchronous or batch processing for workloads that do not need immediate responses.
- Configure budget alerts and user, tenant, or project quotas.
Google recommends practical cost and capacity controls such as setting maximum output tokens, monitoring token throughput, using caching, using batch processing for eligible workloads, and using lower-cost models where task quality permits.[docs.cloud.google]
Example: A support assistant may use a smaller model to classify the issue and retrieve relevant knowledge, then use a higher-capability model only when it needs to generate a nuanced response. This model-routing pattern can reduce cost without applying a lower-quality model to every task.
7. Where will it run, and how will it be operated?
Hosting is a product, risk, security, performance, and financial decision—not merely an infrastructure preference.
Choose based on:
- Data classification and data-residency requirements
- Approved vendors and contractual restrictions
- Enterprise identity and access-management integration
- Network isolation and private connectivity requirements
- Latency to users, source systems, and model endpoints
- Availability, disaster recovery, and provider concentration risk
- Model customization needs
- Expected scale and cost profile
- Internal capability to operate GPUs, models, and ML infrastructure
- Monitoring, incident response, and change-management maturity
Hosting and operations decisions
| Decision | When it fits | Key tradeoff | Example metrics |
|---|---|---|---|
| Managed model service | You prioritize speed, access to advanced models, managed scaling, and low platform-operations effort | Less infrastructure control and dependency on provider capacity, regions, and policies | Availability, p95 latency, rate-limit headroom, provider incidents, cost per successful task |
| Private or enterprise cloud AI | You need enterprise identity, approved regions, private networking, governance controls, and managed model operations | More cloud architecture and configuration effort | Compliance-control coverage, private-network coverage, security-review findings, deployment lead time |
| Self-hosted model | Strict data control, specialized customization, air-gapped needs, or predictable high-volume economics justify the operational burden | Requires GPU capacity, model serving, patching, observability, scaling, and security expertise | GPU utilization, tokens per second, queue wait time, inference latency, infrastructure cost, operator effort |
| Hybrid model | Some data must remain internal while selected workloads can use external model services | More routing, data-boundary, and integration complexity | Data-egress volume, policy violations, routing success rate, end-to-end latency |
| Regional deployment | Regulatory requirements, user location, resiliency, or approved-provider regions dictate placement | More regions increase operational and data-consistency complexity | Regional latency, availability, failover time, recovery-point objective, recovery-time objective |
| RAG architecture | The use case depends on current enterprise knowledge and source citations | Retrieval quality and access-control enforcement become critical | Grounded-answer rate, retrieval precision, stale-content rate, authorization failures |
| Fine-tuning | Prompting and retrieval cannot consistently produce the required behavior, format, or domain performance | Higher data, evaluation, governance, and model-lifecycle burden | Quality lift versus RAG, training cost, regression rate, evaluation pass rate |
| Human-in-the-loop | Decisions are high-impact, irreversible, regulated, or difficult to validate automatically | Lower automation rate but stronger control | Review coverage, approval time, override rate, escaped-defect rate |
Operating model
Before launch, assign named owners for:
- Product outcomes and business KPIs
- Data ownership and content lifecycle
- Security, privacy, and compliance review
- Model selection and vendor management
- Prompt, workflow, and evaluation changes
- Infrastructure reliability and incident response
- Cost management and quota enforcement
- User feedback and support
- Model, retrieval, and policy monitoring
Production monitoring should combine product, quality, safety, operational, and financial signals. Azure’s generative-AI monitoring guidance includes recurring monitoring for quality measures such as groundedness, coherence, fluency, and relevance, with configurable thresholds and alerts.[learn.microsoft][learn.microsoft]
Metrics
| Metric | What it answers |
|---|---|
| Mean time to detect | How quickly is an incident or degraded-quality condition discovered? |
| Mean time to recover | How quickly can service be restored? |
| Deployment failure rate | Are releases introducing instability? |
| Change failure rate | How often do prompt, model, retrieval, or policy changes cause regressions? |
| Incident count and severity | Is the platform becoming more or less stable? |
| Alert coverage | Are critical quality, security, reliability, and cost signals monitored? |
| Compliance-control coverage | Are required controls implemented and verified? |
| Backup and recovery success | Can service and required data be restored? |
| Model-provider dependency | How exposed is the product to a single model, region, or vendor? |
| Owner assignment coverage | Does every critical control have an accountable owner? |
Bottom Line
An AI pilot is ready for production when it has more than a good demo. It needs an approved business case, governed data, measurable quality thresholds, tested security controls, realistic capacity planning, token and cost forecasts, a hosting decision aligned with risk, and an operating model with accountable owners.
The production question is not:
“Does the model work?”
It is:
“Can this end-to-end AI product deliver a measurable outcome securely, reliably, at the expected scale, with controlled cost and clear operational ownership?”
References
- NIST AI Risk Management Framework (AI RMF 1.0). National Institute of Standards and Technology.
A practical framework for managing AI risks across governance, mapping, measurement, and ongoing management. It is useful for defining quality, safety, security, monitoring, and accountability metrics. NIST AI RMF 1.0[nvlpubs.nist] - NIST AI RMF: Generative AI Profile. National Institute of Standards and Technology.
Provides guidance for generative-AI-specific risks, including confabulation, data privacy, harmful content, security, and evaluation. NIST Generative AI Profile[nvlpubs.nist] - General Data Protection Regulation (GDPR). European Union.
Relevant when an AI solution processes personal data, particularly for lawful processing, transparency, security, retention, deletion rights, and cross-border data transfers. GDPR information portal - OWASP Top 10 for Large Language Model Applications. Open Worldwide Application Security Project.
Useful for assessing LLM-specific security risks such as prompt injection, sensitive-information disclosure, insecure output handling, excessive agency, and supply-chain vulnerabilities. OWASP LLM Top 10 - ISO/IEC 42001: Artificial Intelligence Management System. International Organization for Standardization.
Establishes a management-system approach for responsible AI governance, including risk management, accountability, controls, and continuous improvement. ISO/IEC 42001 overview - NIST Privacy Framework. National Institute of Standards and Technology.
Helps organizations identify and manage privacy risk in systems that collect, process, retain, or share personal data. NIST Privacy Framework - NIST Cybersecurity Framework 2.0. National Institute of Standards and Technology.
Relevant for the broader security controls around AI hosting, identity and access management, encryption, monitoring, incident response, and third-party risk. NIST CSF 2.0