Deployment

Taking RAG to production

Work through this before opening the system to real users. Tick each item — your progress is kept in this browser, and it is shared with the Vietnamese version of this page.

Data & indexing

• The list of data sources, and who owns each one, is settled.
• The cleaning pipeline strips headers and footers, handles tables, and normalises Unicode to NFC.
• Every chunk carries metadata: source, section, version, last-updated date, permission groups.
• The chunking strategy was chosen from evaluation numbers, not from intuition.
• Changed and deleted documents reach the index automatically, and the sync lag is measured.
• The embedding model version is recorded, and there is a plan for re-embedding when it changes.

Retrieval

• Hybrid search (vectors + keywords) has been compared against vectors alone.
• Top-k, the score threshold and reranking are configured and measured on the golden set.
• When nothing clears the threshold the system declines politely, instead of calling the LLM with empty context.
• Follow-up questions in a conversation are rewritten before retrieval.

Security & access control

• The permission filter lives inside the retrieval query — a user never receives a passage they may not read.
• There is an automated test: a user in group A asks about a document only group B may see.
• Cache keys, where caching exists, include the permission scope.
• The prompt separates data from instructions, and the model cannot take consequential actions from retrieved content.
• Question and answer logging complies with the personal-data policy, with a stated retention period.

Answer quality

• The golden set has at least 50 questions, includes unanswerable ones, and has been reviewed by a person.
• There is a baseline for recall@k, faithfulness and correctness, plus a minimum bar for release.
• Every answer carries citations the reader can click through to the source.
• The evaluation runs in CI whenever the prompt, the model or the retrieval configuration changes.

Operations

• Full per-stage logging: question, transformed query, passages & scores, prompt, answer.
• A dashboard: p95 latency, tokens per question, refusal rate, 👍/👎 feedback.
• Rate limits and a per-user daily cost budget are in place.
• There is a fallback for when the LLM provider or the vector store fails.
• A weekly process moves 👎 questions into the evaluation set.