Engineering Private RAG Systems That Pass Security & Accuracy Audits

Anyone can wire an OpenAI key to a PDF loader in an afternoon. Shipping internal document search into a regulated Irish or European company is a different engineering problem: Support staff must never retrieve Board annexes, exact contract clause numbers can't blur into semantic neighbours, and every prompt change needs an objective regression score before deployment.

1. Hybrid pgvector + BM25 Beats Dense Vectors Alone

Pure embedding similarity struggles with exact SKU codes, statutory section numbers (like Stamp 1G vs Stamp 4), and contract clause identifiers (§8.2(b)). Combining PostgreSQL pgvector cosine distance with lexical BM25 ranking via Reciprocal Rank Fusion (RRF) captures both conceptual queries and exact clause lookups inside your existing relational database stack.

2. Enforce Role Permissions in SQL—Never in the System Prompt

Telling an LLM “do not mention confidential salary figures to non-HR users” is not access control. Every ingested chunk should carry tenant and role ACL metadata so PostgreSQL filters unauthorized rows before context assembly. If zero permitted chunks match, the FastAPI service returns a deterministic abstention in under 50 milliseconds without invoking the model at all.

3. Never Let an LLM Do Statutory or Financial Arithmetic

Language models summarize policies well, but they shouldn't calculate fee caps or day-count thresholds in free text. Separate retrieval from deterministic calculation. You can inspect this pattern live in our open-source public build, Ask Ireland, which pairs grounded .gov.ie / Irish Immigration source citations with a deterministic TypeScript residency calculator:

4. Gate Every Chunking or Model Change Behind a Golden Eval Set

Before shipping a 2-week RAG pilot to staging, we build a 30–50 question golden dataset with domain owners—covering straightforward lookups, multi-clause comparisons, out-of-scope questions that must abstain, and adversarial cross-role probes. Running that suite in CI tracks citation precision@5, p95 latency, and monthly token spend before any change reaches production.