Production RAG system — live on AWS

Answers that show their work.

Verity retrieves, scores, and cites its evidence before it generates a sentence — and says so plainly when the evidence isn't there. Hybrid BM25 + vector retrieval, two-stage reranking, and a calibrated hallucination fallback, running on Kubernetes.

~180ms median query latency 0.35 min. evidence threshold Llama 3.1 via Groq k3s on AWS EC2
01 · Source document
"This RAG system uses hybrid retrieval combining BM25 keyword search with dense vector embeddings, followed by a reranking stage to reorder the most relevant chunks before generation."
02 · Retrieved & scored
vector_score0.4673
bm25_score0.0000
rerank_score0.3902
confidence: medium
03 · Grounded answer
"This system uses hybrid retrieval combining BM25 keyword search with dense vector embeddings, then reranks the results before generation.[1]"
The pipeline

Six stages between a document and a sentence.

01

Ingest & chunk

Documents are split with recursive chunking and tagged with source metadata, so every downstream answer can be traced back to where it came from.

02

Hybrid retrieve

Dense vector similarity is blended with BM25 keyword scoring, so exact terms and acronyms surface even when their embedding match is weak.

03

Rerank

A second, query-aware pass reorders the top candidates — a check that's too slow to run on the whole corpus but cheap on twenty finalists.

04

Confidence score

Every chunk is labeled high, medium, or low confidence before it's allowed anywhere near the model.

05

Constrained generate

The model is instructed to answer only from what was retrieved, and to cite which source backs each claim.

06

Evidence check

If the strongest match falls under threshold, Verity says the evidence isn't there — instead of guessing.

Under the hood

Built to run like production, not a notebook.

Compute

AWS EC2 running k3s, a lightweight Kubernetes distribution — rolling deploys, health probes, and self-healing pods.

Registry

Multi-stage Docker builds pushed to AWS ECR, keeping runtime images lean and reproducible.

Vector store

ChromaDB for dense retrieval, paired with an in-memory BM25 index rebuilt per query for hybrid search.

Generation

Llama 3.1 8B served via Groq — sub-200ms generation on constrained, citation-aware prompts.

Security

JWT sessions and Google OAuth, served over HTTPS via Let's Encrypt and Traefik ingress.

Observability

Prometheus metrics on every query path, plus a continuous evaluation suite that checks retrieval quality on each deploy.

Why it matters

It would rather say "I don't know" than make something up.

Evidence found
"What is the RAG system built on?"
"The RAG system is built on AWS, using Kubernetes k3s and ChromaDB vector database." — relevance 0.56, confidence: medium
Insufficient evidence
"What is the capital of France?"
"I don't have sufficient evidence in the uploaded documents to answer this question confidently. Please upload relevant documents or rephrase your question."