A Clinical Q&A Assistant Grounded in the Merck Manual

Problem

A bare LLM answering clinical questions is fluent but unaccountable — it cites no sources and quietly invents specifics. The goal: make a local, private assistant whose answers are grounded in an authoritative reference, and measure whether retrieval actually improves answer quality rather than assuming it does.

Approach

Four configurations were evaluated on the same five clinical queries with a scoring rubric (grounding, specificity, coverage): (1) bare Mistral-7B, (2) prompt-engineered, (3) RAG over the Merck Manual, (4) RAG with tuned temperature. Scoring the stages separately shows exactly what each technique buys.

Technology

• Mistral-7B-Instruct (GGUF) running locally via llama-cpp — no patient data leaves the machine
• LangChain + ChromaDB for chunking, embedding, and retrieval
• gte-small sentence embeddings; PyMuPDF for PDF extraction

Result

Grounding rose from 2.0 (bare LLM) to 5.0 once retrieval was attached; without retrieval the model cited zero numeric clinical values across all five queries. The evaluation also surfaced two findings that matter in production: a silent retrieval failure (a leg-fracture query retrieved hip-fracture rehabilitation content and the model answered confidently anyway), and evidence that the LLM scoring its own answers rated everything 5/5 — so human rubric scoring was used instead.