The problem
A language model can produce a convincing answer even when its source material does not support it. For drug-safety information, the goal was a retrieval demonstrator that cites MHRA source passages and refuses unsupported questions.
The approach
I kept a simple fixed-chunk baseline and built an evaluation harness around an 80-question, source-verified benchmark: 64 answerable questions and 16 unanswerable ones. Ground truth used source URLs and verbatim passages rather than chunk IDs, so changing the chunking strategy would not change the target.
How the solution works
MHRA articles are fetched and cleaned, split into fixed token chunks and embedded with BGE. FAISS retrieves semantically similar passages; BM25 retrieves lexical matches. Reciprocal rank fusion combines their rankings. Claude generates an answer from the retrieved context with citations, or refuses when support is missing. FastAPI and Docker package the serving pipeline.
How I made decisions
Before each retrieval experiment, I set a meaningful-improvement threshold: at least two to three questions at rank 1, or 0.02 mean reciprocal rank. A one-question movement was treated as noise. I compared chunking experiments against the dense baseline and embedding changes against the hybrid configuration, rather than mixing comparison points.
What worked
Small-to-big retrieval improved rank-1 results from 45/64 to 47/64. Hybrid dense and keyword retrieval reached 52/64 and became the deployed choice. My final generation evaluation produced 63/64 correct answers, all 16 correct refusals and no false refusals on that benchmark.
What did not work, and how I investigated
Header-aware chunking initially fell to 66% recall at rank 1. Seven of eight rank-1 losses went to shorter chunks, and six to sibling sections in the same article. I found tiny sections that matched a drug name without enough useful context. A 64-token minimum-section rule fixed two examples, but the corrected approach still reached only 44/64, below the baseline. I rejected it. Semantic chunking also regressed to 42/64.
Why I removed the reranker
The reranker added substantial retrieval latency without changing correctness labels across the 64 answerable evaluation questions. Its extra retrieval coverage did not improve the answer-quality scores. I kept hybrid retrieval alone. Fine-tuning the reranker, using a larger BGE model and switching to PubMedBERT also failed to beat the chosen configuration.
How I measured latency
On a 4-CPU machine, I timed retrieval after one warm-up call per configuration, excluding model initialisation. Queries ran sequentially: 20 answerable questions for hybrid retrieval and eight for the reranked configuration. Median retrieval time was 0.19 seconds and 39.01 seconds respectively.
I timed three Claude generation calls separately. Adding their shared median of 2.53 seconds to each retrieval median gives 2.72 seconds versus 41.54 seconds, a ratio of about 15.3×. These are sums of component medians, not medians from complete HTTP requests. They do not measure cold starts or concurrent traffic.
The reranker scores the candidate passages for one query together. Its code leaves the internal inference batch size at the library default; I did not record a fixed batch size for this run. The unequal query counts and small generation sample also limit the precision of this comparison.
Effect and limits
The measured effect was better retrieval than the initial baseline and a faster serving choice than the reranked candidate. It is not evidence of clinical reliability. The benchmark is self-authored, the generation judge was checked against 27 manual labels, and the same benchmark informed iterative engineering decisions. A fresh independent evaluation would strengthen the evidence. This is an information-retrieval demonstration, not medical advice.
Measured retrieval experiments
| Configuration | Rank-1 retrieval | Decision |
|---|---|---|
| Fixed-chunk baseline | 45/64 | Comparison point |
| Header-aware, corrected | 44/64 | Rejected |
| Small-to-big | 47/64 | Improved dense baseline |
| Semantic chunks | 42/64 | Rejected |
| Hybrid retrieval | 52/64 | Deployed |
| Hybrid + reranker | 51/64 | Too slow for no answer-quality gain |
| Fine-tuned reranker | 46/64 | Rejected |
| Larger BGE embedder | 51/64 | No meaningful gain |
| PubMedBERT | 41/64 | Rejected |
Counts are from my evaluation runs. Chunking and embedding experiments use different comparison configurations, as explained above.
Try the demo
Ask a drug-safety question and inspect the cited passages alongside the answer. This is a demonstration, not medical advice. The external demo may take a moment to wake up.
Open interactive demo