Recall@5
Language systems
PharmaLens
Evidence-grounded question answering over pharmaceutical regulatory documents.
The problem
How can retrieval preserve the product context and strength of a regulatory requirement?
A relevant passage can still refer to the wrong product, formulation or administration route—or turn a recommendation into a mandatory requirement. PharmaLens retrieves and verifies answers over EMA EPAR/SmPC documents, FDA product-specific guidance and citizen petitions while preserving that regulatory context.
My role & contributions
I built PharmChunk and the verified hybrid RAG system, and constructed the PharmaEval v2 evaluation.
Supervision: Dr. Ravi Kiran Kummamuru
Method & system
PharmaLens verification pipeline
- Index the evidencePharmChunk: structure + product, formulation, route and modality metadata
- Clarify the queryAmbiguous product or formulation → ask for clarification
- Retrieve and generateBM25 + dense retrieval → reciprocal-rank fusion → evidence-cited answer
- Verify compatibilityCheck product, formulation and route; hard mismatch → abstain
- Verify claim supportSupported → cited answer. Unsupported → strict abstention or soft claim stripping
Documented guard test
A citation can point to the wrong formulation
Reconstructed view of a documented synthetic guard test, separate from the descriptive benchmark.
Question
What release testing is needed for the injection?
Evidence supplied to the guard
Dissolution testing applies to solid oral tablets.
The question specifies a subcutaneous injection. The fixture supplies oral-tablet evidence, so its source refers to a different formulation.
Compatibility and claim checks
The candidate answer says:
The cited source lists dissolution testing.
PharmaGuard flags a dosage-form mismatch and incompatible source formulation.Outcome
Blocked in the saved test result. A citation alone does not make oral-tablet evidence applicable to an injectable-product question.
dissolution_cited_from_wrong_form. Test fixture ↗ · Recorded guard result ↗- A section boundary carries meaning. PharmChunk keeps passages within document structure and attaches product, formulation, route and regulatory-modality metadata, reducing the risk of combining evidence about different products.
- Product ambiguity triggers clarification before generation. Claim support and product/formulation/route compatibility are separate checks; a hard mismatch requires abstention in either verification mode.
- Strict mode withholds answers with unsupported claims. Soft mode removes unsupported claims when there is no hard mismatch. Returned evidence identifies document, section and page.
Evaluation & results
PharmaEval v2 contains 100 extractive-development, 75 classification-development, 40 descriptive-development and 25 held-out descriptive questions: 240 total. A separate typed stress suite contains 75 cases. The held-out questions were not used for prompt or threshold tuning.
100Extractive development
75Classification development
40Descriptive development
25Held-out descriptive
240 benchmark questions total, plus a separate 75-case typed stress suite.
Held-out evaluation
Faithfulness
Key-point coverage
These results come from 25 held-out descriptive questions that were not used for prompt or threshold tuning.
Six-label regulatory-modality classification achieved 98.7% accuracy on the 75-question classification-development split.
Source: manuscript §IV.B–C and Table II.
Faithfulness measures support in the retrieved evidence using a judge-based evaluation. PharmaLens is a research prototype requiring expert oversight, not clinical decision support.
Future work
- Expand the regulatory corpus and assess generalization across new product, formulation and route combinations.
- Complement judge-based source-support scores with evaluation by regulatory experts.