Developing a Retrieval-Augmented Question-Answering System Using Large Language Models

Abstract

Retrieval-augmented generation (RAG) connects large language models with collections that can be inspected, updated, and cited. A dependable system, however, requires coordinated decisions about cleaning, metadata, retrieval, context selection, citation, and abstention. This article presents a lightweight, model-agnostic RAG architecture for small digital-humanities collections. The prototype combines word and character retrieval, curator-defined query expansion, and weighted reciprocal rank fusion. Source-labelled passages are passed to an instruction-tuned model under a prompt contract that permits only evidence-based answers. A reproducible synthetic corpus of 16 archival records and 48 questions was divided into 12 development and 36 held -out questions. On the held-out set, the hybrid method reached Recall@1 of 0.861, Recall@3 of 0.944, and mean reciprocal rank of 0.911. Character retrieval was the strongest individual method for OCR-like noise. These figures measure retrieval, not end-to-end generation.