Question Answering Systems

A user types "When did the first iPhone launch?" and expects "June 29, 2007." Not "Apple's history is long and varied." Not "2007" sitting on its own with no sentence around it. A direct, grounded, correct answer.

Three architectures have dominated QA for the last decade, and each one answers a different question about where the answer comes from.

• Extractive QA. Given a question and a passage known to contain the answer, find the start and end indices of the answer span inside that passage. SQuAD is the canonical benchmark.
• Open-domain QA. No passage is given up front. Retrieve the relevant passage first, then extract or generate an answer from it. This is the bedrock of every RAG pipeline running today.
• Generative / closed-book QA. A large language model answers straight from its parametric memory, no retrieval step involved. Fastest at inference, least reliable on facts.

The trend in 2026 is hybrid: retrieve the best few passages, then prompt a generative model to answer grounded in them. That's RAG, and lesson 14 covers the retrieval half in depth. This lesson builds the QA half.

Extractive. Encode question and passage together with a transformer (BERT family), then train two heads that predict the start and end token indices of the answer. Loss is cross-entropy over valid positions, and the output is a span lifted straight from the passage. It never hallucinates, by construction, and it never handles a question the passage can't answer, by construction either.

Retrieval-augmented (RAG). Two stages. First, a retriever finds the top-k passages from a corpus. Second, a reader, extractive or generative, produces the answer using those passages. Splitting retriever from reader lets you train and evaluate each one independently, and modern RAG usually adds a reranker (a smaller model that re-scores the retrieved passages before the reader sees them) between the two.

Generative. A decoder-only LLM (GPT, Claude, Llama) answers from learned weights with no retrieval step at all. It's excellent on common knowledge and catastrophic on rare or recent facts, because the hallucination rate is inversely correlated with how often a fact showed up in the pretraining data.

Step 1: extractive QA with a pretrained model

from transformers import pipeline
 
qa = pipeline("question-answering", model="deepset/roberta-base-squad2")
 
passage = (
    "Apple Inc. released the first iPhone on June 29, 2007. "
    "The device was announced by Steve Jobs at Macworld in January 2007."
)
question = "When was the first iPhone released?"
 
answer = qa(question=question, context=passage)
print(answer)

{'score': 0.98, 'start': 40, 'end': 53, 'answer': 'June 29, 2007'}

deepset/roberta-base-squad2 is trained on SQuAD 2.0, which includes unanswerable questions. That doesn't mean the pipeline refuses to guess: by default it returns the highest-scoring span even when the model's null score wins, so it does not automatically hand back an empty answer. Pass handle_impossible_answer=True if you want explicit "no answer" behavior; the pipeline then returns an empty answer only when the null score beats every span score. Check the score field regardless.

Step 2: a retrieval-augmented pipeline (sketch)

from sentence_transformers import SentenceTransformer
import numpy as np
 
encoder = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2")
 
corpus = [
    "Apple Inc. released the first iPhone on June 29, 2007.",
    "Macworld 2007 featured the iPhone announcement by Steve Jobs.",
    "Android launched in 2008 as Google's mobile operating system.",
    "The first iPod was released in 2001.",
]
corpus_embeddings = encoder.encode(corpus, normalize_embeddings=True)
 
 
def retrieve(question, top_k=2):
    q_emb = encoder.encode([question], normalize_embeddings=True)
    sims = (corpus_embeddings @ q_emb.T).squeeze()
    order = np.argsort(-sims)[:top_k]
    return [corpus[i] for i in order]
 
 
def answer(question):
    passages = retrieve(question, top_k=2)
    combined = " ".join(passages)
    return qa(question=question, context=combined)
 
 
print(answer("When was the first iPhone released?"))

It's a two-stage pipeline: a dense retriever (Sentence-BERT) finds relevant passages by semantic similarity, and an extractive reader (RoBERTa-SQuAD) pulls the answer span out of the combined top passages. This works fine on small corpora. Once you're past a million documents, reach for FAISS (a library built for fast nearest-neighbor search over embeddings) or a vector database instead.

Step 3: generative with RAG

def rag_generate(question, llm):
    passages = retrieve(question, top_k=3)
    prompt = f"""Context:
{chr(10).join('- ' + p for p in passages)}
 
Question: {question}
 
Answer using only the context above. If the context does not contain the answer, say "I don't know."
"""
    return llm(prompt)

The prompt pattern matters more than people expect. Telling the model explicitly to ground its answer in the context, and to say "I don't know" when the context doesn't cover it, cuts hallucination rates by 40-60% compared to naive prompting. More elaborate patterns go further and add citations, confidence scores, and structured extraction.

Step 4: evaluation that reflects the real world

SQuAD uses Exact Match (EM) and token-level F1. EM is a strict match after normalization (lowercase, strip punctuation, remove articles): either the prediction matches exactly or it scores 0. F1 is computed over token overlap between prediction and reference and gives partial credit. Both under-credit paraphrases. "June 29, 2007" vs "June 29th, 2007" typically gets 0 EM, since the ordinal breaks normalization, but still earns substantial F1 from overlapping tokens.

For production QA:

• Answer accuracy (LLM-judged or human-judged, since metrics don't capture semantic equivalence).
• Citation accuracy. Does the cited passage actually support the answer? That's trivial to check automatically: string-match the generated citations against the retrieved passages.
• Refusal calibration. When the answer isn't in the retrieved passages, does the system correctly say "I don't know"? Track the false confidence rate.
• Retrieval recall. Before you evaluate the reader at all, measure whether the retriever gets the right passage into the top-k. A reader can't fix a passage that was never retrieved.

RAGAS: the 2026 production eval framework

RAGAS is purpose-built for RAG systems, and it's the shipping default in 2026. It scores four dimensions without needing gold references:

• Faithfulness. Does each claim in the answer come from the retrieved context? Measured by NLI-based entailment (natural language inference: whether the context logically supports the claim, or contradicts it, or says nothing about it). This is your primary hallucination metric.
• Answer relevance. Does the answer address the question? Measured by generating hypothetical questions from the answer and comparing them to the real one.
• Context precision. Of the retrieved chunks, what fraction were actually relevant? Low precision means noise sitting in the prompt.
• Context recall. Did the retrieved set contain everything needed to answer? Low recall means the reader never had a chance.

Reference-free scoring means you can evaluate live production traffic without curated gold answers. Layer an LLM-as-judge on top for open-ended questions where exact-match metrics fall apart.

pip install ragas, plug in your retriever and reader, and you get four scalars per query to alert on.

Here's the 2026 stack.

Extractive QA is unfashionable in 2026 because RAG with LLMs simply handles more cases. It still ships wherever literal quotation is non-negotiable: legal research, regulatory compliance, audit tools.

Save as outputs/skill-qa-architect.md:

---
name: qa-architect
description: Choose QA architecture, retrieval strategy, and evaluation plan.
version: 1.0.0
phase: 5
lesson: 13
tags: [nlp, qa, rag]
---
 
Given requirements (corpus size, question type, factuality constraint, latency budget), output:
 
1. Architecture. Extractive, RAG with extractive reader, RAG with generative reader, or closed-book LLM. One-sentence reason.
2. Retriever. None, BM25, dense (name the encoder), or hybrid.
3. Reader. SQuAD-tuned model, LLM by name, or "domain-fine-tuned DistilBERT."
4. Evaluation. EM + F1 for extractive benchmarks; answer accuracy + citation accuracy + refusal calibration for production. Name what you are measuring and how you are measuring it.
 
Refuse closed-book LLM answers for regulatory or compliance-sensitive questions. Refuse any QA system without a retrieval-recall baseline (you cannot evaluate the reader without knowing the retriever surfaced the right passage). Flag questions that require multi-hop reasoning as needing specialized multi-hop retrievers like HotpotQA-trained systems.

1. Easy. Set up the SQuAD extractive pipeline above on 10 Wikipedia passages. Hand-craft 10 questions. Measure how often the answer is correct. You should see 7-9 correct if passages and questions are clean.
2. Medium. Add a refusal classifier. When the top retrieval score is below a threshold (say 0.3 cosine), return "I don't know" instead of calling the reader. Tune the threshold on a held-out set.
3. Hard. Build a RAG pipeline over a 10,000-document corpus of your choice. Implement hybrid retrieval (BM25 + dense) with RRF fusion (see lesson 14). Measure answer accuracy with and without the hybrid step. Document which question types benefit most.

• Rajpurkar et al. (2016). SQuAD: 100,000+ Questions for Machine Comprehension of Text: the benchmark paper.
• Karpukhin et al. (2020). Dense Passage Retrieval for Open-Domain QA — DPR, the canonical dense retriever for QA.
• Lewis et al. (2020). Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks — the paper that named RAG.
• Gao et al. (2023). Retrieval-Augmented Generation for Large Language Models: A Survey: a comprehensive survey of the RAG landscape.