Text Summarization

A 2,000-word news article lands in your feed. You need 120 words that capture it, and you have two options: pick the three sentences that carry the most weight (extractive), or rewrite the whole thing in new words (abstractive). Both get called summarization. They are not the same problem.

Extractive summarization is a ranking problem. Score every sentence, return the top-k, done. Because the output is lifted verbatim, it's always grammatical. The risk is that the content you need is scattered across sentences that never made the cut.

Abstractive summarization is a generation problem: a transformer reads the input and produces new text conditioned on it. The output reads fluently and compresses well, but it can hallucinate facts nobody wrote. Confident fabrication is the risk you're actually managing.

This lesson builds both, and hands you the specific failure mode each one owns.

Extractive. Treat the article as a graph: nodes are sentences, edges are similarity scores. Run PageRank (or something close to it) over that graph so a sentence's score reflects how connected it is to everything else, then take the highest scorers as your summary. The canonical implementation is TextRank (Mihalcea and Tarau, 2004).

Abstractive. Fine-tune a transformer encoder-decoder (BART, T5, Pegasus) on document-summary pairs. At inference, the model reads the document and generates the summary token by token, attending back to the source through cross-attention. Pegasus stands out here: its gap-sentence pretraining objective (mask out whole sentences during pretraining, then train the model to regenerate them) makes it strong at summarization with little to no fine-tuning.

Evaluation runs on ROUGE (Recall-Oriented Understudy for Gisting Evaluation). ROUGE-1 and ROUGE-2 score unigram and bigram overlap; ROUGE-L scores the longest common subsequence. Higher is better, though the scale is compressed: 40 ROUGE-L reads as "good," 50 as "exceptional." Report all three, the way every paper does, using the rouge-score package.

Step 1: TextRank (extractive)

import math
import re
from collections import Counter
 
 
def sentence_split(text):
    return re.split(r"(?<=[.!?])\s+", text.strip())
 
 
def similarity(s1, s2):
    w1 = Counter(s1.lower().split())
    w2 = Counter(s2.lower().split())
    intersection = sum((w1 & w2).values())
    denom = math.log(len(w1) + 1) + math.log(len(w2) + 1)
    if denom == 0:
        return 0.0
    return intersection / denom
 
 
def textrank(text, top_k=3, damping=0.85, iterations=50, epsilon=1e-4):
    sentences = sentence_split(text)
    n = len(sentences)
    if n <= top_k:
        return sentences
 
    sim = [[0.0] * n for _ in range(n)]
    for i in range(n):
        for j in range(n):
            if i != j:
                sim[i][j] = similarity(sentences[i], sentences[j])
 
    scores = [1.0] * n
    for _ in range(iterations):
        new_scores = [1 - damping] * n
        for i in range(n):
            total_out = sum(sim[i]) or 1e-9
            for j in range(n):
                if sim[i][j] > 0:
                    new_scores[j] += damping * sim[i][j] / total_out * scores[i]
        if max(abs(s - ns) for s, ns in zip(scores, new_scores)) < epsilon:
            scores = new_scores
            break
        scores = new_scores
 
    ranked = sorted(range(n), key=lambda k: scores[k], reverse=True)[:top_k]
    ranked.sort()
    return [sentences[i] for i in ranked]

Two things worth flagging. The similarity function uses log-normalized word overlap, the original TextRank variant; cosine similarity over TF-IDF vectors works just as well. The damping factor of 0.85 and the iteration count are both standard PageRank defaults, carried over unchanged.

Step 2: abstractive with BART

from transformers import pipeline
 
summarizer = pipeline("summarization", model="facebook/bart-large-cnn")
 
article = """(long news article text)"""
 
summary = summarizer(article, max_length=120, min_length=60, do_sample=False)
print(summary[0]["summary_text"])

BART-large-CNN is fine-tuned on the CNN/DailyMail corpus, so it produces news-style summaries straight out of the box. Outside news (scientific papers, dialog, legal text), reach for the matching Pegasus checkpoint, or fine-tune on your own data instead.

Step 3: ROUGE evaluation

from rouge_score import rouge_scorer
 
scorer = rouge_scorer.RougeScorer(["rouge1", "rouge2", "rougeL"], use_stemmer=True)
scores = scorer.score(reference_summary, generated_summary)
print({k: round(v.fmeasure, 3) for k, v in scores.items()})

Always turn stemming on. Skip it and "running" and "run" count as different words, so ROUGE quietly undercounts overlap that's actually there.

Beyond ROUGE (2026 summarization eval)

ROUGE has carried summarization evaluation for twenty years, and by 2026 it's not enough on its own. A large-scale meta-analysis of NLG papers backs that up:

• BERTScore (contextual embedding similarity) gained ground through 2023 and now shows up alongside ROUGE in most summarization papers.
• BARTScore treats evaluation as generation: it scores a summary by how likely a pretrained BART model assigns to it, given the source.
• MoverScore (Earth Mover's Distance over contextual embeddings) took the top spot in 2025 summarization benchmarks, because it captures semantic overlap that ROUGE misses.

Those three all score semantic overlap: how close the summary reads to a trusted reference. A second cluster targets faithfulness directly, asking whether the summary said something true rather than something that merely reads well.

• FactCC and QA-based faithfulness were the standard 2021 to 2023, largely replaced now by G-Eval: a GPT-4 prompt chain that scores coherence, consistency, fluency, and relevance with chain-of-thought reasoning.
• G-Eval and similar LLM-judge approaches land within about 80% of human judgment when the rubric is well designed.

For production, report ROUGE-L for legacy comparison, BERTScore for semantic overlap, and G-Eval for coherence and factuality, calibrated against 50 to 100 human-labeled summaries.

Step 4: the factuality problem

Abstractive summaries are prone to hallucination. Extractive summaries carry far less of that risk, since the output is lifted verbatim from the source, though they can still mislead if a sentence gets pulled out of context, goes stale, or gets quoted out of order. That gap is the single biggest reason production systems still lean extractive for compliance-adjacent content.

Name the hallucination types so you know what to look for:

• Entity swap. Source says "John Smith." Summary says "John Brown."
• Number drift. Source says "25,000." Summary says "25 million."
• Polarity flip. Source says "rejected the offer." Summary says "accepted the offer."
• Fact invention. Source does not mention the CEO. Summary says the CEO approved.

A few evaluation approaches actually catch these:

• FactCC. A binary classifier trained on entailment between source sentence and summary sentence, predicting factual or not.
• QA-based factuality. Ask a QA model questions whose answers live in the source. If the summary supports different answers, flag it.
• Entity-level F1. Compare named entities in source against summary. Entities that show up only in the summary are suspect by default.

For anything user-facing where factuality actually matters (news, medical, legal, financial), extractive stays the safer default. If you're shipping abstractive, put a factuality check in the loop, not an afterthought.

Here's the 2026 stack, by use case:

When compute isn't the constraint, LLMs with long context often beat specialized models outright. What you give up is cost and reproducibility: specialized models are cheaper per call and hold their output steadier across runs.

Save as outputs/skill-summary-picker.md:

---
name: summary-picker
description: Pick extractive or abstractive, named library, factuality check.
version: 1.0.0
phase: 5
lesson: 12
tags: [nlp, summarization]
---
 
Given a task (document type, compliance requirement, length, compute budget), output:
 
1. Approach. Extractive or abstractive. Explain in one sentence why.
2. Starting model / library. Name it. `sumy.TextRankSummarizer`, `facebook/bart-large-cnn`, `google/pegasus-pubmed`, or an LLM prompt.
3. Evaluation plan. ROUGE-1, ROUGE-2, ROUGE-L (use rouge-score with stemming). Plus factuality check if abstractive.
4. One failure mode to probe. Entity swap is the most common in abstractive news summarization; flag samples where source entities do not appear in summary.
 
Refuse abstractive summarization for medical, legal, financial, or regulated content without a factuality gate. Flag input over the model's context window as needing chunked map-reduce summarization (not just truncation).

1. Easy. Run TextRank on 5 news articles. Compare the top-3 sentences to a reference summary. Measure ROUGE-L. You should see 30-45 ROUGE-L on CNN/DailyMail-style articles.
2. Medium. Implement entity-level factuality: extract named entities from source and summary (spaCy), compute recall of source entities in summary and precision of summary entities against source. High precision and low recall mean safe but terse; low precision means hallucinated entities.
3. Hard. Compare BART-large-CNN against an LLM (Claude or GPT-4) on 50 CNN/DailyMail articles. Report ROUGE-L, factuality (by entity F1), and cost per summary. Document where each wins.

• Mihalcea and Tarau (2004). TextRank: Bringing Order into Texts: the extractive canonical paper.
• Lewis et al. (2019). BART: Denoising Sequence-to-Sequence Pre-training: the BART paper.
• Zhang et al. (2019). PEGASUS: Pre-training with Extracted Gap-sentences: Pegasus and the gap-sentence objective.
• Lin (2004). ROUGE: A Package for Automatic Evaluation of Summaries: the ROUGE paper.
• Maynez et al. (2020). On Faithfulness and Factuality in Abstractive Summarization: the factuality landscape paper.