TAI BUI
Back to curriculum
BuildPython

Model, System, and Dataset Cards

Three documentation formats structure AI transparency. Model Cards (Mitchell et al. 2019) work like nutrition labels for models: training data, disaggregated quantitative analyses, ethical considerations, caveats. Only 0.3% of Hugging Face model cards document ethical considerations (Oreamuno et al. 2023). Datasheets for Datasets (Gebru et al. 2018, CACM) borrow from electronics datasheets and cover motivation, composition, collection process, labeling, distribution, and maintenance. Data Cards (Pushkarna et al., Google 2022) layer detail at telescopic, periscopic, and microscopic zoom so diverse readers can use one document as a boundary object. 2024-2025 brought automated generation via LLMs (CardGen, Liu et al. 2024), a correlation between model-card detail and up to 29% more downloads on HF (Liang et al. 2024), verifiable attestations (Laminator, Duddu et al. 2024), sustainability reporting for carbon and water (Jouneaux et al. July 2025), and emerging EU/ISO regulatory cards. System Cards (Sidhpurwala 2024; Meta system-level transparency; "Blueprints of Trust" arXiv:2509.20394) document the whole AI system: security capabilities, prompt-injection protection, data-exfiltration detection, and alignment with human values.

TIME: ~60 minutesPREREQUISITES: Phase 18 · 18 (safety frameworks), Phase 18 · 24 (regulatory)

Learning Objectives

  • Describe the original Mitchell et al. 2019 model card and the Gebru et al. 2018 datasheet.
  • Describe Data Cards' telescopic/periscopic/microscopic layering.
  • Describe System Cards and their end-to-end coverage.
  • State three 2024-2025 developments (automated generation, verifiable attestations, sustainability reporting).

The Problem

Regulatory frameworks (Lesson 24) and lab safety policies (Lesson 18) both demand documentation, and the formats grew to match. Model cards came first and describe a model. Datasheets moved upstream to the dataset. System cards moved outward to the whole deployment. Each one covers a different scope of transparency.

Adoption is the part that kept failing: only 0.3% of Hugging Face model cards document ethical considerations. The 2024-2025 work on automated generation and verifiable attestations goes straight at that problem.

Warm-up0/3
1. A lab's safety framework says it evaluates dangerous capabilities before release. What does an outside reviewer need to hold the lab to that claim?Warm-up
A lab's safety framework claiming it evaluates dangerous capabilities before release, with an arrow into an open question asking what the lab must hand over to an outside reviewer.
2. The EU AI Act asks providers of general-purpose models for transparency. Which request fits that duty?Warm-up
A model provider and a regulator connected by an arrow labeled transparency duty, with an open question asking which kind of request fits that duty.
3. A medicine comes with a leaflet listing its intended use, known side effects, and warnings. What is the main purpose of that leaflet?Warm-up
A medicine leaflet listing intended use, side effects, and warnings, with its main purpose left as an open question, noting it is not a claim that every test passed and not a copy-exact production recipe.
Check: The Problem0/2
1. How does the scope of a model card differ from that of a datasheet and a system card?Explain
A pipeline of three boxes, Dataset, Model, and Deployment, each with a blank 'name: ?' chip above it, leaving which document belongs to which scope open.
2. Audits of Hugging Face model cards found that almost none document ethical considerations. What does this show about the model card format?Explain
A single bar showing 0.3% of audited Hugging Face model cards with Ethical Considerations filled in, captioned with the Oreamuno et al. 2023 figure, and an open question asking what this shows about the format.

The Concept

Model Cards (Mitchell et al. 2019)

Nine sections:

  • Model details.
  • Intended use.
  • Factors (relevant demographic or environmental factors for evaluation).
  • Metrics.
  • Evaluation data.
  • Training data.
  • Quantitative analyses (disaggregated by factors).
  • Ethical considerations.
  • Caveats and recommendations.

The adoption problem is visible in the numbers. Oreamuno et al. 2023 audit of Hugging Face model cards found only 0.3% document ethical considerations.

Datasheets for Datasets (Gebru et al. 2018)

The analogy is the datasheet that ships with an electronic component. Sections:

  • Motivation (why was the dataset created).
  • Composition (what is in it).
  • Collection process (how was it assembled).
  • Labeling (if applicable).
  • Uses (intended, prohibited, risks).
  • Distribution.
  • Maintenance.

The paper appeared in CACM in 2021. The datasheet is upstream of the model card, so a model card is only as accurate as the datasheet behind it.

Data Cards (Pushkarna et al., Google 2022)

Modular layered detail. Three zoom levels:

  • Telescopic. High-level summary for non-experts.
  • Periscopic. Middle-level overview for ML practitioners.
  • Microscopic. Detailed feature-level documentation for auditors.

The boundary-object framing means different readers pull different information out of the same document.

System Cards

A system card covers the end-to-end AI system: the model, the safety stack, and the deployment context. Typical sections:

  • Security capabilities.
  • Prompt-injection protection.
  • Data-exfiltration detection.
  • Alignment with stated human values.
  • Incident response.

Sidhpurwala 2024 and Meta system-level transparency work. "Blueprints of Trust" (arXiv:2509.20394) formalizes the System Card as the deployment-layer complement to Model Cards.

2024-2025 developments

  • CardGen (Liu et al. 2024). Automated model-card generation via LLMs; reports higher objectivity than many human-authored cards on the standardized Mitchell 2019 fields.
  • Download correlation (Liang et al. 2024). Detailed model cards correlate with up to 29% higher download rates on HF. Adoption pressure is now market-driven, not only compliance-driven.
  • Laminator (Duddu et al. 2024). Verifiable attestations via hardware TEE / cryptographic signatures. This lets the model card carry a proof-of-claim, not just a claim.
  • Sustainability (Jouneaux et al. July 2025). Additions for carbon, water, and compute-energy footprint; emerging ISO standards.
  • Regulatory cards. EU AI Act (Lesson 24) GPAI Code of Practice Transparency chapter requires model cards as a compliance artifact.

Where this fits in Phase 18

Lessons 24-25 cover the regulatory and CVE layers, and this lesson is the documentation layer. Lesson 27 is training-data governance, which sits upstream of the datasheet. Lesson 28 covers the research ecosystem that produces the evaluations cards cite.

Check: The Concept0/23
1. What is a model card?Define
A real model card's eight section headers, read directly from main.py's model_card() function, laid out in a two-column grid under the title Model Card: ToyClassifier-1.0.
2. What is a datasheet for a dataset?Define
A real datasheet's seven section headers, read directly from main.py's datasheet() function, laid out in a two-column grid under the title Datasheet: ToyBinaryClassification-1.0.
3. What is a Data Card?Define
Three boxes in a row named Telescopic, Periscopic, and Microscopic, captioned that each level holds more detail than the one before it, with no mention of which readers use which level.
4. What does a System Card document?Define
A real system card's five section headers, read directly from main.py's system_card() function, laid out in a two-column grid under the title System Card: ToyClassifier Service.
5. In the Data Cards framing, what is a boundary object?Define
One document box with three arrows fanning out to three reader chips, non-expert reader, ML practitioner, and auditor, captioned that each reader stops at a different depth of the same source.
6. What is a verifiable attestation in model documentation?Define
A claim box reading evaluation passed, an arrow into a seal chip labeled cryptographic signature, and an arrow onward to a separate verifier box that checks it.
7. What does a sustainability field add to a documentation card?Define
A card gaining one more field, fanning out into three chips: carbon, water, and compute-energy.
8. Why does a model card pair a Factors section with disaggregated quantitative analyses?Explain
A single overall-accuracy bar at 85%, with Factors naming two illustrative groups, Group A and Group B, underneath and an open question asking how they compare to each other.
9. A model card reports strong accuracy, but nobody can say how the training labels were produced. Which document covers collection process and labeling, and sits upstream of the model card?Explain
A model card box reporting strong accuracy, with a note that there is no record of how training labels were made, and an open question asking which document covers labels and collection process.
10. In a Data Card (Pushkarna et al. 2022), which zoom level gives auditors feature-level detail?Explain
A small feature table with columns feature, type, and range, captioned as row-by-row feature-level detail, next to an open 'zoom level: ?' chip.
11. Why is a model card alone not enough to document a deployed assistant?Explain
A highlighted Model Card box connected by an arrow to a Deployed assistant box, with the model card's coverage listed as intended use, metrics, and caveats, next to a dashed empty box marked 'not covered by this card.'
12. Detailed model cards on Hugging Face correlate with higher download rates. What does that change about the pressure to document?Explain
Two bars, a minimal card at 100 and a highlighted detailed card at 129, captioned with the Liang et al. 2024 figure of up to 29% higher downloads.
13. A team ships a classifier with a card that reports a single overall accuracy number. Which Mitchell et al. section would expose a gap between demographic groups?Apply
A box reporting an illustrative overall accuracy of 91%, with an arrow into an open question asking which section would expose a gap between demographic groups.
14. You are writing a model card for an image classifier and must state which age groups and lighting conditions the evaluation is split by. Which section holds that?Apply
Two boxes, Age groups and Lighting conditions, both with arrows down into an open question asking which section records that evaluation split.
15. A dataset must not be used for hiring decisions, and its maintainers want that stated clearly. Which datasheet section records it?Apply
A dataset box, an arrow into a chip marked prohibited for hiring decisions, and an arrow onward into an unlabeled '? section' chip.
16. A deployment wraps an unchanged model in a prompt-injection filter and an egress monitor. Which document captures those controls?Apply
An unchanged model box next to a prompt-injection filter box and an egress monitor box, with an open question underneath asking which document captures these controls.
17. A card claims an evaluation result, but a reader has no way to check it. Which 2024 development lets the card carry proof instead of only a claim?Apply
A reported eval-result box, an arrow into a chip reading reader can't check it, and a question asking which 2024 development changes this.
18. A provider must satisfy the transparency chapter of the EU AI Act GPAI Code of Practice. Which documentation artifact does the lesson say that chapter requires?Apply
An EU AI Act GPAI Code of Practice Transparency chapter box, with an arrow into an open 'requires: ?' chip.
19. A model card is thorough, but the model fails on one subgroup because training labels were noisy and the dataset's composition was never recorded. Where is the root documentation gap?Analyze
A fully filled model card box next to a chip reading fails on one subgroup, with a note that the cause is noisy labels and unrecorded composition, and an open 'root gap: ?' chip.
20. A team has a good model card and datasheet. After launch, a prompt injection makes the assistant leak user data. Which documentation gap does this expose?Analyze
A good model card box and a good datasheet box, an arrow down to a chip reading prompt injection leads to a data leak, and an open question asking which document gap this exposes.
21. You deploy a third-party model in a support assistant that retrieves from your own ticket corpus, and an outside auditor will review it. Which documentation set fits?Design
Three boxes, a third-party model, your ticket corpus, and your assistant, with an open question underneath asking which documents an outside auditor needs.
22. A regulator wants to verify a reported evaluation result without trusting your word. Which card design fits?Design
A reported eval-result box, an arrow into a chip reading regulator asks to just trust the number, and an open question asking what design would let them check it themselves.
23. A public dataset will be read by journalists, ML engineers, and auditors. Which documentation design fits?Design
A public-dataset box fanning out to three reader chips, journalist, ML engineer, and auditor, with an open question asking what documentation design fits.

Use It

code/main.py prints a minimal model card, datasheet, and system card for a toy deployment, each following the canonical section structure. Read the three side by side to see how the scope changes from model to data to deployment.

Check: Use It0/4
1. A system card lists an egress rate limit, yet no code in the project enforces one. Why is that worse than a shorter card?Explain
The real Security Capabilities section text from main.py's system_card() function, listing the egress rate limit, next to an open question asking why an unenforced control is worse than a shorter card.
2. In main.py, the datasheet reports 1,500 examples and the model card reports 0.97 accuracy. Where do those numbers come from?Apply
A datasheet box showing 1,500 examples next to a model card box showing 0.97 accuracy, with an open question asking where these numbers come from.
3. Traffic exceeds 100 requests per minute while main.py's system card, which lists that rate limit, is being printed. What does the code do about it?Apply
The real Security Capabilities section text from main.py's system_card() function, with an arrow into an open question asking what main.py does when 150 requests per minute arrive.
4. A model card reports three disaggregated accuracy numbers but gives no group sizes, no confidence intervals, and no pointer to the evaluation data. What is the main weakness?Analyze
The real Quantitative Analysis section text from main.py's model_card() function, with a note that it has no group sizes, confidence intervals, or evaluation-data pointer, and an open question asking the main weakness.

Ship It

This lesson produces outputs/skill-card-audit.md. Hand it a model card, datasheet, or system card and it audits section coverage, numerical disaggregation, and whether verifiable attestations are present.

Check: Ship It0/1
1. You are building a tool that audits submitted cards, in the spirit of the lesson's card-audit skill. Which checks should it run?Design
A submitted-card box feeding into an audit-tool box, with three blank 'check: ?' chips underneath.
Author's Note

I'd run main.py first and read it as three templates rather than three analyses. Each of model_card(), datasheet(), and system_card() returns a fixed Markdown string, and main() just prints them. Nothing is trained or generated. The 1,500 examples in the datasheet, the 0.97 accuracy, and the demographic parity gap are text I typed in, not output from a run.

That is the first thing I'd poke at. The model card's own Ethical Considerations section says its bias metrics are placeholders, which is the kind of weak section Exercise 1 asks you to find. The system card has the same issue: it lists an egress rate limit and 100 requests per minute, but the file contains no server and nothing enforces either one. A card that describes a control the system doesn't have is worse than a short card, and it's easy to write by accident.

My bet is that the Quantitative Analysis section is where you learn the most. It holds three numbers and no group sizes, no confidence intervals, and no evaluation data pointer. When you extend it for Exercise 2, start by deciding what a reader would need to check those numbers, then write the fields.

Exercises

  1. Run code/main.py. Inspect the generated cards. Identify sections that are weak (placeholder-only) and specify what evidence would strengthen them.

  2. Extend the model card with a quantitative disaggregated analysis across two demographic groups (Lesson 20).

  3. Read Oreamuno et al. 2023 on the 0.3% adoption rate. Propose one structural change to the model card specification that would increase ethical-considerations adoption.

  4. Laminator (Duddu et al. 2024) uses TEEs for verifiable attestations. Design a model-card field that carries a cryptographic attestation of an evaluation result and describe the verifier's role.

  5. Write a System Card (System Card, not Model Card) for one of your past projects or a hypothetical deployment. Identify the highest-value section for third-party auditors.

Key Terms

TermWhat people sayWhat it actually means
Model Card"the Mitchell card"Mitchell et al. 2019 standard documentation for ML models
Datasheet"the Gebru datasheet"Gebru et al. 2018 standard documentation for datasets
Data Card"the Pushkarna card"Google 2022 modular layered data documentation
System Card"the deployment card"End-to-end AI system documentation including safety stack
Boundary object"different readers, one doc"Data Cards framing: same document serves diverse audiences
Verifiable attestation"the Laminator attestation"Cryptographic or TEE proof attached to a documentation claim
Sustainability field"carbon / water footprint"Emerging 2025 addition for environmental accounting

Further Reading

Code

Ships With