TAI BUI
Back to curriculum
CapstonePython (backend, learner model), TypeScript (web app), SQL (curriculum graph via Postgres + Neo4j)

Capstone 17 — Personal AI Tutor (Adaptive, Multimodal, with Memory)

Khanmigo (Khan Academy), Duolingo Max, Google LearnLM / Gemini for Education, Quizlet Q-Chat, and Synthesis Tutor all shipped adaptive multimodal tutoring at scale in 2026, and they share one shape. A Socratic policy that never just dumps the answer. A learner model that updates after every interaction (Bayesian knowledge tracing style). Voice, text, and photo-math input, curriculum graph retrieval, spaced-repetition scheduling, and hard safety filters for age-appropriate content. The capstone is to ship a subject-specific tutor (K-12 algebra or intro Python), run a two-week efficacy study with 10 learners, and pass a content-safety audit.

TIME: 30 hoursPREREQUISITES: Phase 5 (NLP), Phase 6 (speech), Phase 11 (LLM engineering), Phase 12 (multimodal), Phase 14 (agents), Phase 17 (infrastructure), Phase 18 (safety)

Learning Objectives

  • Design the four components of an adaptive tutor (Socratic tutor policy, learner model, curriculum graph, memory) and how a single learner turn flows through them.
  • Implement a Bayesian knowledge tracing update from guess, slip, and learn-rate priors, and use the resulting mastery estimate to choose the next scaffold and the next concept.
  • Add text, voice (LiveKit + Whisper), and photo-math (dots.ocr or PaliGemma 2) input behind one tutor policy, with Llama Guard 4, an age-appropriate filter, learner-ID-scoped memory, and a COPPA-aware retention policy.
  • Measure efficacy with a two-week, 10-learner pre-test and post-test against a non-adaptive baseline cohort, and report the learning gain delta.

Problem

A personal AI tutor is a system that teaches one learner at a time. It accepts typed, spoken, or photographed input, asks leading questions before it explains, and keeps a running estimate of what that learner has mastered. Adaptive tutoring used to be an ed-tech research niche. By 2026 it is a consumer product. Khanmigo is deployed across most US school districts, Duolingo Max hit tens of millions of MAUs, and Google's LearnLM / Gemini for Education powers tutoring in Google Classroom. Quizlet Q-Chat sits alongside flashcards, and Synthesis Tutor went viral as a tutor for curious kids. They all share the same elements: multimodal input (type, speak, photograph equations), Socratic pedagogy (ask first, explain later), a learner model that updates after each interaction, and strict age-appropriate safety.

You will build one for a specific cohort, and the bar is a real efficacy study: pre-test and post-test scores over two weeks with 10 learners. The voice loop has to feel natural (it reuses the capstone 03 sub-stack). Memory has to respect privacy. The safety filter has to pass a COPPA-aware red-team for K-12.

Warm-up0/4
1. A voice assistant has to turn a learner's spoken answer into text before anything else can process it. Which component does that job?Warm-up
2. Why do production LLM apps run a safety classifier on the model's output and not only on the user's input?Warm-up
3. A system stores records for many users. Which approach best stops one user from reading another's records?Warm-up
4. In a graph-based agent workflow such as LangGraph, what do the nodes and edges represent?Warm-up

Concept

A Socratic tutor policy is a loop that asks a leading question instead of handing over the answer. It is one of four components. The tutor policy asks a leading question when the learner asks for the answer, moves to the next concept when they get it right, and offers a scaffolded hint when they are stuck. The learner model is Bayesian knowledge tracing (or a simple variant), which updates a mastery probability for each curriculum node after every interaction. The curriculum graph is a Neo4j graph of concepts joined by prerequisite edges, and the policy walks it to pick the next concept. Memory is an episodic plus semantic store (agentmemory-style) holding past interactions, mistakes, and preferences.

The UX is multimodal. Learners type answers, speak them through LiveKit + Whisper (reuse capstone 03), or photograph a math problem and let dots.ocr or PaliGemma 2 read it. Voice output comes from Cartesia Sonic-2. Safety pairs Llama Guard 4 with an age-appropriate filter that blocks adult content, violence, and self-harm, and memory follows a COPPA-aware retention policy.

The efficacy study is the deliverable. Run 10 learners through a pre-test, two weeks of tutoring, and a post-test, then report the learning gain delta and its confidence interval. Compare against a non-adaptive baseline: the same content delivered linearly, without the tutor policy.

Check: Concept0/14
1. What is a Socratic policy in a tutor?Define
2. What does Bayesian knowledge tracing give the tutor for each learner?Define
3. What is the curriculum graph?Define
4. What is episodic memory in this tutor?Define
5. What is semantic memory in this tutor?Define
6. What is COPPA, and why does it shape this tutor's memory?Define
7. What does the efficacy study report as its result?Define
8. Which input paths does the tutor accept from the learner?Define
9. Why does the tutor ask a leading question when the learner asks for the answer?Explain
10. What does the tutor policy use the prerequisite edges for?Explain
11. Transcripts show the tutor giving the final answer after one request. Which component is failing?Analyze
12. The tutor teaches isolating variables to a learner who lacks addition-subtraction-equality. Which part of the design should have prevented this?Analyze
13. The adaptive cohort improves more than the baseline cohort. What must hold for that to show the tutor policy helped?Analyze
14. The adaptive cohort gains slightly more than the baseline, and the confidence interval on the difference includes zero. What does that mean?Analyze

Architecture

learner device
  |
  +-- text         -> web app
  +-- voice        -> LiveKit Agents (ASR + TTS)
  +-- photo math   -> dots.ocr / PaliGemma 2
       |
       v
  tutor policy (LangGraph)
       - Socratic decision head
       - next-concept chooser (curriculum graph walk)
       - hint scaffolder
       - mastery update
       |
       v
  learner model (BKT / item-response theory)
       - per-concept mastery probability
       - spaced-repetition scheduler (SM-2 or FSRS)
       |
       v
  memory (agentmemory-style)
       - episodic: every interaction
       - semantic: learned mistakes, preferences
       - retention policy: COPPA / GDPR aware
       |
       v
  curriculum graph (Neo4j)
       - prerequisite edges
       - OER content attached
       |
       v
  safety:
    Llama Guard 4 + age-appropriate filter
    memory access guarded by learner ID scope
Check: Architecture0/1
1. Learners who mastered a concept two weeks ago now miss questions on it. Which existing piece would you extend to handle this?Design

Stack

  • Subject choice: K-12 algebra or intro Python (pick one for depth)
  • Tutor policy: LangGraph over Claude Sonnet 4.7 (with prompt caching)
  • Learner model: Bayesian knowledge tracing (classic) or FSRS for spacing
  • Curriculum graph: Neo4j of concepts + prerequisite edges + OER content
  • Memory: agentmemory-style persistent vector + episodic + semantic store
  • Voice: LiveKit Agents 1.0 + Cartesia Sonic-2 (reuse capstone 03 sub-stack)
  • Photo math: dots.ocr or PaliGemma 2 for equation recognition
  • Safety: Llama Guard 4 + custom age-appropriate filter
  • Eval: Bloom-level question generation, pre/post test harness, efficacy study tooling
Check: Stack0/1
1. What is FSRS used for in the tutor?Define

Build It

  1. Curriculum graph. Build a Neo4j graph of 50-150 concept nodes (K-12 algebra from "number line" to "quadratic formula", for example) with prerequisite edges. Attach OER content to each node (Open Textbook, OpenStax).

  2. Learner model. Initialize Bayesian knowledge tracing with priors for guess, slip, and learn rate. Update per-concept mastery after each interaction and persist it per learner.

  3. Tutor policy. Build a LangGraph with four nodes: read_signal (was the learner's answer correct, partial, or stuck?), select_concept (walk the curriculum graph and pick the highest-priority concept), scaffold (the Socratic prompt), and update_mastery.

  4. Memory. Write every interaction to an episodic store, and promote mistakes and preferences to semantic memory. The COPPA-aware retention policy auto-deletes after 1 year, and the stored data stays accessible to parents.

  5. Voice path. Attach a LiveKit Agents worker to the tutor policy, with Whisper-v3-turbo for ASR and Cartesia Sonic-2 for TTS. Barge-in is supported (reuse the capstone 03 mechanics).

  6. Photo-math path. Accept an uploaded or captured image, run dots.ocr or PaliGemma 2 to recognize the equation, and feed the result to the tutor as structured input.

  7. Safety. Every model output passes Llama Guard 4 and an age-appropriate filter that blocks self-harm, adult content, and violence. Memory access is scoped by learner ID, and parents get a surface for deletion.

  8. Efficacy study. Give 10 learners a standardized 30-question pre-test, two weeks of tutor interaction (3 sessions/week), and a post-test. Compare against a non-adaptive baseline cohort of 10 learners on the same content.

  9. Weekly progress reports. Auto-generate a PDF per learner summarizing topics explored, mastery trajectories, and recommended next steps.

Check: Build It0/18
1. Which four nodes make up the tutor policy graph?Define
2. Which three priors initialize the Bayesian knowledge tracing update?Define
3. How is the efficacy study structured?Define
4. Which two decisions does the mastery estimate drive?Explain
5. Why does the photo path give the tutor the recognized equation as structured input?Explain
6. Which policy matches the memory retention rule for K-12 learners in Build It step 4?Explain
7. What gates every tutor output before it reaches the learner?Explain
8. On the voice path, which tool handles which direction?Explain
9. The adaptive cohort has 10 learners at 3 sessions per week for two weeks. How many tutoring sessions does that cohort get in total?Apply
10. A learner makes the same sign error in three different sessions. Where does it end up as a lasting pattern?Apply
11. On the 30-question test, a learner scores 12 before tutoring and 21 after. What is that learner's pre/post delta?Apply
12. A learner answers correctly all week, but every concept's mastery stays at its starting value. What is the most likely cause?Analyze
13. During a stress test, learner A's session surfaces a mistake that belongs to learner B. Which control is missing?Analyze
14. The tutor writes a violent story for a young learner, even though the learner's own message was harmless. What was missing?Analyze
15. A team stores logs of young learners for years, and parents cannot view or delete them. Which requirement was skipped?Analyze
16. You retarget the tutor from K-12 algebra to intro Python. Which piece has to be rebuilt because it encodes the subject itself?Design
17. You have limited time and the Socratic loop, voice path, and photo path are all unbuilt. Which do you build first?Design
18. You are about to start the 10-learner study. Which decision should be settled before the first session?Design

Use It

learner: "I don't understand why 3x + 6 = 12 means x = 2"
[signal]   stuck
[concept]  'isolating variables' (prerequisite: addition-subtraction-equality)
[scaffold] "what number would you subtract from both sides to start?"
learner: "6"
[signal]   correct
[mastery]  addition-subtraction-equality: 0.62 -> 0.92
[concept]  continue 'isolating variables'
[scaffold] "great. now what is 3x / 3 equal to?"
Check: Use It0/4
1. In the transcript, the learner says they do not understand why 3x + 6 = 12 means x = 2. What does the tutor do?Apply
2. In the Use It transcript, the learner answers "6" correctly and mastery of addition-subtraction-equality moves from 0.62 to 0.92. What is the BKT update doing?Apply
3. After the correct answer "6", what does the trace show the tutor doing next?Apply
4. In the trace, whose mastery estimate moves from 0.62 to 0.92 after the correct answer?Apply

Ship It

outputs/skill-ai-tutor.md is the deliverable: a subject-specific adaptive tutor with multimodal input, a learner model, memory, safety, and measured efficacy, scored against the rubric below.

WeightCriterionHow it is measured
25Learning gain deltaPre/post-test delta in a 10-learner two-week study
20Socratic fidelityRubric score on transcript samples
20Multimodal UXVoice + photo + text coherence end to end
20Safety + privacy postureLlama Guard 4 pass rate + COPPA-aware retention
15Curriculum breadth and graph qualityConcept coverage + prerequisite graph consistency
100
Check: Ship It0/2
1. In the ship-it rubric, how is Socratic fidelity measured?Define
2. A tutor earns full marks on every rubric criterion except multimodal UX, where it earns nothing. What is its score out of 100?Apply
Author's Note

I'd get the text-only Socratic loop and the mastery update running before touching voice or photo. The voice path reuses capstone 03, and the photo path feeds the recognized equation to the tutor as structured input, so the policy is the part both of them go through. The Socratic fidelity row is scored on transcript samples, and I'd read real transcripts before adding another input path.

I'd also recruit both cohorts early. The Concept section calls the efficacy study the deliverable, it takes two weeks of 3 sessions/week, and the learning gain delta carries the largest weight in the rubric at 25 points. Decide the 1-year retention and parent-access policy from step 4 before the first learner session, so no learner data exists without it.

Exercises

  1. Run the efficacy study with and without the adaptive learner model (random concept order) and report the delta. Expect adaptive to win — the size of the win is the interesting number.

  2. Add a multimodal probe: deliver the same concept question as text, voice, and photo. Measure whether learners converge faster with the modality they prefer.

  3. Build a parent dashboard showing topics practiced, mastery trajectories, upcoming concepts, and safety events (any guardrail hits). Keep it COPPA-aligned.

  4. Add a language-switch mode where the tutor accepts Spanish input and teaches in Spanish. Measure X-Guard coverage.

  5. Stress the memory privacy. Verify that learner A cannot see learner B's data, even through a voice-clip re-ingest attack, and log and alert on the attempted access.

Key Terms

TermWhat people sayWhat it actually means
Socratic policy"Ask, do not dump"Tutor asks a leading question rather than giving the answer
Bayesian knowledge tracing"BKT"Classic learner-model equations for mastery probability per concept
FSRS"Free Spaced Repetition Scheduler"2024 spaced-repetition scheduler, better than SM-2
Curriculum graph"Concept DAG"Neo4j of concepts with prerequisite edges
Episodic memory"Per-interaction log"Every interaction stored for later retrieval
Semantic memory"Learned pattern store"Compacted mistakes and preferences promoted from episodic
COPPA"Kids privacy law"US law restricting data collection from children under 13

Further Reading

Code

Ships With