Few-Shot, Chain-of-Thought, Tree-of-Thought

You build a math tutoring app. Your prompt says: "Solve this word problem." GPT-5 gets 94% of GSM8K, the standard grade-school math benchmark. That looks like a ceiling, but chain-of-thought still adds 3-4 points on the same model (97% zero-shot CoT, 98% few-shot CoT, in the table below).

The older numbers show the effect more sharply. On GPT-4o, adding five words ("Let's think step by step") moves GSM8K from 78% to 91%, and a few worked examples bring it to 95%. Same model, same temperature, a few extra tokens. The only difference is that you gave the model scratch paper.

This is not a hack. Humans do not solve multi-step problems in one mental leap, and neither do transformers. When you force a model to generate intermediate tokens, those tokens become context for the next token, so each reasoning step feeds the next. The model computes its way to the answer.

"Think step by step" is where this starts, not where it ends. What if you sampled five reasoning paths and took a majority vote? What if you let the model explore a tree of possibilities, evaluating and pruning branches? What if you interleaved reasoning with tool use? These are published techniques with measured improvements, and you will build all of them in this lesson.

Zero-Shot vs Few-Shot: When Examples Beat Instructions

Zero-shot prompting gives the model a task and nothing else. Few-shot prompting gives it examples first.

Wei et al. (2022) measured this across 8 benchmarks. For simple tasks like sentiment classification, zero-shot and few-shot performed within 2% of each other. For complex tasks like multi-step arithmetic and symbolic reasoning, few-shot improved accuracy by 10-25%.

The intuition: examples are compressed instructions. Instead of describing the output format, you show it. Instead of explaining the reasoning process, you demonstrate it. The model pattern-matches on the examples more reliably than it interprets abstract instructions.

When few-shot wins: format-sensitive tasks, classification, structured extraction, domain-specific jargon, any task where the model needs to match a specific pattern.

When zero-shot wins: simple factual questions, creative tasks where examples constrain creativity, tasks where finding good examples is harder than writing good instructions.

Example Selection: Similar Beats Random

Not all examples are equal. Choosing examples similar to the target input outperforms random selection by 5-15% on classification tasks (Liu et al., 2022). Three principles:

1. Semantic similarity: pick examples closest to the input in embedding space
2. Label diversity: cover all output categories in your examples
3. Difficulty matching: match the complexity level of the target problem

The best number of examples for most tasks is 3-5. With fewer than 3, the model may not get enough signal to extract the pattern. Past 5, returns diminish and you spend context window tokens for little gain. For classification with many labels, use one example per label.

Chain-of-Thought: Giving Models Scratch Paper

Chain-of-Thought (CoT) prompting was introduced by Wei et al. (2022) at Google Brain. The idea is simple: instead of asking the model for just the answer, ask it to show its reasoning steps first.

Why does this work mechanically? Each token a transformer generates becomes context for the next token. Without CoT, the model must compress all reasoning into the hidden state of a single forward pass. With CoT, the model externalizes intermediate computations as tokens. Each reasoning token extends the effective computation depth.

GSM8K benchmarks (grade-school math, 8.5K problems):

Note on reasoning models. Models like OpenAI's o-series (o3, o4-mini) and DeepSeek-R1 run chain-of-thought internally before emitting their answer. Adding "Let's think step by step" to a reasoning model is redundant and sometimes counterproductive, since they have already done it.

Two flavors of CoT:

Zero-shot CoT: append "Let's think step by step" to the prompt. No examples needed. Kojima et al. (2022) showed this single sentence improves accuracy across arithmetic, commonsense, and symbolic reasoning tasks.

Few-shot CoT: provide examples that include reasoning steps. More effective than zero-shot CoT because the model sees the exact reasoning format you expect.

When CoT hurts: simple factual recall ("What is the capital of France?"), single-step classification, tasks where speed matters more than accuracy. CoT adds 50-200 tokens of reasoning overhead per query. For high-throughput, low-complexity tasks, that is wasted cost.

Self-Consistency: Sample Many, Vote Once

Wang et al. (2023) introduced self-consistency. The insight: a single CoT path might contain reasoning errors. But if you sample N independent reasoning paths (using temperature > 0) and take the majority vote on the final answer, errors tend to cancel out.

Self-consistency improved GSM8K accuracy from 56.5% (single CoT) to 74.4% with N=40 on the original PaLM 540B experiments. On GPT-5 the improvement is small (97% to 98%) because base accuracy is already saturated. The technique shines most on models with 60-85% base CoT accuracy, the sweet spot where single-path errors are frequent but not systematic. For reasoning models (o-series, R1) self-consistency is subsumed by the built-in internal sampling.

The tradeoff: N samples means Nx the API cost and latency. In practice, N=5 often captures most of the benefit. N=3 is the minimum for a meaningful vote. N > 10 has diminishing returns for most tasks.

Tree-of-Thought: Branching Exploration

Yao et al. (2023) introduced Tree-of-Thought (ToT). Where CoT follows one linear reasoning path, ToT explores multiple branches and evaluates which are most promising before continuing.

ToT has three components:

1. Thought generation: produce multiple candidate next-steps
2. State evaluation: score each candidate (can use the LLM itself as evaluator)
3. Search algorithm: BFS or DFS through the tree, pruning low-scoring branches

On the Game of 24 task (combine 4 numbers using arithmetic to make 24), GPT-4 with standard prompting solves 7.3% of problems. With CoT, 4.0% (CoT actually hurts here because the search space is wide). With ToT, 74%.

ToT is expensive. Each node in the tree requires an LLM call. A tree with branching factor 3 and depth 3 requires up to 39 LLM calls. Use it only for problems where the search space is large but evaluatable: planning, puzzle solving, creative problem-solving with constraints.

ReAct: Thinking + Doing

Yao et al. (2022) combined reasoning traces with actions. The model alternates between thinking (generating reasoning) and acting (calling tools, searching, computing).

ReAct outperforms pure CoT on knowledge-intensive tasks because it can ground its reasoning in real data. On HotpotQA (multi-hop question answering), ReAct with GPT-4 achieves 35.1% exact match vs 29.4% for CoT alone. The real power is that reasoning errors get corrected by observations, so the model can update its plan mid-execution.

ReAct is the foundation of modern AI agents. Agent frameworks like LangChain, CrewAI, and AutoGen implement a variant of the Thought-Action-Observation loop. You will build full agents in Phase 14. This lesson covers the prompting pattern.

Structured Prompting: XML Tags, Delimiters, Headers

As prompts get complex, structure prevents the model from confusing sections. Three approaches:

XML tags (works best with Claude, solid everywhere):

<context>
You are reviewing a pull request.
The codebase uses TypeScript and React.
</context>

<task>
Review the following diff for bugs, security issues, and style violations.
</task>

<diff>
{diff_content}
</diff>

<output_format>
List each issue with: file, line, severity (critical/warning/info), description.
</output_format>

Markdown headers (universal):

Senior security engineer at a fintech company.

Analyze this API endpoint for vulnerabilities.

{api_code}

• Focus on OWASP Top 10
• Rate each finding: critical, high, medium, low
• Include remediation steps

**Delimiters** (minimal but effective):

---INPUT--- {user_text} ---END INPUT---

---INSTRUCTIONS--- Summarize the above in 3 bullet points. ---END INSTRUCTIONS---

### Prompt Chaining: Sequential Decomposition

Some tasks are too complex for a single prompt. Prompt chaining breaks them into steps, where the output of one prompt becomes the input of the next.

![Mermaid diagram](/curriculum-assets/11-llm-engineering/02-few-shot-cot/mermaid-diagram-216.svg)

Chaining beats single-prompt for three reasons:

1. **Each step is simpler**: the model handles one focused task instead of juggling everything
2. **Intermediate outputs are inspectable**: you can validate and correct between steps
3. **Different steps can use different models**: use a cheap model for extraction, an expensive one for reasoning

### Performance Comparison

| Technique | Best For | GSM8K Accuracy (GPT-5) | API Calls | Token Overhead | Complexity |
|-----------|----------|------------------------|-----------|----------------|------------|
| Zero-Shot | Simple tasks | 94% | 1 | None | Trivial |
| Few-Shot | Format matching | 96% | 1 | 200-500 tokens | Low |
| Zero-Shot CoT | Quick reasoning boost | 97% | 1 | 50-200 tokens | Trivial |
| Few-Shot CoT | Maximum single-call accuracy | 98% | 1 | 300-600 tokens | Low |
| Self-Consistency (N=5) | High-stakes reasoning | 98.5% | 5 | 5x token cost | Medium |
| Reasoning model (o4-mini) | Drop-in CoT replacement | 97% | 1 | hidden (2-10x internal) | Trivial |
| Tree-of-Thought | Search/planning problems | N/A (74% on Game of 24) | 10-40+ | 10-40x token cost | High |
| ReAct | Knowledge-grounded reasoning | N/A (35.1% on HotpotQA) | 3-10+ | Variable | High |
| Prompt Chaining | Complex multi-step tasks | 96% (pipeline) | 2-5 | 2-5x token cost | Medium |

The right technique depends on three factors: accuracy requirement, latency budget, and cost tolerance. For most production systems, few-shot CoT with a 3-sample self-consistency fallback covers 90% of use cases.

We will build a math problem solver that combines few-shot prompting, chain-of-thought reasoning, and self-consistency voting into a single pipeline. Then we will add tree-of-thought for hard problems.

The full implementation is in code/advanced_prompting.py. Here are the key components.

Step 1: Few-Shot Example Store

The first component manages few-shot examples and selects the most relevant ones for a given problem.

GSM8K_EXAMPLES = [
    {
        "question": "Janet's ducks lay 16 eggs per day. She eats three for breakfast every morning and bakes muffins for her friends every day with four. She sells every remaining egg at the farmers' market for $2. How much does she make every day at the farmers' market?",
        "reasoning": "Janet's ducks lay 16 eggs per day. She eats 3 and bakes with 4, using 3 + 4 = 7 eggs. So she has 16 - 7 = 9 eggs left. She sells each for $2, so she makes 9 * 2 = $18 per day.",
        "answer": "18"
    },
    ...
]

Each example has three parts: the question, the reasoning chain, and the final answer. The reasoning chain is what transforms a regular few-shot example into a CoT few-shot example.

Step 2: Chain-of-Thought Prompt Builder

The prompt builder assembles a system message, few-shot examples with reasoning chains, and the target question into a single prompt.

def build_cot_prompt(question, examples, num_examples=3):
    system = (
        "You are a precise math problem solver. "
        "For each problem, show your step-by-step reasoning clearly. "
        "After your reasoning, state your final answer on the last line "
        "in exactly this format: 'The answer is [number]'."
    )
 
    example_text = ""
    for ex in examples[:num_examples]:
        example_text += f"Q: {ex['question']}\n"
        example_text += f"A: {ex['reasoning']} The answer is {ex['answer']}.\n\n"
 
    user = f"{example_text}Q: {question}\nA:"
    return system, user

The format constraint ("The answer is [number]") matters. Without it, pulling out answers to compare across samples gets unreliable.

Step 3: Self-Consistency Voting

Sample N reasoning paths and take the majority answer.

def self_consistency_solve(question, examples, client, model, n_samples=5):
    system, user = build_cot_prompt(question, examples)
 
    answers = []
    reasonings = []
    for _ in range(n_samples):
        text = call_llm(client, model, system, user, temperature=0.7)
        reasonings.append(text)
        answer = extract_answer(text)
        if answer is not None:
            answers.append(answer)
 
    if not answers:
        return None, 0.0, reasonings, Counter()
 
    vote_counts = Counter(answers)
    best_answer = vote_counts.most_common(1)[0][0]
    confidence = vote_counts[best_answer] / len(answers)
 
    return best_answer, confidence, reasonings, vote_counts

Temperature 0.7 is important. At temperature 0.0, all N samples would come out near-identical, defeating the purpose. You need enough randomness for diverse reasoning paths but not so much that the model produces gibberish.

Step 4: Tree-of-Thought Solver

For problems where linear reasoning fails, ToT explores multiple approaches and evaluates which direction is most promising.

def tree_of_thought_solve(question, client, model, breadth=3, depth=3):
    thoughts = generate_initial_thoughts(question, client, model, breadth)
    scored = [(t, evaluate_thought(t, question, client, model)) for t in thoughts]
    scored.sort(key=lambda x: x[1], reverse=True)
 
    for current_depth in range(1, depth):
        next_thoughts = []
        top_k = min(2, len(scored))
        for thought, score in scored[:top_k]:
            extensions = extend_thought(thought, question, client, model, breadth)
            for ext in extensions:
                ext_score = evaluate_thought(ext, question, client, model)
                next_thoughts.append((ext, ext_score))
        if next_thoughts:
            scored = sorted(next_thoughts, key=lambda x: x[1], reverse=True)
 
    best_thought = scored[0][0] if scored else ""
    return extract_answer(best_thought), best_thought

The evaluator is itself an LLM call. You ask the model: "On a scale of 0.0 to 1.0, how promising is this reasoning path for solving the problem?" This is the key insight of ToT: the model evaluates its own partial solutions.

Step 5: Full Pipeline

The pipeline combines all techniques with an escalation strategy.

def solve_with_escalation(question, examples, client, model):
    system, user = build_cot_prompt(question, examples)
    single_response = call_llm(client, model, system, user, temperature=0.0)
    single_answer = extract_answer(single_response)
 
    sc_answer, confidence, _, _ = self_consistency_solve(
        question, examples, client, model, n_samples=5
    )
 
    if confidence >= 0.8:
        return sc_answer, "self_consistency", confidence
 
    tot_answer, _ = tree_of_thought_solve(question, client, model)
    return tot_answer, "tree_of_thought", None

The escalation logic: run the 5-sample vote, and if self-consistency confidence is below 0.8 (fewer than 4 of 5 samples agree), escalate to ToT. Easy problems stop at the vote, and hard ones get more compute.

With LangChain

LangChain provides built-in support for prompt templates and output parsing that simplify few-shot and CoT patterns:

from langchain_core.prompts import FewShotPromptTemplate, PromptTemplate
from langchain_openai import ChatOpenAI
 
example_prompt = PromptTemplate(
    input_variables=["question", "reasoning", "answer"],
    template="Q: {question}\nA: {reasoning} The answer is {answer}."
)
 
few_shot_prompt = FewShotPromptTemplate(
    examples=examples,
    example_prompt=example_prompt,
    suffix="Q: {input}\nA: Let's think step by step.",
    input_variables=["input"]
)
 
llm = ChatOpenAI(model="gpt-4o", temperature=0.7)
chain = few_shot_prompt | llm
result = chain.invoke({"input": "If a train travels 120 km in 2 hours..."})

LangChain also has ExampleSelector classes for semantic similarity selection:

from langchain_core.example_selectors import SemanticSimilarityExampleSelector
from langchain_openai import OpenAIEmbeddings
 
selector = SemanticSimilarityExampleSelector.from_examples(
    examples,
    OpenAIEmbeddings(),
    k=3
)

With DSPy

DSPy treats prompting strategies as optimizable modules. Instead of handcrafting CoT prompts, you define a signature and let DSPy optimize the prompt:

import dspy
 
dspy.configure(lm=dspy.LM("openai/gpt-4o", temperature=0.7))
 
class MathSolver(dspy.Module):
    def __init__(self):
        self.solve = dspy.ChainOfThought("question -> answer")
 
    def forward(self, question):
        return self.solve(question=question)
 
solver = MathSolver()
result = solver(question="Janet's ducks lay 16 eggs per day...")

DSPy's ChainOfThought automatically adds reasoning traces. dspy.majority implements self-consistency:

result = dspy.majority(
    [solver(question=q) for _ in range(5)],
    field="answer"
)

Comparison: From-Scratch vs Frameworks

This lesson produces two artifacts.

1. Reasoning Chain Prompt (outputs/prompt-reasoning-chain.md): a production-ready prompt template for few-shot CoT with self-consistency. Plug in your examples and problem domain.

2. CoT Pattern Selection Skill (outputs/skill-cot-patterns.md): a decision framework for choosing the right reasoning technique based on task type, accuracy requirements, and cost constraints.

1. Measure the gap: Take 10 GSM8K problems. Solve each with zero-shot, few-shot, zero-shot CoT, and few-shot CoT. Record accuracy for each. Which technique gives the biggest lift on your model?

2. Example selection experiment: For the same 10 problems, compare random example selection vs hand-picked similar examples. Measure accuracy difference. At what point does example quality matter more than example quantity?

3. Self-consistency cost curve: Run self-consistency with N=1, 3, 5, 7, 10 on 20 GSM8K problems. Plot accuracy vs cost (total tokens). Where is the knee of the curve for your model?

4. Build a ReAct loop: Extend the pipeline with a calculator tool. When the model generates a math expression, execute it with Python's eval() (in a sandbox) and feed the result back. Measure if tool-grounded reasoning outperforms pure CoT.

5. ToT for creative tasks: Adapt the Tree-of-Thought solver for a creative writing task: "Write a 6-word story that is both funny and sad." Use the LLM as evaluator. Does branching exploration produce better creative outputs than single-shot generation?

• Chain-of-Thought Prompting Elicits Reasoning in Large Language Models: Wei et al. 2022. The original CoT paper from Google Brain. Read sections 2-3 for the core results.
• Self-Consistency Improves Chain of Thought Reasoning in Language Models: Wang et al. 2023. The self-consistency paper. Table 1 has all the numbers you need.
• Tree of Thoughts: Deliberate Problem Solving with Large Language Models: Yao et al. 2023. ToT paper. The Game of 24 results in section 4 are the highlight.
• ReAct: Synergizing Reasoning and Acting in Language Models: Yao et al. 2022. The foundation of modern AI agents. Section 3 explains the Thought-Action-Observation loop.
• Large Language Models are Zero-Shot Reasoners: Kojima et al. 2022. The "Let's think step by step" paper. Surprisingly effective for how simple it is.
• DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines: Khattab et al. 2023. Treats prompting as a compilation problem. Read if you want to move beyond manual prompt engineering.
• OpenAI: Reasoning models guide: vendor guidance on when chain-of-thought becomes an internal, priced-per-token "reasoning" mode versus a prompt-level trick.
• Lightman et al., "Let's Verify Step by Step" (2023): process reward models (PRM) that grade each step of a chain; the reasoning supervision signal that succeeds outcome-only rewards.
• Snell et al., "Scaling LLM Test-Time Compute Optimally" (2024): systematic study of CoT length, self-consistency sampling, and MCTS; where "think step by step" goes when accuracy matters more than latency.