BuildPython

Audio Generation

Audio is a 1-D signal at 16-48 kHz, so a five-second clip is 80-240k samples. No transformer attends to that sequence directly. The solution for every production audio model in 2026 is the same: a neural codec (Encodec, SoundStream, DAC) compresses audio to discrete tokens at 50-75 Hz, and a transformer or diffusion model generates the tokens.

TIME: ~45 minutesPREREQUISITES: Phase 6 · 02 (Audio Features), Phase 6 · 04 (ASR), Phase 8 · 06 (DDPM)

Learning Objectives

Explain why raw audio is tokenized by a neural codec (Encodec, SoundStream, DAC) before a transformer or diffusion model touches it.
Describe how residual vector quantization turns each timestep into K codebook indices, and compute the resulting token rate (8 codebooks at 75 Hz is 600 tokens/sec).
Build a count-based bigram token predictor in stdlib Python and use it for style-conditioned sampling and VALL-E-style prompt continuation.
Choose between token-autoregressive and flow-matching generators for a TTS, music, or sound-effects brief based on streaming and latency needs.
Name the main pitfalls of audio generation: codec quality caps, RVQ error accumulation, long-sequence structure, training-data licensing, and voice cloning abuse.

The Problem

Three audio generation tasks:

Text-to-speech. Given text, produce speech. Clean speech is narrow-band and has strong phonetic structure, so transformer-over-tokens works well. VALL-E (Microsoft), NaturalSpeech 3, ElevenLabs, OpenAI TTS.
Music generation. Given a prompt (text, melody, chord progression, genre), produce music. The distribution is much broader. MusicGen (Meta), Stable Audio 2.5, Suno v4, Udio, Riffusion.
Audio effects / sound design. Given a prompt, produce ambient sound or Foley. AudioGen, AudioLDM 2, Stable Audio Open.

These three share a substrate: a neural audio codec plus a token-AR or diffusion generator.

Warm-up0/3

1. Which representation of audio shows how energy is spread across frequency bands over time on a perceptual scale?Warm-up

A time ruler from 0 to 1.5 seconds with three labelled stretches of a made-up recording: a low hum from 0 to 0.5 s, a high whistle from 0.5 to 1.0 s, and both at once from 1.0 to 1.5 s. — Made-up recording, described in words only.

2. In a DDPM, what does the trained network predict at each reverse step?Warm-up

Three small plots of a 1-D signal during the reverse process: pure noise at the start, a sine wave with noise at a middle step t, and a clean sine wave at the end. A box below the middle plot says the trained network at step t gets the sample at step t and outputs a question mark. — Toy signal with noise added by the script, not a trained model.

3. A language model writes text one piece at a time. What does it predict at each step?Warm-up

Five rows, one per step, showing the sentence The cat sat on the mat growing by one word each step: The; The cat; The cat sat; The cat sat on; The cat sat on the. — Typed example sentence; the words are made up.

Check: The Problem0/1

1. Which tasks does the Problem section describe as sharing one codec-plus-generator substrate?Define

A wide box labelled neural audio codec plus token-AR or diffusion generator, with three dashed boxes above it labelled task 1, task 2 and task 3, each followed by a question mark. — Schematic of the question; the three task names are left out.

The Concept

Audio generation: codec tokens + transformer or diffusion

Neural audio codecs

Encodec (Meta, 2022), SoundStream (Google, 2021), Descript Audio Codec (DAC, 2023). A convolutional encoder compresses the waveform to a per-timestep vector, and residual vector quantization (RVQ) converts each vector to a cascade of K codebook indices. The decoder reverses it. 24 kHz audio at 6 kbps using 8 RVQ codebooks (1024 entries each) at 75 Hz = 600 tokens/sec.

waveform (24000 samples/sec)
    └─ encoder conv ─┐
                     ├─ RVQ layer 1 → indices at 75 Hz
                     ├─ RVQ layer 2 → indices at 75 Hz
                     ├─ ...
                     └─ RVQ layer 8

Two generative paradigms on top

Token-autoregressive. Flatten RVQ tokens into a sequence, run a decoder-only transformer. MusicGen uses "delayed parallel" to emit K codebook streams in parallel with per-stream offsets. VALL-E generates speech tokens from a text prompt + 3-second voice sample.

Latent diffusion. Pack codec tokens as continuous latents or model them with categorical diffusion. Stable Audio 2.5 uses flow matching on continuous audio latents. AudioLDM 2 uses text-to-mel-to-audio diffusion.

The 2024-2026 trend: flow matching is winning for music (faster inference, cleaner samples) while token-AR still dominates speech, because it is naturally causal and streams well.

Check: The Concept0/10

1. What is a neural audio codec?Define

A table of the lesson's example codec setup: 24 kHz audio in (24000 samples per second), 8 codebooks, 1024 entries per codebook, 75 frames per second, and a bit rate of 6 kbps. A line below names Encodec, SoundStream and DAC. — Numbers from the lesson's Concept section.

2. What does RVQ (residual vector quantization) do in a neural audio codec?Define

Seven columns t1 to t7 of coloured cells stand for encoder output vectors, one per timestep at 75 per second, followed by a box labelled RVQ with K = 8 codebooks of 1024 entries. Below it, the RVQ output for one timestep is shown as a single question mark. — Colours of the vector cells are made up.

3. In audio generation, what is a token?Define

Twelve boxes holding the numbers 0, 2, 2, 3, 3, 4, 4, 5, 6, 6, 6, 7 from the lesson's main.py. A line notes the demo's vocabulary is 16 entries numbered 0 to 15. — Real output of the lesson's main.py (seed 42).

4. What is delayed parallel generation, as used by MusicGen?Define

Schematic with K = 4 and 6 frames to keep it small.

5. What is flow matching, as used by Stable Audio 2.5?Define

A card for Stable Audio 2.5 from the lesson's table: task Music / SFX, backbone DiT plus flow matching on audio latents, latency about 10 seconds for a 1-minute clip. A note lists NaturalSpeech 3 and AudioCraft 2 as other table rows that use flow matching. — Copied from the lesson's table.

6. What is a voice prompt in a system like VALL-E?Define

A terminal excerpt of the lesson's main.py output under the heading VALL-E-style prompt continuation, with one line: 3-second voice prompt (tokens): [0, 2, 2, 3, 5]. — Real output of main.py (seed 42), cut after the prompt line.

7. What is a mel spectrogram, as used by many TTS systems?Define

A heat map with time from 0 to 1.5 s on the x axis and bands 1 to 32 on the y axis, showing a bright diagonal streak that rises from the lowest bands to the upper bands over the clip. — Computed with numpy from a made-up rising tone, not from a lesson system.

8. Why can't a transformer directly attend over a raw audio waveform?Explain

A made-up one-second waveform with a narrow stretch marked and zoomed in below, where the signal appears as a row of 40 separate dots, each one stored sample. — Made-up waveform; the zoom shows 40 dots only to show the idea of samples.

9. Why does token-autoregressive generation still dominate speech even though flow matching is winning for music?Explain

Two cards. Music: flow matching is winning, with faster inference and cleaner samples. Speech: token-AR still dominates, followed by the word why and a question mark. — Wording from the lesson's trend sentence, minus its reason.

10. A codec runs 8 RVQ codebooks at 75 Hz. How many tokens per second does it produce?Apply

A grid of 8 codebook rows by 10 visible frame columns followed by dots, marked as one second of audio equal to 75 frames. — Only 10 of the 75 columns are drawn.

Production landscape

System	Task	Backbone	Latency
ElevenLabs V3	TTS	Token-AR + neural vocoder	~300ms first token
OpenAI GPT-4o audio	Full-duplex speech	End-to-end multimodal AR	~200ms
NaturalSpeech 3	TTS	Latent flow matching	Non-streaming
Stable Audio 2.5	Music / SFX	DiT + flow matching on audio latents	~10s for 1-minute clip
Suno v4	Full songs	Undisclosed; token-AR suspected	~30s per song
Udio v1.5	Full songs	Undisclosed	~30s per song
MusicGen 3.3B	Music	Token-AR on Encodec 32kHz	Real-time
AudioCraft 2	Music + SFX	Flow matching	~5s for 5s clip
Riffusion v2	Music	Spectrogram diffusion	~10s

Check: Production landscape0/3

1. What is a vocoder?Define

A card for ElevenLabs V3 from the lesson's table: task TTS, backbone Token-AR plus neural vocoder, latency about 300ms to first token. — Copied from the lesson's table.

2. Which system does the Production landscape table list at about 200ms latency as an end-to-end multimodal AR model for full-duplex speech?Define

One table row with the system name hidden as a question mark: task full-duplex speech, backbone end-to-end multimodal AR, latency about 200ms. — Row copied from the lesson's table with the name removed.

3. Which backbone does the Production landscape table list for MusicGen 3.3B?Define

One table row: MusicGen 3.3B, task music, backbone hidden as a question mark, latency real-time. — Row copied from the lesson's table with the backbone removed.

Build It

code/main.py simulates the core idea with a count-based bigram model standing in for the transformer. It trains on synthetic "audio token" sequences from two styles: style 0 counts up by one token at a time, style 1 counts up by three, both modulo a 16-token vocabulary and both with a random 0 or 1 jitter on each position. Condition on style and sample.

Step 1: synthetic audio tokens

python

def make_tokens(style, length, vocab_size, rng):
    if style == 0:  # "speech-like": alternating
        return [i % vocab_size for i in range(length)]
    # "music-like": ramp
    return [(i * 3) % vocab_size for i in range(length)]

The snippet is the simplified form without the jitter; main.py adds rng.randint(0, 1) to each position.

Step 2: train a tiny token predictor

The predictor is a table of next-token counts per style, starting at 1 for every entry and updated from 500 sequences of 20 tokens per style. A transformer would replace the table in a real system. The pattern is the same: codec tokens, a likelihood-style training signal, then autoregressive sampling.

Step 3: sample conditionally

Given the style and a starting token, sample the next token from the predicted distribution and repeat. main.py generates 20 tokens per style at temperature 0.7 and 1.0, then continues a 5-token prompt for 15 more tokens, which is the VALL-E-style prompt continuation pattern in miniature.

Check: Build It0/6

1. Why does the Build It demo use a count table in place of a transformer?Explain

The code of generate() from main.py: it starts a list out with the start token, loops length minus 1 times, calls probs(counts, style, out[-1]) (highlighted), optionally applies temperature, then appends sample_from(p, rng), and returns out. — Verbatim from main.py; the highlight is added.

2. Why does the VALL-E-style prompt continuation work so cleanly in the toy demo?Explain

A terminal excerpt of the lesson's demo output: the 5-token voice prompt [0, 2, 2, 3, 5] and the continuation [0, 2, 2, 3, 5, 6, 6, 7, 8, 8, 9, 10, 12, 14, 14, 15, 12, 13, 13, 14]. — Real output of the lesson's main.py (seed 42).

3. Right after init_counts() runs with VOCAB = 16, what does probs() return for any row?Apply

Code from main.py: VOCAB = 16, init_counts() building 2 styles of 16 by 16 tables filled with 1.0, and probs() dividing a row by its total. The last two lines call init_counts() and then probs(counts, 0, 5). — From main.py (one long line wrapped to fit); the final call is the question's setup.

4. generate() applies p ** (1 / temperature) and renormalizes. At temperature 0.7 compared with 1.0, what happens to the most likely next token's probability?Apply

A bar chart of the trained next-token row for style 0 after token 5 at temperature 1.0: token 6 at 0.48, token 7 at 0.25, token 5 at 0.24, others near zero. Below it, the code line that raises each probability to the power 1 / temperature and renormalizes. — Real counts from main.py after training (seed 42).

5. In main.py the VALL-E-style prompt continuation starts from a 5-token prompt and loops 15 times. What does the printed continuation contain?Apply

Seven lines of main.py: a 5-token prompt from make_tokens, a print of the prompt, continuation = list(prompt), a loop of 15 that samples from probs(counts, 0, continuation[-1]) and appends, and a print of continuation. — Verbatim from main.py.

6. After training on style 0 sequences, which next token is most likely after token 5?Apply

The style 0 line of make_tokens, which adds 0 or 1 to i modulo 16, followed by three sample training sequences of 20 tokens each produced by that function. — Code from main.py; sequences generated with the lesson's make_tokens (seed 7).

Pitfalls

Codec quality caps output quality. If the codec can't represent a sound faithfully, no amount of generator quality helps. DAC is the current open best.
RVQ error accumulation. Each RVQ layer models the residual of the previous. Errors on layer 1 propagate to later layers. Sampling with temperature 0 on higher layers helps.
Musical structure. 30 seconds at 75 Hz is 2,250 frames per codebook, or 18,000 tokens across 8 codebooks. That is hard for transformers. MusicGen uses sliding window + prompt continuation; Stable Audio uses shorter clips + crossfading.
Artifacts at boundaries. Crossfading between generated clips needs careful overlap-add.
Clean-data appetite. Music generators need tens of thousands of hours of licensed music. The Suno / Udio RIAA lawsuit (2024) brought this to the surface.
Voice cloning ethics. A 3-second sample plus a text prompt is enough for VALL-E / XTTS / ElevenLabs to clone a voice, so production models need abuse detection + opt-out lists.

Check: Pitfalls0/10

1. Which strategies does the Pitfalls section name for the very long token sequences in music?Explain

Eight equal-length bars, one per codebook, spanning a 30 second track, labelled 2,250 frames per codebook, with the sum 30 x 75 = 2,250 frames, times 8 codebooks = 18,000 tokens. — Computed from the Pitfalls section's numbers.

2. What does the 2024 Suno / Udio RIAA lawsuit highlight about music generators?Explain

Two cards. Left: Suno v4 and Udio v1.5, full songs, music generators. Right: the 2024 RIAA lawsuit against Suno / Udio. A line below asks what the lawsuit highlights about music generators. — Facts from the lesson's table and Pitfalls section.

3. How are codec quality and generator quality related in the quality of the final audio?Explain

A three-box chain: generator, then codec, then audio you hear, each box with quality: ? underneath. A note says the lesson names DAC as the current open best codec. — Schematic; no measured quality.

4. What problem does RVQ error accumulation cause, and what helps with it?Explain

A vertical stack of eight boxes RVQ layer 1 to RVQ layer 8 joined by arrows. Notes say each layer models the residual of the previous layer and indices come out of all eight layers for each timestep. — Schematic from the Concept and Pitfalls sections.

5. How much reference audio is enough to clone a voice with VALL-E, XTTS or ElevenLabs, and what follows?Explain

Schematic of the question.

6. A team swaps in a much bigger transformer, but the generated speech is still muffled and loses fine detail. The codec drops high-frequency detail. What is the diagnosis?Analyze

A plot of energy against frequency from low to high with two curves: a dashed grey curve for natural speech that decays smoothly, and an orange curve for the output with the bigger transformer that follows it until the middle of the range and then drops sharply, losing the high frequencies. — Illustrative curves, not measurements.

7. A music model produces a convincing first ten seconds, but over a 30-second track the song structure drifts. Which pitfall is the best explanation?Analyze

A 30 second timeline: 0 to 10 s is labelled convincing in green, and 10 to 30 s is labelled song structure drifts in red. — Drawn from the stem's description.

8. A system builds a 30-second track by stitching shorter generated clips, and listeners hear clicks at every join. Which fix does the lesson point to?Analyze

A waveform made of three clips A, B and C joined end to end, with dashed lines at the two joins where the waveform jumps. — Illustrative waveform drawn by the script.

9. A voice clone sounds noisy, and the team samples every RVQ layer at a high temperature. Which change targets the cause named in the lesson?Analyze

A table of RVQ layers 1 to 8 with sampling temperature high on every layer, and a note on the right that the result is a noisy voice. — Drawn from the stem; no numeric temperature is given there.

10. You are adding voice cloning to a product, and users will upload a short sample. Which design is best supported by the lesson?Design

A three-box flow: a user uploads a short sample, the voice cloning feature, and speech in the cloned voice, with the open question of which safeguards and rules the product should have. — Schematic drawn from the stem.

Use It

Task	2026 stack
Commercial TTS	ElevenLabs, OpenAI TTS, or Azure Neural
Voice cloning (consent-verified)	XTTS v2 (open) or ElevenLabs Pro
Background music, fast	Stable Audio 2.5 API, Suno, or Udio
Music with lyrics	Suno v4 or Udio v1.5
Sound effects / Foley	AudioCraft 2, ElevenLabs SFX, or Stable Audio Open
Real-time voice agent	GPT-4o realtime or Gemini Live
Open-weights music research	MusicGen 3.3B, Stable Audio Open 1.0, AudioLDM 2
Dubbing / translation	HeyGen, ElevenLabs Dubbing

Check: Use It0/1

1. Which stack does the Use It table recommend for a real-time voice agent?Define

The Use It table's eight tasks, with the stack column hidden. The Real-time voice agent row is highlighted with a question mark, and the other rows show dots. — Task column copied from the lesson's table.

Ship It

Save outputs/skill-audio-brief.md. Skill takes an audio brief (task, duration, style, voice, license) and outputs: model + hosting, prompt format (genre tags, style descriptors, structural markers), codec + generator + vocoder chain, seed protocol, and eval plan (MOS / CLAP score / CER for TTS / user A/B).

Author's Note

I'd run main.py before reading the codec material twice. It prints 20 generated tokens per style at temperature 0.7 and again at 1.0, so you can see by eye that style 0 climbs by roughly one token per step and style 1 by roughly three, wrapping around the 16-token vocabulary. Lower temperature visibly tightens the pattern.

The predictor is a count table, not a transformer. I did that on purpose: with no network in the way, the whole loop is visible in a few lines. Count transitions, turn a row into probabilities, sample, append. A transformer swaps out the table and leaves the rest alone.

The part I'd look at longest is the prompt continuation at the end. It takes five tokens from style 0, treats them as a voice prompt, and samples 15 more. That is the VALL-E shape in miniature, and it works here only because the toy tokens are so predictable.

The toy also skips everything hard about real audio. There is no RVQ, so each step has one token instead of a stack of K, and no codec, so nothing here can lose fidelity. When I need to judge a real system, the Pitfalls list is where I'd start: codec quality and long-sequence structure.

Exercises

Easy. Run code/main.py and set style explicitly. Verify the generated sequences match the style's pattern.
Medium. Add delayed parallel decoding: simulate 2 streams of tokens that must stay offset by 1 step. Train a joint predictor.
Hard. Use HuggingFace transformers to run MusicGen-small locally. Generate a 10-second clip with three different prompts; A/B for style adherence.

Key Terms

Term	What people say	What it actually means
Codec	"Neural compression"	Encoder / decoder for audio; typical output is 50-75 Hz tokens.
RVQ	"Residual VQ"	Cascade of K quantizers; each models the residual of the previous.
Token	"One codec symbol"	Discrete index into a codebook; 1024 or 2048 typical.
Delayed parallel	"Offset codebooks"	Emit K token streams with staggered offsets to reduce sequence length.
Flow matching	"The 2024 win for audio"	Straighter-path alternative to diffusion; faster sampling.
Voice prompt	"3-second sample"	Speaker embedding or token prefix that steers the cloned voice.
Mel spectrogram	"The visual"	Log-magnitude perceptual spectrogram; used by many TTS systems.
Vocoder	"Mel to wave"	Neural component that converts mel spectrograms back to audio.

Production note: audio is a streaming problem

Audio is the one output modality users expect to arrive as it is generated, not all-at-once. In production terms this means TPOT matters (Time Per Output Token) because the target throughput is the user's listening speed, not their reading speed. For 24 kHz audio tokenized by Encodec at 75 frames/second with 8 codebooks, the server has to generate at least 600 tokens/sec per user (75 steps/sec with delayed parallel decoding) to keep playback smooth.

One architectural consequence:

Flow-matching audio models don't stream easily. Stable Audio 2.5 and AudioCraft 2 render a fixed clip length in one pass. To stream, you chunk the clip and overlap the boundaries (think sliding-window diffusion), which adds 100-300ms of latency overhead vs a codec AR model.

If the product is "live voice chat" or "real-time music continuation", pick the codec AR path. If it is "render a 30-second clip on submit", flow-matching wins on quality and total latency.

Check: Production note: audio is a streaming problem0/4

1. For streamed audio, why must a server generate tokens at least as fast as playback consumes them?Explain

A plot of tokens against seconds since playback starts, with a straight line for playback rising from 0 to 3,000 tokens over 5 seconds, labelled 600 tokens/sec. A label says server generation rate: question mark. — Computed from the lesson's 75 x 8 = 600 tokens per second.

2. Why can't flow-matching audio models such as Stable Audio 2.5 stream trivially?Explain

Four rows of the lesson's table: ElevenLabs V3 (token-AR plus neural vocoder, about 300ms first token), NaturalSpeech 3 (latent flow matching, non-streaming), Stable Audio 2.5 (DiT plus flow matching on latents, about 10s for a 1-minute clip) and AudioCraft 2 (flow matching, about 5s for a 5s clip). — Rows copied from the lesson's table (the latents wording is shortened).

3. Your team is shipping a live voice-chat assistant that must start speaking within a few hundred milliseconds while the reply is still being generated. Which generator design fits best?Design

A schematic of one voice chat turn: the user speaks, then a reply is being generated, and the assistant starts speaking a few hundred milliseconds after the user stops, while generation of the reply continues. — Schematic from the stem; not to scale.

4. A product renders a 30-second background track when the user presses submit, with no live playback. Which design fits best?Design

A three-step flow: the user presses submit, no live playback while the track renders, and then a 30-second background track is delivered. — Schematic from the stem.