Neural Audio Codecs — EnCodec, SNAC, Mimi, DAC and the Semantic-Acoustic Split

Language models work on discrete tokens. Audio is continuous. Any LLM-style model for speech or music (MusicGen, Moshi, Sesame CSM, VibeVoice, Orpheus) needs a neural audio codec first: a learned encoder that discretizes audio into a small vocabulary of tokens, plus a matching decoder that reconstructs the waveform.

Two families have emerged:

1. Reconstruction-first codecs, like EnCodec and DAC, optimize for perceptual audio quality. Their tokens are "acoustic": they capture everything, including speaker identity, timbre, and background noise.
2. Semantic-first codecs, like Mimi (Kyutai) and SpeechTokenizer, force the first codebook to encode linguistic and phonetic content, often by distilling from WavLM. The rest of the codebooks handle acoustic detail.

The 2024-2026 insight: a pure reconstruction codec gives you blurry speech when you try to generate from text. The LLM over codec tokens has to learn both language structure and acoustic structure in the same codebook, and that doesn't scale. Splitting them into a semantic codebook 0 and acoustic codebooks 1-N is what makes Moshi and Sesame CSM work.

The core trick: Residual Vector Quantization (RVQ)

One giant codebook would need millions of codes to sound good, so modern audio codecs skip that entirely and use RVQ instead: a cascade of small codebooks. The first codebook quantizes the encoder output, the second quantizes what's left over (the residual), and so on down the cascade. Each codebook holds 1024 codes, so 8 codebooks give you an effective vocabulary of 1024^8 = 10^24.

At inference time, the decoder just sums every chosen code per frame to reconstruct the audio.

The four codecs that matter in 2026

EnCodec (Meta, 2022). The baseline: an encoder-decoder over the waveform with an RVQ bottleneck. Runs at 24 kHz, supports up to 32 codebooks, and defaults to 4 codebooks at 1.5 kbps. Architecture is 1D conv + transformer + 1D conv. MusicGen is built on it.

DAC (Descript, 2023). RVQ with L2-normalized codebooks, periodic activation functions, and improved losses. It has the highest reconstruction fidelity of any open codec: with 12 codebooks it's sometimes indistinguishable from the original speech. Runs at 44.1 kHz full-band.

SNAC (Hubert Siuzdak, 2024). Multi-scale RVQ, where the coarse codebooks operate at a lower frame rate than the fine ones. That models audio hierarchically: a coarse "sketch" at roughly 12 Hz plus detail at 50 Hz. Orpheus-3B uses it because that hierarchical structure maps well onto LM-based generation.

Mimi (Kyutai, 2024). The most consequential of the four for 2026. 12.5 Hz frame rate (extremely low), 8 codebooks at 4.4 kbps. Codebook 0 is distilled from WavLM, trained to predict WavLM's speech-content features, while codebooks 1-7 are acoustic residuals. This split powers Moshi (Lesson 15) and Sesame CSM.

Frame rates matter for language modeling

Lower frame rate = shorter sequence = faster LM.

At 12.5 Hz, a 10-second utterance comes out to only 125 codec frames, a length a transformer predicts easily.

Semantic vs acoustic tokens

frame_t → [semantic_token_t, acoustic_token_0_t, acoustic_token_1_t, ..., acoustic_token_6_t]

• Semantic token (codebook 0 in Mimi). Encodes what was said: phonemes, words, content. Distilled from WavLM via an auxiliary prediction loss.
• Acoustic tokens (codebooks 1-7). Encode timbre, speaker identity, prosody, background noise, fine detail.

An AR LM predicts the semantic token first (conditioned on text), then predicts acoustic tokens (conditioned on semantic + speaker reference). This factorization is why modern TTS can zero-shot-clone voices: the semantic model handles content; the acoustic model handles timbre.

2026 reconstruction quality (bits per sec, lower bitrate is better)

PESQ and ViSQOL are automated scores (roughly 1-5) that estimate how a human listener would rate reconstructed audio, without needing an actual listening panel.

Traditional codecs like Opus still win per bit on perceptual quality. Neural codecs win somewhere else entirely: discrete tokens (which Opus never produces) and generative-model quality (what an LM can actually do once it has those tokens).

Step 1: encode with EnCodec

from encodec import EncodecModel
import torch
 
model = EncodecModel.encodec_model_24khz()
model.set_target_bandwidth(6.0)  # kbps
 
wav = torch.randn(1, 1, 24000)
with torch.no_grad():
    encoded = model.encode(wav)
codes, scale = encoded[0]
# codes: (1, n_codebooks, n_frames), dtype=int64

n_codebooks=8 at 6 kbps. Each code is 0-1023 (10-bit).

Step 2: decode and measure reconstruction

with torch.no_grad():
    wav_recon = model.decode([(codes, scale)])
 
from torchaudio.functional import compute_deltas
import torch.nn.functional as F
 
mse = F.mse_loss(wav_recon[:, :, :wav.shape[-1]], wav).item()

Step 3: the semantic-acoustic split (Mimi-style)

from moshi.models import loaders
mimi = loaders.get_mimi()
 
with torch.no_grad():
    codes = mimi.encode(wav)  # shape (1, 8, [email protected])
 
semantic = codes[:, 0]
acoustic = codes[:, 1:]

Semantic codebook 0 is WavLM-aligned. You can train a text-to-semantic transformer with a much smaller vocabulary than going direct-to-audio, then let a separate acoustic-to-waveform decoder condition on a speaker reference.

Step 4: why AR LM over codec tokens works

For a 10 s speech clip at Mimi's 12.5 Hz × 8 codebooks:

N_tokens = 10 * 12.5 * 8 = 1000 tokens

1000 tokens is a trivial context for a transformer. A 256M-parameter transformer can generate 10 seconds of speech in milliseconds on a modern GPU.

Map problem → codec:

Rule of thumb: if you're building a generative model, start with Mimi or SNAC. If you're building a compression pipeline, use Opus.

• Too many codebooks. Adding codebooks increases fidelity linearly but LM sequence length linearly too. Stop at 8-12.
• Frame-rate mismatch. Training LM on 12.5 Hz Mimi then fine-tuning on 50 Hz EnCodec fails silently.
• Assuming all codebooks equal. In Mimi, codebook 0 carries content; losing it destroys intelligibility. Losing codebook 7 is barely noticeable.
• Using reconstruction quality as the only metric. A codec can have great reconstruction but be useless for LM-based generation if the semantic structure is bad.

Save as outputs/skill-codec-picker.md. Pick a codec for a given generative or compression task.

1. Easy. Run code/main.py. It implements a toy scalar + residual quantizer and measures reconstruction error as you add codebooks.
2. Medium. Install encodec and compare 1, 4, 8, 32 codebooks on a held-out speech clip. Plot PESQ or MSE vs bitrate.
3. Hard. Load Mimi. Encode a clip. Replace codebook 0 with random integers; decode. Then replace codebook 7 similarly. Compare the two corruptions: codebook 0 corruption should destroy intelligibility, while codebook 7 corruption should barely change anything.

• Défossez et al. (2023). EnCodec: the RVQ baseline.
• Kumar et al. (2023). Descript Audio Codec (DAC): highest-fidelity open.
• Siuzdak (2024). SNAC: multi-scale RVQ.
• Kyutai (2024). Mimi codec: semantic-acoustic split, WavLM distillation.
• Borsos et al. (2023). AudioLM: the two-stage semantic/acoustic paradigm.
• Zeghidour et al. (2021). SoundStream: the original streamable RVQ codec.