Audio Classification — From k-NN on MFCCs to AST and BEATs

Audio classification is the task of mapping a clip to a label, and by 2026 the baseline architecture for doing it is mature: log-mel → CNN or Transformer → softmax. Hand a model a 10-second clip and ask "what is it?" and you're doing this task, whatever the domain: urban sound (siren, drill, dog), speech command (yes/no/stop), language ID (en/es/ar), speaker emotion (angry/neutral), environmental sound (indoor/outdoor, babble).

The bottleneck here is data, not model architecture. Audio datasets carry brutal class imbalance, strong domain shift (clean vs noisy), and label noise nobody agrees on (is that "urban babble" or "restaurant noise"?). Curation, augmentation, and evaluation make up 80% of the actual work; swapping a CNN for a Transformer barely touches it.

k-NN on MFCCs (the 1990s baseline). Flatten the MFCCs for a clip, compute cosine similarity against a labeled bank, and return the majority vote of the top K. It's surprisingly strong on clean, small datasets like Speech Commands and ESC-50, and it runs with no GPU at all.

2D CNN on log-mels (2015-2019). Treat the (T, n_mels) log-mel as an image, run it through a ResNet-18 or VGG-style stack, global-mean-pool the time axis, and softmax over classes. Still the baseline you'll see in most 2026 Kaggle competitions.

Audio Spectrogram Transformer, AST (2021-2024). Patchify the log-mel (16×16 patches are typical), add position embeddings, and feed the result to a ViT. State of the art on AudioSet for supervised learning, at mAP 0.485.

BEATs and WavLM-base (2024-2026). Self-supervised pretraining on millions of hours of audio, then fine-tune on your task with 1-10% of the supervised data you'd otherwise need. This is the default starting point for non-speech audio in 2026: BEATs-iter3 beats AST by 1-2 mAP on AudioSet while using 1/4 the compute.

Whisper-encoder as a frozen backbone (2024). Take Whisper's encoder, drop the decoder, attach a linear classifier. That's it. It lands near-SOTA on language ID and simple event classification with zero audio augmentation, which is why it's the "free lunch" baseline.

Class imbalance is the real challenge

ESC-50 keeps things easy: 50 classes, 40 clips each, balanced. UrbanSound8K is already lopsided, 10 classes imbalanced 10:1. AudioSet is the extreme case: 632 classes with a 100,000:1 long tail. A few techniques actually move the needle:

• Balanced sampling during training (not evaluation).
• Mixup: linearly interpolate two clips, and their labels, as augmentation.
• SpecAugment: mask random time and frequency bands. Simple, and critical.

Evaluation

• Multiclass exclusive (Speech Commands): top-1 accuracy, top-5 accuracy.
• Multiclass multi-label (AudioSet, UrbanSound-style): mean average precision (mAP), plus AUC (area under the ROC curve) as the companion metric AudioSet leaderboards report alongside it.
• Heavily imbalanced: per-class recall plus macro F1.

The 2026 numbers worth memorizing:

Step 1: featurize

def featurize_mfcc(signal, sr, n_mfcc=13, n_mels=40, frame_len=400, hop=160):
    mag = stft_magnitude(signal, frame_len, hop)
    fb = mel_filterbank(n_mels, frame_len, sr)
    mels = apply_filterbank(mag, fb)
    log = log_transform(mels)
    return [dct_ii(frame, n_mfcc) for frame in log]

Step 2: fixed-length summary

def summarize(mfcc_frames):
    n = len(mfcc_frames[0])
    mean = [sum(f[i] for f in mfcc_frames) / len(mfcc_frames) for i in range(n)]
    var = [
        sum((f[i] - mean[i]) ** 2 for f in mfcc_frames) / len(mfcc_frames) for i in range(n)
    ]
    return mean + var

Simple but strong: mean + variance across time turns a 13-coef MFCC into a fixed 26-dim embedding. It runs instantly, and this exact trick beat state-of-the-art NN baselines on ESC-50 as recently as 2017.

Step 3: k-NN

def cosine(a, b):
    dot = sum(x * y for x, y in zip(a, b))
    na = math.sqrt(sum(x * x for x in a)) or 1e-12
    nb = math.sqrt(sum(x * x for x in b)) or 1e-12
    return dot / (na * nb)
 
def knn_classify(q, bank, labels, k=5):
    sims = sorted(range(len(bank)), key=lambda i: -cosine(q, bank[i]))[:k]
    votes = Counter(labels[i] for i in sims)
    return votes.most_common(1)[0][0]

Step 4: upgrade to CNN on log-mels

In PyTorch:

import torch.nn as nn
 
class AudioCNN(nn.Module):
    def __init__(self, n_mels=80, n_classes=50):
        super().__init__()
        self.body = nn.Sequential(
            nn.Conv2d(1, 32, 3, padding=1), nn.ReLU(), nn.MaxPool2d(2),
            nn.Conv2d(32, 64, 3, padding=1), nn.ReLU(), nn.MaxPool2d(2),
            nn.Conv2d(64, 128, 3, padding=1), nn.ReLU(),
            nn.AdaptiveAvgPool2d(1),
        )
        self.head = nn.Linear(128, n_classes)
 
    def forward(self, x):  # x: (B, 1, T, n_mels)
        return self.head(self.body(x).flatten(1))

3M parameters, trains in ~10 min on ESC-50 with a single RTX 4090, and clears 80%+ accuracy.

Step 5: fine-tune BEATs (the 2026 default)

from transformers import ASTFeatureExtractor, ASTForAudioClassification
 
ext = ASTFeatureExtractor.from_pretrained("MIT/ast-finetuned-audioset-10-10-0.4593")
model = ASTForAudioClassification.from_pretrained(
    "MIT/ast-finetuned-audioset-10-10-0.4593",
    num_labels=50,
    ignore_mismatched_sizes=True,
)
 
inputs = ext(audio, sampling_rate=16000, return_tensors="pt")
logits = model(**inputs).logits

For BEATs, swap in microsoft/BEATs-base via the beats library. The transformers API keeps the same shape.

The 2026 stack:

Decision rule: start with a frozen backbone, not a fresh model. Fine-tuning a BEATs head gets you 95% of SOTA in hours, not weeks.

This lesson ships outputs/skill-classifier-designer.md, a skill for picking architecture, augmentations, class-balance strategy, and eval metric for a given audio classification task.

1. Easy. Run code/main.py. It trains the k-NN MFCC baseline on a 4-class synthetic dataset (pure tones at different pitches). Report confusion matrix.
2. Medium. Replace summarize with [mean, var, skew, kurtosis]. Does 4-moment pooling beat mean+var on the same synthetic dataset?
3. Hard. Using torchaudio, train a 2D CNN on ESC-50 fold 1. Report 5-fold cross-validation accuracy. Add SpecAugment (time mask = 20, freq mask = 10) and report the delta.

• Gong, Chung, Glass (2021). AST: Audio Spectrogram Transformer: the architecture of record from 2021–2024.
• Chen et al. (2022, rev. 2024). BEATs: Audio Pre-Training with Acoustic Tokenizers: the 2024+ default.
• Park et al. (2019). SpecAugment: the dominant audio augmentation.
• Piczak (2015). ESC-50 dataset: 50-class benchmark that lives on.
• Gemmeke et al. (2017). AudioSet: 632-class YouTube taxonomy; still the gold standard.