Audio-Language Models — Qwen2.5-Omni, Audio Flamingo, GPT-4o Audio

You have 5 seconds of audio: a dog barks, someone yells "stop!", then silence. A handful of very different questions apply to that same clip:

• Transcription. "What was said?" That's plain ASR.
• Semantic reasoning. "Is the person in danger?" Answering that takes joint understanding of the bark, the yell, and the silence together.
• Music reasoning. "What instruments play the melody?"
• Long-audio retrieval. "Where in this 90-minute lecture did the instructor explain gradient descent?"

A single model that answers all of these from one prompt is an audio-language model (LALM, or ALM). That's what separates it from pure ASR: LALMs produce free-form natural-language answers, not just transcripts.

The three-component template

Every 2026 LALM has the same skeleton:

1. Audio encoder. Whisper encoder · BEATs · CLAP · WavLM · or a custom encoder per model. (BEATs, CLAP, and WavLM are other self-supervised audio encoders alongside Whisper's; which one a given LALM uses is an implementation detail, not something worth memorizing.)
2. Projector. Linear or MLP bridging audio-encoder features into the LLM's token embedding space.
3. LLM. Llama / Qwen / Gemma-based decoder. Takes interleaved text and audio tokens, generates text.

Training runs in stages:

• Stage 1. Freeze the encoder and LLM, and train only the projector, on ASR and captioning data.
• Stage 2. Full / LoRA fine-tune on instruction-following audio tasks (QA, reasoning, music understanding).
• Stage 3 (optional). Voice-in / voice-out adds a speech decoder. Qwen2.5-Omni and AF3-Chat do this.

The 2026 model map

Benchmark reality check (2026)

MMAU-Pro is 1800 QA pairs spanning speech, sound, music, and mixed audio, with a multi-audio subset included.

The multi-audio column is damning for everyone. Random chance on a four-option multiple-choice question is 25%, and most models land right around there. LALMs still can't reliably compare two clips against each other.

Where LALMs are useful in 2026

• Compliance audit of call-center recordings. "Did the agent mention the required disclosure?"
• Accessibility. Describe sound events to deaf users (not just transcription).
• Content moderation. Detect violent language, a threatening tone, and background context, together.
• Podcast / meeting chaptering. Semantic summary, not just speaker turns.
• Music catalog analysis. "Find all tracks with a B-section key change."

Where they are NOT (yet) useful

• Fine-grained music theory (below chord-level).
• Speaker-attributed reasoning over long conversations (degrades past 10 minutes).
• Multi-audio comparison (22-26% is barely above random).
• Real-time streaming reasoning (most are offline batch inference).

Step 1: query Qwen2.5-Omni

Set the internals aside for a moment and look at the interaction from the outside: an LALM call takes audio in and hands back a plain-English answer, the same shape as any chat completion you've already sent.

from transformers import AutoModelForCausalLM, AutoProcessor
 
processor = AutoProcessor.from_pretrained("Qwen/Qwen2.5-Omni-7B")
model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-Omni-7B", torch_dtype="auto")
 
audio, sr = load_wav("clip.wav", sr=16000)
messages = [{
    "role": "user",
    "content": [
        {"type": "audio", "audio": audio},
        {"type": "text", "text": "What sounds do you hear, and what's happening?"},
    ],
}]
inputs = processor.apply_chat_template(messages, tokenize=True, return_tensors="pt")
output = model.generate(**inputs, max_new_tokens=200)
print(processor.decode(output[0], skip_special_tokens=True))

Step 2: the projector pattern

The projector is the one piece of this architecture you actually write and train yourself. The encoder and the LLM both arrive pretrained; only this part is yours to build.

import torch.nn as nn
 
class AudioProjector(nn.Module):
    def __init__(self, audio_dim=1280, llm_dim=4096):
        super().__init__()
        self.down = nn.Linear(audio_dim, llm_dim)
        self.act = nn.GELU()
        self.up = nn.Linear(llm_dim, llm_dim)
 
    def forward(self, audio_features):
        return self.up(self.act(self.down(audio_features)))

That's the whole projector: usually 1-3 linear layers. Training it on ASR pairs (audio → transcript) is the Stage-1 pretext task.

Step 3: benchmarking MMAU / LongAudioBench

None of the benchmark numbers earlier in this lesson mean anything until you can reproduce them yourself, so here's the loop that scores a model against MMAU-Pro directly.

from datasets import load_dataset
mmau = load_dataset("MMAU/MMAU-Pro")
 
correct = 0
for item in mmau["test"]:
    answer = call_model(item["audio"], item["question"], item["choices"])
    if answer == item["correct_choice"]:
        correct += 1
print(f"Accuracy: {correct / len(mmau['test']):.3f}")

Report per category (speech, sound, music, multi-audio) separately: aggregate numbers hide where the model fails.

Picking a model here is mostly a lookup, not a debate: match the task on the left to the pick on the right.

• Over-trust on multi-audio. If your task needs "which clip has X," random-chance-level performance is real.
• Long-audio degradation. Past 10 minutes, most models' speaker attribution breaks. Diarize first (Lesson 6), then summarize.
• Hallucinations on silence. Same Whisper-style issue inherited by LALMs that use Whisper encoder. VAD-gate.
• Benchmark cherry-picking. Vendor blog posts highlight best-case categories. Run MMAU-Pro multi-audio subset yourself.

Save as outputs/skill-alm-picker.md. Pick the LALM, benchmark subset, and output modality (text vs. speech) for a given audio-understanding task.

1. Easy. Run code/main.py to see a toy projector pattern and a fake LALM routing (audio-embedding, text-tokens) → output tokens.
2. Medium. Score Qwen2.5-Omni-7B on 100 MMAU-Pro speech items. Compare to the paper's reported number.
3. Hard. Build a minimal audio-captioning baseline: a BEATs encoder, a 2-layer projector, and a frozen Llama-3.2-1B. Fine-tune only the projector on AudioCaps, then compare it to SALMONN on Clotho-AQA.

• Chu et al. (2024). Qwen2-Audio: reference architecture.
• Alibaba (2025). Qwen2.5-Omni: speech-in-speech-out.
• NVIDIA (2025). Audio Flamingo 3: the open long-audio leader.
• NVIDIA (2026). Audio Flamingo Next: LongAudioBench SOTA.
• Tang et al. (2023). SALMONN: dual-encoder pioneer.
• MMAU-Pro leaderboard: live 2026 rankings.