Music Generation — MusicGen, Stable Audio, Suno, and the Licensing Earthquake

Music generation takes a text prompt and turns it into a finished clip, anywhere from 30 seconds to 4 minutes, complete with lyrics, vocals, and structure. That splits into three sub-problems:

1. Instrumental generation. Text like "lo-fi hip-hop drums with warm keys" becomes audio directly. MusicGen, Stable Audio, and AudioLDM live here.
2. Song generation (with vocals and lyrics). "Country song about rainy Texas nights" becomes a full song, vocals included. Suno, Udio, YuE, and ACE-Step do this.
3. Conditional / controllable generation. Extend an existing clip, regenerate just the bridge, swap genre, separate stems, or inpaint a section. Udio's inpainting plus stem separation is the feature every other 2026 platform is racing to match.

Every music-generation model comes down to one of two architecture families, or a closed-weight hybrid of both: autoregressive token language models predicting neural-codec tokens, and diffusion models running over mel spectrograms or latents.

Token LM over neural-codec tokens

Meta's MusicGen (2023, MIT) and its many derivatives condition on text or melody embeddings and autoregressively predict EnCodec tokens (32 kHz, 4 codebooks), then decode with EnCodec. Parameter counts run 300M to 3.3B. It's a strong baseline, but it struggles past 30 seconds.

ACE-Step (open-source, 4B XL, released April 2026) extends the same approach to full-song, lyric-conditioned generation. It's the open community's closest thing to Suno.

Diffusion over mels or latents

Stable Audio (2023) and Stable Audio Open (2024) run latent diffusion on compressed audio. Both excel at loops, sound design, and ambient textures, and neither is great at structured full songs.

AudioLDM / AudioLDM2 does text-to-audio through T2I-style latent diffusion, generalized to music, sound effects, and speech alike.

Hybrid (production): Suno, Udio, Lyria

These are closed weights, likely an autoregressive codec LM paired with a diffusion-based vocoder and specialized voice, drum, and melody heads. Suno v5 (2026) leads on quality at ELO 1293. Udio v4 adds inpainting plus stem separation, so bass, drums, and vocals download as separate files.

Evaluation

• FAD (Fréchet Audio Distance). Embedding-level distance between the generated and real audio distributions, computed with VGGish or PANNs features. Lower is better: MusicGen small scores 4.5 FAD on MusicCaps, against a SOTA of ~3.0.
• Musicality (subjective). Human preference. Suno v5 leads at ELO 1293.
• Text-audio alignment. The CLAP score, a contrastive text-audio embedding (the audio analog of CLIP), between prompt and output.
• Musicality artifacts. Off-beat transitions, vocal-phrase drift, and loss of structure past 30 seconds.

• Warner Music vs. Suno settlement. $500M. The settlement gives WMG oversight of AI likeness, music rights, and user-generated tracks on Suno, and UMG landed a similar settlement with Udio.
• EU AI Act and California SB 942. Both now require disclosure when music is AI-generated.
• Riffusion and MusicGen, both MIT-licensed, carry no compliance baggage, but they also don't do commercial-grade vocals.

Safe-to-ship patterns:

1. Generate instrumental only (MusicGen, Stable Audio Open, MIT/CC0 outputs).
2. Use commercial APIs (Suno, Udio, ElevenLabs Music) with a per-generation license.
3. Train on an owned or licensed catalog, which is where most enterprises end up.
4. Tag every generation with watermarks and metadata.

Step 1: generate with MusicGen

from audiocraft.models import MusicGen
import torchaudio
 
model = MusicGen.get_pretrained("facebook/musicgen-small")
model.set_generation_params(duration=10)
wav = model.generate(["upbeat synthwave with driving drums, 128 BPM"])
torchaudio.save("out.wav", wav[0].cpu(), 32000)

Three sizes ship: small (300M, fast), medium (1.5B), and large (3.3B). small is enough to answer "does the idea land."

Step 2: melody conditioning

MusicGen-melody takes a chromagram (a 12-dimensional, per-frame pitch representation) as input and preserves the tune while it swaps out the timbre, so "give me this melody as a string quartet" is exactly the kind of request it's built for.

melody, sr = torchaudio.load("humming.wav")
wav = model.generate_with_chroma(
    ["jazz piano cover"],
    melody.squeeze(),
    sr,
)

Step 3: FAD evaluation

from frechet_audio_distance import FrechetAudioDistance
fad = FrechetAudioDistance()
 
fad.get_fad_score("generated_folder/", "reference_folder/")

It computes VGGish-embedding distance, which is useful for genre-level regression tests, but it's no substitute for human listeners.

Step 4: adding to the LLM-music workflow

Combine this with the ideas from Lessons 7-8:

prompt = "Write a 30-second jazz loop. Describe the drums, bass, and piano voicing."
description = llm.complete(prompt)
music = musicgen.generate([description], duration=30)

• Copyright-laundering prompts. Commercial platforms filter out prompts like "song in the style of Taylor Swift" now; Suno and Udio both do it. Open models don't, so add your own filter list.
• Repetition / drift past 30 s. AR models loop past that point. Crossfade multiple generations, or use ACE-Step for structural coherence.
• Tempo drift. Models wander off the target BPM. Tag the BPM in the prompt and post-filter with librosa's beat_track.
• Vocal intelligibility. Suno is excellent at this; open models are often mushy on words. If the lyrics matter, reach for a commercial API or fine-tune.
• Mono output. Open models generate mono, or fake stereo. Fix it with a proper stereo reconstruction step, like ezst or Cartesia's stereo diffusion.

Save it as outputs/skill-music-designer.md. Pick the model, the license strategy, a length and structure plan, and the disclosure metadata your deployment needs.

1. Easy. Run code/main.py. It produces a "generative" chord progression and drum pattern as ASCII symbols, a music-gen cartoon you can play back through any MIDI renderer if you want.
2. Medium. Install audiocraft, generate 10-second clips across 4 genre prompts with MusicGen-small, and measure FAD against a reference genre set.
3. Hard. Using ACE-Step (or MusicGen-melody), generate three variations of the same tune with different timbre prompts, then compute CLAP similarity to the prompt to verify alignment.

• Copet et al. (2023). MusicGen: the open autoregressive benchmark.
• Evans et al. (2024). Stable Audio Open: the sound-design default.
• ACE-Step: open 4B full-song generator, April 2026.
• Suno v5 platform docs: the commercial quality leader.
• AudioLDM2: latent diffusion for music + sound effects.
• WMG-Suno settlement coverage: Nov 2025 precedent.