Audio Fundamentals — Waveforms, Sampling, Fourier Transform

A microphone produces a pressure-vs-time signal. Your neural net consumes tensors. What sits between them is a stack of conventions, and violating any one of them produces a silent bug: the model trains fine but the WER (word error rate, the standard ASR accuracy metric) doubles, TTS ships a hiss, or a voice-cloning system memorizes the microphone instead of the speaker.

Every bug in speech systems traces back to one of three questions:

1. What sample rate was the data recorded at, and what does the model expect?
2. Is the signal aliased?
3. Are you operating on raw samples or a frequency representation?

Get these three right and the rest of Phase 6 is tractable. Get any one wrong and even Whisper-Large-v4 produces garbage.

Waveform. A one-dimensional array of floats in [-1.0, 1.0], indexed by sample number. Divide the index by the sample rate to convert to seconds: t = n / sr. A 10-second clip at 16 kHz is an array of 160,000 floats.

Sampling rate (sr). How many samples per second. Common rates in 2026:

Nyquist-Shannon. A sample rate of sr can unambiguously represent frequencies up to sr/2, a boundary known as the Nyquist frequency. Energy above it gets aliased: folded down into lower frequencies, corrupting the signal. Always low-pass filter before downsampling.

Bit depth. 16-bit PCM (signed int16, range ±32,767) is the universal exchange format. Music runs 24-bit; internal DSP runs 32-bit float. Libraries like soundfile read int16 but expose float32 arrays in [-1, 1].

Fourier Transform. Any finite signal is a sum of sinusoids at different frequencies. The Discrete Fourier Transform (DFT) turns N samples into N complex coefficients, one per frequency bin. bin k maps to frequency k · sr / N Hz: magnitude is amplitude at that frequency, angle is phase.

FFT. Fast Fourier Transform: an O(N log N) algorithm for the DFT when N is a power of 2, and the one every audio library actually uses under the hood. A 1024-sample FFT at 16 kHz gives 512 usable frequency bins spanning 0–8 kHz at 15.6 Hz resolution.

Framing + window. You don’t FFT an entire clip. Instead, chop it into overlapping frames (typically 25 ms with 10 ms hop), multiply each frame by a window function (Hann, Hamming) to kill edge discontinuities, then FFT each frame. This is the Short-Time Fourier Transform (STFT). Lesson 02 picks up from here.

Step 1: read a clip and plot the waveform

code/main.py uses only the stdlib wave module to keep the demo dependency-free. For production you will use soundfile or torchaudio.load (both return (waveform, sr) tuples):

import soundfile as sf
waveform, sr = sf.read("clip.wav", dtype="float32")  # shape (T,), sr=int

Step 2: synthesize a sine wave from first principles

import math
 
def sine(freq_hz, sr, seconds, amp=0.5):
    n = int(sr * seconds)
    return [amp * math.sin(2 * math.pi * freq_hz * i / sr) for i in range(n)]

A 440 Hz sine (concert A) at 16 kHz for 1 second is 16,000 floats. Write it out with wave.open(..., "wb") using 16-bit PCM encoding.

Step 3: compute the DFT by hand

def dft(x):
    N = len(x)
    out = []
    for k in range(N):
        re = sum(x[n] * math.cos(-2 * math.pi * k * n / N) for n in range(N))
        im = sum(x[n] * math.sin(-2 * math.pi * k * n / N) for n in range(N))
        out.append((re, im))
    return out

O(N²): fine to confirm correctness at N=256, useless for real audio. Real code calls numpy.fft.rfft or torch.fft.rfft.

Step 4: find the dominant frequency

Magnitude peak index k_star maps to frequency k_star * sr / N. Running this on the 440 Hz sine should return a peak at bin 440 * N / sr.

Step 5: demonstrate aliasing

Sample a 7 kHz sine at 10 kHz (Nyquist = 5 kHz). The 7 kHz tone is above Nyquist and folds to 10 − 7 = 3 kHz. The FFT peak appears at 3 kHz. This is the classic aliasing demo, and the reason every DAC/ADC ships with a brick-wall low-pass filter.

The stack you’ll actually ship in 2026:

Decision rule: match sample rate before you match anything else. Whisper expects 16 kHz mono float32. Pass it 44.1 kHz stereo and you’ll get garbage that looks like a model bug.

Save it as outputs/skill-audio-loader.md: a skill that checks whether your audio input matches what the downstream model expects, and resamples it correctly when it doesn’t.

1. Easy. Synthesize a 1-second mix of 220 Hz + 440 Hz + 880 Hz at 16 kHz. Run DFT. Confirm three peaks at the expected bins.
2. Medium. Record a 3-second WAV of your voice at 48 kHz. Downsample to 16 kHz using torchaudio.transforms.Resample (with anti-aliasing), then to 16 kHz using naive decimation (every third sample). FFT both. Where does the aliasing appear?
3. Hard. Build the STFT from scratch using only math and the DFT from Step 3. Frame size 400, hop 160, Hann window. Plot magnitudes with matplotlib.pyplot.imshow. This is the spectrogram of Lesson 02.

• Shannon (1949). Communication in the Presence of Noise: the paper behind the sampling theorem.
• Smith: The Scientist and Engineer's Guide to Digital Signal Processing: free, canonical DSP textbook.
• librosa docs: audio primer: practical walkthrough with code.
• Heinrich Kuttruff: Room Acoustics (6th ed.): reference for why real-world audio is not a clean sinusoid.
• Steve Eddins: FFT Interpretation notebook: frequency bin intuition cleared up in 10 minutes.