TAI BUI
Back to curriculum
LearnPython

Edge Inference — Apple Neural Engine, Qualcomm Hexagon, WebGPU/WebLLM, Jetson

The core edge constraint is memory bandwidth, not compute. Mobile DRAM sits at 50-90 GB/s and datacenter HBM3 clears 2-3 TB/s, a 30-50x gap, and because decode is memory-bound that gap decides throughput. In 2026 the landscape splits four ways. Apple M4/A18 Neural Engine peaks at 38 TOPS with unified memory (no CPU↔NPU copy). Qualcomm Snapdragon X Elite / 8 Gen 4 Hexagon hits 45 TOPS. WebGPU + WebLLM runs Llama 3.1 8B (Q4) at ~41 tok/s on M3 Max (roughly 70-80% of native), with 17.6k GitHub stars, an OpenAI-compatible API, and ~70-75% mobile coverage. NVIDIA Jetson Orin Nano Super (8GB) fits Llama 3.2 3B / Phi-3, AGX Orin runs gpt-oss-20b via vLLM at ~40 tok/s, and Jetson T4000 (JetPack 7.1) is 2x AGX Orin. TensorRT Edge-LLM supports EAGLE-3, NVFP4, and chunked prefill, and was shown at CES 2026 by Bosch, ThunderSoft, and MediaTek.

TIME: ~60 minutesPREREQUISITES: Phase 17 · 04 (vLLM Serving Internals), Phase 17 · 09 (Production Quantization)

Learning Objectives

  • Explain why mobile LLM inference is memory-bandwidth-bound and compute is secondary.
  • Enumerate the four edge targets (Apple ANE, Qualcomm Hexagon, WebGPU/WebLLM, NVIDIA Jetson) and match each to a use case.
  • Name the 2026 WebGPU coverage gap (Firefox Android catching up) and the Safari iOS 26 landing.
  • Pick a quantization format per target (Core ML INT4 + FP16 for ANE, QNN INT8/INT4 for Hexagon, WebGPU Q4 for browser, NVFP4 for Jetson Thor).

The Problem

A customer wants an on-device chatbot: voice-first, private by default, works offline. On a MacBook Pro M3 Max, Llama 3.1 8B Q4 runs at ~55 tok/s, which is fine. On an iPhone 16 Pro the same model runs at 3 tok/s, which is not. A mid-range Android with Snapdragon 8 Gen 3 gets 7 tok/s. In the browser via WebGPU on Chrome Android v121+, expect 4-8 tok/s depending on the device.

The variance is not a porting issue. It comes from the bandwidth gap, the quantization format, and whether the NPU is reachable from user-space at all. Edge inference in 2026 is four different problems with four different solutions.

Warm-up0/3
1. A dense LLM decodes one sequence at batch size 1 on a single accelerator. What usually caps tokens per second?Warm-up
A single accelerator producing token 1, then token 2, then token 3, then token 4 in a loop, with no network or second machine drawn.
Setup: one accelerator, one sequence, no host in the picture.
2. You requantize a model from BF16 to 4-bit weights and change nothing else. What happens to a bandwidth-bound decode ceiling?Warm-up
Two equal-size boxes side by side: a BF16 weight at 16 bits and a 4-bit weight at 4 bits, with an arrow between them and no size comparison drawn.
Setup: the two weight formats the stem names, same box size.
3. Which statement about the KV cache is correct for a long-running chat session?Warm-up
Five message bars for turns 1 through 5 of a chat, each a little wider than the last, with no memory or cache drawn.
Setup: a chat session getting longer, turn by turn.

The Concept

Bandwidth is the real ceiling

Decode reads the full set of weights for every token. A 7B model in Q4 is 3.5 GB. Reading 3.5 GB at 50 GB/s takes 70 ms, which caps you at ~14 tok/s. At 90 GB/s (high-end mobile DRAM) the cap moves to ~25 tok/s. Faster compute does not lift it.

Datacenter HBM3 at 3 TB/s clears the same 3.5 GB in 1.2 ms, a ceiling of 830 tok/s. Same model, same weights, different memory subsystem.

Apple Neural Engine (M4 / A18)

  • Up to 38 TOPS. Unified memory (CPU and ANE share the same pool), so there is no copy overhead.
  • Access via Core ML + .mlmodel compiled models, or via Metal Performance Shaders (MPS) through PyTorch.
  • The llama.cpp Metal backend uses MPS, not the ANE directly. Native ANE use requires Core ML conversion.
  • Best practical path for iOS apps in 2026: Core ML with INT4 weights + FP16 activations.

Qualcomm Hexagon (Snapdragon X Elite / 8 Gen 4)

  • Up to 45 TOPS. Integrated with CPU and GPU in the SoC but separate memory domain.
  • QNN (Qualcomm Neural Network) SDK and AI Hub provide conversion from PyTorch/ONNX.
  • Chat templates, Llama 3.2, Phi-3 all ship as first-class artifacts on AI Hub.

Intel / AMD NPUs (Lunar Lake, Ryzen AI 300)

  • 40-50 TOPS. Software lags behind Apple/Qualcomm; OpenVINO is improving but niche.
  • Best for Windows ARM copilot apps; native on AMD/Intel desktops for local-first.

WebGPU + WebLLM

  • Run models in the browser via WebGPU compute shaders, with no install.
  • Llama 3.1 8B Q4 at ~41 tok/s on M3 Max, roughly 70-80% of native via the same backend.
  • 17.6k GitHub stars on WebLLM; OpenAI-compatible JS API; Apache 2.0.
  • 2026 coverage: Chrome Android v121+, Safari iOS 26 GA, Firefox Android still catching up. Overall ~70-75% mobile coverage.

NVIDIA Jetson family

  • Orin Nano Super (8GB): fits Llama 3.2 3B, Phi-3 at good tok/s.
  • AGX Orin: runs gpt-oss-20b via vLLM at ~40 tok/s.
  • Thor / T4000 (JetPack 7.1): 2x AGX Orin performance, EAGLE-3 and NVFP4 supported.
  • TensorRT Edge-LLM (2026) supports EAGLE-3 speculative decoding, NVFP4 weights, and chunked prefill. These are the datacenter optimizations ported to edge.

Quantization choice per target

TargetFormatNotes
Apple ANEINT4 weights + FP16 activationsCore ML conversion path
Qualcomm HexagonQNN INT8 / INT4AI Hub converters
WebGPU / WebLLMQ4 MLC (q4f16_1)Use mlc_llm convert_weight + compiled .wasm; GGUF is not supported
Jetson Orin NanoQ4 GGUF or TRT-LLM INT4Memory-bound
Jetson AGX / ThorNVFP4 + FP8 KVEdge-LLM path

The long-context trap on edge

Llama 3.1's 128K context is a datacenter feature. On a phone with 8 GB RAM, 4 GB model + 2 GB KV cache for 32K tokens + OS overhead = OOM. Edge deployments keep context at 4K-8K unless you accept aggressive KV quantization (Q4 KV).

Voice is the killer app

Voice agents are latency-sensitive (first token < 500 ms), and local inference removes the network round trip from that budget. Pair it with speech-to-text (Whisper Turbo variants run on edge) and you have a production-quality voice loop.

Numbers you should remember

  • Apple M4 / A18 ANE: 38 TOPS.
  • Qualcomm Hexagon SD X Elite: 45 TOPS.
  • WebLLM M3 Max: ~41 tok/s on Llama 3.1 8B Q4.
  • AGX Orin: ~40 tok/s on gpt-oss-20b via vLLM.
  • Datacenter-edge bandwidth gap: 30-50x.
  • WebGPU mobile coverage: ~70-75% (Firefox Android lagging).
Check: The Concept0/28
1. What is the Apple Neural Engine (ANE)?Define
A chip boundary labelled Apple M4 / A18 containing one highlighted block, Neural Engine, rated at 38 TOPS.
Setup: the Apple Neural Engine's real 2026 spec, from the lesson's Numbers you should remember.
2. What is Hexagon in Qualcomm's chips?Define
A chip boundary labelled Snapdragon X Elite / 8 Gen 4 containing one highlighted block, Hexagon NPU, rated at 45 TOPS.
Setup: the Hexagon NPU's real 2026 spec, from the lesson's Numbers you should remember.
3. What is WebGPU?Define
A three-step chain: a web page running JavaScript, then WebGPU, then GPU hardware, with no browser names or coverage figures shown.
Setup: where WebGPU sits between a web page and the GPU.
4. What is WebLLM?Define
A browser tab running a chatbot locally, labelled with its real 2026 numbers, 17.6k GitHub stars and an Apache 2.0 license.
Setup: the project's real open-source numbers from the lesson.
5. What is NVIDIA Jetson?Define
One real Jetson board, AGX Orin, labelled with its memory bandwidth, 205 GB/s, from main.py's own target table.
Setup: one real Jetson board's spec, from the lesson's simulator.
6. What does unified memory mean for an NPU?Define
The CPU and the Neural Engine on the same chip, each with a line running down into one RAM pool box below them.
Setup: how the CPU and ANE connect to RAM, on the real M4 chip.
7. What does it mean to call decode bandwidth-bound?Define
An arrow from a box labelled all weights in memory into a box labelled one decode step, with a note that the read repeats for every token.
Setup: one decode step's read from memory, repeated every token.
8. What is Core ML?Define
A three-step chain: a PyTorch model converted to a Core ML .mlmodel file, which then runs on the ANE.
Setup: the real Apple conversion path named in the lesson.
9. What is QNN?Define
A three-step chain: a PyTorch or ONNX model converted through QNN and AI Hub, which then runs on Hexagon.
Setup: the real Qualcomm conversion path named in the lesson.
10. Which set of datacenter optimizations does TensorRT Edge-LLM bring to Jetson-class devices?Define
TensorRT Edge-LLM, shown at CES 2026 by Bosch, ThunderSoft and MediaTek, running on Jetson-class devices, with a box asking which datacenter optimizations it ports over.
Setup: the 2026 announcement context. The ported techniques are left out.
11. Why does Apple's Neural Engine avoid CPU-to-NPU copy overhead?Explain
A typical NPU with a copy step drawn between CPU and NPU, above Apple's ANE path with the same two boxes but a question mark where the copy step would be.
Setup: a typical separate-memory NPU's extra step, next to Apple's path with that step left unresolved.
12. A vendor doubles an NPU's TOPS but keeps the same memory system. What happens to decode speed for a 4.7 GB model?Explain
Two NPU generations, 40 TOPS and 80 TOPS, both using the same memory system, with a question about decode speed for the same model.
Setup: a doubled TOPS figure, illustrative, with the memory system unchanged.
13. Why does the same 3.5 GB model have a decode ceiling near 830 tok/s on datacenter HBM3 but near 14 tok/s at 50 GB/s?Explain
One 3.5 GB weight block read over two different memory buses, 3 TB/s and 50 GB/s, with the read time left as a question for each.
Setup: the same weights, two real memory bus speeds from the lesson.
14. Roughly what share of mobile devices can run WebGPU in 2026?Explain
A single blank meter box with a question mark inside it, with no browser names and no filled fraction, asking for the share of mobile devices.
Setup: the question mark stands for the share. No browser breakdown is shown.
15. A web app needs WebLLM on phones. Which mobile browsers does the lesson say support it in 2026?Explain
Three mobile browsers, Chrome Android, Safari iOS and Firefox Android, each marked support unknown, with no browser singled out.
Setup: the three browsers the stem names. Support status is left open for each.
16. Why does running llama.cpp's Metal backend on an iPhone not mean the model runs on the ANE?Explain
llama.cpp running its Metal backend on an iPhone, with whether that path reaches the ANE left as a question mark.
Setup: the path the stem describes. Whether it reaches the ANE is left open.
17. One product needs gpt-oss-20b at about 40 tok/s and another only needs a 3B model. Which Jetson boards does the lesson match to them?Explain
Two products: product A needing gpt-oss-20b at about 40 tok/s, product B needing only a 3B-class model. Which board fits each is left open.
Setup: the two needs from the stem. No board is named yet.
18. Why is voice called the killer app for edge inference?Explain
Two paths for a voice agent: speech, network, server, reply for the cloud path, and speech, on-device, reply for the local path, with no timing numbers shown.
Setup: the two paths a voice agent's audio can take, hop by hop.
19. A team converts a PyTorch Llama for a Snapdragon 8 Gen 4 phone through Qualcomm AI Hub. Which format does the lesson's table pick?Apply
A PyTorch Llama converted through Qualcomm AI Hub for a Snapdragon 8 Gen 4 phone, with the output format left as a question mark.
Setup: the conversion path the stem names. The target format is left open.
20. A 7B model in Q4 is 3.5 GB and the device reads memory at 50 GB/s. What decode ceiling does the worked example give?Apply
The worked example's two given numbers, 3.5 GB and 50 GB/s, in the ceiling formula, with the result left as a question mark.
Setup: the two numbers the stem gives. The decode ceiling is left open.
21. A 7B Q4 model (3.5 GB) runs on a Snapdragon 8 Gen 3 at 77 GB/s. What is the bandwidth ceiling for decode?Apply
The same 3.5 GB model, this time at 77 GB/s, in the ceiling formula, with the result left as a question mark.
Setup: the stem's own numbers for this device. The decode ceiling is left open.
22. A product ships on a Jetson Orin Nano (8GB), where decode is memory-bound. Which weight format does the lesson's table list for it?Apply
Jetson Orin Nano, 8GB, noted as memory-bound, with a box asking which weight format it uses. No other Jetson board is shown.
Setup: one board from the quantization table. The format is left open.
23. You plan an 8 GB phone app that keeps 32K tokens of chat history on a 4 GB model. What is the most likely outcome, and what is the fix?Analyze
A memory bar drawn to scale: weights at 4 GB and KV cache for 32K tokens at about 2 GB already reach 6 of the 8 GB line, the 8 GB capacity marked with a dashed line, and a text note that OS overhead adds further on top, its size not drawn to scale.
Setup: the stem's own given sizes, drawn to scale. OS overhead is noted but not sized. The outcome and the fix are left open.
24. A developer converts Llama to GGUF Q4_K_M and loads it into WebLLM, but it fails. What was the wrong assumption?Analyze
A GGUF Q4_K_M model being loaded into WebLLM, with the reason it fails left as a question mark.
Setup: the attempted format and the failure. The reason is left open.
25. You ship one assistant to iPhone, Snapdragon Android and Chrome. Which format per target fits the lesson?Design
Three targets, iPhone ANE, Snapdragon Android and Chrome WebGPU, each marked format unknown, in no particular order.
Setup: the three targets from the stem. No format is named yet.
26. A robotics product targets Jetson Thor and wants a 20B-class model at the best speed. Which stack fits the lesson?Design
Jetson Thor targeted for a 20B-class model at the best speed, with a box asking which stack fits. No other Jetson board is shown.
Setup: one board from the quantization table. The stack is left open.
27. Your WebLLM chatbot must also serve users whose browsers lack WebGPU. Which fallback fits?Design
A WebLLM app meeting a browser without WebGPU, with the fallback left as a question mark.
Setup: the stem's own scenario. The fallback is left open.
28. A product must offer 32K context on an 8 GB phone anyway. Which compromise does the lesson allow?Design
32K context required on an 8 GB phone regardless, with a box asking which compromise the lesson allows.
Setup: the requirement from the stem. The compromise is left open.

Use It

code/main.py computes theoretical decode throughput ceilings from bandwidth-bound math across nine edge targets, using Llama 3.1 8B Q4 at 4.7 GB. It prints each ceiling next to an observed benchmark and an efficiency percentage, then reruns the math on an iPhone 16 for BF16, INT8, Q4, and Q3 weights. Where observed throughput sits far below the ceiling, the runtime is the problem. Where it sits near the ceiling, bandwidth is.

Check: Use It0/9
1. With `ceiling()` and `model_gb = 4.7`, what does `main.py` compute for the Apple A18 (iPhone 16) at 60 GB/s?Apply
The real ceiling() function from main.py, with the Apple A18 iPhone 16 row's bandwidth, 60 GB/s, and model_gb 4.7 given, result left as a question mark.
Code copied from the lesson's main.py.
2. In the quantization loop, `main.py` sets the iPhone 16 bandwidth to 60 GB/s. What ceiling does it print for BF16 weights at 18.8 GB?Apply
The real quantization loop from main.py, with iphone_bw 60 and the BF16 row's size, 18.8 GB, given, result left as a question mark. Other rows are not shown.
Code copied from the lesson's main.py. Scoped to the BF16 row only.
3. The Q3 GGUF row in the `main.py` loop uses 3.6 GB on the iPhone 16 at 60 GB/s. What ceiling does it print?Apply
The real quantization loop from main.py, with iphone_bw 60 and the Q3 GGUF row's size, 3.6 GB, given, result left as a question mark. Other rows are not shown.
Code copied from the lesson's main.py. Scoped to the Q3 GGUF row only.
4. `main.py` gives the Apple M3 Max (400 GB/s) a ceiling near 85 tok/s on a 4.7 GB model and lists 55 observed. What efficiency does `efficiency()` print?Apply
The Apple M3 Max row's ceiling, about 85 tok/s, and observed figure, 55 tok/s, with efficiency left as a question mark.
Setup: the row's two given figures. Efficiency is left open.
5. `main.py` lists the H100 at 3350 GB/s with a 4.7 GB model. Which ceiling does `ceiling()` return?Apply
The H100 row's bandwidth, 3350 GB/s, and model_gb 4.7, passed into ceiling(), with the result left as a question mark.
Setup: this row's two given figures. The observed figure is not shown.
6. An iPhone 16 (60 GB/s) runs Llama 3.1 8B Q4 (4.7 GB) at 8 tok/s. A teammate wants a faster NPU. What do you recommend?Analyze
An iPhone 16 observed at 8 tok/s next to a teammate's suggestion to buy a faster NPU, with a question about what is actually limiting the device.
Setup: the device's observed figure and the teammate's proposal. The ceiling is left out.
7. `main.py` shows the H100 at only about 24% of its ceiling, the lowest efficiency in the table. What is the most reasonable reading?Analyze
The H100 row's ceiling, observed figure and resulting efficiency, about 24 percent, with a box asking for the most reasonable reading of that gap.
Setup: the row's figures, already given in the lesson's own Author's Note.
8. The AGX Orin row has 45 tok/s observed against a ceiling near 43.6, so efficiency prints 103%. What should you conclude?Analyze
The AGX Orin row's ceiling, observed figure, and efficiency printed as 103 percent, with a box asking what should be concluded from a ratio above 100.
Setup: the row's figures, as given in the stem.
9. Users on an 8K-context phone chat see lower speed than the ceiling `main.py` prints for the same model. Which omission in the script explains the gap?Analyze
The real ceiling() function from main.py, which takes only model_gb and bandwidth, next to a note that real 8K-context speed sits below what it prints.
Setup: the formula's only two inputs. What it omits is left open.

Ship It

This lesson produces outputs/skill-edge-target-picker.md. Given platform (iOS/Android/browser/Jetson), model, and latency/memory budget, picks a quantization format and conversion pipeline.

Author's Note

I'd run main.py and read the efficiency column before anything else. The ceiling is just model_gb / bandwidth_gb_s inverted, so the interesting part is how far each target lands below it. The H100 row is lowest at 24%, then WebGPU on Pixel 9 at 37%, Snapdragon 8 Gen 3 at 43%, and WebGPU on M3 Max at 48%. For the browser and Android rows I'd look at the runtime before blaming the hardware.

Don't take the table as measured truth. The Jetson AGX Orin row prints 103%, which is above its own ceiling, so one of the bandwidth or observed figures is loose. The script also prices only the weights. It ignores the KV cache, which is the same blind spot the long-context section describes, so on a phone the real ceiling at 4K or 8K context is lower than what it prints.

One mismatch to watch for: the script uses an 8B model at 4.7 GB, while the worked example in the concept section uses a 7B at 3.5 GB. Both are Q4, and the arithmetic is the same. My bet is that the quantization loop at the bottom is the most useful output, because it shows on one device how much of the ceiling you buy back by shrinking the weights.

Exercises

  1. Run code/main.py. For a 7B model in Q4 on a Snapdragon 8 Gen 3 (~77 GB/s bandwidth), compute the decode ceiling. Compare to observed 6-8 tok/s. Is the runtime efficient?
  2. WebGPU on Android requires Chrome v121+. Design a fallback for older browsers: server-side via the same OpenAI-compatible API.
  3. Your iOS app needs 4K-context streaming. Which model/format combination lets you stay under 4 GB active memory on an iPhone 16?
  4. Jetson AGX Orin runs gpt-oss-20b at 40 tok/s. Jetson Nano fits only a 3B. If your product targets both, how do you unify the inference stack?
  5. Argue whether "WebLLM is production-ready in 2026." Cite the coverage, performance, and the Firefox Android gap.

Key Terms

TermWhat people sayWhat it actually means
ANE"Apple neural engine"On-device NPU in M-series and A-series; unified memory
Hexagon"Qualcomm NPU"Snapdragon NPU; QNN SDK for access
WebGPU"browser GPU"W3C-standardized browser GPU API; Chrome/Safari 2026
WebLLM"browser LLM runtime"MLC-LLM project; Apache 2.0; OpenAI-compatible JS
Jetson"NVIDIA edge"Orin Nano / AGX / Thor / T4000 family
TRT Edge-LLM"edge TensorRT"2026 edge port of TensorRT-LLM; EAGLE-3 + NVFP4
Unified memory"shared pool"CPU and NPU see same RAM; no copy overhead
Bandwidth-bound"memory limited"Decode gated by bytes/sec reading weights
Core ML"Apple conversion"Apple framework for ANE-native models
QNN"Qualcomm stack"Qualcomm Neural Network SDK

Further Reading

Code

Ships With