3D Generation

3D content is painful for four reasons:

• Representation. Meshes, point clouds, voxel grids, signed distance fields (SDFs), neural radiance fields (NeRFs), 3D Gaussians. Each one trades something away.
• Data scarcity. ImageNet has 14M images. The largest clean 3D dataset (Objaverse-XL, 2023) has ~10M objects, and most are low quality.
• Memory. A 512³ voxel grid is 128M voxels, and a useful scene NeRF needs 1M samples/ray. Generating is harder than reconstructing.
• Supervision. A 2D image gives you the pixels. For 3D you usually get a handful of 2D views and have to lift them into 3D yourself.

The 2026 stack splits the problem in two. First, generate 2D multi-view images with a diffusion model. Second, fit a 3D representation (usually Gaussian splatting) to those images.

Representation: 3D Gaussian Splatting (Kerbl et al., 2023)

A scene becomes a cloud of ~1M 3D Gaussians. Each has 59 parameters: position (3), covariance (6, or quaternion 4 + scale 3), opacity (1), spherical-harmonics color (48 at degree 3, 3 at degree 0).

Rendering is projection plus alpha-compositing. It is fast (~100 fps at 1080p on a 4090) and differentiable, so you fit it by gradient descent against ground-truth photos. A scene fits in 5-30 minutes on a consumer GPU.

Two 2023-2024 ideas build on this:

• Generative Gaussian splats. LGM predicts a Gaussian cloud directly from a few images. LRM and InstantMesh use the same feed-forward idea but output a triplane and a mesh instead (see the table below).
• 4D Gaussian Splatting. Gaussians with per-frame offsets, for dynamic scenes.

Multi-view diffusion

Fine-tune a pretrained image diffusion model so it generates several consistent views of one object from a text prompt or a single image. Examples: Zero123 (Liu et al., 2023), MVDream (Shi et al., 2023), SV3D (Stability, 2024), CAT3D (Google, 2024). These usually output 4-16 views around the object, which you lift to 3D with Gaussian splatting or a NeRF.

Text-to-3D pipelines

The 2025-2026 direction is direct text-to-mesh models with PBR materials that game engines can use. For general objects, the multi-view diffusion intermediate step remains the best-performing recipe.

NeRF (for context)

Neural Radiance Field (Mildenhall et al., 2020). A tiny MLP takes (x, y, z, view direction) and outputs (color, density), and you render by integrating along rays. Quality beats mesh-based novel-view synthesis, but rendering is 100-1000x slower. Gaussian splatting has replaced it for most real-time use, though NeRF still dominates research.

code/main.py is a toy 2D "Gaussian splatting" fit. The target is a 12x12 image with a bright blob in the upper-left and a dimmer one in the lower-right. Each splat has a position, a single sigma (so it is round, not a full covariance), and a color, and the render is the sum of all splats. The fit nudges those parameters to cut the MSE against the target. That gives you the two core operations: forward render and fit. One difference from the snippets below: main.py gets its gradients by finite differences (nudge a parameter by eps, re-render, compare the loss), because there is no autograd in it. Real 3DGS gets exact gradients through the differentiable renderer.

Step 1: 2D Gaussian splat

def gaussian_at(x, y, gaussian):
    px, py = gaussian["pos"]
    sigma = gaussian["sigma"]
    d2 = (x - px) ** 2 + (y - py) ** 2
    return math.exp(-d2 / (2 * sigma * sigma))

Step 2: render by summing splats

def render(image_size, gaussians):
    img = [[0.0] * image_size for _ in range(image_size)]
    for g in gaussians:
        for y in range(image_size):
            for x in range(image_size):
                img[y][x] += g["color"] * gaussian_at(x, y, g)
    return img

Real 3D Gaussian splatting sorts Gaussians by depth and alpha-composites them in order. The 2D toy just sums.

Step 3: fit by gradient descent

for step in range(steps):
    pred = render(size, gaussians)
    loss = mse(pred, target)
    gradients = compute_grads(pred, target, gaussians)
    update(gaussians, gradients, lr)

• View inconsistency. Generate 4 views independently and they can disagree about the object's structure, which leaves the 3D fit blurry. Fix: multi-view diffusion with shared attention.
• Back-side hallucination. Single-image → 3D has to invent the unseen side, and quality varies wildly.
• Gaussian splat explosion. Unconstrained training can grow to 10M splats and overfit. You need the densification and pruning heuristics from the original 3D-GS paper.
• Topology issues. Meshes from implicit fields (SDFs) often have holes or self-intersections. Run a remesher (e.g. Blender's voxel remesh) before shipping.
• License of training data. Objaverse has mixed licenses, and what you can use commercially varies per model.

For production 3D in a game or e-commerce pipeline, Meshy 4 and Rodin Gen-1.5 output PBR meshes that go straight into Unity / Unreal.

Save outputs/skill-3d-pipeline.md. The skill takes a 3D brief (input: text, one image, or a few images; output: mesh, splat, or NeRF; usage: render, game, or VR) and returns a pipeline (multi-view diffusion plus fit, or a direct mesh model), a base model, an iteration budget, topology post-processing, and the material channels needed.

1. Easy. In code/main.py, change the for n in [2, 4, 8] loop to use 4, 16, 64 Gaussians. Report final MSE vs target.
2. Medium. Extend to color Gaussians (RGB). Confirm reconstruction matches the target color pattern.
3. Hard. Using gsplat or Nerfstudio, reconstruct a real object from a 50-photo capture. Report fit time and final SSIM on held-out views.

Image has latent diffusion and DiT, and video has the spatiotemporal DiT. 3D has no single dominant runtime in 2026, so the production decision tree forks on the representation:

• NeRF / triplane. Inference is ray-marching plus an MLP forward per sample. A 512² render needs millions of MLP forwards, so batch the ray samples aggressively. SDPA/xformers applies.
• Multi-view diffusion + LRM reconstruction. Two stages. Stage 1 (multi-view DiT) is a diffusion server just like Lesson 07. Stage 2 (LRM transformer) is a one-shot forward pass over the views. The latency profile is "diffusion + one-shot", so pick serving primitives per stage.
• SDS / DreamFusion. This is per-asset optimization, not inference. Build jobs, not request handlers.

For most 2026 products, run a multi-view diffusion model on request, reconstruct to 3DGS asynchronously, and serve the 3DGS for real-time viewing. That splits the workload cleanly between a fast GPU-inference server and a slow offline optimizer.

• Mildenhall et al. (2020). NeRF: Representing Scenes as Neural Radiance Fields: NeRF.
• Kerbl et al. (2023). 3D Gaussian Splatting for Real-Time Radiance Field Rendering: 3DGS.
• Poole et al. (2022). DreamFusion: Text-to-3D using 2D Diffusion: SDS.
• Liu et al. (2023). Zero-1-to-3: Zero-shot One Image to 3D Object: Zero123.
• Shi et al. (2023). MVDream: multi-view diffusion.
• Hong et al. (2023). LRM: Large Reconstruction Model for Single Image to 3D: LRM.
• Gao et al. (2024). CAT3D: Create Anything in 3D with Multi-View Diffusion Models: CAT3D.
• Stability AI (2024). Stable Video 3D (SV3D): SV3D.