3D Vision — Point Clouds & NeRFs

A camera gives you a 2D image. A LIDAR gives you a set of 3D points with no ordering at all. A structure-from-motion pipeline gives you a sparse cloud of 3D keypoints, and a NeRF reconstructs an entire scene from a handful of posed photos. Call all four of those "vision" and you'd be right, but none of them look anything like the dense grid a CNN expects.

That mismatch matters because almost every high-value robotics task runs in 3D: grasping, obstacle avoidance, navigation, AR occlusion, capturing real-world content. If you only know how to process 2D images, you're locked out of the fastest-growing part of the field: AR/VR content, robotics, autonomous driving stacks, NeRF-based reconstruction for real estate or construction.

Two representations dominate, and for different reasons. Point clouds are what sensors hand you for free. NeRFs, and everything downstream of them like 3D Gaussian splatting and neural SDFs, are what you get when you ask a neural network to learn the scene instead.

Explicit vs. implicit representations

Every 3D representation is either explicit or implicit. An explicit representation stores geometry directly: a point cloud is a literal list of 3D points, a mesh is a set of vertices connected into triangular faces, and a voxel grid is a 3D array of cells marking occupied space. Query any of them and you get back exactly what was stored, nothing more. An implicit representation stores a function instead: a signed distance field (SDF) returns the distance from any query point to the nearest surface, and a NeRF returns a density and colour for any query point and viewing direction. You don't read geometry off an implicit representation, you query it.

The trade-off is memory against resolution. A point cloud or voxel grid is bounded by however many points or cells got stored, so more detail costs more memory, and any gap between samples is just a gap. A function has no such ceiling: an SDF or a NeRF MLP can be queried at any resolution, including one finer than anything it trained on, because the function is continuous rather than a fixed set of samples. That continuity is also why implicit representations are slower to read from. Rendering a pixel means running a network, not indexing an array.

Meshes and voxel grids matter mostly for context here. The rest of this lesson works the two ends readers hit most often in practice: point clouds, what a sensor hands you directly, and NeRFs, what a network learns to fill in when you only have photos.

Point clouds

A point cloud is an unordered set of N points in R^3, optionally each with features (colour, intensity, normal).

cloud = [
  (x1, y1, z1, r1, g1, b1),
  (x2, y2, z2, r2, g2, b2),
  ...
  (xN, yN, zN, rN, gN, bN),
]

No grid, no connectivity. Two properties make this hard for neural networks:

• Permutation invariance: the output must not depend on point order.
• Variable N: a single model must handle clouds of different sizes.

PointNet (Qi et al., 2017) solved both with one idea: apply a shared MLP to every point, then aggregate with a symmetric function (max pool). The result is a fixed-size vector that does not depend on order.

f(P) = max_{p in P} MLP(p)

This is the entire core of PointNet. Deeper variants (PointNet++, Point Transformer) add hierarchical sampling and local aggregation, but the symmetric-function trick never changes.

The PointNet architecture

"Shared MLP" means the same MLP runs on every point independently, implemented in practice as a 1x1 conv over the point dimension for speed.

Neural Radiance Fields (NeRFs)

A NeRF is a neural network that stores a 3D scene as a function: it maps a 3D position and viewing direction, (x, y, z, viewing_direction), to a density and a colour, (density, colour). Mildenhall et al. (2020) built it to answer one question: can you reconstruct a 3D scene from a handful of photos? Rendering a new view means running a ray-casting loop over that function.

NeRF MLP:  (x, y, z, theta, phi) -> (sigma, r, g, b)

To render a pixel (u, v) of a new view:
  1. Cast a ray from the camera through pixel (u, v)
  2. Sample points along the ray at distances t_1, t_2, ..., t_N
  3. Query the MLP at each point
  4. Composite the colours weighted by (1 - exp(-sigma * dt))
  5. The sum is the rendered pixel colour

A loss compares the rendered pixel to the ground-truth pixel in the training photos. Backprop through the rendering step updates the MLP. No 3D ground truth, no explicit geometry: the scene lives in the MLP's weights.

Positional encoding in NeRF

Positional encoding maps each coordinate into a Fourier feature vector, sine and cosine at multiple frequencies, before it ever reaches the MLP:

gamma(p) = (sin(2^0 pi p), cos(2^0 pi p), sin(2^1 pi p), cos(2^1 pi p), ...)

NeRF needs this because a plain MLP fed raw (x, y, z) values can't represent high-frequency detail: MLPs are spectrally biased toward low frequencies, so without the encoding, renders come out blurry. NeRF uses up to L=10 frequency levels. Transformers use the same trick for positions, and it shows up again in diffusion time conditioning (Lesson 10).

Volumetric rendering

C(r) = sum_i T_i * (1 - exp(-sigma_i * delta_i)) * c_i

T_i  = exp(- sum_{j<i} sigma_j * delta_j)
delta_i = t_{i+1} - t_i

T_i is transmittance: how much light survives to reach point i. (1 - exp(-sigma_i * delta_i)) is the opacity at point i, and c_i is its colour. The final pixel is a weighted sum along the ray.

What replaced NeRFs

Pure NeRFs are slow to train (hours) and slow to render (seconds per image). The lineage since:

• Instant-NGP (2022): hash-grid encoding replaces the MLP's position input; trains in seconds.
• Mip-NeRF 360: handles unbounded scenes and anti-aliasing.
• 3D Gaussian Splatting (2023): replaces the volumetric field with millions of 3D Gaussians; trains in minutes, renders in real time. The current production default.

Almost every NeRF product shipping in 2026 is actually 3D Gaussian splatting under the hood. The mental model is still NeRF.

Datasets and benchmarks

• ShapeNet: classification and segmentation of 3D CAD models as point clouds.
• ScanNet: real indoor scans for segmentation.
• KITTI: outdoor LIDAR point clouds for autonomous driving.
• NeRF Synthetic / Blended MVS: posed-image datasets for view synthesis.
• Mip-NeRF 360 dataset: unbounded real scenes.

Step 1: PointNet classifier

Same shared-MLP-then-max-pool idea from the Concept section, written out as a full classifier:

import torch
import torch.nn as nn
 
class PointNet(nn.Module):
    def __init__(self, num_classes=10):
        super().__init__()
        self.mlp1 = nn.Sequential(
            nn.Conv1d(3, 64, 1),    nn.BatchNorm1d(64),   nn.ReLU(inplace=True),
            nn.Conv1d(64, 64, 1),   nn.BatchNorm1d(64),   nn.ReLU(inplace=True),
        )
        self.mlp2 = nn.Sequential(
            nn.Conv1d(64, 128, 1),  nn.BatchNorm1d(128),  nn.ReLU(inplace=True),
            nn.Conv1d(128, 1024, 1), nn.BatchNorm1d(1024), nn.ReLU(inplace=True),
        )
        self.head = nn.Sequential(
            nn.Linear(1024, 512),   nn.BatchNorm1d(512),  nn.ReLU(inplace=True),
            nn.Dropout(0.3),
            nn.Linear(512, 256),    nn.BatchNorm1d(256),  nn.ReLU(inplace=True),
            nn.Dropout(0.3),
            nn.Linear(256, num_classes),
        )
 
    def forward(self, x):
        # x: (N, 3, num_points) — transposed for Conv1d
        x = self.mlp1(x)
        x = self.mlp2(x)
        x = torch.max(x, dim=-1)[0]       # (N, 1024)
        return self.head(x)
 
pts = torch.randn(4, 3, 1024)
net = PointNet(num_classes=10)
print(f"output: {net(pts).shape}")
print(f"params: {sum(p.numel() for p in net.parameters()):,}")

About 1.6M parameters. Runs on 1,024 points per cloud.

Step 2: Positional encoding

The Fourier-feature transform from the Concept section, as actual code:

def positional_encoding(x, L=10):
    """
    x: (..., D) -> (..., D * 2 * L)
    """
    freqs = 2.0 ** torch.arange(L, dtype=x.dtype, device=x.device)
    args = x.unsqueeze(-1) * freqs * 3.141592653589793
    sinc = torch.cat([args.sin(), args.cos()], dim=-1)
    return sinc.reshape(*x.shape[:-1], -1)
 
x = torch.randn(5, 3)
y = positional_encoding(x, L=10)
print(f"input:  {x.shape}")
print(f"encoded: {y.shape}     # (5, 60)")

Multiplying by 2^l * pi gives progressively higher frequencies.

Step 3: Tiny NeRF MLP

Positional encoding feeding straight into the density-and-colour MLP described above:

class TinyNeRF(nn.Module):
    def __init__(self, L_pos=10, L_dir=4, hidden=128):
        super().__init__()
        self.L_pos = L_pos
        self.L_dir = L_dir
        pos_dim = 3 * 2 * L_pos
        dir_dim = 3 * 2 * L_dir
        self.trunk = nn.Sequential(
            nn.Linear(pos_dim, hidden), nn.ReLU(inplace=True),
            nn.Linear(hidden, hidden),  nn.ReLU(inplace=True),
            nn.Linear(hidden, hidden),  nn.ReLU(inplace=True),
            nn.Linear(hidden, hidden),  nn.ReLU(inplace=True),
        )
        self.sigma = nn.Linear(hidden, 1)
        self.color = nn.Sequential(
            nn.Linear(hidden + dir_dim, hidden // 2), nn.ReLU(inplace=True),
            nn.Linear(hidden // 2, 3), nn.Sigmoid(),
        )
 
    def forward(self, x, d):
        x_enc = positional_encoding(x, self.L_pos)
        d_enc = positional_encoding(d, self.L_dir)
        h = self.trunk(x_enc)
        sigma = torch.relu(self.sigma(h)).squeeze(-1)
        rgb = self.color(torch.cat([h, d_enc], dim=-1))
        return sigma, rgb
 
nerf = TinyNeRF()
x = torch.randn(128, 3)
d = torch.randn(128, 3)
s, c = nerf(x, d)
print(f"sigma: {s.shape}   rgb: {c.shape}")

Tiny next to the original NeRF, which runs two MLP trunks eight layers deep, but it's enough to see the architecture clearly.

Step 4: Volumetric rendering along a ray

The compositing formula from the Volumetric rendering section, run over one ray:

def volumetric_render(sigma, rgb, t_vals):
    """
    sigma: (..., N_samples)
    rgb:   (..., N_samples, 3)
    t_vals: (N_samples,) distances along the ray
    """
    delta = torch.cat([t_vals[1:] - t_vals[:-1], torch.full_like(t_vals[:1], 1e10)])
    alpha = 1.0 - torch.exp(-sigma * delta)
    trans = torch.cumprod(torch.cat([torch.ones_like(alpha[..., :1]), 1.0 - alpha + 1e-10], dim=-1), dim=-1)[..., :-1]
    weights = alpha * trans
    rendered = (weights.unsqueeze(-1) * rgb).sum(dim=-2)
    depth = (weights * t_vals).sum(dim=-1)
    return rendered, depth, weights
 
 
N = 64
t_vals = torch.linspace(2.0, 6.0, N)
sigma = torch.rand(N) * 0.5
rgb = torch.rand(N, 3)
rendered, depth, weights = volumetric_render(sigma, rgb, t_vals)
print(f"rendered colour: {rendered.tolist()}")
print(f"depth:           {depth.item():.2f}")

One ray, 64 samples, composite to a single RGB pixel and a depth.

For real work:

• nerfstudio (Tancik et al.): the current reference library for NeRF / Instant-NGP / Gaussian Splatting, with a command-line workflow plus a web viewer.
• pytorch3d (Meta): differentiable rendering, point-cloud utilities, mesh ops.
• open3d: point cloud processing, registration, visualisation.

For deployment, 3D Gaussian splatting has largely replaced pure NeRFs: it renders about 100x faster, and the reconstruction quality holds up just as well.

This lesson produces:

• outputs/prompt-3d-task-router.md, a prompt that routes to the right 3D representation (point cloud, mesh, voxel, NeRF, Gaussian splat) based on task and input data.
• outputs/skill-point-cloud-loader.md, a skill that writes a PyTorch Dataset for .ply / .pcd / .xyz files with correct normalisation, centring, and point sampling.

1. (Easy) Show that PointNet is permutation-invariant: run the same cloud through twice, once with points shuffled. Verify outputs are identical up to floating-point noise.
2. (Medium) Implement a minimal ray-generation function that, given camera intrinsics and pose, produces ray origins and directions for every pixel of an H x W image.
3. (Hard) Train a TinyNeRF on a synthetic dataset of rendered views of a coloured cube (generated via differentiable rendering or a simple ray tracer). Report rendering loss at epoch 1, 10, and 100. At what epoch does the model produce recognisable views?

• PointNet (Qi et al., 2017): the permutation-invariant classifier
• NeRF (Mildenhall et al., 2020): the paper that made 3D reconstruction from photos a neural-net problem
• Instant-NGP (Müller et al., 2022): hash grids, 1000x speedup
• 3D Gaussian Splatting (Kerbl et al., 2023): the architecture that replaced NeRFs in production