Monocular Depth & Geometry Estimation

Depth is the missing axis in 2D computer vision. RGB tells you where something sits in the image plane, not how far away it is. Depth sensors (stereo rigs, LiDAR, time-of-flight) solve that directly, but they're expensive, fragile, and limited in range.

Monocular depth estimation, predicting depth from a single RGB frame, used to produce blurry, unreliable output. By 2026 large pretrained encoders changed that. Depth Anything V3 uses a frozen DINOv2 backbone and produces depth maps that generalise across indoor, outdoor, medical, and satellite domains. Marigold reframes depth as a conditional diffusion problem, and ZoeDepth regresses true metric distances.

Depth is also the bridge between 2D detection and 3D understanding: multiply a detected box's pixels by its depth value and you've lifted a 2D object into a 3D point cloud. That's the core of every AR occlusion system, every obstacle-avoidance pipeline, and every "pick up the cup" robot.

Relative vs metric depth

• Relative depth: ordered z values without a real-world unit. "Pixel A is closer than pixel B" holds, but the ratio between them isn't anchored to metres.
• Metric depth: absolute distance in metres from the camera. The model has to have learnt the statistical relationship between image cues and real-world distance.

MiDaS and Depth Anything V3 produce relative depth, and so does Marigold. ZoeDepth, UniDepth, and Metric3D produce metric depth instead. Metric models are sensitive to camera intrinsics; relative models aren't.

The encoder-decoder pattern

Depth Anything V3 freezes the encoder and trains only a DPT-style decoder (DPT: Dense Prediction Transformer, a convolutional head built for exactly this job). The encoder supplies rich features, and the decoder interpolates them back to image resolution and regresses depth.

Why a single image produces depth at all

A 2D image carries several monocular cues that correlate with depth:

• Perspective: parallel lines in 3D converge in 2D.
• Texture gradient: surfaces far away have smaller, denser texture.
• Occlusion order: nearer objects occlude farther ones.
• Size constancy: known objects (cars, humans) give approximate scale.
• Atmospheric perspective: distant objects look hazier and bluer in outdoor scenes.

A ViT trained on billions of images internalises these cues on its own. Given enough data and a strong backbone, monocular depth hits reasonable accuracy with no explicit 3D supervision at all.

What monocular depth cannot do

• Absolute metric scale without intrinsics or a known object in the scene. The network can predict "the cup is twice as far as the spoon" without knowing whether the cup is 1 m or 10 m away.
• Occluded geometry: the back of a chair is unseen and can't be inferred reliably.
• Truly untextured or reflective surfaces: mirrors, glass, uniform walls. The network reports a plausible depth, and it's wrong.

Depth Anything V3 in 2026

• Vanilla DINOv2 ViT-L/14 as encoder (frozen).
• DPT decoder.
• Trained on posed image pairs from diverse sources (no explicit depth supervision needed beyond photometric consistency).
• Predicts spatially consistent geometry from an arbitrary number of visual inputs, with or without known camera poses.
• SOTA across monocular depth, any-view geometry, visual rendering, camera pose estimation.

This is the model to reach for by default when you need depth in 2026.

Marigold: diffusion for depth

Marigold (Ke et al., CVPR 2024) reframes depth estimation as conditional image-to-image diffusion: it conditions on RGB and denoises toward a depth map, using a pretrained Stable Diffusion 2 U-Net as its backbone. The payoff is depth maps that are exceptionally sharp at object boundaries. The cost is slower inference than feed-forward models: 10-50 denoising steps instead of one forward pass.

Intrinsics and the pinhole camera

To lift a pixel (u, v) with depth d to a 3D point (X, Y, Z) in camera coordinates:

fx, fy, cx, cy = camera intrinsics
X = (u - cx) * d / fx
Y = (v - cy) * d / fy
Z = d

Intrinsics come from EXIF metadata, a calibration pattern, or a monocular intrinsics estimator (Perspective Fields, UniDepth). Without intrinsics, you can still render a point cloud by assuming a 60-70° FOV and moderate-resolution principals. That's fine for visualisation, but not for measurement.

Evaluation

Two standard metrics:

• AbsRel (absolute relative error): mean(|d_pred - d_gt| / d_gt). Lower is better. Production models land around 0.05-0.1.
• delta < 1.25 (threshold accuracy): fraction of pixels where max(d_pred/d_gt, d_gt/d_pred) < 1.25. Higher is better. SOTA models clear 0.9+.

For relative depth (Depth Anything V3, MiDaS), evaluation uses scale-and-shift invariant versions of both metrics.

Step 1: Depth metrics

import torch
 
def abs_rel_error(pred, target, mask=None):
    if mask is not None:
        pred = pred[mask]
        target = target[mask]
    return (torch.abs(pred - target) / target.clamp(min=1e-6)).mean().item()
 
 
def delta_accuracy(pred, target, threshold=1.25, mask=None):
    if mask is not None:
        pred = pred[mask]
        target = target[mask]
    ratio = torch.maximum(pred / target.clamp(min=1e-6), target / pred.clamp(min=1e-6))
    return (ratio < threshold).float().mean().item()

Always mask invalid depth pixels (zero, NaN, saturated) before evaluation.

Step 2: Scale-and-shift alignment

For relative-depth models, align prediction to ground truth before computing metrics. Least-squares fit of a * pred + b = target:

def align_scale_shift(pred, target, mask=None):
    if mask is not None:
        p = pred[mask]
        t = target[mask]
    else:
        p = pred.flatten()
        t = target.flatten()
    A = torch.stack([p, torch.ones_like(p)], dim=1)
    coeffs, *_ = torch.linalg.lstsq(A, t.unsqueeze(-1))
    a, b = coeffs[:2, 0]
    return a * pred + b

Run align_scale_shift before abs_rel_error when evaluating MiDaS / Depth Anything.

Step 3: Lift depth to a point cloud

import numpy as np
 
def depth_to_point_cloud(depth, intrinsics):
    H, W = depth.shape
    fx, fy, cx, cy = intrinsics
    v, u = np.meshgrid(np.arange(H), np.arange(W), indexing="ij")
    z = depth
    x = (u - cx) * z / fx
    y = (v - cy) * z / fy
    return np.stack([x, y, z], axis=-1)
 
 
depth = np.random.uniform(0.5, 4.0, (240, 320))
intr = (320.0, 320.0, 160.0, 120.0)
pc = depth_to_point_cloud(depth, intr)
print(f"point cloud shape: {pc.shape}  (H, W, 3)")

One function, every 3D-lifted application. Export the point cloud to .ply and open in MeshLab or CloudCompare.

Step 4: Smoke test with a synthetic depth scene

def synthetic_depth(size=96):
    yy, xx = np.meshgrid(np.arange(size), np.arange(size), indexing="ij")
    # Floor: linear gradient from near (top) to far (bottom)
    depth = 1.0 + (yy / size) * 4.0
    # Box in the middle: closer
    mask = (np.abs(xx - size / 2) < size / 6) & (np.abs(yy - size * 0.6) < size / 6)
    depth[mask] = 2.0
    return depth.astype(np.float32)
 
 
gt = torch.from_numpy(synthetic_depth(96))
pred = gt + 0.3 * torch.randn_like(gt)  # simulated prediction
aligned = align_scale_shift(pred, gt)
print(f"before align  absRel = {abs_rel_error(pred, gt):.3f}")
print(f"after align   absRel = {abs_rel_error(aligned, gt):.3f}")

Step 5: Depth Anything usage (reference)

import numpy as np
import torch
from transformers import pipeline
from PIL import Image
 
pipe = pipeline(task="depth-estimation", model="depth-anything/Depth-Anything-V2-Large-hf")
 
image = Image.open("street.jpg").convert("RGB")
out = pipe(image)
depth_np = np.array(out["depth"])

Three lines. out["depth"] is a PIL grayscale; convert to numpy for math. The snippet loads Depth Anything V2, the release the transformers depth pipeline loads directly. V3 is a separate release with its own loading code, so this snippet does not cover it.

• Depth Anything V3 (Meta AI / ByteDance, 2024-2026): the default for relative depth, and the fastest ViT-large-backbone model in production.
• Marigold (ETH, 2024): the highest visual quality, at the cost of slow inference.
• UniDepth (ETH, 2024): metric depth with camera intrinsics estimation built in.
• ZoeDepth (Intel, 2023): metric depth, older than the others but still reliable.
• MiDaS v3.1: legacy at this point, but stable enough to use as a baseline for comparison.

Typical integration pattern:

1. RGB frame arrives.
2. Depth model produces a depth map.
3. Detector produces boxes.
4. Lift box centroids through depth to 3D, then merge with a point cloud if one's available.
5. Downstream: AR occlusion, path planning, object-size estimation, stereo replacement.

For real-time use, Depth Anything V2 Small (INT8 quantised) hits roughly 30 fps on a consumer GPU at 518x518.

This lesson produces:

• outputs/prompt-depth-model-picker.md: picks between Depth Anything V3, Marigold, UniDepth, and MiDaS given latency, metric-vs-relative need, and scene type.
• outputs/skill-depth-to-pointcloud.md: a skill that builds point clouds from depth maps with correct intrinsics handling and exports them to .ply.

1. (Easy) Run Depth Anything V2 on any 10 images of your desk. Save depth as grayscale PNGs and inspect. Identify one object whose predicted depth looks wrong and explain why the monocular cues failed.
2. (Medium) Given RGB + depth from Depth Anything V2, lift to a point cloud and render with open3d. Compare two scenes (indoor / outdoor) and note which looks more believable.
3. (Hard) Take five pairs of images that differ only by a known object's position (e.g. bottle moved 30 cm closer). Use UniDepth to predict metric depth on both. Report the predicted distance delta vs the true 30 cm.

• Depth Anything V3 paper page: SOTA monocular depth with a DINOv2 encoder
• Marigold (Ke et al., CVPR 2024): diffusion-based depth estimation
• UniDepth (Piccinelli et al., 2024): metric depth with intrinsics
• MiDaS v3.1 (Intel ISL): the canonical relative-depth baseline
• DINOv3 blog post (Meta): the encoder family behind the accuracy gains in depth