Production-Ready Camera Tracking via LiDAR-Constrained Deep Optical Flow
TL;DR
A LiDAR-Constrained Deep Visual Odometry system, a robust tracking architecture designed to solve these "impossible" shots by fusing pre-existing LiDAR geometry with modern Deep Learning, and introduces a "Leapfrogging" architecture that automatically detects and corrects temporal drift by re-anchoring to the geometry from trusted keyframes.
Abstract
Matchmoving is the bedrock of visual effects, yet it remains a fragile bottleneck when footage contains heavy motion blur, low texture, or dynamic occlusion. While physical on-set camera tracking (e.g., encoded cranes) exists, it is often impractical for handheld interior shots and prone to mechanical slippage, leaving post-production software to solve the gap. This talk presents a LiDAR-Constrained Deep Visual Odometry system, a robust tracking architecture designed to solve these "impossible" shots by fusing pre-existing LiDAR geometry with modern Deep Learning. Unlike traditional commercial solvers that hunt for sparse, high-contrast corners, our approach uses Deep Optical Flow (RAFT) to track the entire dense image context, locking the camera directly to the set’s 3D mesh. We introduce a "Leapfrogging" architecture that automatically detects and corrects temporal drift by re-anchoring to the geometry from trusted keyframes. By prioritizing geometric truth over feature quantity, this standalone Python tool reduces days of manual hand-tracking and rotoscoping to minutes of automated computation, achieving high median precision on sequences where standard algorithms fail entirely.