Design and Optimization for Interactive Spatial-Temporal Multi-Modal Gaussian Splatting

Abstract

With the advances of 3D representations, Gaussian Splatting has quickly evolved into the most promising volumetric video representation. A great amount of research efforts has been conducted in many perspectives for better spatial-temporal multi-modal scene reconstruction based on Gaussian Splatting. In this paper, we will first introduce the fundamental representation of Gaussian Splatting. Then, we will overview the recent development of Gaussian Splatting along the end-to-end ecosystem from content capture, content creation, content delivery, to content consumption. On the content capture stage, we will address critical issues and solutions to help good quality model building. On the content creation stage, we will discuss methods to enrich the multi-modal experience via embedding multimodal info into the Gaussian Splatting primitives. On the content delivery stage, we will discuss methods to reach high photorealistic quality and streamable bit rate. On the content consumption stage, we will discuss multiple approaches to enable more interactive and immersive experience, such as language and semantics guided navigation, interaction with multi-surface-part articulable 3D Gaussian Splatting. We will discuss the latest international standardization efforts and highlight future research trends.