Tutorials

Abstract: 3D Gaussian Splatting (3DGS) has emerged as a powerful approach to photorealistic 3D scene reconstruction and real-time novel-view synthesis. By representing scenes with explicit anisotropic Gaussian primitives, it supports efficient rendering for applications such as free-viewpoint video and immersive telepresence. However, its explicit representation introduces considerable redundancy and high storage and transmission costs, which become more significant in dynamic 4D Gaussian Splatting (4DGS).

This tutorial introduces the foundations of 3DGS and examines representation and coding strategies for static and dynamic Gaussian scenes. It discusses unstructured and anchor-based representations, rate-distortion optimization, and the trade-offs between per-scene optimized and feed-forward coding paradigms. The tutorial then extends these concepts to 4DGS, covering spatio-temporal modeling, dynamic-scene coding, streaming, and real-time considerations. It concludes with an overview of MPEG standardization activities, emerging applications, and open research challenges.

Description: 3D Gaussian Splatting (3DGS) has established itself as a transformative paradigm for photorealistic 3D scene reconstruction, representing scenes utilizing a set of explicit, anisotropic Gaussian primitives. By optimizing these primitives through differentiable tile-based rasterization, 3DGS overcomes the computational intensity of implicit neural fields, enabling high-quality novel-view synthesis at real-time speeds. This explicit nature is particularly advantageous for applications requiring six degrees of freedom (6DoF) and interactive navigation, such as Free-Viewpoint Video (FVV) and immersive telepresence. Despite these advantages, the explicit nature of Gaussian representations introduces a critical challenge: large data redundancy and lack of compact structure, resulting in substantial storage and transmission costs. This issue becomes even more pronounced when extending to dynamic scenes in 4D Gaussian Splatting (4DGS), where temporal redundancy further amplifies the problem.

This tutorial equips participants to understand and tackle these efficiency challenges through the lens of advanced representation and coding strategies. We will explore the technical evolution of 3DGS, comparing unstructured models with topologically-aware structures like Scaffold-GS. Building upon these static foundations, a primary focus will be placed on Rate-Distortion Optimization (RDO) frameworks and their expansion into the 4DGS domain. This includes the specific representation of 4DGS and the corresponding coding architectures required to manage the spatio-temporal data. We will further contrast Per-scene Optimized workflows with modern Feed-forward coding paradigms, which are essential for generalized, low-latency applications. Extending into the temporal domain, the tutorial addresses the architectural trade-offs between 4DGS streaming and dynamic per-scene coding. Finally, we discuss emerging standardization activities in MPEG to align these advances with future media ecosystems.

Recent advances in visual sensing, computational imaging, and multimodal large language models (MLLMs) have substantially expanded the scope of machine vision, from low-level signal acquisition and reconstruction to high-level visual understanding and reasoning. Despite this progress, effectively bridging visual signals with multimodal intelligence remains challenging due to limitations in sensing fidelity, reconstruction quality, representation efficiency, and reasoning reliability. This tutorial reviews recent developments across these interconnected areas and highlights how advances at different stages of the visual processing pipeline contribute to more robust and efficient visual intelligence.

The first part focuses on neuromorphic visual sensing, reconstruction, and perceptual representation. We introduce bio-inspired spike cameras for high-speed visual acquisition and discuss key techniques for spike-based optical flow estimation and image reconstruction under noisy and challenging conditions. We further present open-source frameworks, datasets, and benchmarks that support reproducible research in spike-based vision, and examine just noticeable difference modeling for deep visual features as a means of improving representation efficiency, compression, and quantization.

The second part turns to robust multimodal understanding and reasoning. We discuss vision-language representation learning under visual perturbations and fine-grained visual evidence tracking for more reliable multimodal reasoning. We then explore RAW image understanding as a way to leverage sensor-level information beyond conventional RGB representations, as well as egocentric video understanding for modeling human activities, temporal relationships, and procedural knowledge. Together, these topics offer a unified perspective on the progression from visual sensing and reconstruction to robust multimodal perception, understanding, and reasoning.

Neural spatial computing, the ability to perceive, reason about, and manipulate 3D spatial/geometric data using neural network-based algorithms, is emerging as a foundational component of modern artificial intelligence. Its impact spans a wide range of domains, including robotics, autonomous systems, immersive telepresence, and the metaverse.

This tutorial offers a comprehensive overview of recent advances in neural spatial computing, focusing on two key pillars: geometric modeling and representation learning of 3D data. Participants will master advanced explicit and implicit representations of 3D geometry, understand neural techniques for computational geometry, and explore methods for enhancing geometric data and learning 4D representations from monocular video. They will also gain insights into how 2D modalities can augment 3D representation learning through techniques such as representation translation, neural calibration, contrastive distillation, and cross-modal interaction.

Generative models have substantially advanced visual restoration and compression, while Physical AI motivates representations that capture the underlying structure and dynamics of the physical world. These developments call for visual signal processing methods that jointly consider perceptual quality, semantic fidelity, physical consistency, and human preferences.

This tutorial presents recent advances in preference-aligned generative low-level visual signal processing, compression, and quality assessment, organized around four complementary themes:

(i) generative low-level visual processing, covering image and video restoration and enhancement, with emphasis on the trade-offs between perceptual realism and content fidelity;

(ii) generative image and video compression, including compact visual representations, low-bitrate reconstruction, and rate-distortion-perception optimization;

(iii) revisiting Physical AI through disentangled representation learning, examining how separating underlying factors of variation can support structured representations and the modeling of physical environments; and

(iv) quality assessment and preference alignment, covering perceptual metrics, reward modeling, human preference optimization, and multimodal feedback for evaluating and guiding visual processing and compression.

By connecting representation, generation, and evaluation, the tutorial highlights key challenges and future opportunities for efficient, controllable, and preference-aligned visual communication systems.

Learning-based image and video compression has achieved remarkable rate-distortion performance, outperforming many traditional, hand-designed codecs. However, their deployment in real-world scenarios is often hindered by excessive computational overhead and limited applicability. While existing research has primarily focused on visual image and video coding, multi-sensor multimedia systems increasingly demand compression solutions for a broader range of structured signals, from visual content to audio, tactile and spike data. There remains a clear need to systematically consolidate recent advancements and detail the inherent performance-complexity trade-offs across these diverse modalities; this tutorial aims to address that need.

This tutorial surveys state-of-the-art, learning-driven visual compression techniques, with a focus on achieving competitive reconstruction quality under constrained computational budgets. We begin by establishing the foundational background of learning-based visual compression, then extend these neural coding principles to three representative, emerging families of modalities. The first part covers novel learning-based coding for visual, auditory modalities and even multimodal codecs, discussing their unique statistical properties, codec designs, and remaining practical challenges. The second part focuses on tactile data compression, examining modality-specific signal attributes, end-to-end neural codec architectures, perceptual evaluation metrics, and downstream applications for tactile data transmission. In the third part, we delve into spike cameras, a new type of bio-inspired neuromorphic imaging devices and discuss the ultra-high-speed spike data coding and understanding.

Our tutorial summarizes cutting-edge research in both visual and non-visual neural compression, highlights unresolved technical hurdles, and offers practical insights for bridging the gap between academic research and real-world implementation. It provides a comprehensive perspective for attendees from diverse backgrounds, helping them navigate this rapidly evolving field of research.

The rapid progress of generative media and editing tools has made it increasingly difficult to assess whether a given video is authentic, unaltered, and originates from a trusted source. The Digitally Signed Content (DSC) Supplemental Enhancement Information (SEI) messages, recently added to the H.274/VSEI standard, and the Coalition for Content Provenance and Authenticity (C2PA) are addressing this challenge by specifying end to end workflows for trustworthy media. This spans from content creation to distribution and verification, combining cryptographic signatures, secure metadata, and standard compliant video coding tools.

This tutorial will introduce the foundations, standards, and practical implementation aspects of media authentication for video, with a focus on:

• Digitally signed content for compressed video streams with SEI messages specified in the H.274/VSEI standard

• Authenticity and provenance with C2PA

• Integration into typical video delivery pipelines (capture, encoding, distribution, playback)

• Implementation walk throughs using open-source tools, including live demonstrations of signing and verifying video content with open-source tooling