Video Temporal-Spatial Decomposition Model


  • We propose a robust temporal-spatial decomposition (RTSD) model and discuss its applications in video processing. A video sequence usually possesses high correlations among and within its frames. Fully exploiting the temporal and spatial correlations enables efficient processing and better understanding of the video sequence. Considering that the video sequence typically contains slowly changing background and rapidly changing foreground as well as noise, we propose to decompose the video frames into three parts: the temporal-spatially correlated part, the feature compensation part, and the sparse noise part. Accordingly, the decomposition problem can be formulated as the minimization of a convex function, which consists of a nuclear norm, a TV (total variation)-like norm, and an ℓ1  norm.  We develop a two-stage strategy to solve this decomposition problem,

  • The RTSD model treats video frames as a unity from both the temporal and spatial point of view, and demonstrates robustness to noise and certain background variations. It is then widely applied in video denoising, video defect detection, video error concealment, video frame interpolation, and more.

[An Example of Video Decomposition and Denoising by RTSD]

[An Example of Video Defect Detection by RTSD]

[An Example of Video Error Concealment by RTSD]

Zhangyang (Atlas) Wang​