Deep Video Generation, Prediction and Completion of Human Action Sequences

Deep Video Generation, Prediction and Completion of Human Action Sequences

Video Result Demonstration

This video result demonstration is divided into two parts: Quanlitative Results and Illustration of Our Pipeline

Quanlitative Results

Note: Each following section corresponds to a generation task, namely video generation, video prediction and video completion. Columns named "Real" stands for real data (for your reference). Columns named "Input-n" stands for input frames where n is the frame number used (e.g. “Input-1” means the 1st frame in a video is used as input/constraint). The other columns show the qualitative results of each method. For our method we also show our pose sequence results, denoted as “Ours-Pose”. Each row corresponds to an action class, from top to bottom: Walking, Direction, Greeting, Sitting, Sitting Down.

Video Generation

Real

VGAN

Ours

Ours-Pose

Real

VGAN

Ours

Ours-Pose

Real

VGAN

Ours

Ours-Pose

Real

VGAN

Ours

Ours-Pose

Real

VGAN

Ours

Ours-Pose

Video Prediction

Input-1

Input-3

Input-2

Input-4

PredNet

PoseVAE

MS-GAN

Ours

Ours-Pose

Input-1

Input-3

Input-2

Input-4

PredNet

PoseVAE

MS-GAN

Ours

Ours-Pose

Input-1

Input-3

Input-2

Input-4

PredNet

PoseVAE

MS-GAN

Ours

Ours-Pose

Input-1

Input-3

Input-2

Input-4

PredNet

PoseVAE

MS-GAN

Ours

Ours-Pose

Input-1

Input-3

Input-2

Input-4

PredNet

PoseVAE

MS-GAN

Ours

Ours-Pose

Video Completion

Input-1

Input-50

cond-VGAN

Ours

Ours-Pose

Input-1

Input-50

cond-VGAN

Ours

Ours-Pose

Input-1

Input-50

cond-VGAN

Ours

Ours-Pose

Input-1

Input-50

cond-VGAN

Ours

Ours-Pose

Input-1

Input-50

cond-VGAN

Ours

Ours-Pose

Illustration of Our Pipeline

Video Generation Pipeline

Video Prediction Pipeline

Video Completion Pipeline

$z_0 \sim \mathcal{U}(-1, 1)$

$z \sim \mathcal{N}(0, 1)$

Input-1

Input-2

Input-3

Input-4

Input-1

Input-50

$z_0 + z\: (concatenation)$

stacked hourglass pose estimation

stacked hourglass pose estimation

$z_0 \in \mathbb{R}^{8}, z \in \mathbb{R}^{24}$

Pose-1

Pose-2

Pose-3

Pose-4

Pose-1

Pose-50

Pose Sequence Generation Process

Constrained Pose Sequence Generation Process

Constrained Pose Sequence Generation Process

Skeleton to Image

Skeleton to Image

Skeleton to Image