Final Project Report

Jian Yu, Xiaoyu Zhu, Xinzhe Wei, Zimo Fan

Our video(click it!):

1 Abstract

Our project aimed to create high-fidelity 3D head avatars using a combination of neutral 3D Gaussians and a fully leaned MLP-based deformation field. our method can capture intricate dynamic details while maintaining expression precision. What's more, for the stability and convergence of the training procedure, we devise a well-designed initialization strategy guided by geometry, leveraging implicit SDF and Deep Marching Tetrahedra.

2 Technical Approach

We begin by removing the background from each image and simultaneously estimating the 3DMM model, 3D facial landmarks, and expression coefficients for every frame. During the initialization phase in 2.1, we reconstruct a neutral geometry based on Signed Distance Fields (SDF). Additionally, we refine a deformation MLP and a color MLP using training data to create a guidance model. Subsequently, we initialize the neutral Gaussians using the neutral mesh extracted through DMTet, while the deformation and color MLPs are also inherited from this stage.

Above is the preprocessing on Jian Yu's face.

Moving on to the training phase in Section 2.2, we utilize the dynamic generator to deform the neutral Gaussians to the target expression, leveraging the driving expression coefficients as conditions.

Ultimately, with a specified camera perspective, the expressive Gaussians undergo rendering to produce a feature map. This feature map subsequently serves as input to the convolutional super-resolution network, tasked with generating high-resolution avatar images. The optimization of the entire model occurs under the guidance of supervision from multi-view RGB videos.

2.1 Geometry-guided Initialization

We are familiar with how to start neural networks but Gaussians are quite different. Random initialization will be hard to converge.

To overcome this, we propose utilizing an implicit Signed Distance Field (SDF) representation and Deep Marching Tetrahedra (DMTet) to reconstruct a neutral mesh for Gaussian position initialization. Furthermore, rough optimization is applied to the color and deformation MLPs.

We build MLPs to initialize parameters $f_{col}^{exp},f_{col}^{pose},f_{def}^{exp},f_{def}^{pose}$$f_{col}^{exp}, f_{col}^{pose}, f_{def}^{exp}, f_{def}^{pose}$ using a deformation MLP and a color MLP. During training, constraints are introduced to prevent overfitting:

1. Offset regularization $L_{offset}$$L_{offset}$ penalizes non-zero displacements to prevent learning a global constant offset;

2. Landmark regularization $L_{lmk}$$L_{lmk}$restricts SDF values near 3D landmarks to be close to zero, ensuring landmarks are on the mesh surface;

3. Laplacian regularization $L_{lap}$$L_{lap}$​ maintains mesh smoothness.

The overall loss function is formulated as:

Where $\lambda$$\lambda$ represents the weight of each term. The MLPs, along with the neutral 3D landmarks $P_0$$P_0$, are jointly optimized until convergence.

Then, we use the optimized results to attributes:$\{f_{col}^{exp},f_{col}^{pose},f_{def}^{exp},f_{def}^{pose}\}$$\{f_{col}^{exp},f_{col}^{pose}, f_{def}^{exp},f_{def}^{pose}\}$. And other attributes we just follow the original strategy.

2.2 Training

2.2.1 Training Pipeline

To compute loss, in each iteration, we first generate the head avatar, then we render a 32-channel image with 512 resolution $I_C\in R^{512\ast 512\ast 32}$$I_C\in R^{512*512*32}$. Thirdly, we feed the image to a super-resolution network $\mathrm{\Psi }$$\Psi$ generating an image $I_{hr}\in R_{2048\ast 2048\ast 3}$$I_{hr}\in R_{2048*2048*3}$ which can show more details. All parameters including $\{X_0,F_0,Q_0,S_0,A_0\}$$\{X_0,F_0,Q_0,S_0,A_0\}$, $\{f_{col}^{exp},f_{col}^{pose},f_{def}^{exp},f_{def}^{pose},f_{att}^{exp},f_{att}^{pose}\}$$\{f_{col}^{exp},f_{col}^{pose}, f_{def}^{exp},f_{def}^{pose}, f_{att}^{exp},f_{att}^{pose}\}$, and $\mathrm{\Psi }$$\Psi$.

2.2.2 Loss Function

$\lambda$$\lambda$ is weight. $I_{lr}$$I_{lr}$ is the first three channels of the 32-channel image $I_C$$I_C$. $VGG$$VGG$ is the VGG loss and $I_{gt}$$I_{gt}$ is the ground truth.

2.3 Avatar Representation

Our goal is to generate a dynamic head avatar controlled by expression coefficients. To achieve this, we represent the head avatar as dynamic 3D Gaussians conditioned on expressions. To accommodate dynamic changes, we incorporate expression coefficients and head pose as inputs to the head avatar model, which then outputs the position and other attributes of the Gaussians accordingly.

This is the pipeline of the paper we refer to:

Firstly, we build a neutral Gaussian model with expression-independent attributes: $\{X_0,F_0,Q_0,S_0,A_0\}$$\{X_0,F_0,Q_0,S_0,A_0\}$. While $X_0\in R^{N\ast 3}$$X_0\in R^{N*3}$ is the position of the Gaussians in the canonical space, $F_0\in R^{N\ast 128}$$F_0\in R^{N*128}$ is the point-wise feature vectors as their inherent attribute, $Q_0\in R^{N\ast 4}$$Q_0\in R^{N*4}$ is neutral rotation, $S_0\in R^{N\ast 3}$$S_0\in R^{N*3}$ is the scale of the Gaussians, and $A_0\in R^{N\ast 1}$$A_0\in R^{N*1}$ is opacity. Then, using MPL-based deformation field, we generate a dynamic head avatar with expression coefficients: $\{X,F,Q,S,A\}=\varphi (X_0,F_0,Q_0,S_0,A_0;\theta ,\beta )$$\{X,F,Q,S,A\}=\phi(X_0,F_0,Q_0,S_0,A_0;\theta,\beta)$. $\theta$$\theta$ represents the expression coefficients, and $\beta$$\beta$ is the head pose. During training, we optimize the parameters of the model to minimize the difference between the generated head avatar and the ground truth.

Then, we will further explain how to update $\{X,F,Q,S,A\}$$\{X,F,Q,S,A\}$ through $\varphi$$\phi$.

For $X$$X$,we assume the displacement is related to the expression and the head pose, which show as $f_{def}^{exp}\in \varphi$$f_{def}^{exp}\in\phi$ and $f_{def}^{pose}\in \varphi$$f_{def}^{pose}\in\phi$respectively. Then we add them to the neutral position :

The two $\lambda$$\lambda$ are the weights of the expression and the head pose. We can calculate ${\lambda }_{exp}$$\lambda_{exp}$ as follows.

and ${\lambda }_{pose}(x)=1-{\lambda }_{exp}(x)$$\lambda_{pose}(x)=1-\lambda_{exp}(x)$. $dist(x,P_0)$$dist(x,P_0)$ is the minimum distance between $x$$x$ and the the 3D landmarks $P_0$$P_0$. $t_1$$t_1$ and $t_2$$t_2$​ are the predefined hyperparqmeters when the length of the head is set to approximately 1.

Instead of using linear function, we also tried to add some sigmoid-like function on it. That's because:

1. Speaking from experience with masking and weight drawing, the general weight matrix is very sparse, with epidermal vertices overwhelmingly influenced by the nearest bone. Skeleton manipulation of meshes is similar to mesh manipulation of Gaussians.

2. It may be more in keeping with the body's natural.

For $C$$C$, color will be directly predicted by two color MLPs:

For $Q$$Q$, $S$$S$ and $A$$A$, we similarly use another two MLPs to predict the rotation, scale and opacity:

Lastly, we need to transform from canonical space to real world space. Only direction related variables need to be changed. So we have

2.4 Problems

• The training process was too slow, and although our results were trained on an existing dataset, it was too late to train it after we created our own data.

• The work we referenced is so advanced (CVPR 2024) that it's hard for us to make big improvements on it. We only modified some parameters to make them more reasonable and tried to create our own dataset.

2.5 Lessons Learned

For the final project, our team has extensively studied the relevant content. First, we explored the grid-based human body model, and we researched the classic work of SMPL from the Max Planck Institute. SMPL appears to be a complex work with many abstract mathematical notations and involves a lot of prerequisite knowledge in computer graphics. However, combining the knowledge we learned in the course, we gradually understood that it is essentially a mesh model implemented using blend shapes technology, considering the influence of Shape and Pose on the standard Template through a technique called linear blend skinning (LBS). The following image is the slide we prepared for our discussion. These are the footprints of our learning process👣👣👣.

Next is the novel scene representation technique: 3D Gaussian splatting (3DGS). 3DGS expresses the scene by optimizing several small Gaussian distributions and renders them using splatting, a rasterization technique. Each Gaussian distribution not only has its mean (position) and variance (size and shape) attributes but also carries opacity (α) and spherical harmonic functions to represent the three-channel color values in different views. Since the rendering process of 3DGS involves rasterization instead of ray tracing and the original author utilized CUDA and parallelized the rendering process using multiple threads, the efficiency of rendering 3DGS scenes has been significantly optimized, resulting in real-time performance. Moreover, from an effectiveness perspective, 3DGS exhibits a strong capability in scene representation.

We tried to reproduce the latest CVPR work. Even though it was difficult, it was fun. We created COOL avatars and found our joy in code and graphics.

When making adjustments to parameters, thorough deliberation is paramount, particularly given the extensive training times characteristic of a model of such magnitude. Deviating from the optimal direction could lead to significant time wastage, as rectifying errors and rerunning experiments can be exceptionally time-consuming.

Furthermore, it's imperative to craft a comprehensive and flexible plan with generous time allocations. Unforeseen complications are inevitable, and they often demand more time than initially anticipated. For example, delving into the intricacies of the original codebase can prove to be far more challenging than initially imagined. Additionally, there can be a considerable disparity between the current state of the art in research papers and the knowledge base at our disposal. Such disparities may necessitate additional time for learning and adaptation, underscoring the importance of a well-padded timeline.

3 Results

We first use images and cameras to model the head in existing dataset:

Then convert it into Gaussian model:

Use another person's head to reenactment our avatar:

The right side is the trained avatar, the left side is making a new movement making the right head mimic the left one.

4 References

• Papers

Xu, Yuelang, et al. "Gaussian head avatar: Ultra high-fidelity head avatar via dynamic Gaussians." arXiv preprint arXiv:2312.03029 (2023).

Gerig, Thomas, et al. "Morphable face models-an open framework." 2018 13th IEEE international conference on automatic face & gesture recognition (FG 2018). IEEE, 2018.

• Websites

https://github.com/YuelangX/Gaussian-Head-Avatar

https://github.com/YuelangX/Multiview-3DMM-Fitting/tree/main

https://flame.is.tue.mpg.de/

5 Contributions from each team member

Jian Yu: Do research on the topic, Reproduction of possible work, Try to make our own dataset, Polish the final report

Xinzhe Wei: Do research on the topic, Reproduction of possible work, Train on the exist dataset, Polish the final report

Xiaoyu Zhu: Do research on the topic, Reproduction of possible work, Try to make our own dataset, Polish the final report

Zimo Fan: Do research on the topic, Reproduction of possible work, Drafting of the first version of the final report, Polish the final report