Project 3: Face Morphing

Austin Zhu

This project largely dealing with algorithms regarding facial morphing, computation of population face means, and the caricatures we can create using them.

Part 1: Defining Correspondences

For the next three parts, I will focus on morphing my face and my friend Joe's face (also taking 180!). The original images to morph as shown below.
My picture.
Joe's picture.
In order to start the morphing process, we need to define the point correspondences between our two faces, which I did using the provided labeling tool. These correspondences will allow us to clearly define what sets of points should morph into each other to have a smooth transition. Then, I define a consistent triangulation for both sets of points by averaging the points together, and finding a triangulation using scipy.spatial.Delaunay in order to get the Delaunay triangulation. The results for both images are shown below:
Triangulation on my image.
Triangulation on Joe's image.

Part 2: Computing the "Mid-Way" Face

Next, given these triangulations, we need to define a method to find the affine transformation from one triangle to another triangle. This affine transformation will be a \(3\times 3\) matrix mapping homgenous coordinates (of the form (x,y,1)). Call the matrix \(A\), then we can define it by setting up a system of equations, or more easily, by solving a matrix equation. So, the transformation is: $$A = YX^{-1}$$ where \(X\) and \(Y\) are \(3 \times 3\) matrices with columns of the form (x,y,1) with the x and y coordinates of the vertex of each triangle of interest. \(X\) corresponds to the original triangle and \(Y\) corresponds to the triangle that the transformation is mapping \(X\) onto.

Then, using this, we can calculate the transformation going from the mean triangulation of mine and Joe's images back to the original images. We then use these transformations to find the original pixels that correspond to each pixel within the triangle of the averaged triangulation, and then set the pixel value of the average image to the average of the mapped pixel values of the two original images. Implementing this, we get the following result:
Mid-Way face between the two faces.

Part 3: The Morph Sequence

In order to get the morph sequence between two images, we can repeat the previous steps from part 2, but now introduce two new parameters to slightly alter it: warp_frac and dissolve_frac. These two variables are between 0 and 1, and simply change the averaging that we've been doing into weighted averages. warp_frac alters the averaging of the reference points, so image 1 is weighted by warp_frac, while image 2 is weighted by 1-warp_frac. dissolve_frac behaves very similarly, instead altering the averaging between the original pixel values.

By varying these parameters (running the algorithm with both set to values in the array [0, 1/50, 2/50, ..., 49/50, 1],), we can create a smooth transition between the two images. My result is shown below:
Morph sequence between the two images.

Part 4: The "Mean-Face" of a Population

For my project, I used the Danish dataset from the IMM Face Database. In order to obtain the mean population face, I did the following steps:
  1. Read in all of the face correspondences from the .asf files and add correspondences in the corners and halfway along the edges.
  2. Calculate the mean of face correspondences.
  3. Morph each image into the mean face shape defined by the mean face correspondences, done by setting the warp_frac to 1 and the dissolve_frac to 0.
  4. Calculate the mean of the morphed images.
Here are a few examples of the Danish faces in the database morphed into the mean face shape (3 male and 3 female):
(a)
(b)
(c)
(d)
(e)
(f)
My results of the mean faces are outlined below:
Mean face of the whole population (40 images).
Mean face of the male population (33 images).
Mean face of the female population (7 images).
Then, we can see the results of morphing my face into the average face, and the average face into my face.
My face morphed into the mean face.
The mean face morphed into my face.

Part 5: Caricatures: Extrapolating from the Mean

Using these mean images, we can take it further by creating "caricatures" of my face by setting the warp_frac value outside of 0 and 1. Results are shown below:
warp_frac = 1.75
warp_frac = -0.5

Bells and Whistles

Face-Morphing Video Between Classmates

For this bells and whistles, we created a morphing gif between four of us in the class: myself, Rebecca Feng, Joe Alarcon, and Samyak Tiwari. The original images are shown below:
Me.
Rebecca Feng.
Joe Alarcon.
Samyak Tiwari
The resulting gif is shown below:
Morph sequence between the four of us.

PCA

I did PCA on the Dane dataset of the morphed images for the male images. The following are the eigenfaces for the five greatest eigenvalues.