Category Archives: Face Swapping Iterations

My Final Code

As required, here is my final code.

The one I tested with can be found here on my Github

https://github.com/kuuurttyy/Face-Swapping

My new, updated and enhanced code which I wrote about here can also be found on Github

https://github.com/kuuurttyy/newFaceSwappingShortCode

As a back up I’ll post them both here (better safe than sorry, right).

Continue reading

Processing: Mega Clean up

In a previous post where I was cleaning up my code, I mentioned how I wanted to work out how to store the captured faces in an array of PImages rather than individual PImages like I am currently doing. This will significantly reduce the number of lines in my code and make it look a lot neater.

Well, after a long day of staring at my screens, and a few mental breakdowns I finally worked it out! Below is the code which does the magic:

//empty array to store captured faces for swapping
  PImage[] captured = new PImage[faces.length];

  //fill the array with the faces
  for (int i = 0; i < faces.length; i++) {
    captured[i] = get(faces[i].x*scale, faces[i].y*scale, faces[i].width*scale, faces[i].height*scale);
  }

  //swap over the faces
  for (int s = 0; s < captured.length-1; s++) {
    captured[s].resize(faces[s+1].width*scale, faces[s+1].height*scale);

    mask.resize(faces[s+1].width*scale, faces[s+1].height*scale);
    captured[s].mask(mask);

    image(captured[s], faces[s+1].x*scale, faces[s+1].y*scale);
  }

  //swap the last detected face with the first detected face
  if (captured.length >= 2) {
    int c = captured.length-1;
    captured[c].resize(faces[0].width*scale, faces[0].height*scale);

    mask.resize(faces[0].width*scale, faces[0].height*scale);
    captured[c].mask(mask);

    image(captured[c], faces[0].x*scale, faces[0].y*scale);
  }

The new code shortens my old code significantly as there is no longer the need to grab the faces individually, then for each instance where there could be 2, 3 or 4 faces for example, it doesn’t need another if statement to separate them all. I’ll try to summarise the above code with words as unfortunately while it is shorter, its much harder to read through than before.

It starts by creating a PImage array which is the same length as the array of detected faces. The array is then filled with a for loop by iterating the value of i from 0 up to the length of the array. The number is then used in place of the array index so that it fills that spot in the array with the pixel information it is grabbing with the get function. To then swap the faces over, another for loop is used which uses the 4 functions needed from before; resize to match the host face, resize the mask to match, apply the mask, then place the face down in its new location. The for loop works by swapping the face with the next one along in the array. For example, face 1 is in place of face 2, face 2 in place of face 3, and so on until there aren’t any more faces. This loop leaves a problem with the last face in the array as it has nowhere to go. Logically it needs to loop back around to the beginning of the array and be in place of the first one. To do this I used an if statement which checks to see of there are 2 or more faces detected. If there are, it creates a variable called c which is the length of the array of captured faces minus one. This is so it can be used in the array index to always access the last face in the array (as arrays start at 0 not 1). It then uses c to resize and swap the face in place of the first one in the array to complete the cycle.

This new method has cut the length of my code in over a half making it much more concise. There is also another benefit which I didn’t even consider while making it. Before, my code was limited to swapping over 4 faces. This was then increased to 6 during testing as 4 wasn’t enough. With this new code there isn’t a set limit of faces it can swap. Using these loops I can swap every face that is detected which would work much better in a public space with a lot of people.

While I’ve already done the testing, I much prefer this final version of my code as it is much tidier and actually works better than it did before due to there not being a limit to the faces it can swap.

Processing: Final Touches

As I’m doing my testing tomorrow in the foyer, I took it upon myself to do my final touches to my sketch to make sure its as good as possible before I test it. The code at the moment doesn’t need any changes so I decided to perfect my the mask image to make sure it cuts the face out as good as possible.

Before my final touches I was on mask 9, this slightly changed soon after to mask 10 which i made a bit longer to try and not cut off the chin as much (this was changed a long time ago). I came back to it today and started afresh with the mask shape. Using an image taken from the face tracking, I drew around the face to get the shape and it’s more oval than before, as it should be as faces aren’t rectangular. After testing it with mask 9 it was a bit too wide at the top and was showing some of the background so I trimmed it even further and ended with mask 12.

Mask 10

Mask 10

Mask 11

Mask 11

Mask 12 (final mask)

Mask 12 (final mask)

Here are some screenshots using the (currently) final mask shape. The first shows how it works at a distance which is pretty good. While not perfect, the masking does do a pretty good job for cutting out faces, especially as they’re not always the same shape and size.

21When I do some testing I expect to need to make some alterations to my project as I’m sure there will be some slight variations when in the actual space which will mean the mask might not work as expected for example. I might need to change the shape of the mask when testing in the space. A lot of my testing has been done while sitting directly in front of the camera so it always captures full frontal faces. In the space the position of the camera and the angle of people could vary hugely and mean that the mask and swapping doesn’t work as well as it does at the moment. Only time will tell…

Processing: Clean up

Currently I’m very happy with the stage my face swapping project is at, and at the moment it is a matter of cleaning up my code while I think of further developments and improvements I can make on it.

Before my setup I have 4 lines similar to this ‘PImage face0 = createImage(0, 0, RGB);’ which is the PImages used to store the captured faces from the face tracking. I’ve now learned that it doesn’t require a blank image to store it in and therefore can be simplified to just ‘PImage face0;’.

I’ve also re-ordered the swapping for when there are 4 detected faces on screen. In a previous post I drew a table to keep track of my swapping while writing it but I now realise that the order doesn’t make sense for the 4 face swap. The faces are swapped with the next one along in the array (i.e face 1 in place of, 2 moved to 3 etc.) with the exception of the last face which is moved to the beginning of the array to be swapped with the first face (hopefully this description makes sense). This is a first progression to a rather confusing idea I had to try to streamline my code a bit more.

I want to try to store the captured faces in an array rather than 4 individual PImages, as it would hopefully eliminate a lot of the repeated lines of code which have very small differences. I could then capture the faces and swap them using for loops to access various faces in the array and work from there. I’ve tried to get my head around doing this but having the masking and resizing has made it significantly harder than I originally expected so I’m yet to blog about any progress until/ if I work it out.

These small changed might not seem that significant but I feel it is good practice to keep the code as simple as possible without any excess clutter to confuse things or slow them down.

Processing: Masking Fix

I mentioned in my last Processing post that there was a problem with the mask being resized constantly. As the mask was being resized multiple times each frame it soon lost its quality and its masking abilities slowly got worse as the mask got more stretched and blurred.

To get around this I moved the ‘mask = loadImage(“mask.jpg”);’ line from the setup to the draw. With the line in the setup, the mask was loaded once at the beginning and then constantly edited as time passed. Putting it in the draw meant that every time the code runs for each frame, it would reload the original mask so that the quality doesn’t diminish over time and makes sure it doesn’t get blurrier.

Processing: Face Swap Masking

One thing that was bothering me with my face swapping was that it was just placing the whole rectangle of each face on top of the other. This meant that if the rectangle included any of the background behind the face that would be picked up and moved too, giving a very crude face swap. I wanted to get something smoother, something that could make this work just that little bit better.

In case you’ve forgotten what the swapping looks like, heres an example which clearly shows how the whole rectangle is picked up and moved, including background.

Screen-Shot-2014-12-05-at-14.20.13

I was introduced to masking in Processing which masks a part of an image by loading another image and using it to control the alpha channel (opacity). The mask is in greyscale with black being transparent and white being opaque. The in-between grey values are all the opacity values in-between making it possible to get a nice gradual fade.

I created A PImage before the setup to store the mask, and then loaded in the mask file in the setup. Below is a snippet of the code showing how the mask is implemented when swapping the faces over.

  //swap three faces over
  if (faces.length == 3) {
    //resize images to current tracked faces
    face0.resize(faces[2].width*scale, faces[2].height*scale);
    face1.resize(faces[0].width*scale, faces[0].height*scale);
    face2.resize(faces[1].width*scale, faces[1].height*scale);

    //resize and mask images to smooth transition
    mask.resize(faces[2].width*scale, faces[2].height*scale);
    face0.mask(mask);
    mask.resize(faces[0].width*scale, faces[0].height*scale);
    face1.mask(mask);
    mask.resize(faces[1].width*scale, faces[1].height*scale);
    face2.mask(mask);

    //place swapped faces
    image(face0, faces[2].x*scale, faces[2].y*scale);
    image(face1, faces[0].x*scale, faces[0].y*scale);
    image(face2, faces[1].x*scale, faces[1].y*scale);
  }

A mask image is loaded in the setup and each time it’s used it is resized appropriately to fit the face. Currently i’m not really sure if theres a better way of doing this, there probably is but I haven’t yet figured it out. This has made my code look a lot longer and messier but at least it does the job. In the swapping sections. the face images are resized to fit the swapped face, then the mask is resized to match the image and then applied to it before the now masked image is placed down in its new place.

There is one problem currently as the mask is being resized multiple times each frame it gets blurrier and blurrier and its masking properties get worse and worse so that the image end up looking more square again. I need to work out a way around this at some point soon (fixed here).

You may have noticed that the file loaded is called ‘mask9.jpg’, this is because it took a few attempts to get a mask that was roughly right for the faces. Annoyingly, because of the basic nature of the face tracking, the faces don’t always fit exactly the same in the face square so its pretty much impossible to get a mask that will work perfectly on all faces due to the faces being as slightly different angles and such.

Below is a timeline of sorts of my development of trying to find a good shaped mask for the faces. I started with an oval shape as I thought thats what shape faces were but I soon realised that a more rectangular shaped mask was needed. The shape on the far right is mask9, and currently the best fit. I may play around to try and further refine the shape some more but at the moment it does a good enough job for whats needed.masks

 

Here’s a screenshot of the swapping using one of the masks, unfortunately I didn’t write down which mask iteration it was but you get the idea. The swap looks so much better now that the background has been cut out and it fits the face a lot better. Using the mask further benefits my aim of playing with the idea of identity and representation. The swap looks far more realistic now and helps further blur the boundaries between the body and the new face imposed on it. From my experience, the better effect also makes it a lot more fun to play with, which will hopefully make it more interesting for the audience in the Weymouth House foyer.
Screen-Shot-2014-12-05-at-14.52.50

And here’s a bonus image of me face swapped with a picture of my dog using a mask:

Screen-Shot-2014-12-07-at-12.15.04

 

Processing: Face Swap Frame Rate Fix

I mentioned in my last progress update that there was a problem with the frame rate being too slow (~5fps) instead of a more acceptable level around 20/25fps. I spent a few hours trying to work out how to do it by scaling the video using the scale() function, but alas I couldn’t find a way that worked. Fortunately I stumbled across another GitHub repository of Processing examples by Daniel Schiffman. The repository is from a workshop he did about face detection, recognition and tracking methods. In here there are examples using the OpenCV library to detect and track faces. One of the examples is a scaled down version to improve performance – Exactly what I needed!

The example creates a scaled down version of the image to do the tracking on while having another version which is just displayed, which improves the resolution and frame rate of the sketch.

OpenCV does its tracking on an image half the size of the original (as the scale is currently set to 2) which means there are a lot less pixels for it to check for faces which improves performance vastly. When grabbing, resizing and displaying each of the tracked faces now, I had to multiply the coordinates and sizes by the scale so that they would appear as the correct size.

Initially I tried using a scale of 4 (so the tracked image is 4 times smaller) in an effort to maximise performance as much as possible, but I found when the image was that small it struggled to find the faces in the image when they were more distant from the screen. With the resolution i’m working at at the moment, the halved scale works perfectly for what I need.

Here is an example from my code showing the two images. The smaller one in the top corner (which is usually hidden) is what OpenCV tracks to find the faces, then the larger image which is the size of the window is displayed separately.

Screen Shot 2014-12-07 at 11.27.40

 

Below is my updated code to work in the scaling to fix the frame rate. To summarise the changes, a smaller image is created before the setup along with a variable for the scale, so I can adjust it as needed. The OpenCV object is created at the same size as the scaled down size, and the smaller image is loaded into it to be tracked in the background. The faces for the swapping are grabbed from the larger video, so therefore the coordinates and dimensions needed to be multiplied by the scale to match it.

//import libraries
import gab.opencv.*;
import processing.video.*;
import java.awt.*;

Capture video;
OpenCV opencv;

//PImages to store captured faces
PImage face0 = createImage(0, 0, RGB);
PImage face1 = createImage(0, 0, RGB);
PImage face2 = createImage(0, 0, RGB);
PImage face3 = createImage(0, 0, RGB);

//scaled down image for tracking
PImage smaller;
int scale = 2;

void setup() {
  size(640, 480);
  video = new Capture(this, 640, 480, 20);
  opencv = new OpenCV(this, video.width/scale, video.height/scale);
  opencv.loadCascade(OpenCV.CASCADE_FRONTALFACE);  
  frameRate(20);
  video.start();
  //make scaled down image for openCV to track
  smaller = createImage(opencv.width, opencv.height, RGB);
}

void draw() {
  //load smaller image into OpenCV for tracking, better framerate
  opencv.loadImage(smaller);
  //display video feed
  image(video, 0, 0 );


  //style face rectangle
  noFill();
  stroke(0, 255, 0);
  noStroke();
  strokeWeight(3);
  Rectangle[] faces = opencv.detect();
  println(faces.length);

  //draw rectangle around seen faces
  //NOT BEING USED
  for (int i = 0; i < faces.length; i++) {
    println(faces[i].x + "," + faces[i].y);
    rect(faces[i].x, faces[i].y, faces[i].width, faces[i].height);
  } 

  //stores detected faces in images
  //iterate through to check how many faces there are
  if (faces.length >= 1) {
    face0 = get(faces[0].x*scale, faces[0].y*scale, faces[0].width*scale, faces[0].height*scale);
    if (faces.length >= 2) {
      face1 = get(faces[1].x*scale, faces[1].y*scale, faces[1].width*scale, faces[1].height*scale);
      if (faces.length >= 3) {
        face2 = get(faces[2].x*scale, faces[2].y*scale, faces[2].width*scale, faces[2].height*scale);
        if (faces.length >= 4) {
          face3 = get(faces[3].x*scale, faces[3].y*scale, faces[3].width*scale, faces[3].height*scale);
        }
      }
    }
  }

  //swap two faces over
  if (faces.length == 2) {
    //resize images to current tracked faces
    face0.resize(faces[1].width*scale, faces[1].height*scale);
    face1.resize(faces[0].width*scale, faces[0].height*scale);

    //place swapped faces
    image(face1, faces[0].x*scale, faces[0].y*scale);
    image(face0, faces[1].x*scale, faces[1].y*scale);
  }

  //swap three faces over
  if (faces.length == 3) {
    //resize images to current tracked faces
    face0.resize(faces[2].width*scale, faces[2].height*scale);
    face1.resize(faces[0].width*scale, faces[0].height*scale);
    face2.resize(faces[1].width*scale, faces[1].height*scale);

    //place swapped faces
    image(face0, faces[2].x*scale, faces[2].y*scale);
    image(face1, faces[0].x*scale, faces[0].y*scale);
    image(face2, faces[1].x*scale, faces[1].y*scale);
  }

  //swap four faces over
  if (faces.length >= 4) {
    //resize images to current tracked faces
    face0.resize(faces[2].width*scale, faces[2].height*scale);
    face1.resize(faces[0].width*scale, faces[0].height*scale);
    face2.resize(faces[3].width*scale, faces[3].height*scale);
    face3.resize(faces[1].width*scale, faces[1].height*scale);

    //place swapped faces
    image(face0, faces[2].x*scale, faces[2].y*scale);
    image(face1, faces[0].x*scale, faces[0].y*scale);
    image(face2, faces[3].x*scale, faces[3].y*scale);
    image(face3, faces[1].x*scale, faces[1].y*scale);
  }
  
  
} // close draw


void captureEvent(Capture c) {
  c.read();
  
  // Make smaller image
  smaller.copy(video, 0, 0, video.width, video.height, 0, 0, smaller.width, smaller.height);
  smaller.updatePixels();
}

With this new scaled version for tracking, the frame rate is drastically improved and it as a whole runs a lot better than it did before. It is now running at my capped frame rate at 20 fps rather than the 5 fps it was running at before the fix. This is helpful as while making this, i’ve been working on a small version of it, usually 640 x 480px whereas when it is displayed in the foyer it will need to be a much bigger size to fit the screen. With this new method I can increase the size of the window and video, and appropriately adjust the scale so that I can maximise appearance/resolution and performance.

Processing: Face Swap Next Iteration

My first iteration had some very messy code so my first step was to clean out the bits that weren’t being used to cut it down to the barebones of the parts I needed. The next step was to work out how I was going to swap the faces over for when there were 3 and 4 faces. At the moment I’m limiting it to 4 faces due to the stress it puts on my laptop, but for all intents and purposes I feel that 4 faces is enough. If the tracking detects any more than that it wont swapping the faces over and they’ll stay looking normal.

As a reference I drew out a table so I could keep track of which faces I was swapping. I tried doing it all in my head but kept getting it wrong so this is what I had to resort to. The faces along the top represent the faces in the array and the ‘swaps’ are which faces they will be swapped with for 2, 3, and 4 faces.

Screen Shot 2014-12-06 at 11.13.37

From here it was a simple matter of writing the if statements needed for up to 4 faces, resizing and placing the images appropriately:

//swap two faces over
  if (faces.length == 2) {
    //resize images to current tracked faces
    face0.resize(faces[1].width, faces[1].height);
    face1.resize(faces[0].width, faces[0].height);

    //place swapped faces
    image(face1, faces[0].x, faces[0].y);
    image(face0, faces[1].x, faces[1].y);
  }

  //swap three faces over
  if (faces.length == 3) {
    //resize images to current tracked faces
    face0.resize(faces[2].width, faces[2].height);
    face1.resize(faces[0].width, faces[0].height);
    face2.resize(faces[1].width, faces[1].height);

    //place swapped faces
    image(face0, faces[2].x, faces[2].y);
    image(face1, faces[0].x, faces[0].y);
    image(face2, faces[1].x, faces[1].y);
  }

  //swap four faces over
  if (faces.length >= 4) {
    //resize images to current tracked faces
    face0.resize(faces[2].width, faces[2].height);
    face1.resize(faces[0].width, faces[0].height);
    face2.resize(faces[3].width, faces[3].height);
    face3.resize(faces[1].width, faces[1].height);

    //place swapped faces
    image(face0, faces[2].x, faces[2].y);
    image(face1, faces[0].x, faces[0].y);
    image(face2, faces[3].x, faces[3].y);
    image(face3, faces[1].x, faces[1].y);
  }

The last if statement is greater than or equal to 4 faces so that if the camera picks up more than 4 faces, the first 4 will still swap but anything over that won’t be effected. One thing that bothers me with this code is that there is a lot of repetition of very similar lines of code for each statement. While it still works fine I would like to try to clean it up if at all possible to make it look a bit neater.

One thing I didn’t mention in my first post is the frame rate when it runs. In the example which comes with the OpenCV library (my basis for building this), the video is halved and then scaled back up to fit the window. This is so that the tracking is working on a lower resolution version of the video so that it can run faster and improve performance. In my attempts, the scaling kept confusing me while trying to grab the faces so I turned it off. This means that it is doing the tracking on a relatively high resolution video which takes longer to check for faces due to the larger number of pixels to check and affects the frame rate. My sketches currently run at about 4/5 frames per second which isn’t exactly very good. If it was run on a more powerful computer it could work a little better but I need to find a way to optimise the code and improve the frame rate up to a more reasonable level around 20/25 fps.

Below is it swapping 3 faces which works as expected. What I did notice is that sometimes it sees the faces in a different order, causing them to jump about a bit from face to face which could either be a good or bad thing. It could be considered a good thing as it makes it a bit more interesting as you have to keep track of where your face is going. Although it could get quite annoying and detract from the effect if the face doesn’t stay in one place long enough. Also the tracking isn’t exactly perfect. As you can see in the picture below there are 4 faces but one of them is being ignored for some reason. It could be a combination of lighting, positioning, the angle of the face or something completely different which is stopping it from being seen as a face by the sketch. At the moment I don’t think this is a problem with my code but rather the limitations of the Haar Cascade which is currently being used to detect the faces.

Screen-Shot-2014-12-05-at-14.20.13

Processing: Face Swap Start

To get started on face swapping I began by altering one of my old sketches which captured and saved faces as images. Before, I was trying to get the image out of Processing to do something with it but now it can just say within the Processing environment and I can work from there. Below is my code, commented so that hopefully it makes sense to whomever is reading it. A lot of the code in there isn’t being used (like the timer for example) but I left it in there to show how it was a development upon one of my old ideas where I was experimenting with what I could do.

Quick note:  I’ve recently noticed that the plugin i’ve used for inserting code has sometimes changed my greater than (>) and less than symbols (<) to &gt; and &lt; instead for some reason. 

(Theres a summary of what the code does below)

import gab.opencv.*;
import processing.video.*;
import java.awt.*;

Capture video;
OpenCV opencv;

//PImages to store captured faces
PImage face0 = createImage(0, 0, RGB);
PImage face1 = createImage(0, 0, RGB);
PImage face2 = createImage(0, 0, RGB);
PImage face3 = createImage(0, 0, RGB);

//values for timer
int savedTime;
int totalTime = 5000; // 5 second delay
int number = 0; //number of picture

void setup() {
  size(640, 480);
  video = new Capture(this, 640, 480, 20);
  opencv = new OpenCV(this, 640, 480);
  opencv.loadCascade(OpenCV.CASCADE_FRONTALFACE);  
  frameRate(20);
  video.start();
  savedTime = millis();
}

void draw() {
  // Calculate how much time has passed
  int passedTime = millis() - savedTime;

  //should be scale 2 but made it 1 so it looks better, but makes it laggy
  scale(1);

  //display video feed
  opencv.loadImage(video);
  image(video, 0, 0 );

  //style face rectangle
  //NOT BEING USED
  noFill();
  stroke(0, 255, 0);
  noStroke();
  strokeWeight(3);
  Rectangle[] faces = opencv.detect();
  println(faces.length);

  //draw rectangle around seen faces
  //NOT BEING USED
  for (int i = 0; i < faces.length; i++) {
    println(faces[i].x + "," + faces[i].y);
    rect(faces[i].x, faces[i].y, faces[i].width, faces[i].height);
  }

  // only runs if a face is seen
  // captures a face in the box
  // start timer
  
  //if (passedTime > totalTime) {
  number++; // adds number to picture so they count up and not overwrite
  if (faces.length >= 1) {
    face0 = get(faces[0].x, faces[0].y, faces[0].width, faces[0].height);
    //String number0 = "0_" + number + ".jpg";
    //face1.save(number1);
    if (faces.length >= 2) {
      face1 = get(faces[1].x, faces[1].y, faces[1].width, faces[1].height);
      //String number1 = "1_" + number + ".jpg";
      //face2.save(number2);
      if (faces.length >= 3) {
        face2 = get(faces[2].x, faces[2].y, faces[2].width, faces[2].height);
        //String number2 = "2_" + number + ".jpg";
        //face3.save(number3);
        if (faces.length >= 4) {
          face3 = get(faces[3].x, faces[3].y, faces[3].width, faces[3].height);
          //String number3 = "3_" + number + ".jpg";
          //face4.save(number4);
        }
      }
    }
  }

  //println( " 5 seconds have passed! " );
  //savedTime = millis(); // Save the current time to restart the timer!
  //}

  //swap two faces over
  if (faces.length == 2) {
    //resize images to current tracked faces
    face0.resize(faces[1].width, faces[1].height);
    face1.resize(faces[0].width, faces[0].height);

    //place swapped faces
    image(face1, faces[0].x, faces[0].y);
    image(face0, faces[1].x, faces[1].y);
  }

} // close draw


void captureEvent(Capture c) {
  c.read();
}

A summary of the code above is that it initialises the face tracking using OpenCV and stores the information about the rectangle drawn around each face (their x & y coordinates and their width & height) in an array called faces. The sketch then checks to see how many faces there are, up to 4 so far, and using the get() function in Processing, grabs the pixels within the rectangle and saves it in a blank PImage created before the setup. Once it has done that it checks to see of there are two faces on the screen, if there are it resizes face0 (the first face in the array) to be the same size as face1 (the second face) and vice versa. After that it just places the image on top of the video using the coordinates of the swapped faces.

As this was just my test i’ve only written enough to swap over 2 faces (even though it’s capturing up to 4). As I develop this idea I will write the extra code for it to swap over 3 & 4 faces appropriately.

Here is a small example of it working:

[gfycat data_id=”MinorFoolhardyCaterpillar”]

As the whole rectangle taken from each face is being swapped over, it creates a very crude swap which isn’t the best of fits but definitely makes it clear that the faces are being swapped. My initial testing as I was making it had a good response as everyone found the results to be quite funny. Some people had said that they weren’t sure whether to be watching their face or their body which lead to some rather interesting and awkward interactions as people tried to change their behaviour.