Processing: Face Swap Masking

One thing that was bothering me with my face swapping was that it was just placing the whole rectangle of each face on top of the other. This meant that if the rectangle included any of the background behind the face that would be picked up and moved too, giving a very crude face swap. I wanted to get something smoother, something that could make this work just that little bit better.

In case you’ve forgotten what the swapping looks like, heres an example which clearly shows how the whole rectangle is picked up and moved, including background.


I was introduced to masking in Processing which masks a part of an image by loading another image and using it to control the alpha channel (opacity). The mask is in greyscale with black being transparent and white being opaque. The in-between grey values are all the opacity values in-between making it possible to get a nice gradual fade.

I created A PImage before the setup to store the mask, and then loaded in the mask file in the setup. Below is a snippet of the code showing how the mask is implemented when swapping the faces over.

  //swap three faces over
  if (faces.length == 3) {
    //resize images to current tracked faces
    face0.resize(faces[2].width*scale, faces[2].height*scale);
    face1.resize(faces[0].width*scale, faces[0].height*scale);
    face2.resize(faces[1].width*scale, faces[1].height*scale);

    //resize and mask images to smooth transition
    mask.resize(faces[2].width*scale, faces[2].height*scale);
    mask.resize(faces[0].width*scale, faces[0].height*scale);
    mask.resize(faces[1].width*scale, faces[1].height*scale);

    //place swapped faces
    image(face0, faces[2].x*scale, faces[2].y*scale);
    image(face1, faces[0].x*scale, faces[0].y*scale);
    image(face2, faces[1].x*scale, faces[1].y*scale);

A mask image is loaded in the setup and each time it’s used it is resized appropriately to fit the face. Currently i’m not really sure if theres a better way of doing this, there probably is but I haven’t yet figured it out. This has made my code look a lot longer and messier but at least it does the job. In the swapping sections. the face images are resized to fit the swapped face, then the mask is resized to match the image and then applied to it before the now masked image is placed down in its new place.

There is one problem currently as the mask is being resized multiple times each frame it gets blurrier and blurrier and its masking properties get worse and worse so that the image end up looking more square again. I need to work out a way around this at some point soon (fixed here).

You may have noticed that the file loaded is called ‘mask9.jpg’, this is because it took a few attempts to get a mask that was roughly right for the faces. Annoyingly, because of the basic nature of the face tracking, the faces don’t always fit exactly the same in the face square so its pretty much impossible to get a mask that will work perfectly on all faces due to the faces being as slightly different angles and such.

Below is a timeline of sorts of my development of trying to find a good shaped mask for the faces. I started with an oval shape as I thought thats what shape faces were but I soon realised that a more rectangular shaped mask was needed. The shape on the far right is mask9, and currently the best fit. I may play around to try and further refine the shape some more but at the moment it does a good enough job for whats needed.masks


Here’s a screenshot of the swapping using one of the masks, unfortunately I didn’t write down which mask iteration it was but you get the idea. The swap looks so much better now that the background has been cut out and it fits the face a lot better. Using the mask further benefits my aim of playing with the idea of identity and representation. The swap looks far more realistic now and helps further blur the boundaries between the body and the new face imposed on it. From my experience, the better effect also makes it a lot more fun to play with, which will hopefully make it more interesting for the audience in the Weymouth House foyer.

And here’s a bonus image of me face swapped with a picture of my dog using a mask:




Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s

%d bloggers like this: