Processing: Clean up

Currently I’m very happy with the stage my face swapping project is at, and at the moment it is a matter of cleaning up my code while I think of further developments and improvements I can make on it.

Before my setup I have 4 lines similar to this ‘PImage face0 = createImage(0, 0, RGB);’ which is the PImages used to store the captured faces from the face tracking. I’ve now learned that it doesn’t require a blank image to store it in and therefore can be simplified to just ‘PImage face0;’.

I’ve also re-ordered the swapping for when there are 4 detected faces on screen. In a previous post I drew a table to keep track of my swapping while writing it but I now realise that the order doesn’t make sense for the 4 face swap. The faces are swapped with the next one along in the array (i.e face 1 in place of, 2 moved to 3 etc.) with the exception of the last face which is moved to the beginning of the array to be swapped with the first face (hopefully this description makes sense). This is a first progression to a rather confusing idea I had to try to streamline my code a bit more.

I want to try to store the captured faces in an array rather than 4 individual PImages, as it would hopefully eliminate a lot of the repeated lines of code which have very small differences. I could then capture the faces and swap them using for loops to access various faces in the array and work from there. I’ve tried to get my head around doing this but having the masking and resizing has made it significantly harder than I originally expected so I’m yet to blog about any progress until/ if I work it out.

These small changed might not seem that significant but I feel it is good practice to keep the code as simple as possible without any excess clutter to confuse things or slow them down.

Processing: Masking Fix

I mentioned in my last Processing post that there was a problem with the mask being resized constantly. As the mask was being resized multiple times each frame it soon lost its quality and its masking abilities slowly got worse as the mask got more stretched and blurred.

To get around this I moved the ‘mask = loadImage(“mask.jpg”);’ line from the setup to the draw. With the line in the setup, the mask was loaded once at the beginning and then constantly edited as time passed. Putting it in the draw meant that every time the code runs for each frame, it would reload the original mask so that the quality doesn’t diminish over time and makes sure it doesn’t get blurrier.

Processing: Face Swap Masking

One thing that was bothering me with my face swapping was that it was just placing the whole rectangle of each face on top of the other. This meant that if the rectangle included any of the background behind the face that would be picked up and moved too, giving a very crude face swap. I wanted to get something smoother, something that could make this work just that little bit better.

In case you’ve forgotten what the swapping looks like, heres an example which clearly shows how the whole rectangle is picked up and moved, including background.

Screen-Shot-2014-12-05-at-14.20.13

I was introduced to masking in Processing which masks a part of an image by loading another image and using it to control the alpha channel (opacity). The mask is in greyscale with black being transparent and white being opaque. The in-between grey values are all the opacity values in-between making it possible to get a nice gradual fade.

I created A PImage before the setup to store the mask, and then loaded in the mask file in the setup. Below is a snippet of the code showing how the mask is implemented when swapping the faces over.

  //swap three faces over
  if (faces.length == 3) {
    //resize images to current tracked faces
    face0.resize(faces[2].width*scale, faces[2].height*scale);
    face1.resize(faces[0].width*scale, faces[0].height*scale);
    face2.resize(faces[1].width*scale, faces[1].height*scale);

    //resize and mask images to smooth transition
    mask.resize(faces[2].width*scale, faces[2].height*scale);
    face0.mask(mask);
    mask.resize(faces[0].width*scale, faces[0].height*scale);
    face1.mask(mask);
    mask.resize(faces[1].width*scale, faces[1].height*scale);
    face2.mask(mask);

    //place swapped faces
    image(face0, faces[2].x*scale, faces[2].y*scale);
    image(face1, faces[0].x*scale, faces[0].y*scale);
    image(face2, faces[1].x*scale, faces[1].y*scale);
  }

A mask image is loaded in the setup and each time it’s used it is resized appropriately to fit the face. Currently i’m not really sure if theres a better way of doing this, there probably is but I haven’t yet figured it out. This has made my code look a lot longer and messier but at least it does the job. In the swapping sections. the face images are resized to fit the swapped face, then the mask is resized to match the image and then applied to it before the now masked image is placed down in its new place.

There is one problem currently as the mask is being resized multiple times each frame it gets blurrier and blurrier and its masking properties get worse and worse so that the image end up looking more square again. I need to work out a way around this at some point soon (fixed here).

You may have noticed that the file loaded is called ‘mask9.jpg’, this is because it took a few attempts to get a mask that was roughly right for the faces. Annoyingly, because of the basic nature of the face tracking, the faces don’t always fit exactly the same in the face square so its pretty much impossible to get a mask that will work perfectly on all faces due to the faces being as slightly different angles and such.

Below is a timeline of sorts of my development of trying to find a good shaped mask for the faces. I started with an oval shape as I thought thats what shape faces were but I soon realised that a more rectangular shaped mask was needed. The shape on the far right is mask9, and currently the best fit. I may play around to try and further refine the shape some more but at the moment it does a good enough job for whats needed.masks

 

Here’s a screenshot of the swapping using one of the masks, unfortunately I didn’t write down which mask iteration it was but you get the idea. The swap looks so much better now that the background has been cut out and it fits the face a lot better. Using the mask further benefits my aim of playing with the idea of identity and representation. The swap looks far more realistic now and helps further blur the boundaries between the body and the new face imposed on it. From my experience, the better effect also makes it a lot more fun to play with, which will hopefully make it more interesting for the audience in the Weymouth House foyer.
Screen-Shot-2014-12-05-at-14.52.50

And here’s a bonus image of me face swapped with a picture of my dog using a mask:

Screen-Shot-2014-12-07-at-12.15.04

 

Processing: Face Swap Frame Rate Fix

I mentioned in my last progress update that there was a problem with the frame rate being too slow (~5fps) instead of a more acceptable level around 20/25fps. I spent a few hours trying to work out how to do it by scaling the video using the scale() function, but alas I couldn’t find a way that worked. Fortunately I stumbled across another GitHub repository of Processing examples by Daniel Schiffman. The repository is from a workshop he did about face detection, recognition and tracking methods. In here there are examples using the OpenCV library to detect and track faces. One of the examples is a scaled down version to improve performance – Exactly what I needed!

The example creates a scaled down version of the image to do the tracking on while having another version which is just displayed, which improves the resolution and frame rate of the sketch.

OpenCV does its tracking on an image half the size of the original (as the scale is currently set to 2) which means there are a lot less pixels for it to check for faces which improves performance vastly. When grabbing, resizing and displaying each of the tracked faces now, I had to multiply the coordinates and sizes by the scale so that they would appear as the correct size.

Initially I tried using a scale of 4 (so the tracked image is 4 times smaller) in an effort to maximise performance as much as possible, but I found when the image was that small it struggled to find the faces in the image when they were more distant from the screen. With the resolution i’m working at at the moment, the halved scale works perfectly for what I need.

Here is an example from my code showing the two images. The smaller one in the top corner (which is usually hidden) is what OpenCV tracks to find the faces, then the larger image which is the size of the window is displayed separately.

Screen Shot 2014-12-07 at 11.27.40

 

Below is my updated code to work in the scaling to fix the frame rate. To summarise the changes, a smaller image is created before the setup along with a variable for the scale, so I can adjust it as needed. The OpenCV object is created at the same size as the scaled down size, and the smaller image is loaded into it to be tracked in the background. The faces for the swapping are grabbed from the larger video, so therefore the coordinates and dimensions needed to be multiplied by the scale to match it.

//import libraries
import gab.opencv.*;
import processing.video.*;
import java.awt.*;

Capture video;
OpenCV opencv;

//PImages to store captured faces
PImage face0 = createImage(0, 0, RGB);
PImage face1 = createImage(0, 0, RGB);
PImage face2 = createImage(0, 0, RGB);
PImage face3 = createImage(0, 0, RGB);

//scaled down image for tracking
PImage smaller;
int scale = 2;

void setup() {
  size(640, 480);
  video = new Capture(this, 640, 480, 20);
  opencv = new OpenCV(this, video.width/scale, video.height/scale);
  opencv.loadCascade(OpenCV.CASCADE_FRONTALFACE);  
  frameRate(20);
  video.start();
  //make scaled down image for openCV to track
  smaller = createImage(opencv.width, opencv.height, RGB);
}

void draw() {
  //load smaller image into OpenCV for tracking, better framerate
  opencv.loadImage(smaller);
  //display video feed
  image(video, 0, 0 );


  //style face rectangle
  noFill();
  stroke(0, 255, 0);
  noStroke();
  strokeWeight(3);
  Rectangle[] faces = opencv.detect();
  println(faces.length);

  //draw rectangle around seen faces
  //NOT BEING USED
  for (int i = 0; i < faces.length; i++) {
    println(faces[i].x + "," + faces[i].y);
    rect(faces[i].x, faces[i].y, faces[i].width, faces[i].height);
  } 

  //stores detected faces in images
  //iterate through to check how many faces there are
  if (faces.length >= 1) {
    face0 = get(faces[0].x*scale, faces[0].y*scale, faces[0].width*scale, faces[0].height*scale);
    if (faces.length >= 2) {
      face1 = get(faces[1].x*scale, faces[1].y*scale, faces[1].width*scale, faces[1].height*scale);
      if (faces.length >= 3) {
        face2 = get(faces[2].x*scale, faces[2].y*scale, faces[2].width*scale, faces[2].height*scale);
        if (faces.length >= 4) {
          face3 = get(faces[3].x*scale, faces[3].y*scale, faces[3].width*scale, faces[3].height*scale);
        }
      }
    }
  }

  //swap two faces over
  if (faces.length == 2) {
    //resize images to current tracked faces
    face0.resize(faces[1].width*scale, faces[1].height*scale);
    face1.resize(faces[0].width*scale, faces[0].height*scale);

    //place swapped faces
    image(face1, faces[0].x*scale, faces[0].y*scale);
    image(face0, faces[1].x*scale, faces[1].y*scale);
  }

  //swap three faces over
  if (faces.length == 3) {
    //resize images to current tracked faces
    face0.resize(faces[2].width*scale, faces[2].height*scale);
    face1.resize(faces[0].width*scale, faces[0].height*scale);
    face2.resize(faces[1].width*scale, faces[1].height*scale);

    //place swapped faces
    image(face0, faces[2].x*scale, faces[2].y*scale);
    image(face1, faces[0].x*scale, faces[0].y*scale);
    image(face2, faces[1].x*scale, faces[1].y*scale);
  }

  //swap four faces over
  if (faces.length >= 4) {
    //resize images to current tracked faces
    face0.resize(faces[2].width*scale, faces[2].height*scale);
    face1.resize(faces[0].width*scale, faces[0].height*scale);
    face2.resize(faces[3].width*scale, faces[3].height*scale);
    face3.resize(faces[1].width*scale, faces[1].height*scale);

    //place swapped faces
    image(face0, faces[2].x*scale, faces[2].y*scale);
    image(face1, faces[0].x*scale, faces[0].y*scale);
    image(face2, faces[3].x*scale, faces[3].y*scale);
    image(face3, faces[1].x*scale, faces[1].y*scale);
  }
  
  
} // close draw


void captureEvent(Capture c) {
  c.read();
  
  // Make smaller image
  smaller.copy(video, 0, 0, video.width, video.height, 0, 0, smaller.width, smaller.height);
  smaller.updatePixels();
}

With this new scaled version for tracking, the frame rate is drastically improved and it as a whole runs a lot better than it did before. It is now running at my capped frame rate at 20 fps rather than the 5 fps it was running at before the fix. This is helpful as while making this, i’ve been working on a small version of it, usually 640 x 480px whereas when it is displayed in the foyer it will need to be a much bigger size to fit the screen. With this new method I can increase the size of the window and video, and appropriately adjust the scale so that I can maximise appearance/resolution and performance.

Interaction Design

Interaction design is concerned primarily with interactions between computers and users, this is often referred to as human-computer interaction (or HCI). Interaction design helps to determine the initial user experience such as navigation or how to use something. Good interaction design means that it is intuitive for the user, there isn’t a steep learning curve to work out how the technology and they can pick it up in seconds. For my face swapping I am designing a human-computer interaction for people walking through the Weymouth Hose foyer space.

From the user’s perspective, the experience is continuous as the environment, the user, the screen and whats on the screen all feedback to one another (Kuniavsky, 2003, p.43). With my real-time face swapping the feedback needs to be instantaneous, seeing their faces swapped as soon as they notice whats on screen, and having the swapping keep up with their movements and actions as they pass through the space.

There are four main pieces of information an interaction designer (myself in this case) needs to know during the development process either about whether the designs are on the right track or whether people can actually do what they’re supposed to be able to.

  • Task flows
    • Task flows are the actions which are needed for something interesting to happen. For my face swapping the users need to be facing the camera and screen with their heads relatively straight for it to be able to detect them. It then needs at least 2 visible faces for any interaction to happen so that is a part of the task flow too.
  • Predictability and consistency
    • The predictability determines how comfortable users will feel with the task flows; does the face swapping work intuitively or is there a really complex method to go around it. For my work, I feel the only thing that might not be obvious initially is that it is supposed to be swapping faces, especially if there is only one person present at a time, they wouldn’t be able to understand what its supposed to do and why it isn’t working.
  • The relationship between features and emphasis on specific elements
    • This regards the relationship between the normal video feed, and the swapped faces. The face swapping needs to be obvious enough that it gets noticed, but subtle enough that it blends back into the video to look relatively seamless. While this seems quite paradoxical, I feel the key to creating a good user experience is making the swapped faces look as real as possible, so that possibly it isn’t noticed initially until they go in for closer inspection and see their their on-screen representation is different.
  • Different audiences
    • My face swapping has to work for all the different types of people which would be passing through the foyer space. This means the camera needs to be in a place where it can detect tall people and short people, while still capturing, masking and swapping their faces. It’s important for all ages to understand how it works and what is happening on screen too. One thing that has become apparent in my testing is that it doesn’t always recognise people wearing glasses as haar-cascade for the face tracking is made for people without.

References

Kuniavsky, M., 2003. Observing the User Experience: A Practitioner’s Guide to User Research. Morgan Kaufmann.

http://www.uxbooth.com/articles/complete-beginners-guide-to-interaction-design/

Performance Theory

(Mentions of face swapping have been highlighted for skim-reading)

In 1959. an American Sociologist, Erving Goffman, published a book titled The Presentation of Self in Everyday Life. In this book he uses imagery of theatre to portray the importance of human and social action and interaction. He refers to it as ‘the dramaturgical model of social life’. The model relates social interactions to a theatre, and the people you interact with in everyday life as actors on stage who each play a varying role. The audience is other people who observe the roleplaying and in turn react to the performance.

Goffman uses the term ‘performance’ to describe all of the activities of an individual in front of an audience or set of observers. It is through this performance that the individual can give social meaning to themselves, to others, and their context. The audience is not always aware of the performance but they are constantly attributing meaning to it and the actor themselves. This idea can be related to one I mentioned in a previous post about how our behaviour changes when we are being watched. The audience is always affecting the performance whether the actor is aware of it or not. It is also important for the actor to stay ‘in character’. The performance has to conform to the correct set of signals and behaviours, and anything outside of this detracts from the performance and could mislead the audience. All of our actions form part of our identity, who we, and other people think we are. Our behaviour needs to conform our previous patterns of behaviour (our character) or it seems out-of-place and weird. 

The appearance of the actor or individual functions to portray social statuses and people’s roles in society. These can include gender, class, status, age, occupation etc. Appearance includes clothes, body language, hair style etc. The way we chose to present ourselves plays a big part in the way others view us. My face swapping idea plays with the idea of appearance as the onscreen representation is altered, disrupting the interactors sense of appearance and self. The persons face (which is usually unique to a person) is then associated with a different sartorial discourse, age and possibly gender, creating tension between the performance and the actor’s front.

The actor’s front’, as defined by Goffman, is the part of the performance which defines the situation for the audience. It is the image or impression they are trying to give off with their appearance and performance. The front can be seen as a standardised mask for the performer to control the way in which they’re perceived by the audience. Goffman likens a front to a script for the actor containing stereotyped expectations of how they should behave. A personal front contains all the items needed to perform and is usually identifiable by the audience as a representation of the specific actor. Face swapping could be seen as a way of altering these personal fronts by interchanging pieces between actors. When the appearance is disrupted, the script is also disrupted as there is a contrast between the head and the new body it’s imposed upon. As the actors watch an altered version of themselves, they have to try and manage two separate discourses of the self rather than just one. Certain situations and scenarios have social scripts that define how the actor should behave in the given situation. When the actor is put into a new situation they or establishes a new role, they usually construct a new front or script from a combination of past fronts, rarely creating something completely new. The actor has to use their past experience to try to react to the environment to find/ create a front which best suits it.

In a staged performance there are three main locations for interactions; front-stage, back-stage and off-stage. Front-stage is where the audience is watching. The actor needs to conform to their performance, appearance and front, following social conventions which have meaning to the audience. The actor is often aware they’re being watched and therefore acts accordingly. Back stage the actor  may be able to act a little differently and is able to step out of character. It is a place where no members of the general audience can see. It is usually where the actor can be representative of their true self and get rid of roles they have to play in public. The backstage area can occur at home with a close group of friends for example, where people can be more informal and act completely differently to what is usually expected. It has been argued that there is no true back stage as there will always be members of the back-stage audience who aren’t as trusted and stand on the fringes of the group. Finally off-stage is when the actor isn’t involved with the performance, when they can interact with members of the audience directly and independently of their performance on stage. This is where a specific performance can be given as the audience is selected and segmented. For me, an example of this could be when interacting with the employee at the till in a shop. When you go to the till to buy a product, you briefly step out of your usual performance and front and put on a new one specifically for interactions with the person at the till. The new front is usually more polite and courteous than the usual self and is put on specifically for the interaction with a certain person in the audience.

Face swapping playfully alters front stage performances, creating two stages with different audiences – one in the foyer space and one on-screen. As people become engrossed with their altered performance on-screen, hopefully they forget about their performance in the actual space as they adapt their performance to fit the face swapped reality. The aim is to change the way people behave, trying to deviate from the social norms people often try and follow when out in public. For people who are in the space but can’t see the screen, the behaviour of the interactors of the piece would appear to sit outside of the norms, creating an inconsistency and contractions with everyone else on the stage. It would be interesting to see if this does actually happen or if people aren’t interested in their face-swapped performance and ignore it and carry on walking.

References

http://sociology.about.com/od/Works/a/Presentation-Of-Self-Everyday-Life.htm

http://en.wikipedia.org/wiki/Dramaturgy_(sociology)

http://www.csulb.edu/~hmarlowe/SOC335/Goffman_Dramaturgical_Model.pdf

Goffman, E. (1956). The Presentation of Self in Everyday Life. New York: Doubleday.

 

 

 

Processing: Face Swap Next Iteration

My first iteration had some very messy code so my first step was to clean out the bits that weren’t being used to cut it down to the barebones of the parts I needed. The next step was to work out how I was going to swap the faces over for when there were 3 and 4 faces. At the moment I’m limiting it to 4 faces due to the stress it puts on my laptop, but for all intents and purposes I feel that 4 faces is enough. If the tracking detects any more than that it wont swapping the faces over and they’ll stay looking normal.

As a reference I drew out a table so I could keep track of which faces I was swapping. I tried doing it all in my head but kept getting it wrong so this is what I had to resort to. The faces along the top represent the faces in the array and the ‘swaps’ are which faces they will be swapped with for 2, 3, and 4 faces.

Screen Shot 2014-12-06 at 11.13.37

From here it was a simple matter of writing the if statements needed for up to 4 faces, resizing and placing the images appropriately:

//swap two faces over
  if (faces.length == 2) {
    //resize images to current tracked faces
    face0.resize(faces[1].width, faces[1].height);
    face1.resize(faces[0].width, faces[0].height);

    //place swapped faces
    image(face1, faces[0].x, faces[0].y);
    image(face0, faces[1].x, faces[1].y);
  }

  //swap three faces over
  if (faces.length == 3) {
    //resize images to current tracked faces
    face0.resize(faces[2].width, faces[2].height);
    face1.resize(faces[0].width, faces[0].height);
    face2.resize(faces[1].width, faces[1].height);

    //place swapped faces
    image(face0, faces[2].x, faces[2].y);
    image(face1, faces[0].x, faces[0].y);
    image(face2, faces[1].x, faces[1].y);
  }

  //swap four faces over
  if (faces.length >= 4) {
    //resize images to current tracked faces
    face0.resize(faces[2].width, faces[2].height);
    face1.resize(faces[0].width, faces[0].height);
    face2.resize(faces[3].width, faces[3].height);
    face3.resize(faces[1].width, faces[1].height);

    //place swapped faces
    image(face0, faces[2].x, faces[2].y);
    image(face1, faces[0].x, faces[0].y);
    image(face2, faces[3].x, faces[3].y);
    image(face3, faces[1].x, faces[1].y);
  }

The last if statement is greater than or equal to 4 faces so that if the camera picks up more than 4 faces, the first 4 will still swap but anything over that won’t be effected. One thing that bothers me with this code is that there is a lot of repetition of very similar lines of code for each statement. While it still works fine I would like to try to clean it up if at all possible to make it look a bit neater.

One thing I didn’t mention in my first post is the frame rate when it runs. In the example which comes with the OpenCV library (my basis for building this), the video is halved and then scaled back up to fit the window. This is so that the tracking is working on a lower resolution version of the video so that it can run faster and improve performance. In my attempts, the scaling kept confusing me while trying to grab the faces so I turned it off. This means that it is doing the tracking on a relatively high resolution video which takes longer to check for faces due to the larger number of pixels to check and affects the frame rate. My sketches currently run at about 4/5 frames per second which isn’t exactly very good. If it was run on a more powerful computer it could work a little better but I need to find a way to optimise the code and improve the frame rate up to a more reasonable level around 20/25 fps.

Below is it swapping 3 faces which works as expected. What I did notice is that sometimes it sees the faces in a different order, causing them to jump about a bit from face to face which could either be a good or bad thing. It could be considered a good thing as it makes it a bit more interesting as you have to keep track of where your face is going. Although it could get quite annoying and detract from the effect if the face doesn’t stay in one place long enough. Also the tracking isn’t exactly perfect. As you can see in the picture below there are 4 faces but one of them is being ignored for some reason. It could be a combination of lighting, positioning, the angle of the face or something completely different which is stopping it from being seen as a face by the sketch. At the moment I don’t think this is a problem with my code but rather the limitations of the Haar Cascade which is currently being used to detect the faces.

Screen-Shot-2014-12-05-at-14.20.13

Processing: Face Swap Start

To get started on face swapping I began by altering one of my old sketches which captured and saved faces as images. Before, I was trying to get the image out of Processing to do something with it but now it can just say within the Processing environment and I can work from there. Below is my code, commented so that hopefully it makes sense to whomever is reading it. A lot of the code in there isn’t being used (like the timer for example) but I left it in there to show how it was a development upon one of my old ideas where I was experimenting with what I could do.

Quick note:  I’ve recently noticed that the plugin i’ve used for inserting code has sometimes changed my greater than (>) and less than symbols (<) to &gt; and &lt; instead for some reason. 

(Theres a summary of what the code does below)

import gab.opencv.*;
import processing.video.*;
import java.awt.*;

Capture video;
OpenCV opencv;

//PImages to store captured faces
PImage face0 = createImage(0, 0, RGB);
PImage face1 = createImage(0, 0, RGB);
PImage face2 = createImage(0, 0, RGB);
PImage face3 = createImage(0, 0, RGB);

//values for timer
int savedTime;
int totalTime = 5000; // 5 second delay
int number = 0; //number of picture

void setup() {
  size(640, 480);
  video = new Capture(this, 640, 480, 20);
  opencv = new OpenCV(this, 640, 480);
  opencv.loadCascade(OpenCV.CASCADE_FRONTALFACE);  
  frameRate(20);
  video.start();
  savedTime = millis();
}

void draw() {
  // Calculate how much time has passed
  int passedTime = millis() - savedTime;

  //should be scale 2 but made it 1 so it looks better, but makes it laggy
  scale(1);

  //display video feed
  opencv.loadImage(video);
  image(video, 0, 0 );

  //style face rectangle
  //NOT BEING USED
  noFill();
  stroke(0, 255, 0);
  noStroke();
  strokeWeight(3);
  Rectangle[] faces = opencv.detect();
  println(faces.length);

  //draw rectangle around seen faces
  //NOT BEING USED
  for (int i = 0; i < faces.length; i++) {
    println(faces[i].x + "," + faces[i].y);
    rect(faces[i].x, faces[i].y, faces[i].width, faces[i].height);
  }

  // only runs if a face is seen
  // captures a face in the box
  // start timer
  
  //if (passedTime > totalTime) {
  number++; // adds number to picture so they count up and not overwrite
  if (faces.length >= 1) {
    face0 = get(faces[0].x, faces[0].y, faces[0].width, faces[0].height);
    //String number0 = "0_" + number + ".jpg";
    //face1.save(number1);
    if (faces.length >= 2) {
      face1 = get(faces[1].x, faces[1].y, faces[1].width, faces[1].height);
      //String number1 = "1_" + number + ".jpg";
      //face2.save(number2);
      if (faces.length >= 3) {
        face2 = get(faces[2].x, faces[2].y, faces[2].width, faces[2].height);
        //String number2 = "2_" + number + ".jpg";
        //face3.save(number3);
        if (faces.length >= 4) {
          face3 = get(faces[3].x, faces[3].y, faces[3].width, faces[3].height);
          //String number3 = "3_" + number + ".jpg";
          //face4.save(number4);
        }
      }
    }
  }

  //println( " 5 seconds have passed! " );
  //savedTime = millis(); // Save the current time to restart the timer!
  //}

  //swap two faces over
  if (faces.length == 2) {
    //resize images to current tracked faces
    face0.resize(faces[1].width, faces[1].height);
    face1.resize(faces[0].width, faces[0].height);

    //place swapped faces
    image(face1, faces[0].x, faces[0].y);
    image(face0, faces[1].x, faces[1].y);
  }

} // close draw


void captureEvent(Capture c) {
  c.read();
}

A summary of the code above is that it initialises the face tracking using OpenCV and stores the information about the rectangle drawn around each face (their x & y coordinates and their width & height) in an array called faces. The sketch then checks to see how many faces there are, up to 4 so far, and using the get() function in Processing, grabs the pixels within the rectangle and saves it in a blank PImage created before the setup. Once it has done that it checks to see of there are two faces on the screen, if there are it resizes face0 (the first face in the array) to be the same size as face1 (the second face) and vice versa. After that it just places the image on top of the video using the coordinates of the swapped faces.

As this was just my test i’ve only written enough to swap over 2 faces (even though it’s capturing up to 4). As I develop this idea I will write the extra code for it to swap over 3 & 4 faces appropriately.

Here is a small example of it working:

[gfycat data_id=”MinorFoolhardyCaterpillar”]

As the whole rectangle taken from each face is being swapped over, it creates a very crude swap which isn’t the best of fits but definitely makes it clear that the faces are being swapped. My initial testing as I was making it had a good response as everyone found the results to be quite funny. Some people had said that they weren’t sure whether to be watching their face or their body which lead to some rather interesting and awkward interactions as people tried to change their behaviour.

Some Inspiration

After coming up with my idea I looked for some inspiration to see how people were using face detection and swapping at the moment. By far the best and closest thing to my idea was this: (Faces from arturo castro on Vimeo.)

//player.vimeo.com/video/29279198

This face swapping looked a little more advanced than what i’m probably capable of and was made using OpenFrameworks. Instead of OpenCV like I am going to use, this uses ofxFaceTracker, an addon for OpenFrameworks which does more complicated face tracking. The premise of this example is a different to mine, which in reality is a good thing as I don’t want to be making something thats already been done. With this, A picture is used to overlay on to a face, and in real time is warped to fit the actual face as moves; like a puppet mask type thing. While this is a very interesting effect its not quite what I wanted to achieve but is still good inspiration.