Processing: Face Swap Frame Rate Fix

I mentioned in my last progress update that there was a problem with the frame rate being too slow (~5fps) instead of a more acceptable level around 20/25fps. I spent a few hours trying to work out how to do it by scaling the video using the scale() function, but alas I couldn’t find a way that worked. Fortunately I stumbled across another GitHub repository of Processing examples by Daniel Schiffman. The repository is from a workshop he did about face detection, recognition and tracking methods. In here there are examples using the OpenCV library to detect and track faces. One of the examples is a scaled down version to improve performance – Exactly what I needed!

The example creates a scaled down version of the image to do the tracking on while having another version which is just displayed, which improves the resolution and frame rate of the sketch.

OpenCV does its tracking on an image half the size of the original (as the scale is currently set to 2) which means there are a lot less pixels for it to check for faces which improves performance vastly. When grabbing, resizing and displaying each of the tracked faces now, I had to multiply the coordinates and sizes by the scale so that they would appear as the correct size.

Initially I tried using a scale of 4 (so the tracked image is 4 times smaller) in an effort to maximise performance as much as possible, but I found when the image was that small it struggled to find the faces in the image when they were more distant from the screen. With the resolution i’m working at at the moment, the halved scale works perfectly for what I need.

Here is an example from my code showing the two images. The smaller one in the top corner (which is usually hidden) is what OpenCV tracks to find the faces, then the larger image which is the size of the window is displayed separately.

Screen Shot 2014-12-07 at 11.27.40


Below is my updated code to work in the scaling to fix the frame rate. To summarise the changes, a smaller image is created before the setup along with a variable for the scale, so I can adjust it as needed. The OpenCV object is created at the same size as the scaled down size, and the smaller image is loaded into it to be tracked in the background. The faces for the swapping are grabbed from the larger video, so therefore the coordinates and dimensions needed to be multiplied by the scale to match it.

//import libraries
import gab.opencv.*;
import java.awt.*;

Capture video;
OpenCV opencv;

//PImages to store captured faces
PImage face0 = createImage(0, 0, RGB);
PImage face1 = createImage(0, 0, RGB);
PImage face2 = createImage(0, 0, RGB);
PImage face3 = createImage(0, 0, RGB);

//scaled down image for tracking
PImage smaller;
int scale = 2;

void setup() {
  size(640, 480);
  video = new Capture(this, 640, 480, 20);
  opencv = new OpenCV(this, video.width/scale, video.height/scale);
  //make scaled down image for openCV to track
  smaller = createImage(opencv.width, opencv.height, RGB);

void draw() {
  //load smaller image into OpenCV for tracking, better framerate
  //display video feed
  image(video, 0, 0 );

  //style face rectangle
  stroke(0, 255, 0);
  Rectangle[] faces = opencv.detect();

  //draw rectangle around seen faces
  for (int i = 0; i < faces.length; i++) {
    println(faces[i].x + "," + faces[i].y);
    rect(faces[i].x, faces[i].y, faces[i].width, faces[i].height);

  //stores detected faces in images
  //iterate through to check how many faces there are
  if (faces.length >= 1) {
    face0 = get(faces[0].x*scale, faces[0].y*scale, faces[0].width*scale, faces[0].height*scale);
    if (faces.length >= 2) {
      face1 = get(faces[1].x*scale, faces[1].y*scale, faces[1].width*scale, faces[1].height*scale);
      if (faces.length >= 3) {
        face2 = get(faces[2].x*scale, faces[2].y*scale, faces[2].width*scale, faces[2].height*scale);
        if (faces.length >= 4) {
          face3 = get(faces[3].x*scale, faces[3].y*scale, faces[3].width*scale, faces[3].height*scale);

  //swap two faces over
  if (faces.length == 2) {
    //resize images to current tracked faces
    face0.resize(faces[1].width*scale, faces[1].height*scale);
    face1.resize(faces[0].width*scale, faces[0].height*scale);

    //place swapped faces
    image(face1, faces[0].x*scale, faces[0].y*scale);
    image(face0, faces[1].x*scale, faces[1].y*scale);

  //swap three faces over
  if (faces.length == 3) {
    //resize images to current tracked faces
    face0.resize(faces[2].width*scale, faces[2].height*scale);
    face1.resize(faces[0].width*scale, faces[0].height*scale);
    face2.resize(faces[1].width*scale, faces[1].height*scale);

    //place swapped faces
    image(face0, faces[2].x*scale, faces[2].y*scale);
    image(face1, faces[0].x*scale, faces[0].y*scale);
    image(face2, faces[1].x*scale, faces[1].y*scale);

  //swap four faces over
  if (faces.length >= 4) {
    //resize images to current tracked faces
    face0.resize(faces[2].width*scale, faces[2].height*scale);
    face1.resize(faces[0].width*scale, faces[0].height*scale);
    face2.resize(faces[3].width*scale, faces[3].height*scale);
    face3.resize(faces[1].width*scale, faces[1].height*scale);

    //place swapped faces
    image(face0, faces[2].x*scale, faces[2].y*scale);
    image(face1, faces[0].x*scale, faces[0].y*scale);
    image(face2, faces[3].x*scale, faces[3].y*scale);
    image(face3, faces[1].x*scale, faces[1].y*scale);
} // close draw

void captureEvent(Capture c) {;
  // Make smaller image
  smaller.copy(video, 0, 0, video.width, video.height, 0, 0, smaller.width, smaller.height);

With this new scaled version for tracking, the frame rate is drastically improved and it as a whole runs a lot better than it did before. It is now running at my capped frame rate at 20 fps rather than the 5 fps it was running at before the fix. This is helpful as while making this, i’ve been working on a small version of it, usually 640 x 480px whereas when it is displayed in the foyer it will need to be a much bigger size to fit the screen. With this new method I can increase the size of the window and video, and appropriately adjust the scale so that I can maximise appearance/resolution and performance.


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: