Category Archives: Theory

Augmented Reality

On my first day of user testing in the space, One of the people passing by commented saying ‘I think its augmented reality’ which is something I never really considered while making it. Augmented Reality takes computer generated information such as images, audio and video, and overlaying them over a real-time environment (Kipper & Rampolla, 2012, p.1). Augmented Reality is often confused with Virtual Reality which immerses the user into synthetic, digital world and can’t see the real world around them. Augmented Reality allows digital objects to be superimposed with or composited with the real world and can be used to supplement and enhance reality.

In the context of my face swapping, an Augmented Reality is created on screen where people have their faces swapped over. It is using a video feed of the space in front of the screen and superimposing the faces it sees into different locations. The faces are digitally altered before they’re placed back down on screen by blurring the edges of them in an attempt to blend them into their new location. The face swapping also managed to keep up with the video in real time, be it a bit jittery due to capturing a new face, resizing and masking it 20 times a second.

A real world implementation of Augmented Reality is the Magic Mirror. The Magic Mirror is a digital screen which allows you to try on different clothes and outfits in an Augmented Reality space. It uses a Kinect sensor to track body movements so it can superimpose 3D clothes onto you and you move and rotate to see how it would look while you’re wearing it. Using the Kinect it is able to recognise gestures in order to change the clothes being modelled by swiping to the sided, or raising a hand to take a photo. Obviously this technology is far more advanced than what I’m doing with my face swapping but it goes to show how it does have real world implementations.

I feel that using Augmented Reality is a good idea for creating an interactive information graphic display as it makes it much harder for people to resist looking at it. I feel that most people are naturally quite narcissistic in that they can’t resist looking at their own reflection, be it in a mirror, shop window or on a screen from a video camera. The Augmented Reality element of this reflection then makes it more engaging for the user/ audience as their appearance and representation has been manipulated without their explicit consent.
References

Kipper, G., Rampolla, J., 2012. Augmented Reality: An Emerging Technologies Guide to AR. Elsevier.

Advertisements

The Hawthorne Effect

I have recently been introduced to a psychological theory known as the Hawthorne effect, observer effect. The Hawthorn effect is a type of reactivity in which individuals improve or modify their behaviour in response to them knowing they’re being watched. The original study took place in a business setting where they were experimenting with different levels of light to see if it made the workers more or less productive. Though the results of the study showed an increase in productivity no matter what the lighting, but when the experiment was over (and they were no longer being observed) productivity dropped again. They concluded that the increase in productivity was due to a motivational effect as the workers were being watched and interest was showed in them and their activities.

I have touched on an idea like this in a previous blog post about a psychological experiment about how being watched changes behaviour. While this isn’t directly related to my product (as I’m not measuring productivity or things like that) it does help to exemplify my idea of people changing behaviour and reacting to being observed. In my face swapping project, the participants (willing or otherwise) are being observed by a camera and it is being shown on one of the screens. In theory, this in itself should change people’s behaviour as they become aware of them being watched, especially as it is out of the norm for the media foyer space as there isn’t usually a camera watching them. With their representations being altered (i.e face swapped) it should further influence their behaviour as notice and hopefully play around with it.

Arguably, none of this may be true as the experiment involved a person (or group of people) doing the observing rather than a camera and that is far more obtrusive and would have a larger effect on people’s behaviour in my opinion. However I have high hopes for it working providing the interactive nature of the piece is obvious enough and people are interested.

 

http://psychology.about.com/od/hindex/g/def_hawthorn.htm

 

Interaction Design

Interaction design is concerned primarily with interactions between computers and users, this is often referred to as human-computer interaction (or HCI). Interaction design helps to determine the initial user experience such as navigation or how to use something. Good interaction design means that it is intuitive for the user, there isn’t a steep learning curve to work out how the technology and they can pick it up in seconds. For my face swapping I am designing a human-computer interaction for people walking through the Weymouth Hose foyer space.

From the user’s perspective, the experience is continuous as the environment, the user, the screen and whats on the screen all feedback to one another (Kuniavsky, 2003, p.43). With my real-time face swapping the feedback needs to be instantaneous, seeing their faces swapped as soon as they notice whats on screen, and having the swapping keep up with their movements and actions as they pass through the space.

There are four main pieces of information an interaction designer (myself in this case) needs to know during the development process either about whether the designs are on the right track or whether people can actually do what they’re supposed to be able to.

  • Task flows
    • Task flows are the actions which are needed for something interesting to happen. For my face swapping the users need to be facing the camera and screen with their heads relatively straight for it to be able to detect them. It then needs at least 2 visible faces for any interaction to happen so that is a part of the task flow too.
  • Predictability and consistency
    • The predictability determines how comfortable users will feel with the task flows; does the face swapping work intuitively or is there a really complex method to go around it. For my work, I feel the only thing that might not be obvious initially is that it is supposed to be swapping faces, especially if there is only one person present at a time, they wouldn’t be able to understand what its supposed to do and why it isn’t working.
  • The relationship between features and emphasis on specific elements
    • This regards the relationship between the normal video feed, and the swapped faces. The face swapping needs to be obvious enough that it gets noticed, but subtle enough that it blends back into the video to look relatively seamless. While this seems quite paradoxical, I feel the key to creating a good user experience is making the swapped faces look as real as possible, so that possibly it isn’t noticed initially until they go in for closer inspection and see their their on-screen representation is different.
  • Different audiences
    • My face swapping has to work for all the different types of people which would be passing through the foyer space. This means the camera needs to be in a place where it can detect tall people and short people, while still capturing, masking and swapping their faces. It’s important for all ages to understand how it works and what is happening on screen too. One thing that has become apparent in my testing is that it doesn’t always recognise people wearing glasses as haar-cascade for the face tracking is made for people without.

References

Kuniavsky, M., 2003. Observing the User Experience: A Practitioner’s Guide to User Research. Morgan Kaufmann.

http://www.uxbooth.com/articles/complete-beginners-guide-to-interaction-design/

Performance Theory

(Mentions of face swapping have been highlighted for skim-reading)

In 1959. an American Sociologist, Erving Goffman, published a book titled The Presentation of Self in Everyday Life. In this book he uses imagery of theatre to portray the importance of human and social action and interaction. He refers to it as ‘the dramaturgical model of social life’. The model relates social interactions to a theatre, and the people you interact with in everyday life as actors on stage who each play a varying role. The audience is other people who observe the roleplaying and in turn react to the performance.

Goffman uses the term ‘performance’ to describe all of the activities of an individual in front of an audience or set of observers. It is through this performance that the individual can give social meaning to themselves, to others, and their context. The audience is not always aware of the performance but they are constantly attributing meaning to it and the actor themselves. This idea can be related to one I mentioned in a previous post about how our behaviour changes when we are being watched. The audience is always affecting the performance whether the actor is aware of it or not. It is also important for the actor to stay ‘in character’. The performance has to conform to the correct set of signals and behaviours, and anything outside of this detracts from the performance and could mislead the audience. All of our actions form part of our identity, who we, and other people think we are. Our behaviour needs to conform our previous patterns of behaviour (our character) or it seems out-of-place and weird. 

The appearance of the actor or individual functions to portray social statuses and people’s roles in society. These can include gender, class, status, age, occupation etc. Appearance includes clothes, body language, hair style etc. The way we chose to present ourselves plays a big part in the way others view us. My face swapping idea plays with the idea of appearance as the onscreen representation is altered, disrupting the interactors sense of appearance and self. The persons face (which is usually unique to a person) is then associated with a different sartorial discourse, age and possibly gender, creating tension between the performance and the actor’s front.

The actor’s front’, as defined by Goffman, is the part of the performance which defines the situation for the audience. It is the image or impression they are trying to give off with their appearance and performance. The front can be seen as a standardised mask for the performer to control the way in which they’re perceived by the audience. Goffman likens a front to a script for the actor containing stereotyped expectations of how they should behave. A personal front contains all the items needed to perform and is usually identifiable by the audience as a representation of the specific actor. Face swapping could be seen as a way of altering these personal fronts by interchanging pieces between actors. When the appearance is disrupted, the script is also disrupted as there is a contrast between the head and the new body it’s imposed upon. As the actors watch an altered version of themselves, they have to try and manage two separate discourses of the self rather than just one. Certain situations and scenarios have social scripts that define how the actor should behave in the given situation. When the actor is put into a new situation they or establishes a new role, they usually construct a new front or script from a combination of past fronts, rarely creating something completely new. The actor has to use their past experience to try to react to the environment to find/ create a front which best suits it.

In a staged performance there are three main locations for interactions; front-stage, back-stage and off-stage. Front-stage is where the audience is watching. The actor needs to conform to their performance, appearance and front, following social conventions which have meaning to the audience. The actor is often aware they’re being watched and therefore acts accordingly. Back stage the actor  may be able to act a little differently and is able to step out of character. It is a place where no members of the general audience can see. It is usually where the actor can be representative of their true self and get rid of roles they have to play in public. The backstage area can occur at home with a close group of friends for example, where people can be more informal and act completely differently to what is usually expected. It has been argued that there is no true back stage as there will always be members of the back-stage audience who aren’t as trusted and stand on the fringes of the group. Finally off-stage is when the actor isn’t involved with the performance, when they can interact with members of the audience directly and independently of their performance on stage. This is where a specific performance can be given as the audience is selected and segmented. For me, an example of this could be when interacting with the employee at the till in a shop. When you go to the till to buy a product, you briefly step out of your usual performance and front and put on a new one specifically for interactions with the person at the till. The new front is usually more polite and courteous than the usual self and is put on specifically for the interaction with a certain person in the audience.

Face swapping playfully alters front stage performances, creating two stages with different audiences – one in the foyer space and one on-screen. As people become engrossed with their altered performance on-screen, hopefully they forget about their performance in the actual space as they adapt their performance to fit the face swapped reality. The aim is to change the way people behave, trying to deviate from the social norms people often try and follow when out in public. For people who are in the space but can’t see the screen, the behaviour of the interactors of the piece would appear to sit outside of the norms, creating an inconsistency and contractions with everyone else on the stage. It would be interesting to see if this does actually happen or if people aren’t interested in their face-swapped performance and ignore it and carry on walking.

References

http://sociology.about.com/od/Works/a/Presentation-Of-Self-Everyday-Life.htm

http://en.wikipedia.org/wiki/Dramaturgy_(sociology)

http://www.csulb.edu/~hmarlowe/SOC335/Goffman_Dramaturgical_Model.pdf

Goffman, E. (1956). The Presentation of Self in Everyday Life. New York: Doubleday.

 

 

 

Some Inspiration

After coming up with my idea I looked for some inspiration to see how people were using face detection and swapping at the moment. By far the best and closest thing to my idea was this: (Faces from arturo castro on Vimeo.)

//player.vimeo.com/video/29279198

This face swapping looked a little more advanced than what i’m probably capable of and was made using OpenFrameworks. Instead of OpenCV like I am going to use, this uses ofxFaceTracker, an addon for OpenFrameworks which does more complicated face tracking. The premise of this example is a different to mine, which in reality is a good thing as I don’t want to be making something thats already been done. With this, A picture is used to overlay on to a face, and in real time is warped to fit the actual face as moves; like a puppet mask type thing. While this is a very interesting effect its not quite what I wanted to achieve but is still good inspiration.

My Final Idea

My experimentation with processing has led me down the route of face-tracking based interactions. My ideas all stemmed from the basic example with the OpenCV library which put a green rectangle around the faces it saw on screen. With this example I am able to get the x & y coordinates, the width & height of the faces and the number of faces it sees in the room. It was all a matter of experimenting what could be done with these parameters to create an interesting interaction.

My final idea for the project is to do face swapping of the faces tracked on screen. This idea builds upon one of my previous ideas where I was capturing the pixels of the tracked face and saving it as an image. The process can initially be broken down into a few steps of what I will need to write to get me started in the right direction:

  1. Track the faces on the video feed.
  2. Capture the tracked faces and save them as an image within Processing.
  3. Resize the images to match the face they will be swapped with.
  4. Display the resized faces on top of the video feed in the appropriate location.

The piece is going to be linked with ideas about changing behaviour based on ideas around identity and representation, looking at audience theories (i.e. reaction to being watched), and playing with their sense of self via altered representations.

My initial thoughts here are that it’s going to be interrupting and altering peoples self image as they will see their face and their body but not together as they should be which will change their behaviour as they will have to consider their facial expression and body language more separately as they will be torn between watching one or the other. The aim is to get some playful interactions where people actually stop to look at the piece rather than just a quick glance in passing.

Face swapping doesn’t really seem like that much of an original idea as I’m sure its something anyone with photoshop has done before in the past, including me. e.g:

252576_2132802882275_4629589_n

The process for face swapping involved choosing a face, cutting it out and placing it down where there are other faces in the image. However this process has never really been quick or easy and requires a decent degree of accuracy to look good.

I then thought if there were any apps which did this. I knew there were apps which would do face swapping on image static images but I was looking for something which would do it on real time video. The closest thing I could find was Face Stealer. However this swapped the face with another face from an image using a series of control points to match up the eyes, mouth and jawline. From my searching I couldn’t find anything that swapped the faces of 2 people (or more) in a video feed real time while they’re watching it and this only pushed me further to give it a try.

How being watched changes you

While researching ideas I found this interesting BBC article about how being watched changes the way we behave and interact without us knowing. Humans have become sensitive to the presence of others and it influences how we behave when we know or think we’re being watched. When we feel like we’re being watched even if its just a drawing, a painting, or a pair of eyes, it influences the decisions we make trying to adjust our self presentation. The article gives an example of a psychology experiment which took place in the 1970s.

It was Halloween night, and children were out knocking on doors collecting candy. Psychologists positioned themselves inside 18 different homes, and prepared themselves for the stream of costumed children seeking sweets. After opening the door and chatting with the children for a minute or two, they’d tell them to take a single piece of candy from a bowl chockfull with treats, and no more. The researchers then left the children alone with the candy bowl and, half the time, with a mirror. A second hidden experimenter covertly recorded the kids’ behavior. The researchers reasoned that children might be less likely to take a sneaky handful of sweets if they could see their own reflection in the mirror.

And that’s just what they found. When faced with a reflection of their own faces, even masked by a Halloween costume, the kids were more likely to behave.

When the children felt like they were being watched, even by themselves via a mirror, they feel the pressure of their actions and appearance being scrutinised and are more likely to behave an an acceptable way, i.e not stealing extra sweets.

This could be an interesting idea to play with (as mentioned at the end of my panopticon post) by basing interactions about watching the audience, in an attempt to change their behaviour. This could be doing something that actually displays the camera feed so that it is blatantly obvious they’re being watched, or something more subtle that does the tracking in the background and watches them in a more abstract way, for example a pair of eyes on screen that follows people as they move.

UPDATE: With my face tracking route i’ve opted for the blatant watching of the audience as the video is being shown on screen. The interactors of the piece are likely to notice it and change their behaviour as they know they’re being watched. Their representation on screen is also altered which may cause them to feel even more conscious of their appearance and overall self presentation as their self image has been manipulated without their direct consent.

Panopticon

The Panopticon is a type of prison designed by English philosopher and social theorist Jeremy Bentham in the late 18th century. The idea was that by using a circular design with the watchmen in the middle and the inmates in cells around the outside, a single watchman would be able to watch over all of the inmates without the inmates being able to tell whether or not they were being watched. Though the watchman can’t physically watch all the inmates at once, the fact inmates cannot know when they’re being watched means that they act as though they are, altering and controlling their behaviour at all times assuring the automatic functioning of power.

Panopticon diagram

Another philosopher and social theorist, Michel Foucault, in his book Discipline and Punish used the idea of the Panopticon as a metaphor for modern disciplinary societies and their inclination to observe and normalise. He believes that the Panopticon is the ultimate architectural figure of disciplinarily power as it uses a consciousness of permanent visibility to instil power rather than bars and chains like traditional prisons.

Building upon this idea, modern technology has brought this idea of constant surveillance to current society with the deployment of panoptic structures with CCTV cameras on them in public spaces. When people feel that they are constantly being watched they change their actions and behaviour with the idea that it will cut down on crime. Even I’ve noticed this when walking around campus for example, when I spot the camera I instantly become more conscious of my actions and appearance thinking that I’m being watched. Panopticism provides us with a model of a self-disciplined society, in which we govern ourselves and control behaviours without the need for constant surveillance and intervention by an external agency.

Similarly Panopticism has been linked to internet usage too, as ISPs (Internet Service Providers) are able to track our every move online and view our data (dataveillance). Many websites also use cookies to track data about the sites we visit and the products we look at to provide targeted advertising to try and persuade us to buy products or visit websites similar to ones we’ve looked at in the past. One prime example of this that people are becoming more aware of is when booking hotels or flights. The booking websites use cookies that track how many times you’ve been on the website to check prices, and each time you do it raises the prices to make you think that the rooms or flights are being booked up and to get you to spend your money now. By using Private mode on your browser it blocks these cookies so your visits can’t be tracked and the prices aren’t pushed up allowing you to get a better deal.


The idea of the Panopticon and panopticism could be used in my project by playing with the idea of the audience being watched but not knowing. In a sense it is a role reversal. As the audience they’re watching one thing on the screen when in reality it is watching them without them knowing and potentially reacting to their presence or movements.

UPDATE: Looking back at this post after i’ve advanced a bit further with my project has made me consider whether its still relevant. Above I mention how the Panopticon is ‘a metaphor for modern disciplinary societies and their inclination to observe and normalise’ however in relation to my face swapping idea, its using observation to do the opposite of ‘normalise’. While the cameras are observing the audience/ participants, instead of changing their behaviour to conform to what would be considered ‘normal’ it does the opposite by playing with their representations on screen in an attempt to alter their behaviour and deviate even further from the norm by invoking play.

Requirements Gathering

The Dorset independence poster task we just finished was a task to get us thinking about the requirements gathering and the iterative design process. Our requirements gathering within the space also acts as the first step towards gathering the requirements for this project too.

Rather than scouting out good spots for the project to go, it need to go on one (or more) of the screens in the foyer space. There are two choices of placements which will determine the style of the piece I produce. There is one where the screen is a long strip of monitors and next to that there is the usual style screen.

IMG_0977

Here are the screens. Theres the 2 normal shaped screens, one above the other, and the long screen can be seen on the left (turned off as usual).


 

UPDATE: Many weeks after writing this original post, I now know a lot more about how we’re going about displaying our work. With the route I’ve chosen I know I’m using one of the normal shaped screens (whichever one the computer gets hooked up to) as it fits the idea of my work as it uses a camera feed and they’re usually that shape. The location of the screens is pretty good for my project I feel as my face swapping requires the audience to be looking directly towards the camera (which I presume will be near the screen) and they should be when they’re walking to leave the foyer as the exit/entrance is to the right of the screens.

The position also limits it to only working on people exiting the building as it can see their faces, and only the backs of heads of people who are entering. While this does initially seem rather limiting, it could be beneficial during busy periods so there aren’t too many faces that the camera sees to confuse the sketch.


 

The screens are placed relatively central in the foyer space with the screens facing towards the Costa area. When entering the building the screens likely wont be seen due to the angle of their placement (the long strip screen probably will though as it’s not angled) however when exiting the building people would be walking towards them which will give a good frontal view for interaction. It could also be noted that this angle may be beneficial for interactions as people are less likely to be rushing out of the building as they are in as people tend to rush to lectures & workshops and not away.

From observing the space in the Media School foyer we have gained the following insights about the audience and space:

The Space: 

  • A foyer.
  • Seating on the sides of the space.
    • Screens above some of the seating.
  • Costa.
  • Toilets.
  • Mostly light coloured walls.
    • Orange accent walls.
  • Posters & media related displays.

The Audience:

  • Students.
    • Young people (18-25),
    • Mixed gender,
    • Majority on Media related courses.
  • Staff.
    • Lecturers,
    • Tutors,
    • Other staff.
  • Usually rush 10 minutes before & after the hour (Session change over time).
    • Not really looking around or at the screens,
    • A lot stop & queue to get coffee, may look at the screens to the left of Costa.
  • Other times people move slower/
    • Often still distracted by phones or conversation,
    • more likely to notice whats on the screen.
  • People stopping in the space.
    • Using the seating on either side,
    • Standing & waiting for other people,
    • Still don’t notice the screens that much.
  • People passing through to the toilets either side of Costa.

One thing I did notice while requirements gathering for the space is that not all of the screens were turned on, which made it difficult to determine which screens people actually notice and pay attention to. For example, my favourite screens in the space, the long row to the left of Costa from the entrance haven’t been turned on in the last 3+ weeks I’ve been looking. I think these screens would be good for an interactive piece as the long layout gives a good canvas for camera interactions and these are screens that you have to walk past if you’re going through the space. Other screens often just display the news or something similar, but with no sound or subtitles these are usually disregarded too.

The information graphic will need to be different and eye-catching & different to stand out against all the other media in the room. It has become routine for people to not bother looking at the screens or any of the other static displays on the walls as they’ve become old and uninteresting. Feedback from the interactive nature of the piece should be quite obvious to try to get the attention of people walking past, the prompt them to play around with it rather than just quickly passing through and disregarding it as is usually the case.

If I go down the camera based interaction route, the area around the screen which the camera sees will need to be taken into consideration. Having a busy and cluttered background could make tracking interactions harder, and if need be a plain background may have to be put up to block out the clutter in the background, making it easier to track the actions of people.

 

Weymouth House Public Screens Brief

For the Design Iterations unit we have been given a brief that is marked through this blog. The brief is to:

Create a piece of interactive information design for a shared public space, which is intended to elucidate/explain some an idea or concept you perceive as key to our 21st century media experience.

The task here is to create a camera based interactive information graphic using Processing to be displayed on one of the public screens in the foyer of Weymouth House (the Media building). The aim of the piece is to display one of the media related concepts we have been learning about last year and during this term.

Media concepts could be related to technologies, for example displaying sorting algorithms or network connections, or they could be more media theory related looking into semiotics or identity & representation. It can be a literal/ direct piece of information or can be a more abstract/artistic interpretation of an idea.

Currently, I am leaning towards the more abstract interpretation based on my current understanding of Processing. Processing is geared towards creating artistic pieces and is very capable of displaying information in an abstract format.

Taken from the brief:

The work will be developed iteratively through a process of testing and analysis, through at four-stage design cycle:

1) requirements gathering 
2) analysis
3) design
4) user testing
Hopefully on this blog I demonstrate my iterative design process while tackling this task, showing each stage of my progression as I find solutions to design problems to ultimately meet the needs of the brief.