Facial Recognition Software Distinguishes Between Real And Phony Smiles

Con-artists, deceivers, and fakers take note: feigning emotion to manipulate others is about to get a lot harder. Researchers at the MIT Media Lab have developed software that can differentiate between a genuinely delighted smile and one born from frustration. It turns out that the majority of people unknowingly smile to cope with frustration, and others may interpret those smiles as genuine. But what’s the real difference?
By analyzing video, the researchers discovered it’s all in the timing: genuine smiles develop gradually, but frustrated smiles emerge rapidly and dissipate fast. In the study, humans and computers were equally capable of identifying genuine smiles, but when it came to frustrated smiling, people misread the coping smile half the time whereas the algorithm correctly identified the smile with 92 percent accuracy.
The motive behind the research is to help individuals who have difficulty interpreting face-to-face communication, such as those on the autistic spectrum, but the research also has some profound implications for artificial intelligence.
The study, which was recently published in the IEEE Transactions on Affective Computing, (unfortunately, the article is behind a paywall, but you can access the abstract here) details two different experiments. First, male and female participants were asked to act out delight and frustration with an activity. Second, subjects were to fill out an online form that was designed to elicit natural frustration (using commonly aggravating computer experiences such as time outs, CAPTCHAs, and disabled copy-and-paste) followed by delight (by showing a popular YouTube video of a baby laughing).
In both experiments, a webcam recorded facial expressions throughout the activity. Then the video data were analyzed using Google’s facial feature tracker (from its 2006 acquisition of Neven Vision) that measures 22 different points and processed mathematically into an interpretative model to differentiate genuine and phony smiles.
The study revealed some curious findings about smiling. For instance, there is a clear difference in speed and intensity of smiles. Delighted smiles lasted an average of 13.8 seconds from beginning to end, whereas frustrated smiles took nearly half the time (7.5 seconds). Furthermore, delighted smiles were 60 percent more intense than frustrated ones.
Another revelation is it seems that the majority of people are unaware that they smile to cope with frustration, which would explain why 90 percent smiled naturally when faced with a frustrating task but when they were asked to feign frustration, 90 percent didn’t smile…almost as if the memory of someone being frustrated ignores the presence of a smile because it is deemed inconsistent with the interpreted emotion. This is also supported by how easily human’s misidentified frustrated smiles.
The researchers concluded that the most helpful component of determining emotion is the dynamic pattern in a smile’s evolution over time, rather than whether a smile is present or not.