Using cameras and a microphone, the Socially Aware Robot Assistant is able to not only listen to your queries but take in how you are feeling. A normal answer coupled with detection of a smile will make S.A.R.A. reply differently than someone frowning and offering a response that doesn’t comply with social norms.
"She looks at your behavior and her behavior," says Justine Cassell, Carnegie Mellon professor and director of the project, "and calculates on the basis of the intersection of that behavior. Which no one has ever done before."
The S.A.R.A. project consists of three elements never before used, says Cassell: conversational strategy classifiers, a rapport estimator, and a social reasoner. "The conversational strategy classifiers are five separate recognizers that can classify any one of five conversational strategies with over 80 percent accuracy."
Simultaneously, S.A.R.A.’s rapport estimator gives the virtual assistant a readout of how well you and her are getting along. "That one uses a kind of A.I. algorithm that’s brand new called a temporal association rule." The algorithm works in real time, calculating in the moment.
Then the social reasoner "decides what to say based on the level of rapport, the conversational strategies last used, and the non-verbal behavior." This is where S.A.R.A.’s ability to recognize smiling and social norms comes in.
When I chatted with S.A.R.A. about the conference, its answers were not too much different than what I’d get from something like Siri. As my smile increased, S.A.R.A.’s responses got more conversational. When stating my goal for the Frontiers Conference was "to meet Obama" S.A.R.A. responded saying "I bet you’re not the only one with those goals." After S.A.R.A. showed me what panels the president would be a part of, I tried to steer the conversation towards other queries, but S.A.R.A. coldly brought it back around to the topic of Frontiers Conference panels.
Cassell hopes AI assistants like S.A.R.A. become prevalent. No word on if S.A.R.A.’s abilities will ever come to Siri or the Google Assistant, but the virtual bot is already branching out. There’s a demo installed in the conference lobby, and Carnegie Mellon has even made use of S.A.R.A. for peer tutoring. If a student is struggling more than others, S.A.R.A. picks up on that and offers extra social support.