AI Reads Vocal Tone Well but Stumbles on Body Language

Generative artificial intelligence systems can accurately identify emotional states in a person’s voice, but they fall remarkably short when trying to interpret bodily gestures. A new study published in Scientific Reports reveals that while state-of-the-art AI approaches human levels of proficiency when listening to vocal tone, its ability to read physical posture and movement lags far behind—a blind spot further compounded by a pronounced tendency to overlook negative feelings.

Sharp Hearing, Poor Vision for Nonverbal Cues

As developers race to integrate conversational agents into everyday life, questions remain about whether these systems possess genuine social cognition—the capacity to decode interpersonal signals like body language and vocal inflection. According to the study, advanced generative AI models achieved near-human accuracy when interpreting emotions from vocal tone, demonstrating a strong aptitude for parsing auditory cues.

However, that proficiency did not translate into the visual domain. When tasked with interpreting emotional bodily gestures, the AI systems performed significantly worse than human benchmarks. The disparity suggests that while modern multimodal architectures can effectively decode acoustic patterns, they struggle to make sense of the complex, dynamic spatial movements that humans naturally use to express how they feel.

An Unexpected Positivity Bias

Beyond the gap between hearing and seeing, the researchers uncovered an unexpected asymmetry in how AI processes affect: a distinct positivity bias. Across all tested models, the algorithms were markedly better at recognizing positive emotions than negative ones when analyzing bodily gestures.

This uneven grasp of emotional valence was not limited to movement. The more advanced models carried the same positivity bias into their evaluations of vocal tone, regularly struggling to identify negative emotional states. The authors observed that this creates a non-humanlike cognitive profile; whereas human psychology is often finely tuned to detect distress or threat signals, the tested AI models consistently skewed toward optimistic interpretations.

How the Models Were Tested

To evaluate these nonverbal skills, the research team tested three generative models from Google’s Gemini family: Gemini Pro 1.5, Gemini Pro 2 (Experimental), and Gemini Flash 2. The systems were assessed using the EU-Emotion Stimulus Set, a standardized library of video and audio recordings depicting basic human emotional expressions through posture, gesture, and tone of voice.

The models’ classifications were benchmarked directly against validated human responses gathered during the standardization of the dataset. This setup allowed the researchers to measure whether current AI achieves functional parity with human observers when performing fundamental emotion recognition across different communication channels.

Risks for Mental Health Deployment

These findings carry urgent implications for healthcare, where automated tools are increasingly pitched for psychiatric intake, remote therapy, and crisis triage. The researchers caution that an algorithm’s inability to reliably identify negative affect presents serious clinical risks, particularly if a system misinterprets a patient’s agitation, sorrow, or physical withdrawal as neutral or positive.

Because the social-cognitive profile of generative AI diverges so sharply from human perception, the study authors warn against premature deployment in therapeutic environments. Without extensive validation and safeguards, relying on artificial intelligence to monitor sensitive nonverbal interactions could lead to missed warning signs when vulnerable individuals need support the most.

---