Hidden Emotion Detection through Analyzing Facial Expression
Enhao Gong, Tianhe Wang, Jing Xiong
EE 368 Digital Image Processing
Dept. Electrical Engineering, Stanford University
Introduction
Methods
Motivation:
 Facial expression plays an important role in human
communication and interaction.
 Utilizing technology to detect hard-to-notice expressions
smoothens communication process.
 Extracting and highlighting facial features potentially
assists children with autism in studying facial expressions
and emotions.
We developed an Android App using DROID smart
phones/tablets to detect and characterize human facial
expressions. By using image processing and computer vision
algorithms, simple but hard-to-notice facial features can be
highlighted and emotion state can be further analyzed.
1. Feature Detection – Client (Android with OpenCV)
 Implement Viola-Jones face detection and Haar Cascade
classifier algorithm from OpenCV library.
 Sequentially detect faces and eyes & mouths afterwards
 Smart correction and ROI updates
Results
Image Fusion
 Face Images
 Facial feature localization
 Feature & Contour extraction
 Micro-expression Hotspots
Fused Visualization
With AugmentedReality for Emotion
Figure 2: Kernels and method in Viola- Figure 3: Successfully detected face,
Jones detection Haar Cascade Classifier.
eyes, and mouth.
Figure 1.
Snapshot of App.
2. Facial Features & Contour Extraction Server(php,matlab)
 Canny-filter-based extraction
 Corrected ROI-based adaptively adjusting
 Feature-Background-Ratio-based adjusting
Procedure – Flow Chart
Analysis & Application
Figure 6. Image Fusion for
Augmented Reality and Expression
Decoding.
 Faithful feature extraction and highlighting
 Multiple dimensions of information
Future Work
Figure 4: Facial Feature extraction using corrected
ROI&FBR-based adaptively adjusted Canny Filter
3. MicroExpression Hotspot Extraction–Server(php,matlab)
 Covariance & Tracking based Registration
 Characterize motions using
variance in image domain
 Highlighted with ROI based weights
Figure 2: Flowchart of the App.
Decoding: Neutral 70%
Figure 5: Efficient Micro expression
detection using covariance based
registration
and
cross-frame
variance based motion extraction.
Real-time visualization
Further interpretation on micro-expression hotspots
Experiments and training for more accurate hidden
expression decoding
Robust scoring system for hidden emotion
Extend to detect more facial features, gestures and other
emotional states (stress level, lie detection, etc.)
References
1 P. Viola and M. Jones, CVPR 2001
2 G. Bradski, 2000.
3 Essa, Irfan A., and Alex Paul Pentland. 1997
4 Fasel, Beat, and Juergen Luettin, 2003
5 Ekman, Paul, and Wallace V. Friesen. 2003