Development Virtual Environments for fMRI
Socially to Interact with a virtual avatar
; a pilot study
Hyeongrae, Lee
Dept. of Biomedical Engineering, Hanyang University,
Seoul, Korea
Introduction
Introduction
• The social virtual environments (VEs)
VEs
Virtual
avatar
3
Social VEs
Introduction
• The research and therapy using the social VEs
Neuroscience
Social VEs
Social phobia
Schizophrenia
4
Alcoholism
Introduction
• The functional MRI environments
Conventional device
High magnetic and electronic fields
MR room
Display
device
Tracking
device
Tactile
device
…
 For social VEs, the other method is needed!!!
5
Introduction
•In this study, we purpose that
1.development the social VEs for social cognitive neuroscience
study using MR compatible devices.
2.a pilot study performed on the developed social VEs.
6
Methods
development the social VEs
• fMRI environment for social interaction
Inside of magnet room
Outside of magnet room
Filter panel
MR compatible camera
Computer
Real-time
video image
MR compatible HMD
marker
Marker detection
on the hand
Calculate
maker position
Apply hand position
to virtual environments
8
development the social VEs
• Contents for the social VEs
(a) Rejection
(b) Acceptance
9
development the social VEs
10
A pilot study
• Participants
Normal group
( n = 12 )
Age
20.92 (SD: 1.78)
Sex (M:F)
5:7
11
A pilot study
• fMRI experiment design
12
A pilot study
• Questionnaire (5 point response scale )
• Questionnaires of social cognition about the other avatar
(1) Did you feel positive feeling about the other avatar’s response?
(2) Did you feel negative feeling about the other avatar’s response?
(3) Did you think that other avatar’s response was suitable in the
situation?
(4) Did you feel that the other avatar rejected your offer?
(5) Did you feel that the other avatar hated you or had negative
emotion about you?
(6) Did you feel that the other avatar took threatening attitude to you?
(7) Did you feel that the virtual other ignored or excluded you?
(8) Did you feel that the virtual other was aware of you?
• Questionnaires of social realism about the other avatar
(1) How real the other avatar’s action was?
(2) Did you feel that you really shook hands with (or was rejected by)
the virtual other ?
13
A pilot study
• fMRI data acquisition
- The functional MRI conducted with 1.5T machine (GE Medical System).
• BOLD (Blood Oxygenate Level Dependent)
•EPI sequence, Gradient Echo
•64x64x30 matrix with 3.75x3.75x5-mm spatial resolution, FOV:
240mm,
•Slice thickness: 5mm, TR: 2s, TE: 14.3, FA=90, # of slices: 30
• High resolution anatomical images
•Fast Spoiled Gradient Echo sequence
•256x256x116 matrix with 0.94x0.94x1.50-mm spatial resolution,
FOV: 240mm,
•Slice Thickness: 1.5mm, TR: 8.5s, TE: 1.8s, FA: 12, # of slices:
116).
14
A pilot study
• fMRI analysis
- Analysis of Functional NeuroImages (AFNI, Ver. 2006_06_30_1332)
- subjects whose haed movements are more than 1mm are excluded
Statistical Analysis
Preprocessing
3D construction
General Linear Model
Analysis in each subject
Coregistration
Temporal Normalization
Group Analysis
Temporal Smoothing
(3-point Low pass filter )
Paired t-test within group
(uncorrected P < 0.001)
Spatial Normalization
(MNI N27 templete, 2x2x2mm3)
Spatial Smoothing
(Gaussian filter, 8 mm FWHM)
15
Results &
Discussions
Results & Discussions
• The questionnaire results.
- Social realism about other avatars
5.0
4.0
(1) How real the other avatar’s action
3.0
Acce… (2) really shake hands with (or was
rejected by) the virtual other ?
2.0
1.0
0
1
2
3
- 1 point : not at all
5 point : very much
17
Results & Discussions
• The questionnaire results.
- social cognition about other avatars
(1)
(2)
(3)
(4)
5.0
4.0
3.0
2.0
1.0
012345678
- 1 point : not at all
5 point : very much
positive feeling
negative feeling
other avatar’s response was suitable
the other avatar rejected participant
Acc…
offer
(5) hated participant or had negative
emotion
(6) threatening attitude
(7) ignored or excluded participant
(8) awareness of participant
18
Results & Discussions
• Acceptance > Rejection
R insula lobe (37, -3, 0)
L thalamus (-11, -17, 8)
• Rejection > Acceptance
L prefrontal gyrus (-17, 53, 20)
R rolandic operculum (37, -3, 20)
L cerebelum (VI) (-25, -45, -22)
L superior temporal sulcus (-37, -31, 6)
uncorrected P <0.001, cluster size > 80 microliter
19
Results & Discussions
• Acceptance > Rejection
• mirroring other’s emotion
related to empathy
R insula lobe
• Social cognition related
friendship and teamwork
L thalamus
20
Results & Discussions
• Rejection > Acceptance
L superior temporal sulcus
• understanding other’s
mental states
• Social emotion
(ex. embarrassment)
• Social emotion
(ex. embarrassment)
L prefrontal gyrus
21
Summary
 In this study, we develop the VEs for social cognitive
neuroscience study that it is possible to interact with the
virtual avatars.
 This social VEs would provide social environment to patients
with deficit in social cognition, such as patients with
schizophrenia, social phobia and etc., even though just only
handshake with other.
 The social interaction methods using maker would be
practicable in various social cognitive neuroscience studies.
22
Prof.
Ph.D
candidate
M.S candidate
Hyeongrae Lee
Neuro-mechanism On Virtual Environment Lab.
Dept of Biomedical Engineering
In Hanyang University
E-mail: [email protected]
Homepage: http://bme.hanyang.ac.kr
FAX: +82-2-2296-5943
Tel: +82-2-2290-0691
23