5/3/12 User Experience in Connected TV A usability and eye tracking research Mari-­‐Carmen Marcos. Communica>on Department. Universitat Pompeu Fabra. Verónica Mansilla. Student at Master in Digital Contents Management (UB-­‐UPF) Collaborators: Reinald Besalú
Ariadna Fernández
Mònica Figueras
Cristòfol Rovira Carlos Scolari Javier Díaz Canseco David Hérnández Jaume Ponsa What is a Connected TV 3 features: -­‐ TV as a big screen for browsing the net -­‐  Specific apps (like Youtube, Facebook, etc) -­‐ VoD Internet: Broad banding TDT: Broad cas>ng 1 5/3/12 The global study Lavinia Interac>va à New business models Televisió de Catalunya à New business models Havas Media à New adver>sement formats Universitat Pompeu Fabra à UX challenges UX study 2 5/3/12 Goals •  To know how Connected TV matchs user’s expecta;on •  To detect usability weakness •  To propose improvements for the UX Devices • 
• 
• 
• 
Sony Bravia Engel (HbbTV standard) Play Sta>on 3 Blu:Sens 3 5/3/12 Methods • 
• 
• 
• 
Brainstorming Heuris;c evalua>on User tes;ng in lab + Survey SUS User tes>ng with eye tracking Brainstorming. Methodology •  40 people –  Students of Audiovisual Communica>on at UPF –  They never had used a Connected TV –  Chaeng in groups of 3 people •  Two ques;ons to open de chat: •  What func;onali;es do you think that a Connected TV has got? •  For which type of people do you think that is this kind of device? •  Result: a brief document with expecta;ons 4 5/3/12 Brainstorming. Preliminary Results They think that: •  Connected TV should have some computer features • 
• 
• 
Browse the web and search for informa>on Open audiovisual files from a pen drive Play to online games with friends (like a console) •  Connectec TV should be an “Enriched” TV • 
• 
• 
Programs •  See programs emihed days before (VoD) •  Record programs and programme the recording (without a video recorder) •  Access to foreign TV channels Interac;on •  Buy products adver>sed on TV •  Par>cipate in compe;;ons emiLed on TV Customiza;on •  Customize contents with the user history •  List of programs in favorites Heuris>c evalua>on. Methodology 1)  Crea;on of a check list with 59 fields organized in the following sec;ons Hardware TV / Box Remote control Interface: Menus Set up Keyboard Help Search engine Services TDT VoD Web Apps 5 5/3/12 Heuris>c evalua>on. Results Remote control -­‐  Lack of standard icons -­‐  Lack of an easy keyboard Interac;on -­‐  Slow response of the system -­‐  No feedback while wai>ng the response Naviga;on menus -­‐  Lack of breadcrumbs -­‐  Labels and icons are not intui>ve -­‐  Architecture is not clear Video on Demand (VoD) -­‐  Difficult to scroll in long lists of channels -­‐  Located in different labels of the menu along the 4 devices -­‐  Lack of standardiza>on of the interface of each channel Search engine -­‐  Basic, no advanced features User tes>ng. Methodology • 
Par;cipants: 50 users –  Young and adults –  Alone and in pairs –  One or two devices • 
Ini>al survey –  Demographic data –  TV and Internet consume habits • 
4 tasks –  Free walkthrough –  TV guide in TDT –  Find a program in VoD –  Search in Google or Search in Youtube TDT
VOD
BRAVIA
x
x
ENGEL
x
x
Blu:Sens
x
x
x
x
x
Play Station 3
• 
Final survey –  SUS • 
No recording. 2 moderators take notes in a template –  Timing –  Observa>ons –  Quotes of users’ comments Google
Youtube app
x
x
6 5/3/12 User tes>ng. Results Main problems observed and main comments 1. Expecta;ons. Frustra>on about func>onali>es: •  They expected more integra>on between TV and internet, especially with social networks •  Connec>on speed for Internet func>onali>es is slower than in computers 2. Interac;on. Difficul>es with the remote control •  Not prepared for new func>ons •  Too big, too small, non standard icons, no feedback aper pressing a key 3. Interface. Difficul>es finding the access to func>onali>es •  TDT, VoD and Internet access are not obvious in some devices. Lack of standariza>on SUS sa>sfac>on ques>onnaire –  Between 70% (Bravia) to 50% (Engel) User tes>ng. Results Main problems observed and main comments 1. Expecta;ons. Frustra>on about func>onali>es: •  They expected more integra>on between TV and internet, especially with social networks •  Connec>on speed for Internet func>onali>es is slower than in computers 2. Interac;on. Difficul>es with the remote control •  Not prepared for new func>ons •  Too big, too small, non standard icons, no feedback aper pressing a key 3. Interface. Difficul>es finding the access to func>onali>es •  TDT, VoD and Internet access are not obvious in some devices. Lack of standariza>on SUS sa>sfac>on ques>onnaire –  Between 70% (Bravia) to 50% (Engel) 7 5/3/12 User tes>ng. Results Circles. People pressed the wrong one by mistake. They needed to look at the control to press properly User tes>ng. Results Clic twice Exit to full screen. Necessary to access to the VoD service. Difficult to no>ce. 8 5/3/12 User tes>ng. Results Logo to open keyboard. Necessary for performing a search. “Back” is not a standard icon User tes>ng. Results X to “go” instead for closing Logo to start Cursor and navigator 9 5/3/12 User tes>ng. Results Main problems observed and main comments 1. Expecta;ons. Frustra>on about func>onali>es: •  They expected more integra>on between TV and internet, especially with social networks •  Connec>on speed for Internet func>onali>es is slower than in computers 2. Interac;on. Difficul>es with the remote control •  Not prepared for new func>ons •  Too big, too small, non standard icons, no feedback aper pressing a key 3. Interface. Difficul>es finding the access to func>onali>es •  TDT, VoD and Internet access are not obvious in some devices. Lack of standardiza>on SUS sa>sfac>on ques>onnaire –  Between 70% (Bravia) to 50% (Engel) User tes>ng. Results Access path for Video on Demand Service in the 4 devices: BRAVIA BLU:SENS Play Sta;on 3 Main menu > Video > Choose channel hhp://youtu.be/WyX8jbpxkCc Main menu > Internet > Choose channel Main menu > TV > Choose channel ENGEL Main menu > TV > Full screen > Popup > Press red key 10 5/3/12 User tes>ng. Results Time required to finish each task (in seconds): BRAVIA
ENGEL
BS
PS3
TDT
VOD
Average
1,50
4,10
Google
Youtube app
1,11
St. Dev.
0,71
1,79
0,33
Sample size
10,00
10,00
9,00
Average
1,88
5,18
1,93
St. Dev.
1,11
1,70
1,00
Sample size
17,00
17,00
14,00
Average
1,82
3,27
St. Dev.
0,40
0,79
1,57
Sample size
11,00
11,00
11,00
Average
4,00
1,92
St. Dev.
1,48
1,08
Sample size
12,00
12,00
2,45
User tes>ng. Results Difficul>es for finishing the each tasks. People who required our help: TDT
VOD
100%
50%
Needed help
0%
50%
0%
ENGEL Did it alone
87%
100%
BRAVIA Did it alone
Google
Youtube app
100%
Needed help
12%
31%
68%
BS Did it alone
73%
18%
45%
Needed help
27%
0%
82%
54%
PS3 Did it alone
66%
83%
Needed help
33%
17%
11 5/3/12 User tes>ng. Results TDT
VOD
BRAVIA
x
x
x
ENGEL
x
x
x
Blu:Sens
x
Play Station 3
Google
Youtube app
x
x
x
Annota;ons during the test. Task VoD on Engel device: -­‐  193 annota>ons -­‐  113 from observa>on and 80 quotes -­‐  29 posi>ve and 164 nega>ve -­‐  Figure calculated in % (193=100%) 35.00 32.12 30.00 26.42 25.00 20.00 Interface Remote control 15.00 10.00 5.00 10.36 6.74 4.66 4.15 3.11 1.55 1.04 1.04 0.52 Both 8.29 0.00 posi>ve nega>ve posi>ve Observed nega>ve Quotes User tes>ng. Results Main problems observed and main comments 1. Expecta;ons. Frustra>on about func>onali>es: •  They expected more integra>on between TV and internet, especially with social networks •  Connec>on speed for Internet func>onali>es is slower than in computers 2. Interac;on. Difficul>es with the remote control •  Not prepared for new func>ons •  Too big, too small, non standard icons, no feedback aper pressing a key 3. Interface. Difficul>es finding the access to func>onali>es •  TDT, VoD and Internet access are not obvious in some devices. Lack of standardiza>on SUS sa>sfac>on ques>onnaire –  Between 70% (Bravia) to 50% (Engel) 12 5/3/12 User tes>ng-­‐ Ques>onnaire Average results on SUS
1. 
2. 
3. 
4. 
5. 
6. 
7. 
8. 
9. 
10. 
I think that I would like to use this system frequently
I found the system unnecessarily complex
I thought the system was easy to use
I think that I would need the support of a technical person to be able to use it
I found the various functions in this system were well integrated
I thought there was too much inconsistency in this system
I would imagine that most people would learn to use this system very quickly
I found the system very cumbersome to use
I felt very confident using the system
I needed to learn a lot of things before I could get going with this system
Sony Bravia: 75% Play Sta;on 3: 66,5% Blu:Sens: 62,7% Engel: 50% System Usability Scale (SUS). Developed by John Brooke while working at ©Digital Equipment Corpora>on, 1986 Conclusions so far •  Low sa;sfac;on due to: – 
– 
– 
– 
High expecta;ons about func>onali>es. Frustra;on Difficul>es in naviga;on. Weak informa;on architecture Slow response Difficul>es using the remote controls •  Let’s go further with the Engel device because: –  It uses HbbTV (Hybrid Broadcast Broadband TV, supported by European countries as a industry standard –  It has the lower sa;sfac;on in usability tests and surveys –  People found high difficul>es to access the VoD service, and we wonder what happens during the seconds that the popup in on the screen… An eye tracking study is required! 13 5/3/12 Eye tracking study Eye tracking. Methodology •  Device: Tobii Glasses •  Soaware: Tobii Studio 3.2 •  Room setup at University •  Sofa, TV and boxes •  Users: 20 people •  Devices: 4 connected TV •  Task: find a specific program on VoD –  10 minuts –  5 minuts before the task to try the device 14 5/3/12 Eye tracking. Test session •  Introduc>on and consent form •  Calibra>on •  Test with 4 devices in a shuffled order –  First task + Second task + Interview –  Template for moderators •  Gip Eye tracking. Analysis for the Engel device Step 1 Check the sample recordings Select the good recordings: 14 users -­‐ some users did not use the Engel device -­‐ some recordings were just ahemps -­‐ some>mes the device failed (Internet, control) 15 5/3/12 Eye tracking. Analysis Step 2 Create segments for each recording for each >me that the popup appears on the screen Eye tracking. Analysis Step 3 Select a scena to draw the maps over it Step 4 Create 2 AOIs: -­‐  Full screen -­‐  Popup Engel Full_screen 16 5/3/12 Eye tracking. Analysis Step 5 Sta>s>cs for the AOIs People who fixated on the popup E01 E02 E03 4 of 14 users recorder fixa>ons on the popup E05 x x (no help) E09 E12 1 of 14 users finished the task x E14 E15 E16 E17 E18 x E19 E20 E21 Eye tracking. Analysis Step 5 Sta>s>cs for the AOIs The popup appears between 1 and 7 >mes on the screen Each >me that it appears, it remains on the screen an average of 5 seconds (between 2.3 and 10 seconds) People were able to see it an average of 14 seconds by session (between 5 and 26 seconds) Example user E21: he/she could see the popup 5 >mes for a total of 26 seconds, but he/she did not fixate the gaze on it E01 E02 E03 E05 E09 E12 E14 E15 E16 E17 E18 E19 E20 number of ;mes Average dura;on of Exposure ;me of that popup the popup on screen popup on screen appears each ;me (seconds) (seconds) 3 1 2 1 5 1 2 3 7 3 5 1 3 3 10 7,5 5 3,8 5 3 6 3 2,3 4 5 4 9 10 15 5 19 5 6 18 21 6,9 20 5 12 E21 5 5,2 26 Average 3 4,77 14,31 17 5/3/12 Eye tracking. Analysis Step 5 Sta>s>cs for the AOIs A high number of fixa>on on the screen does not lead to a higher probability of fixa>ons on the popup (Ex. E01, E02) E03 and E18 saw the popup but they did much more fixa;ons on the rest of the screen. E12 saw immedialty the popup and understood it: 2 fixa>ons in the full screen, 1 in the popup E01 E02 E03 E05 E09 E12 E14 E15 E16 E17 E18 E19 E20 E21 Average Number of Number of Fixa;ons Fixa;ons in popup in full screen 0 54 0 46 1 32 3 8 0 34 1 2 0 5 0 17 0 26 0 7 1 18 0 14 0 11 0 25 1,5 21,36 Percentage of Fixa;ons in popup (%) 3,13 37,50 50,00 5,56 24,05 Eye tracking. Analysis Step 5 Sta>s>cs for the AOIs Fixa;on dura;on on the popup in rela;on to the full screen is very low: 3, 4 and 6% of the >me. Just E12 looked to the popup for a longer >me (53% of >me) E01 E02 E03 E05 E09 E12 E14 E15 E16 E17 E18 E19 E20 E21 Average Fixa;on Dura;on Fixa;on Dura;on ni Percentage of in seconds (popup) seconds (Full Screen) ;me on popup (%) 0 27,05 0 24,99 0,47 11,21 4,19 2,21 4,15 53,25 0 18,86 0,27 4,52 5,97 0 1,31 0 6,53 0 12,25 0 5,79 0,33 11,28 2,93 0 4,76 0 7,54 0 9,16 0,82 10,67 16,59 18 5/3/12 Eye tracking. Results Users who did not fixate the gaze in the popup (N=10) Users who fixated the gaze in the popup (N=4) Eye tracking. Results Segment for the user who finished the task Segment for one of the users who did not finish the task 19 5/3/12 Eye tracking. Unexpected problems and challenges* • 
Preparing the test. Unexpected problems –  Lots of cables and connec;ons have to be set up properly. –  A good TDT signal is required for TV. –  A stable internet signal is required for VoD and Internet access. • 
Running the tests. Unexpected problems –  Users used to look below the glasses to see the remote control. We did not obtain valid data for the gaze on the remote control. –  Eye tracker infrared had conflicts with remote control infrared. We put the TDT boxes under a table to avoid they crossed. • 
In the analysis phase –  Big files generated from Tobii Studio. Good RAM and nd fast HD were required. –  Dedicated graphics card was required to visualize videos. –  Accurate manual work is needed to label the beginning and the end of each segment. Very >me consuming. Thanks to Iván del Muro, Alt64 (Tobii distributor in Spain), for his support Thank you for your aLen;on Mari-­‐Carmen Marcos University Pompeu Fabra. Communica>on Department [email protected] Veronica Mansilla Student of Master on Digital Contents Management (UB-­‐UPF) 20