Picking The Right Set of
Mobile Devices
By Brian Kitchener
Software Quality Architect
[email protected]
Overview
•
•
•
•
•
•
•
•
About me
Some Background
The Problem
Understanding Android
How Apps Work
Building a Device Matrix
Example Matrices
Conclusion
About Me
• Software Quality Architect at ProtoTest
• We're a mobile test lab that combines usability testing with quality
assurance to cultivate a great user experience
• Project Architect, Technical Lead, Trainer.
• Started in QA in 2001
• BA in Applied Computing from University of Denver
• Testing background : Functional, Performance, UAT,
Security, API, Database.
• Automation : Selenium, WebDriver, WatiN,
MonkeyTalk, SOASTA, Fitnesse, QTP, EggPlant, Squish
• Languages : C#, Java, Ruby, Javascript
BACKGROUND
INFORMATION
Some Stats for 2012
• Mobile Apps achieved $17 billion in sales
• 5.2 Mobile Subscribers
– 1.2 Billion PC’s
– 4.2 Billion people use a toothbrush
– 1 Billion Smartphones
• 722 Million Smartphones sold
• 1.4 Million iOS + Android Apps
• 25 developers = half of app revenue
iPhone - June 2007
About the iPhone
• Steve Ballmer : Microsoft CEO
– “There’s no chance the iPhone is going to gain significant market
share. No chance.”
• Patrick Stewart:
– “Last Wednesday, I stupidly dropped my iPhone in the bath, and my
life has sort of spiraled almost out of control.”
• Jon Rubinstein – Palm CEO
– Is there a toaster that also knows how to brew coffee? There is no such
combined device, because it would not make anything better than an
individual toaster or coffee machine. It works the same way with the
iPod, the digital camera or mobile phone: it is important to have
specialized devices.
• Mike Lazaridis – Blackberry CEO
– And so what [the iPhone] has actually done is increased our sales.
ANDROID IS THE
PROBLEM
And Then There Were Two…
• Android unveiled November 2007
• First device was sold in October
2008.
• Over 11,000 models have been
released.
• 48 Billion app installs
• Over 1 Billion Android devices
activated
• 8 OS Revisions
OS Fragmentation
Device Fragmentation
Let’s do some math!
•
•
•
•
•
•
•
•
16 device display categories
20 different common resolutions
8 OS versions
6 Hardware Manufacturers
4 Major cellular networks
16 x 20 x 8 x 6 x 4 = 76,800 permutations
Pairwise approach = over 30 permutations
Who can afford to increase testing by 30X?
Our Approach
•
•
•
•
Efficiency, not coverage
Flexible: support small or large number of devices
Understand how apps work to logically select criteria
Use Market research to pick most common
configurations
• Pick minimum and maximum boundary values for each
criteria
• Choose a value that matches an edge case or abnormal
configuration.
• Pick values that stress or tax the system
UNDERSTANDING
ANDROID
Android
•
•
•
•
•
•
Built and Maintained by Google
Open Source
Built on Linux kernel
ARM
X86 Ports
Built to support almost any type of device
– Phones, tablets, phablets, media players, tv’s,
watches, etc.
• Device Manufacturers customize code.
Example: Kindle Fire
• Forked Android 2.3
– Not updateable
•
•
•
•
Customized UI
Separate App store
Not all android apps work
Custom web browser
The Operating System
• Google Releases “stock”
versions
• 10 Major Releases since
2008
– API Level, not Version
• Device manufacturers like
to customize the OS
– Drivers, libraries, UI
• “Stock” OS available in
Nexus devices or an
Emulator
Simulators / Emulators
• Simulator imitates the software layer
– OS and Libraries
– Apple provides a simulator in xCode IDE
• Emulator duplicates the hardware and software
– Processor and Memory
– Cannot mimic GPU, GPS, accelerometer
• Always run stock OS
• Can be used to test some functionality
• Should always test on a physical device too
The Processor
•
•
•
•
•
•
ARM RISC-based instruction set
Specification defined by ARM holdings
32 bit
Same as iOS
X86 patches and ports
It’s only a spec, can be modified.
SoC – System On A Chip
•
•
•
•
•
Main Board
Processor
RAM Bus
GPU
May include :
–
–
–
–
–
Cellular
WiFi
NFC
GPS
Bluetooth
2 Samsung Galaxy S4s
Quallcomm Snapdragon
• Quallcomm Krait 300
– Quad core ARMv7
Cortex A15 Architecture
• Adreno 320 GPU
• Dual Channel 533Mhz
Bus
• Integrated LTE
Samsung Exynos 5 Octa
• Samsung Big.little
processor
– Quad core Cortex A 15
– Quad core Cortex A7
• PowerVR SGX 544 GPU
• Dual Channel 800Mhz
Bus
• No Integrated LTE
Common SoC
Manufacturer
Device Name
Cores
Qualcomm
Snapdragon S4
2 or 4
Nvidia
Tegra 3
4+1
Samsung
Exynos 4
2 or 4
Intel
Medfeld
1
Texas
Instruments
OMAP 4
2+2
ST-Ericcson
NovaThor
2
Processor
GPU
ARM CortexA15
ARM CortexA9
ARM CortexA9
Geforce
Intel x86
PowerVR
ARM Cortex
A9
ARM CortexA9
Adreno
Mali
PowerVR
PowerVR
Resolution is not enough
• Unlimited number of screen sizes available
• Screens range from 3” to 11”
• Each screen has a resolution, same as a
monitor
– If you increase the resolution everything shrinks!
• Pixels per Inch = Density
• Screen Size + Density = Display Bucket
• Resolution is not enough!
The Display Buckets
Size :
• Xlarge : 8” - 10” tablet.
• Large : 5” - 7” tablet.
• Normal : 3.5” - 5” phones.
• Small : 3” - 3.5” phones.
Density :
• ldpi = Low DPI (~120)
• mdpi = Medium DPI (~160)
• hdpi = High DPI (~240)
• xhdpi = Extra High DPI
(~320)
Low density Medium
(120), ldpi
density
(160), mdpi
High
density
(240), hdpi
Extra high
density
(320), xhdpi
Small
screen
Normal
screen
QVGA
(240x320)
WQVGA400 HVGA
(240x400)
(320x480)
WQVGA432
(240x432)
480x640
Large
screen
WVGA800*
*
(480x800)
WVGA854*
* (480x854)
WVGA800*
(480x800)
WVGA854*
(480x854)
600x1024
Extra Large
screen
1024x600
WXGA
1536x1152
†
(1280x800) 1920x1152
1024x768
1920x1200
1280x768
WVGA800
(480x800)
WVGA854
(480x854)
600x1024
640x960
2048x1536
2560x1536
2560x1600
Display Buckets
• Galaxy S3
– 1280 x 720
– Xhdpi density (331ppi)
– Normal screen (4.7”)
• Galaxy Tab 10.1
– 1280 x 800
– ldpi density (149ppi)
– Xlarge sceren (10.1”)
• Galaxy Note LTE
– 1280 x 800
– hdpi density (285ppi)
– Large Screen (5.5”)
Market Analysis
Aspect Ratio
• UI is manipulated from code
• Density Pixels adjust for screen size
– But can use regular pixels!
• Need to take both X and Y into account!
– Easy to overlap or hide things
• Includes orientation
• Some devices include an aspect ratio changer!
(LG Optimus Vu)
Cellular Carrier
• Four Major US Networks
– Verizon, Sprint, AT&T, T-Mobile
– Some phone interoperability
– 2 protocols
• GSM – T-Mobile AT&T
• CDMA – Verizon and Sprint
– Carriers assigned specific frequency bands
– LTE will be new standard - But spectrum issues will
prevent cross-network phones
• So if the phone supports the carrier’s protocol
and band it can theoretically connect.
HOW APPS WORK
How Apps work
• Apps need to work on all screen sizes
– May not be functional
– May be wasted space
– May not make sense
• Apps define XML layouts similar to HTML
– Node structure
– Static Content – Images, etc
– Dynamic Content – Color, Text, etc.
Layouts and Fragments
• XML Fragments are
reusable components
• Layouts stitch together
fragments for a specific
sized device
• App may need different
flow for tablet vs phone
BUILDING THE
DEVICE MATRIX
Our Criteria
• Operating System
– OS customizations, missing libraries, driver issues,
• Screen Size
– Rendering issues, usability, missing layouts
• Pixel Density
– Density Independence, missing layouts.
• Aspect Ratio
– X,Y calculations, overlapping panels, display issues
• SoC
– Hardware performance, Instruction set, battery, signal
• Carrier
– Network protocol, speed, responsiveness, packet loss
The Goal
• Efficiency, not coverage!
• Build a set of devices to be used for app and
website testing.
• Know when to update them
• Define a list of simple categories of devices
• Pick devices that offer broad coverage
• Adjust the number of devices based upon
needed coverage
Categorical Approach
• Define scope
– Android, iOS, phone, tablet, etc.
•
•
•
•
•
•
Understand Testing requirements
Self-descriptive Names
Help to broaden coverage
Will adjust devices chosen to cover our criteria
Should be apparent when to update a device
Spread coverage :
– Usage -> Edge Cases -> Strange -> Stress
Example Categories
• Common
– Matches most common
display configuration
• Newest
– Latest OS version, largest
screen, highest resolution
• Oldest
– Oldest, slowest, smallest
device.
• Abnormal
– Non-standard OS, aspect
ratio, orientation, size
• Popular
– Most popular device in
terms of sales
• Budget
– Low-priced new model.
Tend to have strange specs
• Flagship
– Nexus device running stock
Android OS
• Catch-All
– Cover any missing criteria
Android Phone Matrix
March
2012
Newest
Device Name
HTC Droid DNA
Oldest
HTC Tattoo
OS
Aspe
SoC
ct
4.2 Normal-xhdpi 9:16 Snapdragon S4
1.6
Display
Verizon
Small-ldpi
3:4
Snapdragon S1
AT&T
Normal-hdpi
9:16
TI OMAP 4
Verizon
Exynos 4
Sprint
Nvidia Tegra 3
Tmobile
Common
Motorola Droid 3 2.3
Popular
Samsung Galaxy 4.1 Normal-xhdpi 9:16
S3
LG Optimus VU
4
Large-hdpi
3:4
Abnormal
Carrier
Flagship
LG Nexus 4
4.2 Normal-xhdpi
3:5
Snapdragon S4
TMobile
Budget
Dell Venue
2.2 Normal-mdpi
3:5
Snapdragon S3
AT&T
2.3
9:16 Sony NovaThor
AT&T
Catch-All
Sony Xperia P
Normal-hdpi
iOS Matrix
March
Device
OS Display
Aspect SoC
Carrier
2012
Name
Newest
iPhone 5S
7
4” 1136 x 640 326ppi
9:16
Apple 64bit A7
T-Mobile
Oldest
iPhone 3g
6
3.5” 320 x 480 165ppi
2:3
Apple A3
AT&T
Common
iPhone 5
6
4” 1136 x 640 326ppi
9:16
Apple A5
Verizon
Popular
iPhone 4
6
3.5” 640x960 330ppi
2:3
Apple A4
Sprint
iPad
iPad 3
7
10” 1536x2048 264ppi
3:4
Apple A5X
Verizon
iPod Touch
5
3.5” 640x960 326ppi
2:3
Apple A4
WiFi
6
7” 1024 x 768 162ppi
3:4
Apple A5
AT&T
(Retina)
iPod
(4th gen)
Mini
iPad Mini
Conclusion
• Understanding how everything works allows
us to logically select devices.
• A large number of permutations can be
covered in few devices
• If additional coverage is needed additional
devices can be added
• White Paper :
– http://www.prototest.com :
– Building the Ultimate Device Matrix