Better Data: Better Testing
James Lyndsay
Workroom Productions
© Workroom Productions 2002
Slide 1
www.workroom-productions.com
Overview
Introduction
Classification
Data-driven functional test techniques
Poor data; characteristic problems
Starting out with good data
Improve testing by using good data
Data Load
Data and UAT
Data and non-functional testing
Conclusion
© Workroom Productions 2002
Slide 2
www.workroom-productions.com
Data Classification
Classification by Data Type
• Environmental
• Setup
• Input
• Fixed
• Consumable
© Workroom Productions 2002
Slide 3
www.workroom-productions.com
Data Classification
Environmental data
• Tells system about its technical
environment
• Examples
• Environmental variables
• File names
• Communication addresses
• Permissions
• Time and date
© Workroom Productions 2002
Slide 4
www.workroom-productions.com
Data Classification
Setup data
• Tells the system about its business
environment
• Cause different functionality to apply to
similar data
• Business input to cross-reference tables
• Examples
•
•
•
•
Slide 5
Tax structure
Postcode to location
Location to cost
Customer type to customer treatment ©
Workroom Productions 2002
www.workroom-productions.com
Data Classification
Input data
• Input by day-to-day system functions
• Standard ‘produced and consumed’ data
• Largest set, most frequently changed
• New lines in tables
• Customers
• Orders
• Events
• Changes and additions produce history and
cross-references
© Workroom Productions 2002
Slide 6
www.workroom-productions.com
Data Classification
Input data
• Fixed Input data
• Available at the start of the test
• Consumable
• Input during testing
© Workroom Productions 2002
Slide 7
www.workroom-productions.com
Data Classification
Classification by Data Use
• Uses
•
•
•
•
Logic Flow
Computational
Linkage
Display
• Basic judgement - is this data item used or
not?
• An item of data can have more than one
use
• Classification based on fixed view of
© Workroom Productions 2002
Slide 8
www.workroom-productions.com
Data Classification
Logic-flow data
• Data used for decisions
• aka ‘Predicate’ data use
• Data item used in logic statement
• if
• while
• case
• Clear-box technique; data-flow testing, puse coverage - see ISEB exam / BSI 7925
© Workroom Productions 2002
Slide 9
www.workroom-productions.com
Data Classification
Computational data
• Data used in calculations
• Includes step calculations and loop
counters
• Clear-box technique; data-flow testing, cuse coverage - see ISEB exam / BSI 79252
© Workroom Productions 2002
Slide 10
www.workroom-productions.com
Data Classification
Linkage data
• Data used to link tables
•
•
•
•
•
Keys
Generated internally
Never input - rarely output
Generated ID is exception - often output
Causes test problems
© Workroom Productions 2002
Slide 11
www.workroom-productions.com
Data Classification
Linkage data
• Data used to represent member of limited
set
• Flags; M/F, W/M/Q/Y etc
• Business knows meaning
• Used in business logic
• Often used in summaries /filtering
• Often linked to setup data
• Generally not shown directly to user, but
translated to meaningful output
• Input from pick list
Slide 12
• Default
© Workroom Productions 2002
www.workroom-productions.com
• Set of valid values - everything else is invalid
Data Classification
Display data
• Data used as output to user
• Output is a result of discreet piece of work
• Contents affect functionality
• May be used for calculation / logic
• Contents reflect functionality
• May be a result
• May be chosen
© Workroom Productions 2002
Slide 13
www.workroom-productions.com
Data Classification
Further Data classifications
• Transitional data
• Created and destroyed as part of internal
operation
• Never shown to users
• Hard to test - may only be stored in memory
• Output data
• Result of data use
• End point of unit of work:
• Input data functionality  output data
© Workroom Productions 2002
Slide 14
www.workroom-productions.com
Test Techniques
Equivalence Class Partitioning
Boundary Value Analysis
C-R-U-D
Data Use / Data Flow
Input fields - context and checklists
Empirical Testing
© Workroom Productions 2002
Slide 15
www.workroom-productions.com
Data-Driven Functional Test Techniques
Equivalence Class Partitioning
• Full range of data - what parts are
equivalent?
• Setup Data + Consumable Input Data
• Computational Data
• Logic Flow - each choice is a class
© Workroom Productions 2002
Slide 16
www.workroom-productions.com
Data-Driven Functional Test Techniques
Equivalence Class Partitioning - refinements
•
•
•
•
Invalid equivalence classes
Equivalence classes in results / output
Error Guessing
Category Partition test - Equivalence
Classes with combinations
• Dependency islands - Category Partition
tests with dependency / independence
• Beizer - ‘Black-Box Testing’
© Workroom Productions 2002
Slide 17
www.workroom-productions.com
Data-Driven Functional Test Techniques
Boundary Value Analysis
• At, on, over the edges of equivalence
classes
• Setup Data + Consumable Input Data
• Computational Data
© Workroom Productions 2002
Slide 18
www.workroom-productions.com
Data-Driven Functional Test Techniques
Boundary Value Analysis - refinements
•
•
•
•
•
•
Special Values - Logic / Flow
Special Values - Transitions and Ends
Special Values - Invalid data
Boundaries of results / output
Domain Test - edges of an area
Kaner, Falk, Nguyen - ‘Testing Computer
Software’
© Workroom Productions 2002
Slide 19
www.workroom-productions.com
Data-Driven Functional Test Techniques
C-R-U-D testing
•
•
•
•
•
•
Create, Read, Update, Delete
Rows of data / items in a row / objects
Parent / child relationship
Lifecycle Testing
Black-box, consumable / fixed input
Hans Schaffer, SIGiST Nov 2001
© Workroom Productions 2002
Slide 20
www.workroom-productions.com
Data-Driven Functional Test Techniques
Data Use / Data Flow
•
•
•
•
•
•
•
Data relationships, rather than logic
Internal, clear-box testing
Creation - Use - (Destruction)
Predicate Use - decisions!
Computational Use
Output Use (sometimes forgotten!)
Beizer + BCS Standard for Software
Component Testing / BS 7925-2 (?)
© Workroom Productions 2002
Slide 21
www.workroom-productions.com
Data-Driven Functional Test Techniques
Input fields - context
•
•
•
•
Green-screen
GUI
Browser
CGI
• Context - specific books:
• Beizer and Kaner et al. for green-screen + GUI
• Splaine for Browser
• Dustin, Rashka, McDiarmid for CGI
© Workroom Productions 2002
Slide 22
www.workroom-productions.com
Data-Driven Functional Test Techniques
Input fields - checklists
• Context-driven - application, input method
• Context-driven - equivalent data type
• Pick List
• Default
• What if invalid?
• Input / output text to internal representation
© Workroom Productions 2002
Slide 23
www.workroom-productions.com
Data-Driven Functional Test Techniques
Input fields - checklists
• Context-driven - application, input method
• Context-driven - equivalent data type
• Free format
• Empty / zero length
• Maximum Length
• Odd characters
© Workroom Productions 2002
Slide 24
www.workroom-productions.com
Data-Driven Functional Test Techniques
Input fields - checklists
• Context-driven - application, input method
• Context-driven - equivalent data type
• Fixed format
•
•
•
•
Dates
Postcodes
Credit card details
Numeric
© Workroom Productions 2002
Slide 25
www.workroom-productions.com
Data-Driven Functional Test Techniques
Input fields - checklists
• Context-driven - application, input method
• Context-driven - equivalent data type
• Applies to all fields
© Workroom Productions 2002
Slide 26
www.workroom-productions.com
Data-Driven Functional Test Techniques
Input fields - checklists
• Valid
• Equivalence classes
• Special cases and boundaries
• Invalid
•
•
•
•
•
Special cases and boundaries
Garbage vs. partly invalid
Inconsistent with other data (existing / new)
Wrong format
Control /escape / dodgy characters
© Workroom Productions 2002
Slide 27
www.workroom-productions.com
Data-Driven Functional Test Techniques
Empirical Testing
•
•
•
•
•
•
Exploratory technique
Examines functionality and setup data
Needs business knowledge to set input
Needs business knowledge to verify output
Yields dividends pre-live
Formalises and controls testing on live
system
• Satisfice.com, Kaner.com,
Slide 28
Workroom-productions.com
© Workroom Productions 2002
www.workroom-productions.com
Specific Data Considerations
Dates
Keys
Corruption
One to Many, Many to One, Many to Many
Synchronisation
© Workroom Productions 2002
Slide 29
www.workroom-productions.com
Specific Data Considerations
Dates: Ever-changing environmental data
•
•
•
•
Date-dependent functionality
Working with other systems
Setting the date
Regular date and time changes
© Workroom Productions 2002
Slide 30
www.workroom-productions.com
Specific Data Considerations
Dates in Data
•
•
•
•
•
Using old data
Day of week vs day of month
Month length and ‘Last day’
Clashes - event swaps, child before parent
Expiration
© Workroom Productions 2002
Slide 31
www.workroom-productions.com
Specific Data Considerations
Keys
•
•
•
•
Database key - unique per database row
Duplication in table / data
Linkage to other tables
Key generation
© Workroom Productions 2002
Slide 32
www.workroom-productions.com
Specific Data Considerations
Corruption
• Introduced by bugs
• Write / update bugs
• Bad setup data
• Introduced by testing
• Data load problems
• Test clashes
• On purpose
• Introduced in normal use
• Inappropriate data entry: ‘Dirty Data’
• Hardware failure
Slide 33
© Workroom Productions 2002
www.workroom-productions.com
Specific Data Considerations
Corruption
• Environment specific - ‘Not seen here’
• Intermittent problems
• Introduces new ‘state’
• Failure may be a long way from fault
• Intermittent - but root cause may be systematic
• Hard to spot without data examination tools
• Sanity check
• TOAD / MyAdmin etc
• Commercial migration tools
• Develop ‘Catalogue of Errors’
Slide 34
© Workroom Productions 2002
www.workroom-productions.com
Specific Data Considerations
One to Many, Many to One, Many to Many
• Entity relationships - not just parent-child
• Think 0, 1 or many - on both sides!
• Special cases
None
One
Many
• Maximum?
• Functional change?
None
One
Many
© Workroom Productions 2002
Slide 35
www.workroom-productions.com
Specific Data Considerations
Synchronisation
• Isolated testing / use of stubs and
harnesses
• Process of synchronisation may hurt other
tests
• Synchronisation and large volumes
• Data load tools and shared source
• Use of ‘Translators’
Slide 36
• Custom tools
• Parse and change data
• Particularly useful for dates
© Workroom Productions 2002
www.workroom-productions.com
Problems Caused by Poor Data
Discussion
•
•
•
•
•
•
Problem
Root cause
Worst aspect of problem
How was it resolved? Was it resolved?
Time spent on resolution
Duration of problem
© Workroom Productions 2002
Slide 37
www.workroom-productions.com
Problems Caused by Poor Data
Unreliable test results
• Running the same test twice produces
inconsistent results.
• May be due to:
• Uncontrolled environment
• Unrecognised database corruption
• Failure to recognise all the data that is
influential on the system.
© Workroom Productions 2002
Slide 38
www.workroom-productions.com
Problems Caused by Poor Data
Degradation of test data over time
• Intermittent problems, increasing frequency
• Test-only problems
• Restoring to a clean dataset eliminates the
symptoms
• Temporary fix
• Cause may go undetected
• Evidence of failure is lost
© Workroom Productions 2002
Slide 39
www.workroom-productions.com
Problems Caused by Poor Data
Increased test maintenance budget
• Maintenance cost increases with:
•
•
•
•
Number of independent test sets
Size of test data
Hard to understand
Hard to manipulate
• Need for fewer, smaller, simpler datasets
© Workroom Productions 2002
Slide 40
www.workroom-productions.com
Problems Caused by Poor Data
Reduced flexibility in test execution
•
•
•
•
Tests ‘locked into’ dataset
High cost of data load/reload
Typical in ‘one test, one dataset’ models
Can result in:
• Infrequent tests of some aspects
• No budget for investigatory / diagnostic tests
© Workroom Productions 2002
Slide 41
www.workroom-productions.com
Problems Caused by Poor Data
Obscure results and bug reports
• Testers may understand data only in
context of specific test
• May miss data corruption issues
• May not understand why they are using the
data
• May not be using the right data
• Bug reports reference data
• If data is not clear, bug reports are
compromised
• Harder for designers / coders to understand
©
• Harder for testers to understand on re-test
Workroom Productions 2002
Slide 42
www.workroom-productions.com
Problems Caused by Poor Data
Larger proportion of problems can be traced to poor
data
• Some issues are traced to the test process
• A proportion can be traced to data faults
• Particularly:
• Environmental data
• Setup data
• Corruption in Fixed Input data
• Measure to locate systematic causes
© Workroom Productions 2002
Slide 43
www.workroom-productions.com
Problems Caused by Poor Data
Less time spent hunting bugs
• Finding bugs is a core activity
• Loading / manipulating data is a secondary
activity - but is a necessary infrastructure
step
• Maintaining tests and data is also
secondary
• Spend more time hunting for bugs;
• Spend less time fiddling with data
• Spend less time maintaining tests and setting
up new diagnostics
©
Workroom Productions 2002
Slide 44
www.workroom-productions.com
Problems Caused by Poor Data
Confusion between developers, testers and business
•
•
•
•
•
•
Each has different aims
Each has different skills
Each has different documents
These do not always mesh well together
Illustrate scenarios with input + output
Understand other teams’ data
requirements
© Workroom Productions 2002
Slide 45
www.workroom-productions.com
Problems Caused by Poor Data
Requirements problems can be hidden in inadequate
data
• Requirements can be visualised with
models
• Process models need inputs and outputs
• Inadequate data can lead to ambiguous or
incomplete requirements
• Complete data can help improve
requirements
© Workroom Productions 2002
Slide 46
www.workroom-productions.com
Problems Caused by Poor Data
Simpler to make test mistakes
• Everybody makes mistakes
• Confusing or over-large datasets can make
data selection mistakes more common.
• Poorly controlled data can make clashes
more common
• Number and frequency of mistakes
reduced:
• Control and label data
• Use small data sets
© Workroom Productions 2002
Slide 47
www.workroom-productions.com
Problems Caused by Poor Data
Unwieldy volumes of data
•
•
•
•
•
Long / complicated / intrusive loads
Hard to summarise effectively
Hard to maintain
Take up too much space
Small datasets can be manipulated more
easily than large datasets.
• A few datasets are easier to manage than
many datasets.
© Workroom Productions 2002
Slide 48
www.workroom-productions.com
Problems Caused by Poor Data
Business data not representatively tested
• Test data doesn't reflect the way the
system will be used in practice.
• End users feel distanced from testing and
lose confidence
• Improve communication, rather than
compromise data
• Use illustrative subset
• Test the business’ setup data
© Workroom Productions 2002
Slide 49
www.workroom-productions.com
Problems Caused by Poor Data
Inability to spot data corruption caused by bugs
• Bugs break data - particularly fixed input
data
• The larger the dataset, the harder to check
• Complex datasets hinder diagnosis
• Can be helped by:
• Use a few, well-known datasets
• Use automated checking, summaries and
sanity checks
• An understandable dataset can allow
straightforward diagnosis
©
Workroom Productions 2002
Slide 50
www.workroom-productions.com
Problems Caused by Poor Data
Poor database integrity
• Intermittent, wide-ranging symptoms
• Caused by:
•
•
•
•
•
•
•
•
Slide 51
Bugs
Simultaneous use and test clashes
Bad keys in data
Bad dates
Wrong data in ‘list content’ fields
Empty fields
Bad data load / merge / append
Database + data versioning problems©
Workroom Productions 2002
www.workroom-productions.com
Poor Data: Conclusion
Poor data causes plenty of problems in:
• Performing the tests
• Effectively
• Efficiently
• Accurately
• Trusting the results
• Test maintenance
• Communication - within and outside test
team
© Workroom Productions 2002
Slide 52
www.workroom-productions.com
How data influences a system
System
Testing
• Consumption
• Test conditions
• Input from users
• Input from other
systems
• Manipulation
• Process under test
• Computation
• Extrapolation
• Summary
• Production
• On-screen
• Report
• Database
Slide 53
• Check expectations
© Workroom Productions 2002
www.workroom-productions.com
How data influences a system
Systems greatly influenced by data
• Generally handle databases
• Customer care / billing
• Dynamic content for net
Systems not so influenced by data
• No significant data structures
•
•
•
•
Embedded systems
Utilities and tools
Compilers and interpreters
Interpretation / display
© Workroom Productions 2002
Slide 54
www.workroom-productions.com
Data: Positive influences
A system is programmed by its data
• Functionality can be configured many ways
•
•
•
•
Control flow
Data manipulation
Data presentation
User interface.
• This allows:
• User can deal with change without new
software
• Same package can fit several business models
• Package can work with varied co-operative
©
systems
Workroom Productions 2002
Slide 55
www.workroom-productions.com
Data: Positive influences
Good data is vital to reliable test results
• Characteristic of good functional testing
• Test can be repeated with the same result
• Test can be varied to allow diagnosis.
• If not . . .
• Hard to communicate problems to coders
• Difficult to have confidence in the QA results
• Good data allows
• repeated with confidence
• diagnosis
• effective reporting
Slide 56
© Workroom Productions 2002
www.workroom-productions.com
Data: Positive influences
Good data can help testing stay on schedule
•
•
•
•
•
•
•
Communicate more effectively
Improve accuracy and avoid mistakes
Speedy diagnosis
New tests may not ned new data
Rapid re-testing
Regression testing
Faster test maintenance - particularly for
automated tests
© Workroom Productions 2002
Slide 57
www.workroom-productions.com
Data: Positive influences
Bad data hinders testing
Good data assists testing
© Workroom Productions 2002
Slide 58
www.workroom-productions.com
Data: Positive influences
If your data . . .
• increases data reliability
• reduces data maintenance time
• helps improve the test process
. . . it is good data
© Workroom Productions 2002
Slide 59
www.workroom-productions.com
Good Data
How to make good data?
• Permutations
• Partitioning
• Clarity
© Workroom Productions 2002
Slide 60
www.workroom-productions.com
Good Data: Permutations
All possible combinations:
•
•
•
•
•
Used to develop test scenarios
‘Complete’ testing
Increases exponentially
Rapidly becomes unwieldy
Techniques can be applied to data
© Workroom Productions 2002
Slide 61
www.workroom-productions.com
Good Data: Permutations
All possible combinations:
• Isolate functionality to be tested
• Identify elements that affect functionality
• For each element, identify valid range
• Equivalence classes
• Boundary analysis
• Special values
• Combine items in ranges
• Multiply sizes of ranges to get number of
tests
© Workroom Productions 2002
Slide 62
www.workroom-productions.com
Good Data: Permutations
Small number of permutations
• Element A; 3 possibilities
• Element B; 2 possibilities
• Total number of possible combinations
3x2=6
•
•
•
•
•
•
Slide 63
A1 B1
A2 B1
A3 B1
A1 B2
A2 B2
A2 B2
© Workroom Productions 2002
www.workroom-productions.com
Good Data: Permutations
Large number of permutations, showing expansion
• More ranges, more numbers
• A:10; B:5; C:5; D:5; E:2; F:2
• Possible combinations 10x5x5x5x2x2 = 5000
• Add one possibility to one range; E:3
• Possible combinations 10x5x5x5x3x2 = 7500
• Add one more basic range G:2
• Possible combinations 10x5x5x5x2x2x2 =
10,000
© Workroom Productions 2002
Slide 64
www.workroom-productions.com
Good Data: Permutations
Reducing the impossible bloat
• Are we really interested in all possible
combinations?
• Many problems involve one element/value
• Max combinations = largest range
• Vast majority of other problems involve
interaction of just two elements/values
• Show up in pair combinations
© Workroom Productions 2002
Slide 65
www.workroom-productions.com
Good Data: Permutations
Pair Combinations
• Check all pairs
• One combination can contain many pairs
• Significantly reduces number of
combinations
• Similar numbers for small no. combinations
• Much smaller for large no. combinations
• Size ~ product of two largest ranges
• Needs a tool or formalised approach
© Workroom Productions 2002
Slide 66
www.workroom-productions.com
Good Data: Permutations
Pair Combination Tools
• Hand-cranked
• Tai, Lei. ‘A Test Generation Strategy for
Pairwise Testing’
• Website:
http://www4.ncsu.edu/~kct/paper/c14.pdf
• Perl
• James Bach’s tool. Free to download
• Website:
http://www.satisfice.com/tools/pairs.zip
• Web
© Workroom Productions 2002
Slide 67
• Telecordia’s AETGweb. $2000-$6000 per seat
www.workroom-productions.com
Good Data: Permutations
Pair Combination: Example
• A customer can have
•
•
•
•
one of three tariffs 1,2,3.
Monthly or Quarterly bills
High or Low value
Last bill Paid or Unpaid.
• Combination; 3x2x2x2 = 24
© Workroom Productions 2002
Slide 68
www.workroom-productions.com
Good Data: Permutations
Pair Combination: Example
• Pairs involving tariff;
• M1, M2, M3. Q1, Q2, Q3. H1, H2, H3. L1, L2,
L3. P1, P2, P3. U1, U2, U3.
• Pairs not covered involving customer
value;
• MH, ML, QH, QL,
• Pairs not covered involving bill payment;
• HP, LP, HU, LU
• MP, MU, QP, QU.
• 30 pairs
Slide 69
© Workroom Productions 2002
www.workroom-productions.com
Good Data: Permutations
Pair Combination: Example
• 30 pairs can be fitted into these 6
combinations:
Account
M
M
M
Q
Q
Q
Tariff
1
2
3
1
2
3
Care
H
L
H
L
H
L
Bill
P
P
U
U
U
P
© Workroom Productions 2002
Slide 70
www.workroom-productions.com
Good Data: Permutations
Pair Combination: Example
• You could consider these as tests . . .
Test Account
T1
M
T2
M
T3
M
T4
Q
T5
Q
T6
Q
Tariff
1
2
3
1
2
3
Care
H
L
H
L
H
L
Bill
P
P
U
U
U
P
© Workroom Productions 2002
Slide 71
www.workroom-productions.com
Good Data: Permutations
Pair Combination: Example
• . . . or as data
Customer Account
Cust 1
M
Cust 2
M
Cust 3
M
Cust 4
Q
Cust 5
Q
Cust 6
Q
Tariff
1
2
3
1
2
3
Care
H
L
H
L
H
L
Bill
P
P
U
U
U
P
© Workroom Productions 2002
Slide 72
www.workroom-productions.com
Good Data: Permutations
Pair Combination: Example
Customer Account
Cust 1
M
Cust 2
M
Cust 3
M
Cust 4
Q
Cust 5
Q
Cust 6
Q
Tariff
1
2
3
1
2
3
Care
H
L
H
L
H
L
Bill
P
P
U
U
U
P
• Small, easy to handle, load, read and
maintain
• Supports many tests
• Allows effective diagnostics
© Workroom Productions 2002
Slide 73
www.workroom-productions.com
Good Data: Permutations
Pair combination - uses
• Best used on
• Data has many rows
• Independent fields
• Ideal for systems which use databases
• Setup data
• Fixed input data
• Play one off against the other
• Test new setup data by using known fixed input
• Test new fixed input by using known setup data
© Workroom Productions 2002
Slide 74
www.workroom-productions.com
Good Data: Permutations
Pair Combination helps because:
 Familiar from test planning (?!)
 Good test coverage without massive
datasets
 Allows investigative testing without new
data
 Reduces the impact of change
 Can be used to test other data
© Workroom Productions 2002
Slide 75
www.workroom-productions.com
Good Data: Partitions
Partitions
•
•
•
•
•
Control data access
Help make datasets independent
Allow simultaneous tests
Allow independent testers
Fulfil ‘Exclusive Use’ requirements
© Workroom Productions 2002
Slide 76
www.workroom-productions.com
Good Data: Partitions
Partitions; a well- known technique
• Most test teams use partitions
• Partition by machine / database / instance
• Excellent for partitioning environmental
data
• Problems
• Shared resources aren’t partitioned
• Databases and data
• Servers
• Links to outside
Slide 77
• Configuration management
• Expense
© Workroom Productions 2002
www.workroom-productions.com
Good Data: Partitions
Soft Partitions; a data-based technique
• Partitions are accessible - but labelled
• Labels use free-text fields
• No effect on functionality
• Available on output - can apply filter, sort etc
• Self-enforced discipline
• Three basic partitions
• Safe
• Change
• Scratch
© Workroom Productions 2002
Slide 78
www.workroom-productions.com
Good Data: Partitions
Soft Partitions; Safe area
 No test changes the data, so the area can
be trusted
• Many testers can use simultaneously
 Used for enquiry tests, usability tests etc
© Workroom Productions 2002
Slide 79
www.workroom-productions.com
Good Data: Partitions
Soft Partitions; Change area
 Used for tests which update/change data
 Used by one test/tester at a time
 May need many change areas to support
team
 Data must be reset or reloaded after
testing
© Workroom Productions 2002
Slide 80
www.workroom-productions.com
Good Data: Partitions
Soft Partitions; Scratch area
 Used for investigative update tests and
those which have unusual requirements
 Existing data cannot be trusted
 Data may be reloaded without warning
 Needs direct communication channels
• Used at tester's own risk!
© Workroom Productions 2002
Slide 81
www.workroom-productions.com
Good Data: Partitions
Soft Partitions: Example
• Determine that first line of address has no
effect on functionality
• Apply easy-to-remember differences
•
•
•
•
Safe Area - ‘Safe Street’
Change Area 1 - ‘1 Change Crescent’
Change Area 2 - ‘2 Change Crescent’
Scratch Area - ‘Dangerous Drive’
• Set up three data sets - investigate tools
• TELL THE TEAM!!
© Workroom Productions 2002
Slide 82
www.workroom-productions.com
Good Data: Partitions
Soft Partitions: Problems
•
•
•
•
No good for environmental data
Not always good for setup data
Needs knowledge within team
Needs self-discipline
© Workroom Productions 2002
Slide 83
www.workroom-productions.com
Good Data: Partitions
Soft Partitions: Advantages
•
•
•
•
•
Partitions without overheads
Allows partitioning of single resources
Simpler configuration management
Helps new groups avoid test clashes
Allows exploration and rigour in same
environment
© Workroom Productions 2002
Slide 84
www.workroom-productions.com
Good Data: Clarity
Clarity
• More uses for free-text fields
• Label your data - row by row
• Use meaningful labels
• Correspond to internals of data
• Same format from record to record
© Workroom Productions 2002
Slide 85
www.workroom-productions.com
Good Data: Clarity
Clarity: Examples
• Data from pair combinations . . .
Customer Account
HP1 Monthly
M
LP2 Monthly
M
HU3 Monthly
M
LU1 Qu arterly
Q
HU2 Qua rterly
Q
LP3 Qu arterly
Q
Tariff
1
2
3
1
2
3
Care
H
L
H
L
H
L
Bill
P
P
U
U
U
P
© Workroom Productions 2002
Slide 86
www.workroom-productions.com
Good Data: Clarity
Clarity: Advantages
• Very good for setup data
• Allows sorts, filters etc
• Helps diagnosis
• Simpler to select data for exploratory tests
• Internal characteristics made external - spot
connections
• Helps avoid mistakes
• Helps communicate results
• Avoids poor/legacy choice of names . . .
© Workroom Productions 2002
Slide 87
www.workroom-productions.com
Good Data: Conclusion
Data reliability increased
• Fewer clashes
• Simpler checks
• Reliable choice of data
© Workroom Productions 2002
Slide 88
www.workroom-productions.com
Good Data: Conclusion
Data maintenance time reduced
• Less frequent need for maintenance
• Smaller datasets
• Well-understood data
© Workroom Productions 2002
Slide 89
www.workroom-productions.com
Good Data: Conclusion
Test process improved
•
•
•
•
Wider range of possible tests
More flexible approach
Better communication
Reduced potential for errors
© Workroom Productions 2002
Slide 90
www.workroom-productions.com
Data Load and Maintenance
Issues
Data Sources
Data Load Techniques and Frequencies
Data Maintenance Strategies
© Workroom Productions 2002
Slide 91
www.workroom-productions.com
Data Load and Maintenance
Issues
•
•
•
•
Problems cause fake bugs!
Often omitted from Test Strategy
Setup, Fixed and Consumable must match
Needs tools
• Custom - takes designers / coders time
• Commercial - expensive
• Fixed Input data load needs to be separate
from Setup data load
• Hard to load Environmental data
© Workroom Productions 2002
Slide 92
www.workroom-productions.com
Data Load and Maintenance
Data Sources
• Minimal set
• Expectation of empty fixed input dataset
• Also applies to situations where Fixed Input data is
unreliable
•
•
•
•
Data created for test, by test
Scripted data needs maintenance
Test execution overhead
Test dependency groups
© Workroom Productions 2002
Slide 93
www.workroom-productions.com
Data Load and Maintenance
Data Sources
• Maximum set
• Loaded from ‘production data’
• Rarely loaded - often changed - becomes
unreliable
• Have to find appropriate data
• Needs translation and filtering
•
•
•
•
Slide 94
Production to Test database schema
Setup data
Clean up production data
Data Protection act
• Can be good for performance tests,
comparison tests
© Workroom Productions 2002
www.workroom-productions.com
Data Load and Maintenance
Data Sources
• Custom set
• Generated for test purposes
• Functional testing
• Load / Stress / Volume
• May not match business requirements
• UAT
• Training
• Comparison
• May need specialised tools
• May need large storage area
© Workroom Productions 2002
Slide 95
www.workroom-productions.com
Data Load and Maintenance
By Classification: Environmental data
• Loaded manually at installation
• Loaded / maintained by architects /
technicians
• Rarely uses user-facing application
functionality
• May be contained in installation scripts
© Workroom Productions 2002
Slide 96
www.workroom-productions.com
Data Load and Maintenance
By Classification: Setup data
• Business dependent
• May be loaded manually for testing
• Training issues
• Often uses application functionality
• Maintenance tool may be supplied
• Can be useful for test data
• Don’t rely on supplied tool
• Source may be very large
Slide 97
• Co-operative system
• Too large for test environment
© Workroom Productions 2002
www.workroom-productions.com
Data Load and Maintenance
By Classification: Input data
• Fixed input data
• Generated from existing data / analysis
• Loaded by testers as part of test
• Loaded by test team as part of test
infrastructure
• Consumable input data
• Not really loaded - used by testers as part of
test
• May exist in automated script input
• Must match each other, and match Setup
Slide 98 data
© Workroom Productions 2002
www.workroom-productions.com
Data Load and Maintenance
Data Load Techniques
• Using the system you're trying to test
•
•
•
•
Slow
Good for data integrity
Good for setting up keys and references
Tests input functionality (as consumable input
data)
• Can pinpoint bugs in input functionality
• Can pinpoint bugs in database handling
• Data can be broken by bugs in input functionality
© Workroom Productions 2002
Slide 99
www.workroom-productions.com
Data Load and Maintenance
Data Load Techniques
• Using the system you're trying to test
• Can be entered manually or by using a tool
• Manually entered
• Often used when loading from minimal data set
• Test scripts need data maintenance
• Ad hoc entry is unlikely to result in ‘Good Data’
• Automated tool, making keystrokes into application
• From automated script or from flat file
• Flat file allows simpler maintenance and reuse
• Can be used effectively for loading a customised
data set
© Workroom Productions 2002
Slide 100
www.workroom-productions.com
Data Load and Maintenance
Data Load Techniques
• Using a data load tool
•
•
•
•
Quick to load
Does not test input functionality
Only practical way of loading volume data
May be the only way to test effects of broken
data
• Maintenance simpler - flat files, or using the
tool
© Workroom Productions 2002
Slide 101
www.workroom-productions.com
Data Load and Maintenance
Data Load Techniques
• Data Load Tools
• Customised tools from coders
• Needs extra development effort
• Coders may already be using similar tool
• Coders may find many unit test uses for tool
• Commercial tools
•
•
•
•
Often not made directly for testing
Useful data analysis, cleaning, migration functionality
May be expensive
List - http://www.dwinfocenter.org/clean.html
© Workroom Productions 2002
Slide 102
www.workroom-productions.com
Data Load and Maintenance
Data Load Techniques
• Not loaded
• Already set up for testing / Left in the system
• Known and consistent set - i.e. ‘safe area’
• Partway through execution of test dependency group
• No tools, no data strategy, tradition of failure
• Data load causes problems
• ‘Read’ observation of ongoing tests
• Key / permission problems
• You are working on the live system
© Workroom Productions 2002
Slide 103
www.workroom-productions.com
Data Load and Maintenance
Data Load Techniques
• Append
• Leaves existing data untouched
• Clean new data
• Problems with summaries and dirty data
• Overwrite
• Keeps keys and integrity
• Replaces consumed data in fields
• Delete then write
• Removes keys, history, children
• All-new data
Slide 104
© Workroom Productions 2002
www.workroom-productions.com
Data Load and Maintenance
Data Load Frequencies
• By classification:
• Fixed input data is loaded often
• Environmental and setup data less frequently
• Frequently loaded data
• Higher cost and downtime
• Greater urgency for maintenance
• More productive to need fewer, faster loads
• Different datasets may be loaded by
people outside team, outside team’s
control
© Workroom Productions 2002
Slide 105
www.workroom-productions.com
Data Load and Maintenance
Data Load Frequencies
• In order of increasing frequency
•
•
•
•
•
At the start of testing
With each release
First thing Monday
On demand
Before every test
© Workroom Productions 2002
Slide 106
www.workroom-productions.com
Data Load and Maintenance
Data Load Frequencies
• Our survey said . . .
•
•
•
•
•
“Before I ever got involved”
“The developers left it there”
“The last testers / tests left it”
“Whenever we get enough time”
“After we've found out what shape the
database is”
• “When we know what it means”
© Workroom Productions 2002
Slide 107
www.workroom-productions.com
Data Load and Maintenance
Data Maintenance
• Why do we need to maintain data?
• Replacing consumed data
• Repairing broken data
• Responding to change
• Database schema
• Code
• Requirements
• New test requirements
© Workroom Productions 2002
Slide 108
www.workroom-productions.com
Data Load and Maintenance
Data Maintenance Issues
• Big job - comparable to test maintenance
• Prone to human error
• Performed by more than one group
• Architects / technicians for Environmental
• Designers / business for Setup
• Testers / migration team /users for Fixed input
• Can be hindered by references, dates and
keys
• Synchronisation with other systems
© Workroom Productions 2002
Slide 109
www.workroom-productions.com
Data Load and Maintenance
Data Maintenance Strategies
• Use good data
• Reduce need for maintenance with flexible data
• Reduce volume with small datasets
• Reduce mistakes with comprehensible data
• Recognise and prepare for problems
• Build time to refresh data into daily
activities
• Plan data maintenance time
Slide 110
• With test maintenance
• Around release
© Workroom Productions 2002
www.workroom-productions.com
Data Load and Maintenance
Data Maintenance Strategies
• Measure data change
• Periodic extract
• Data under change control - particularly Setup
data
• Synchronise planned changes
• Identify and react to unplanned changes
© Workroom Productions 2002
Slide 111
www.workroom-productions.com
Data Load and Maintenance
Data Maintenance Strategies
• Use tools
•
•
•
•
Fast - and needs less attention than manual
Repeatable
Less prone to error
Easier to maintain data input to tool
© Workroom Productions 2002
Slide 112
www.workroom-productions.com
Data Load and Maintenance
Data Maintenance Strategies
• Quick win
• Basic tools to clear or do simple load
• Communicate with other data maintainers
• Medium term
• Good data
• Data change measurement
• Data under change control
• Longer term
• Custom tools shared with coders and business
©
• Commercial analysis tools
Workroom Productions 2002
Slide 113
www.workroom-productions.com
Data and the Business
Who is ‘the Business’?
Communication
Testing Setup Data
User Acceptance Testing
Going Live
© Workroom Productions 2002
Slide 114
www.workroom-productions.com
Data and the Business
Who is ‘the Business’?
• Generally used - but not a standard term
• Project initiators
• Product users
• Non-IT stakeholders
© Workroom Productions 2002
Slide 115
www.workroom-productions.com
Data and the Business
Characteristics of the Business
• Good at data
• Already using it
• Tangible
• Requirements often cover turning input into
output
• Not so good at systems
• Busy
• Ahead of the curve in expectations,
requirements and business knowledge
•
Behind
the
curve
in
application
knowledge
Slide 116
© Workroom Productions 2002
www.workroom-productions.com
Data and the Business
Communication
• Data can be compared with existing
systems
• Data is simpler to understand than tests
• Tangible
• Input  Output
• Illustration and modelling
• Use data to communicate effectively
• With ‘the Business’
• With other IT technical staff
• Within the team
Slide 117
© Workroom Productions 2002
www.workroom-productions.com
Data and the Business
Use of data in communication: Advantages
 Increases trust and understanding by
improving ease of communication
 Helps focus when requirements are vague
 Helps early user identification of problems
 Improves familiarity and effectiveness of
User Acceptance Testing
© Workroom Productions 2002
Slide 118
www.workroom-productions.com
Data and the Business
Use of data in communication: Disadvantages
• Data creep
• Pressure to include changes that aren't directly
required by the Test Team
• Training
• Demo
• Leads quickly to large and uncontrolled
datasets
 Vague requirements
 Data changes frequently and fundamentally
© data
 Extra work, and can lead to out-of-date
Workroom Productions 2002
Slide 119
www.workroom-productions.com
Data and the Business
Use of data in communication: Disadvantages
• Incomplete data can lead to incomplete
testing
• Communication around data is a key tool
• Poor data, non-systematic approach
• Reduced scope of testing
• Missed tests
• Feels good, looks rosy - but still broken.
© Workroom Productions 2002
Slide 120
www.workroom-productions.com
Data and the Business
Testing Setup Data
• A system is programmed by its data . . .
• . . . is yours!?
• Part of your job;
TEST THE DATA!
© Workroom Productions 2002
Slide 121
www.workroom-productions.com
Data and the Business
Testing Setup Data
• Test Questions:
• Does the current setup data affect the
application appropriately?
• Will changes made to the setup data have the
desired effect?
• Aims:
•
•
•
•
Improve overall testing
Help get the data right before live operation
Increased confidence in re-configuration
Help pinpoint bugs in live operation
© Workroom Productions 2002
Slide 122
www.workroom-productions.com
Data and the Business
Testing Setup Data
• General tools needed
• Business involvement - they ‘own’ the data
• Set up the Setup data
• Help with the results
• Organised data,
• Right set of Setup data, ‘tester’s’ sets of Fixed Input
data
• Data load tools
• Setup data
• Fixed Input data
• Business information
Slide 123
• What model is it based on?
© Workroom Productions 2002
www.workroom-productions.com
Data and the Business
Testing Setup Data : Current data
• ‘Tester’s’ sets of Fixed Input data
• Allows many scenarios to be played through
• Complete / pair-complete
• Proven in testing
• Define tests to cover:
• Setup data values
• Key model requirements
• Special cases
• Apply business-owned Setup data
• Check it with the Business
Slide 124
© Workroom Productions 2002
www.workroom-productions.com
Data and the Business
Testing Setup Data : Data Changes
• From testing
• ‘Tester’s’ sets of Fixed Input data
• Agreed output - summary report, screen etc.
• Define tests to cover:
• Changes to key setup data
• Boundaries, special cases etc
• Apply Setup data, change, apply again
• Study changes, look for unexpected
changes
© Workroom Productions 2002
Slide 125
www.workroom-productions.com
Data and the Business
User Acceptance Testing
• Environmental data should not match live
operation
• Interaction with Live systems needs
synchronised test data at both ends
• Effective stubs or test systems to replace Live
systems is possible
• Hard to do well
• Still have to synchronise data with stub /test system
• Date and time should be real
© Workroom Productions 2002
Slide 126
www.workroom-productions.com
Data and the Business
User Acceptance Testing
• Setup data should match expected Setup
data for live operation as far as possible
• User functionality must be ‘as live’ as far as is
possible
• UAT should not be used to check functionality
• However, practical considerations may mean that the
setup data is enhanced with test setup data
• Software delivery and fixes
• Problem reproduction and diagnosis
© Workroom Productions 2002
Slide 127
www.workroom-productions.com
Data and the Business
User Acceptance Testing
• Input data should be:
• Comprehensible to users
• Useful to testers
• Communication and organisation issue
•
•
•
•
Explain Partitioning and Clarity to users
Set up safe / change / scratch areas
Set up naming convention for new data
Encourage users /testers to work freely, but
within bounds to help diagnosis and
communication
©
• Resolve bugs caused by data / user
Workroom Productions 2002
Slide 128
www.workroom-productions.com
Data and the Business
Going Live
•
•
•
•
Catalogue of errors
Taking the Setup data live
Live environmental data
Keeping test data on a live system?
© Workroom Productions 2002
Slide 129
www.workroom-productions.com
Data and the Business
Going Live
• Data-related errors will appear in live
operation, just as in testing.
• Don’t throw away bugs that were due to test
data!
• Symptoms exhibited are important
• Excellent source of diagnostic information - live
and test
• Summarise into ‘Catalogue of Errors’
• Give ‘Catalogue of Errors’ to Operations
• Keep it up to date through further testing
© Workroom Productions 2002
Slide 130
www.workroom-productions.com
Data and the Business
Going Live
• Taking the Setup data live
• Test it FIRST!
• Take copy / automated summary before
live
• Compare after it is input to live
• Alert business to discrepancies
© Workroom Productions 2002
Slide 131
www.workroom-productions.com
Data and the Business
Going Live
• Live environmental data
• Identify who is responsible for individual
elements
• Regular snapshots in test . . .
• . . . Regular snapshots in Live
• Can you use the same script?
• Compare at / just before go-live date
• Alert on similarities with test
© Workroom Productions 2002
Slide 132
www.workroom-productions.com
Data and the Business
Going Live
• Keeping test data on a live system?
• Can be very useful
• ‘Not reproducible in test’ - is it reproducible in
Live?
• First through conversion / migration
• Can cause problems
• Exclude / discount from summaries and reports
• Sanity checks for corruption
• Real-world interface; real money, real credit
rating
©
Workroom Productions 2002
Slide 133
www.workroom-productions.com
Data and the Business
Conclusion
• Useful tool
• Needs to be planned
• Important quality enhancement to UAT /
Live
• TEST THE SETUP DATA!
© Workroom Productions 2002
Slide 134
www.workroom-productions.com
Data and Non-functional Testing
Types of Test
Volume: User Profiles
Volume: Avoiding thrashing
Data in Usability testing
Data in Beta Testing
© Workroom Productions 2002
Slide 135
www.workroom-productions.com
Data and Non-functional Testing
Non-functional testing
• Volume - performance, stress
• Usability
• Beta
Not influenced by data (perhaps)
•
•
•
•
Security
Availability
Reliability
Scalability
© Workroom Productions 2002
Slide 136
www.workroom-productions.com
Data and Non-functional Testing
Volume: Data Profiles
• Gathering profiles
• Poor profiles mean useless results
• Non-linear response to volume
• Analysis to see which aspects of data cause
significant use
• Analysis of existing patterns of use
• Modelling of target patterns of use
• User profiles
• User actions
• Co-operative systems
© Workroom Productions 2002
Slide 137
www.workroom-productions.com
Data and Non-functional Testing
Volume: Data Profiles
• Synchronising data
• Poorly synchronised data
•
•
•
•
Some consumable volume data generated randomly
Many inputs rejected on checking against fixed
Error processing and reports make lots of extra load
Inaccurate results
• Well-synchronised data
• Needs to be generated from same source
• May be generated from formula to ensure
synchronisation
• Will have to be stored somewhere
© Workroom Productions 2002
Slide 138
www.workroom-productions.com
Data and Non-functional Testing
Volume: Data Profiles
• Fixed Input data
• Profile to match real / target
• What’s important for high-use functionality?
• History records
• Complex linkage
• Alerts
• Avoid thrashing
© Workroom Productions 2002
Slide 139
www.workroom-productions.com
Data and Non-functional Testing
Volume: Data Profiles
• Consumable Input data
• Profile based on actions over time over typical
period
• User actions
• Co-operative system actions
•
•
•
•
•
Generation may take longer than consumption
Generated to match Fixed Input
May use same source, extract, formula
Include appropriate levels of errors
Date dependent data should match system
©
date
Workroom Productions 2002
Slide 140
www.workroom-productions.com
Data and Non-functional Testing
Volume: Data Profiles
• Batch input and co-operative systems
• Tools for user action may be based on capture /
replay. Not adequate for many volume tests.
• Load may come from batch files
• Data must match
• Tool to generate data - Perl etc
• Tool to synchronise delivery
• Load may come from co-operative system
Slide 141
•
•
•
•
Data must match
Load may come from wait delays
Stub to generate input / reply as required
© Workroom Productions 2002
www.workroom-productions.com
Tool to drive input timing
Data and Non-functional Testing
Volume: Data Profiles
• Importance appreciated in wider test
community due to web testing
• Splaine’s ‘Web Testing Handbook’ for
further User Profile information
© Workroom Productions 2002
Slide 142
www.workroom-productions.com
Data and Non-functional Testing
Volume: Avoiding thrashing
• ‘Thrashing’ is not a standard term
• Testing looks for real bottlenecks. If the
system thrashed, it will find test-only
bottlenecks
• Thrashing occurs when a resource is overused
• Over-use makes a bottle-neck get worse
• Bad data can make a database or disk thrash
• Results in race conditions
• Typical in volume testing with generated
Slide 143
data
© Workroom Productions 2002
www.workroom-productions.com
Data and Non-functional Testing
Volume: Avoiding thrashing
• ‘Thrashing’ due to Setup data
• Setup data inappropriately uses same record
for many entries
• Setup data runs same functionality for many
entries
• ‘Thrashing’ due to Input data
Slide 144
•
•
•
•
•
Capture/replay tools with poor input
Generated batch
Same customer used for many tests
Same setup data used for many test rows
©
Same cross-reference used for many rows
Workroom Productions 2002
www.workroom-productions.com
Data and Non-functional Testing
Data in Usability testing
• Test should reflect user experience
• Construct data to allow wide exploration
• Avoid free-text labels
• Clarity labels may put off new users
• Helpful labels may skew test
© Workroom Productions 2002
Slide 145
www.workroom-productions.com
Data and Non-functional Testing
Data in Beta Testing
• Not strictly non-functional . . . .
• Setup data must be close to live - but users
may need to experiment
• User data is real - and has value: Treat as
live
• Use sanity checks to ensure data integrity
aftre corruption
• Outside test team, ‘Beta Test’ may mean
training, demo, or pre-sales - be on your
guard!
Slide 146
© Workroom Productions 2002
www.workroom-productions.com
Overview
Better Data: Better Testing
•
•
•
•
•
•
•
•
Classification
Data-driven functional test techniques
Poor data; characteristic problems
Starting out with good data
Improve testing by using good data
Data Load
Data and UAT
Data and non-functional testing
© Workroom Productions 2002
Slide 147
www.workroom-productions.com
Better Data : Better Testing
Conclusion
• Plan the data for maintenance and
flexibility
• Know your data, and make its structure
and content transparent
• Use the data to improve understanding
throughout testing and the business
• Test setup data as you would test
functionality
© Workroom Productions 2002
Slide 148
www.workroom-productions.com
Better Data : Better Testing
Contact details
See web site for:
• Information and profile
• Papers
• Brochure
James Lyndsay
Workroom Productions Ltd.
www.workroom-productions.com
[email protected]
© Workroom Productions 2002
Slide 149
www.workroom-productions.com