Contributions Implementation Working Group
End to End Testing
Presented by Ty Winmill
Australian Taxation Office
2 May 2014
End to End Testing
Scope and process links
Process
Steps
Who is
responsible?
Build &
Test
Lead
entity or
Solution
Provider*
Third Party
Certify
End-to-end
Testing
Induction
Lead
entity or
Solution
Provider*
Cohort
groups
ATO to
coordinate
cohorts
ATO to
coordinate
gateway
cohort
* ‘Solution Provider’ as agent for employer or fund
Super Stream
2
End to End Testing – Test plan
Scope of testing includes activities prior to induction



3rd party certification (build shakedown)
–
Populate test data
–
Execute test scenarios
Complete E2E testing (build shakedown)
–
Pre-condition: 3rd party complete (exchange 3rd party certification)
–
Exchange credentials
–
Establish lead test co-ordinators
–
Establish test cohort and populate test matrix
–
Execute pre-induction E2E tests
E2E solution test completion, induction entry ready
–
Solution fit for purpose by meeting entry ready criteria
–
Inclusion of solution in induction process
Super Stream
3
End to End Testing – E2E test process
Example contributions solution scenarios (scenarios 3b1 and 5a)
The employer uses commercial payroll software to extract the payroll data, an external service provider is used to transform the data and deliver the XBRL
message to a contracted gateway service provider who passes the ebMS message to the fund’s gateway service provider (scenario 3b1).
The fund uses an integrated gateway and registry solution (scenario 5a).
Employer Domain
Fund Domain
Employer
Fund
6
7
1
11
Gateway
Gateway
10
2
Registry
solution
Payroll
software
8
3
Service
provider
Payroll
Bureau
9
4
Service
provider
Fund
Administ
rator
Clearing
house
12
SMSF
Service
Provider
5
Payroll data
extraction
Super Stream
Payroll data
transformation
XBRL packaging
13
ebMS
wrapping
Message
transmission
Message
receipt and
delivery
SMSF
Message
unpacking and
processing
4
End to End Testing
Key Organising Principles

Lead coordinator for cohorts

Rationalise number of test iterations for each player

Share results

Applies mostly to the first wave of implementations (solutions focus, not every entity)

Voluntary process
ATO to publish and maintain a
master plan of E2E participants*
* Looking to track top 50-100 implementations only
Super Stream
5
End to End Testing – E2E test process

Complete E2E testing
–
Implementation partners assist with the process
–
Pre-condition: 3rd party complete (exchange 3rd party certification)
–
Exchange credentials
–
Establish lead test co-ordinators
–
–
–
Super Stream
Single point of contact for test purposes for each solution/solution provider
Establish test cohort and populate test matrix
–
Include any lead up activities
–
schedule test execution
Execute pre-induction E2E tests
–
Execute test use cases (includes messaging and production-like payload)
–
In receive mode (fund) score each sending party (employer)
–
Response messages by agreement by sender and receiver
6
End to End Testing – Overview E2E test process (cont)

Test scoring process (best practice is to be used in pre-induction and induction processes
General Instructions
 Test cases will be as per the Pre-Induction Process Test Plan
 Each receiving party is expected to score the respective sending party
 All parties are expected to complete all test cases with their appointed peer group
 Test case run sheets will be used for recording results.
Peer groups for testing
 Employer side solutions and Fund side solutions will be allocated in cohorts as peers in the ‘peer to peer’
testing for their end to end testing activities.
 For each round of testing, an employer solution and a fund solution will be allocated a ‘peer’ to initiate oneon-one testing for each test case
 On successful completion of one-on-one testing, each employer and fund solution progresses to testing the
same cases with multiple peers
Issues resolution
 The parties to each message exchange are responsible for problem diagnosis, remediation and re-testing
 Each employer solution fund solution must successfully complete its one-on-one peer test case before
moving into the test phase for multiple peers
 Where a test case cannot be completed in the allotted period, parties should seek to re-schedule that test at
the next available opportunity
 If an issue arises which simultaneously impacts multiple peers the receiving fund solution should deal with
the issue in order of receipt, while letting other parties know that the issue is under active investigation.
Super Stream
7
End to End Testing – Overview E2E test process (cont)

Test scoring process (best practice is to be used in pre-induction and induction processes
Scoring
• Test scores are measured whilst in receiving mode as follows:
• PASS – all test criteria have been successfully met for each step of the test case
• FAIL – one or more criteria could not be completed due to receiving party issue
• INCOMPLETE – one or more criteria could not be completed due to a sending party issue.
•
Overall scoring for a fund will comprise a ‘scoring matrix’ summarising each test case result for all funds in
their peer group
Super Stream
8
End to End Testing – Overview E2E test process (cont)

Example test scoring sheet
CONTRIBUTIONS PRE-INDUCTION END TO END TESTING
Test Plan Score Sheet
DATE:
dd/mm/yy
SCORING FUND SOLUTION:
FUND SOLUTION AAA
EMPLOYER SOLUTION PARTNER:
EMPLOYER SOLUTION BBB
Test ID
Test Case
Test number
Test description
Test number
Test description
Test number
Test description
Test number
Test description
Result
P Pass
F Fail
I
Super Stream
9
Incom plete
Questions?
1.
Does this process work? Gaps?
2.
Who are the logical cohort leaders?
3.
How do we collaborate in arriving at a sensible number and allocation of test iterations per solution?
© COMMONWEALTH OF AUSTRALIA 2014
This presentation was current in May 2014
Super Stream
10