ACCEPTANCE TESTS
Andrea Modigliani
VLT
2nd
SDD/DFS A. Modigliani
Generation Instrumentation Pipelines, 19 Apr 2007 - 1
OUTLINE
•
Why/When
•
Priorities
•
Acceptance tests
•
Experience & feedback
VLT
2nd
SDD/DFS A. Modigliani
Generation Instrumentation Pipelines, 19 Apr 2007 - 2
Why/When
• Why:
 Need to define a clear and compact list of requirements and tests to verify the
compliance of DRL deliveries to PAE/COM1.
 Standardise and make transparent the review process.
 Possibly distribute the review across a long period.
 Ensure that the DRL not only meets scientific specs but also is easy to maintain,
portable, well documented, efficient, robust.
•



When:
Every three months to have at least 4-6 iterations in the period FDR-PAE.
We encourage the consortium to apply the tests during development.
Tests are performed by ESO after a given DRL version has been certified/provided
on an appropriate data set.
VLT
2nd
SDD/DFS A. Modigliani
Generation Instrumentation Pipelines, 19 Apr 2007 - 3
Priorities
The next slides list several requirements organized by priority:
Priority 1 tests will be checked on every delivery,
Priority 2 tests are performed from time to time,
Priority 3 tests are verified at the end (PAE & end of commissioning)…together
with priority 1 and priority 2
Each test is listed and a possible verification criteria is indicated in parenthesis,
references to other presentations are indicated as [Author]
VLT
2nd
SDD/DFS A. Modigliani
Generation Instrumentation Pipelines, 19 Apr 2007 - 4
Compliance and initial verifications
The acceptance test should be part of the PAE process (executed on DRL 0.5).
Any action should to be completed before the last commissioning run (in DRL 1.0).
See also [R. Palsa, Y. Jung].
Does the pipeline follow the recipe template [Y. Jung]? (static checks).
Is the documentation in English? (manual inspection)
Does the software conform to ESO coding standards [R. Palsa]? (manual
inspection, compilation with no warning with strict compiler options).
Does the pipeline use only CPL and agreed upon libraries [C. Izzo, S. Castro] ?
(static checks).
Does the code contain only CPL functions/objects? If something is missing, ask
[email protected] (compute ratio CPL/SLOC).
Are exported function namespace protected? (static checks).
VLT
2nd
SDD/DFS A. Modigliani
Generation Instrumentation Pipelines, 19 Apr 2007 - 5
Execution tests: to verify the delivery is complete and
installable/executable (1)
Do the recipes exist in the expected version? (esorex –recipes, esorex –man-page)
Is there a test package covering each combination of recipes and major instrument setting?
(manual inspection, compare with DRL doc)
Are test data appropriate to verify each recipe and DRL relevant function? (manual
inspection, compare with DRL doc)
Are recipes executable? (verify on provided test package)
Do the recipes create the expected products [S. Castro]? (compare output versus DRL doc)
Do the recipes create the QC they should [P. Ballester, S. Castro]? (compare output versus
DRL doc)
VLT
2nd
SDD/DFS A. Modigliani
Generation Instrumentation Pipelines, 19 Apr 2007 - 6
Execution tests (2)
Do the expected DRL functions exist? (nm libiiinstrument.so)
Is there a unit test for each DRL function covering each instrument setting and parameter
space [L. Lundin]? (manual inspection, compare with DRL doc)
Are there memory leaks [J. M. Larsen]? (esorex –mem-check recipe_name recipe_name.sof)
Are there memory errors [J.M. Larsen]? (valgrind –tool=memcheck)
Is the recipe interface implemented in a standard way [Y. Jung]? (runtest.pl, test with
Gasgano)
Is the doxygen doc complete? (make html, check results)
Do all source files contain addtogroup/defgroup? (static checks, make html)
Are all recipes documented, including In/Out tags? (esorex –man-page)
VLT
2nd
SDD/DFS A. Modigliani
Generation Instrumentation Pipelines, 19 Apr 2007 - 7
Detailed validation: are results correct?
Do recipes give correct results (FITS products + QC parameters)? (run recipe on
test data and check In/Out, compare with DRL doc)
Do the interfaces fit together (is it possible to run a recipe cascade?)
Are the final results (after running the cascade) correct? (manual inspection,
compare with DRL doc)
Does each recipe terminate in a reasonable time ? (exposure time/execution time
>1). [To profile see J.M. Larsen]
Does the pipeline work and give results as defined in the DRL design doc? (manual
inspection, compare results with DRL doc)
VLT
2nd
SDD/DFS A. Modigliani
Generation Instrumentation Pipelines, 19 Apr 2007 - 8
Detailed validation (2)
Does the unit test implement the quality assessment defined in the DRL doc?
(manual inspection, compare with DRL doc) [L. Lundin]
Does the unit test check the basic validity of the function results (including QC
parameters) and that the accuracy is as required? (manual inspection, compare
with DRL doc)
Does the unit test cover relevant parts of input parameter space, such as
true/false for Boolean parameters? (manual inspection)
Do the unit tests pass? (manual inspection, make check)
Does the recipe verify it receives proper input (check for input TAGs)?
Is the FITS format as described in the DRL doc (extensions/keywords)?
Are the product valid FITS files? (fitsverify)
Is the documentation and code understandable? (manual inspection)
VLT
2nd
SDD/DFS A. Modigliani
Generation Instrumentation Pipelines, 19 Apr 2007 - 9
Experience and feedback
Communication is a crucial factor
Acceptance tests should be part of the implementation schedule presented at FDR.
Acceptance tests may involve additional work in the initial implementation phase but
will save a lot of time for PAE and commissioning and help to provide a better product
to the users.
Acceptance tests are in addition to other tests which may be applied by the
instrument SV team.
VLT
2nd
SDD/DFS A. Modigliani
Generation Instrumentation Pipelines, 19 Apr 2007 - 10