AMORE and QA
Barthélémy von Haller
CERN PH/AID
Outline
•
•
•
•
Reminder on AMORE and its components
Reminder on the QA release procedure
Responsibilities at P2
Status
March 17, 2010 - Offline Week
Barthélémy von Haller – CERN PH/AID
2
MonitorObjects Producers
CAF
Prompt
Reco
HLT
AliEve
Objects
Event
HOMER
GDC
LDC
File
Data
samples
Event
Histograms
AliRoot
QA
March 17, 2010 - Offline Week
Plots, ESD’s
Monitor
Objects
Agent
Under test,
Used to work
before Xmas
Data Pool
Monitor
Objects
Client
Monitor Objects
DA
Barthélémy von Haller – CERN PH/AID
3
Offline Core &
ROOT Teams
CR2
HLT nodes
eLogBook
Detector
code
Detector Team
ACR
DAQ Team
CR1
Detector
GUI
Detector
Shifter
DQM/QA
Shifter
Generic
GUI
Detector
Shifter
DQM/QA
Shifter
DA/DQM nodes
Detector
agents
Histo, objects
DA’s
Detector
QAs (expert
On-Call
Experts
Obj.
Histograms
AMORE
DB
AliEve
Expert
Access
MonALISA
ACR
2 Offline
Shifters
AliEve
ACR
Projector
& shifter)
Prompt
Reco 2x
Histo, ESD’s
CERN CC
CAF
Different types of agents
Expert
Shifter
QA agent
This one or …
DQM custom agent
… this one must
Choose a subset
be working online of histograms
March 17, 2010 - Offline Week
Barthélémy von Haller – CERN PH/AID
Mandatory
If it works for
the shifter
plots it works
for the expert
plots
5
AliRoot release procedure
• Follow the offline release policy
• Must be tested with AMORE to make
sure it works fine online
• Once it is in an official AliRoot tag, we install it at P2
• Weekly time slot (currently on Wed. morning)
– To install AliRoot at P2
– No impact on data taking
– But all QA agents must be stopped and restarted
• Requires some discipline from detector groups
– Code cannot be installed at P2 immediately after it has
been committed (see next slide)
March 17, 2010 - Offline Week
Barthélémy von Haller – CERN PH/AID
6
AliRoot release schedule
• During the week : continuous build and tests of
AMORE modules against the Release branch
Identify problems, contact people, take actions
• Monday : offline weekly meeting
Discuss remaining issues, decide what/when to tag
• Tuesday : when tag ready, prepare the rpm, test it
• Wednesday : install the new RPM at P2
• Note : Paul Kuijer will take the final decision about
deploying or not at P2 in case of doubts
March 17, 2010 - Offline Week
Barthélémy von Haller – CERN PH/AID
7
AliRoot online validation
• Automatic build and tests run on a dedicated machine
– Old machine : >3h for tests, 2h to build aliroot
– New machine : 1h for tests, 11m to build aliroot
We can build and check very often
• Tests are ran with various configuration :
– Current aliroot tag / release branch
– With P2 OCDB snapshot / with default OCDB
– AMORE and modules from trunk or latest releases
• Aim : Run every 2 hours, publish a nice html page
 Identify very quickly problems introduced by a change
in any component (ROOT, AliRoot, AMORE, modules)
March 17, 2010 - Offline Week
Barthélémy von Haller – CERN PH/AID
8
Operations : starting agents (1)
• Agents will be started by the DQM shifter
Filtering
Running agent(s)
Available agent(s)
March 17, 2010 - Offline Week
Barthélémy von Haller – CERN PH/AID
9
Operations : starting agents (2)
• Current situation:
Agents
Started by
QA shifter
QA/DQM shifter
QA expert
Detectors team, or QA/DQM
shifter if properly setup
Detectors’ agents
Detectors team, or QA/DQM
shifter if properly setup
Prompt reco : histogram and
ESD’s publisher
QA/DQM shifter
Prompt reco : publisher for
AliEVE on big screen
Offline shifter
March 17, 2010 - Offline Week
Barthélémy von Haller – CERN PH/AID
10
Operations : checking objects (1)
• Objects must be divided into two sets :
– Expert: check by detectors experts
– Shifter: check by DQM shifters (or detector shifter)
– This is already the case for QA
• Number of shifter’s histograms must be limited
• Shifter’s histograms must be easy to understand
March 17, 2010 - Offline Week
Barthélémy von Haller – CERN PH/AID
11
Operations : checking objects (2)
• Current situation:
Objects published by…
Checked by…
QA shifter
QA/DQM shifter (and detector’s
shifter)
QA expert
Detector’s expert
Detectors’ agents
Detector’s shifter and detector’s
expert
Prompt reco : histogram and
ESD’s publisher
QA/DQM shifter
Prompt reco : publisher for
AliEVE on big screen
Everybody
March 17, 2010 - Offline Week
Barthélémy von Haller – CERN PH/AID
12
Status
A R
C D
O E
CE
a
PM
l
VC
F H M NN P
I
MM U T _ R H S
D
DP
O MT
O
P
M
D
S
D
D
S
P
D
S
S
D
T
0
T
O
F
T
P
C
T
R
D
V
0
Z
D
C
0
2
2
1
1
2
2
1
2
1
2
1
1
6
1
1
4
1
2
1
QA shifter @ P2
QA agents
2
Custom agents
2
1
2
1
7
2
2
• All QA shifter agents able to run
• A few need to be checked again at P2 (following AliRoot’s update)
• Reconstruction agent : was working, has to be checked and
updated for aliroot v4-18
March 17, 2010 - Offline Week
Barthélémy von Haller – CERN PH/AID
13
Conclusion
• No major problems running the QA agents
• Up to 10 days between a commit and its
actual installation at P2
• Be careful with what you commit and test it
extensively !
• ALICE week, DAQ session : 24th March
morning
March 17, 2010 - Offline Week
Barthélémy von Haller – CERN PH/AID
14
March 17, 2010 - Offline Week
Barthélémy von Haller – CERN PH/AID
15
BACKUP
March 17, 2010 - Offline Week
Barthélémy von Haller – CERN PH/AID
16
User Interface
• Generic GUI
– Displays common objects (histograms, numbers,
string,…)
– Gives a tree-like view of the objects
– Indicates the quality of each object
– Used to monitor the QA shifter objects
• Custom GUI
– Displays in a custom way common objects or
home-defined objects
March 17, 2010 - Offline Week
Barthélémy von Haller – CERN PH/AID
17
Generic GUI
Red square: bad quality
Green square: good quality
March 17, 2010 - Offline Week
Barthélémy von Haller – CERN PH/AID
18
QA and DQM experts agents
• QA :
–
–
–
–
Code entirely in AliRoot repository
Released as the same pace as AliRoot
Can be used online and offline
As much as possible in this category
• DQM (custom modules) :
– Released on demand, module per module
– Used for
• Analysis / check that don’t make sense offline
• Quick test of a new QA
March 17, 2010 - Offline Week
Barthélémy von Haller – CERN PH/AID
19
Release procedures (2)
• DQM strict release procedures :
• Detectors : Commit all changes,
update version number, ask for a new version
• DAQ : Validate the module by compiling it, producing
an rpm and running publisher over a set of test files
• Install it at P2 if validation passed successfully
• Set up environment for an easy launch by shifters
March 17, 2010 - Offline Week
Barthélémy von Haller – CERN PH/AID
20
New features and changes
• HLT objects to AMORE (see manual)
• Publications (and un-publications) not only
during BookMonitorObjects() but also at SOR()
 Adapt your GUI to handle it, if needed
• Logbook : access to AMORE objects
• QA generic agent : species Cosmic instead of
Default
• Quality flag : please use it
March 17, 2010 - Offline Week
Barthélémy von Haller – CERN PH/AID
21
DQM – QA – Prompt RECO
ACR
CR2
Detector
Detector
Detector
code
QAs
QAs
HLT
nodes
DQM
node
DQM
node
Detector Team
Detector
GUI
Detector
Shifter
ACR
DQM/QA
Shifter
AMORE
Generic
GUI
ACR
DQM/QA
Shifter
CR1
Detector
Detector
Detector
agents,QA
QAs
QAs
Offline
Core Team
Prompt
Reco
DQM
DQM
node
DQMnodes
node
Plots
DQM node
AMORE
DB
eLogBook
ESDs
DQM node
AliEve
DQM node
CERN CC
CAF
AliEve
On-Call
Experts
ACR
Projector
Expert
Access
ACR
MonALISA
MonaLisa
2 Offline
Shifters
P. VANDE VYVRE CERN-PH
Custom GUI example
March 17, 2010 - Offline Week
Barthélémy von Haller – CERN PH/AID
23
Status : detectors’ modules
and QA expert
A
Code under development or
developed (number of Modules)
Code in SVN
Estimation of needs
Validation/statistics gathering
requested
Test file(s) available
Proper info.db.txt file
Proper rpm.spec file
Validation passed
RPM released and installed at
Point 2 (version deployed)
Test at point 2 (number of agents)
Agent is completed, prod. Ready
E
HM
M
T
C Please, M
tell
U agent
P
meMifU your
is working fine at P2 !
O
C
C
F
P O TO T
H
P
S
S
S
R
P
a
M
I N RN R
O
M
D
P
S
D E
V
l
D
D_ K_ G
S
D
D
D
D
1
1.8
1
1
1
1
(+1)
1
1*
1
1
1
3
1-2
R
T
T
T
T
O
P
R
0
F
C
D
1
1
2
1
Z
DI
H
V
D
AG E
0
C
QG R
L
T
1
1.8 1.8 1.4 1.0 1.3 1.8 1.8 1.4 1.3 1.9 1.8 1.3 1.1 1.2 1.5
1
1
1
1
1
1
3
1
1
1
1
1
1.8
1.0
1
Done
To be done
Won’t be done as the QA is used instead
March 17, 2010 - Offline Week
Barthélémy von Haller – CERN PH/AID
1
24