CSC2.0 Status
CUC Telecon - 4/28/17
G. Fabbiano (Senior Astrophysicist, CXC DS Division Head)
Ian Evans (Astrophysicist, CSC2 Lead Scientist)
Janet Evans (CXC Software Manager)
4/28/17
CUC telecon
1
Chandra Source Catalog Review
• As recommended by the CUC, an Expert Review Panel was
appointed to advise CXC on the current status and prospects for
timely completion of the 2nd Chandra Source Catalog (CSC 2.0).
– Mike Watson (University of Leicester)
• XMM Catalog Lead Scientist
– Andy Lawrence (University of Edinburgh)
• Chair of the ad hoc committee that recommended CSC development
• PI of UKIDSS survey
• Lead of VO/IVOA for UK
• Review Panel met at CfA on March 22-24, 2017
– Documentation circulated in advance of the meeting
– Panel met with key staff in the CSC team
•
•
•
•
4/28/17
General overview and schedule
Master match
Source properties
User interface
CUC telecon
2
CSC Review Panel Findings & Recommendations
(from the panel report, shared with the CUC)-1.
• Ambitious project which aims at producing a very high quality
catalog of Chandra detections and sources, coupled with a rich set
of associated data products.
• Detecting and parameterising X-ray detections in each Chandra
observation is demanding because of the complex pattern of fully
and partially overlapping observations, coupled with the large
change in angular resolution with off-axis angle. This demands a
sophisticated data processing approach to maximise detection
sensitivity and reliability.
• CSC2.0 includes a wide range of important improvements, for
example in implementing more robust statistical analyses, and
better handling of extended sources.
• The CSC project has been afforded a very high priority by both the
NASA Senior Review and the Chandra Users Committee (CUC), a
view that is shared by the current panel.
4/28/17
CUC telecon
3
CSC Review Panel Findings & Recommendations
(from the panel report, shared with the CUC)-2.
• The panel were impressed by the excellent work and
dedication of the catalog team of scientists and
programmers and noted the impressive usage statistics of
the CSC1.1 which demonstrates the extent of community
interest, underlining the importance of a timely completion
of version 2.0.
• The week before our visit the CXC released the core of the
version 2.0 catalog, the detection list (CSC2.0pd1). This is a
major milestone, already scientifically useful, and gave the
panel confidence that the full CSC can be completed soon.
• The panel were impressed with the scientific and technical
rigour applied to the issues concerned with these closing
stages, and the thorough testing being undertaken.
4/28/17
CUC telecon
4
CSC Review Panel Findings & Recommendations
(from the panel report, shared with the CUC)-3.
• The project is behind schedule. Two key roadblocks
were identified
– Master source matching
• Solved by reverting to the CSC1 master match (V1) and reconsider
V2 for a future release
– Aperture photometry
• V2 is likely to be ready but if not we will revert to V1
• User interface
– Committee liked CSCView
– Recommended taking a new look at the web page
• Work in progress. We have revised the web page
– Naming and numbering scheme unclear
• We are now naming the detection lists CSC2.0pd#
4/28/17
CUC telecon
5
CSC Review Panel Findings & Recommendations
(from the panel report, shared with the CUC)-4.
• Completion plan developed by the CSC team and the Panel
– Urgent critical testing of the V2 Match software, with a decision point within
two weeks. The team will hold a telecon involving the panel members to
approve the decision
• DONE – using V1 (CSC1.1)
– Starting the Match processing, and associated QA, within three weeks
• STARTED
– In parallel, final testing and tuning of the source property production
software, most of which should be uncontroversial
• IN PROCESS
– Critical testing of the V2 ApPhot software, with a decision point at one-to-two
weeks before the source property production run is due to start
• IN PROCESS / OPEN
– A State of Readiness review before the final production run starts. The panel
members will be involved in the final go-ahead decision
– Public release of CSC 2.0 by ~Oct 1 (see white board from the meeting)
4/28/17
CUC telecon
6
CSC Review Panel Findings & Recommendations
(from the panel report, shared with the CUC)-5.
• The panel recommended a more staged approach to
future releases, compressing the interval between
new releases
• The CXC will address this recommendation after the
CSC2.0 is fully released and will seek input from the
CUC, expert panel, and user community
4/28/17
CUC telecon
7
Schedule and Milestones
Whiteboard Schedule from Review Panel Meeting
4/28/17
CUC telecon
8
CSC2.0pd1 Detection List
•
•
•
•
CSC2.0pd1 released to public Mar 21, 2017, supersedes CSC2.0pd0 from last year
CSC 2.0 pd1 includes the final detections from ~99.5% of the observation stacks in
CSC 2.0.
The release contains 362,182 positions, likelihoods, extents, a fitted intensity that
is a reasonably good proxy for aperture photometry, along with associated errors
for these quantities.
The CSC 2.0 pd1 data access page provides access to the table as well as a full list
of the properties and caveats.
4/28/17
CUC telecon
9
CSC2.0pd1 First-Time Downloads
•
•
•
•
CSC2.0pd1 released to public Mar 21, 2017, supersedes CSC2.0pd0 from last year
CSC 2.0 pd1 includes the final detections from ~99.5% of the observation stacks in
CSC 2.0.
The release contains 362,182 positions, likelihoods, extents, a fitted intensity that
is a reasonably good proxy for aperture photometry, along with associated errors
for these quantities.
The CSC 2.0 pd1 data access page provides access to the table as well as a full list
of the properties and caveats.
4/28/17
CUC telecon
10
The CSC2.0 Web Page
4/28/17
CUC telecon
11
The CSC2.0 Web Page
http://cxc.cfa.harvard.edu/csc2/
4/28/17
CUC telecon
12
CSC Release 2 Highlights
CXC
Release 2 Highlights
• Completed processing ~99.5% of data through stacker pipeline
– Remaining stacks will be run after bulk of data completes master_match
(includes computationally expensive Sgr A* deep stack)
• Released 2nd edition of preliminary detections list on 2017 Mar 21
– 362,182 detections from 7,247 (~99.5%) observation stacks
• Met with CSC2 Review Panel on 2017 Mar 22-24
– Focused on milestones to completion (Master Match, Source Properties, User
Interface)
– Worked through schedule for completion; planned follow-up telecon ~Jul 01
– Very positive feedback to project so far and community impact
• Fell back to rel.1 algorithm for detection matching due to time constraints
– Rel. 1 algorithm implemented and tested in rel. 2.0 master_match pipeline
– Plan to include more scientifically robust rel. 2 algorithm in a future catalog
release (2.1)
• Put Sgr A* and Crab on HOLD for short-term
– Will pick up again once Master Match running smoothly in production
• Master Match processing started this week
– Run single-stack datasets first, followed by multi-stack datasets
– There is a PipeLine (PL) trigger and QA for complicated matches
Inclusive CSC2 report with contributions from CXCDS development, operations, and SDS
from CXC quarterly review this week
13-1
CXC
CSC Rel. 2 Production Statistics
Release 2 Production Processing Statistics
• Through 2017 April 21(total stacks = 7287)
–
–
–
–
–
–
–
Number of stacks completed thru stacker:
Number of stacks still to complete thru stacker:
Number of stacks waiting for manual QA:
Number of manual QA’s completed since 2016 March 04:
Number of detections classified as TRUE:
Number of detections classified as MARGINAL:
Total number of TRUE/MARGINAL:
7269
16
5
11470
286250
83986
370236
CSC 2.0 pd1 detection locations
from CXC quarterly review this week
14-2
CSC Post-stacker Pipeline Thread
From stacker
pipeline
CXC
From stacker
pipeline
From simulations
from CXC quarterly review this week
15-3
CSC Release Summary (cont.)
CXC
Software Releases Planned
Release
CAT 4.4
Date
April 24
Summary
Master Match (patch release)
• Going with Release 1 Xmatch tool
• Master Match Pipeline and QA
• Includes determination of no detect regions for source
properties
CAT 4.4
Production
April 26
Master Match Production
• Run single cohorts first — 80% of data
• Follow-up with multi-source datasets — 20% of data
• Clean up rest of Stacker datasets during this interval —
Sgr A*, Crab, handful of others; process in MM
• Estimate ~6 weeks*
~June – telecon with Review Committee
to assess the Aperture Photometry status
from CXC quarterly review this week
16-4
CSC Release Summary (cont.)
CXC
Software Releases Planned
Release
CAT 4.5
Date
July 14
Summary
Source Properties tools and pipeline
• Includes production of per source data prod for catalog
• Includes per Obsid and per Bayesian Block processing
(Aperture phot, hardness ,timing, spectral) over several
bands
• Includes upgrade of baseline software to CIAO 4.9 with
no OS upgrade (RH v6.2 rather than RH 6.8)*
• Schedule: (Develop/integrate in 3 stages)
– Stage 1: April 19 — Aperture phot, Data Products
– Stage 2: Jun 01 — ObsId PL (add hardness ratio,
timing, & spectral tools); formal integration start
– Stage 3: Jul 01 — Add Bayesian Blocks tool + PL
* Schedule assumes no OS upgrade on L3 Cluster computer;
(Verified OTS build; will build baseline system in next few weeks)
CAT 4.5
Production
July 18
from CXC quarterly review this week
Source Properties/Limiting Sensitivity Production
• 8 week estimate; completion ~Sept 15
• Plan to populate and view CSCview during production
• Run Limiting Sensitivity
17-5
CSC Release Summary (cont.)
CXC
Software Releases Planned
Release
L3SAP
Date
Summer
Summary
Convex Hull Master Match/Source Properties
• ~200 cases
• Planning to handle procedurally
CAT 5.0
~Oct 01
CSC 2.0 labeled and released
• Catalog labeled CSC 2.0
• CSCview full accessibility
• Simple web interface to tabular data queried with
a crossmatch interface
• Limiting Sensitivity results
from CXC quarterly review this week
18-6
CXC
Chandra Source Catalog
Current Release
• Catalog version: 1.1; Released: 2010 Aug 10
– 106,586 master sources
– 158,071 source detections
– 5,110 observations with at least one detected source
• Subset of master source properties are available via ESASky, HEASARC
Browse, NED, and Vizier services
– Usage statistics reported below do not include accesses via these services
Usage Statistics
Release 1.1
Current Reporting Period
2016 Oct 01 – 2017 Mar 31
Number
% Non-CfA
Previous Reporting Period
2016 Apr 01 – 2016 Sep 30
Number
% Non-CfA
CSCview catalog browser
initializations
158 /month
95%
132 /month
94%
CSCview catalog browser
properties searches
234 /month
96%
320 /month
95%
Command-line (CLI)
searches
4255 /month
89%
8219 /month
97%
VO cone searches
8560 /month
~100%
5231 /month
~100%
Data product downloads
64K /month
~100%
296K /month
~100%
CSC Sky in Google Earth
from CXC quarterly review this week
579 visits/month
629 visits/month
19-7
Catalog Use Example (CSC1.1)
A New, Faint Population of X-ray Transients
Franz E. Bauer et al.
https://arxiv.org/pdf/1702.04422.pdf
from which comes the quote:
"Most critically, the CSC is the only source
catalog that provides easy and
straightforward access to photon event
lists and light curves, which we consider
absolutely essential to characterize
the nature of the variability of each
source.”
Press release:
http://chandra.si.edu/photo/2017/cdfsxt1/
4/28/17
CUC telecon
20