NDPPP - as-ska

NDPPP
• Now we should convert the data to something in casa format.
• We will use the New Default Pre-Processing Pipeline (NDPPP) tool
• Sect. 5 LOFAR Imaging Cookbook
• LOFAR Wiki Page: user=school & password=french
8
http://www.lofar.org/wiki/doku.php?id=start
http://www.lofar.org/wiki/doku.php?id=engineering:software:tools:ndppp
15
Get rid of LofarStMan
• To run NDPPP, you should specify in a parset file which are the operations to
perform on the data and which are the parameters that you want to use.
8>8vim8DPPP?makeplain.parset888#!or!emacs,!nano,!geany,!…8
• Put the following commands in the parset and save it:
msin=L114221_SAP000_SB031_uv.MS8
msout=L114221_SAP000_SB031_uv_plain.MS8
msin.autoweight=true8
steps=[]8
• Apart from copying the input MS, it will also flag NaNs and infinite data values.
Furthermore the second line means that proper data weights are calculated using
the auto-correlations
• Now run DPPP on this parset:
8>8DPPP8DPPP?makeplain.parset8
• Now we have our own copy which can be opened in casa tools
16
Do we need to
flag data ?
• Open the data in casa tools:
8>8use8Casa8
8>8casaplotms8&8
• Open your MS from the GUI. To speed up
plotting, only plot the xx correlation.
• Select only cross correlations by typing
*&* in the antenna field
(*: any antenna, &: cross correlations)
• Note the large spikes: probably RFI.
Adjust the y-scale to find any real signal.
• In the Axes tab, set max y-scale to
something sensible.
Does the scale of the signal make
sense to you? (answer: no)
17
rfigui
• rfigui makes plots to check RFI
8>8rfigui8L114221_SAP000_SB031_uv.MS8
• In the dialog, just choose ‘Open’
• Press ‘Forward’ to see data for the first
baseline: LOFAR8CS001HBA08×8LOFAR8CS001HBA18
• The spikes we spotted are clearly RFI
• Some are broadband, some are not
• Create a power spectrum plot (plot menu)
What is the interference at 134.375 MHz?
• rfigui (aoflagger) can also flag the data:
‘Actions’ —> ‘Execute Strategy’
Offringa et al., MNRAS, Volume 405, Issue 1, pp 155-167
Offringa et al., Proceedings of Science (http://arxiv.org/abs/1007.2089).
before flagging
after flagging
18
aoflagger
• The AOFlagger can be called with DPPP.
8>8cd8/data/scratch/yourname!
8>8vim8DPPP?flag.parset8
• Enter the following contents in the parset:
msin=L114221_SAP000_SB031_uv_plain.MS8
msout=.8
steps=[preflagger,8aoflagger]8
preflagger.baseline=*&&&8#8autocorr’s8
• Now run your first real action:
8>8DPPP8DPPP?flag.parset8
• From the output:
Which station/ant was most affected?
Which channel was most affected?
• Re-examine the data with casaplotms
8>8casaplotms8
Does it look reasonable now?
19
aoflagger
• The AOFlagger can be called with DPPP.
8>8cd8/data/scratch/yourname!
8>8vim8DPPP?flag.parset8
• Enter the following contents in the parset:
msin=L114221_SAP000_SB031_uv_plain.MS8
msout=.8
steps=[preflagger,8aoflagger]8
preflagger.baseline=*&&&8#8autocorr’s8
• Now run your first real action:
8>8DPPP8DPPP?flag.parset8
• From the output:
Which station/ant was most affected?
Which channel was most affected?
• Re-examine the data with casaplotms
8>8casaplotms8
Does it look reasonable now?
20
averaging
• The observation we were working on was far from bright sources, so demixing is not
necessary
• The data has been flagged, so now we can average it down in time and frequency
• As usual, prepare the parset file for NDDD:
vim8DPPP?average.parset8
• The parameters to set:
msin=L114221_SAP000_SB031_uv_plain.MS8#8The8‘plain’8data8
msout=L114221_SAP000_SB031_uv_plain_avg.MS8
steps=[averager]8
averager.timestep=58
averager.freqstep=88
• Run this parset through DPPP
Do you get the compression you expected?
21
All together!
• As the name suggests, DPPP (Default PreProcessing Pipeline) was designed do
pipelines of steps. We could have done our reduction in one go:
msin=L114221_SAP000_SB031_uv.MS8#8The8raw8data8
msin.autoweight=true8
msout=L114221_SAP000_SB031_uv_avg.MS8
steps=[preflagger,aoflagger,averager]8
preflagger.baseline=*&&&8
averager.timestep=58
averager.freqstep=8
22
scripting
• Congratulations on reducing your first data set! Only 79 to go! !
• We are not going to do this by hand
• Write a script in your favorite language that processes a number of subbands
• Tip: you can override keys in your parset by specifying them on the command line:
DPPP8reduction.parset8msin=another.MS8msout=another_avg.MS8
scp -r lhd002:/data/dataschool2014/imaging/t1/DPPP-all.parset .
vim8DPPP?all.parset8
msin.autoweight=true8
msout=L114221_SAP000_SB031_uv_avg.MS8
steps=[preflagger,aoflagger,averager]8
preflagger.baseline=*&&&8
averager.timestep=58
averager.freqstep=8
23
scripting
scp -r lhd002:/data/dataschool2014/imaging/t1/script.sh .
for8inputname8in8$(ssh8lhd0028'ls8/data/dataschool2014/imaging/target');8do8
88888echo8"Copying8${inputname}8from8head8node8to8working8node"8
88888scp8?rq8lhd002:/data/dataschool2014/imaging/target/${inputname}8.8
88888outputname=$(echo8${inputname}8|8sed8"s/\.MS/_processed.MS/")8
88888echo8${outputname}8
88888DPPP8DPPP?all.parset8msin=${inputname}8msout=${outputname}8
88888echo8"Removing8raw8data8${inputname}8from8working8node"8
88888rm8?rf8${inputname}8
done88
24
25
26

Download Report