PYBDSM - AN UPDATE
Blob Detection and Source
Measurement
Niruj Mohan Ramanujam
Sterrewacht, Leiden
Busyweek 11
Monday, May 2, 2011
PyBDSM - Python Blob Detection and Source Measurement
•
Runs as a single command
•
Does not appear to compile fully on daily build.
•
Currently everyone uses the one in David’s account (it builds on its own on compute nodes)
•
You can compile it on your laptop as well
•
Cookbook has some bacic information. Detailed manual not yet done.
•
Ask me or David if you need extra functionality in a hurry
•
Structure may change sometime this week, but will be reflected in the cookbook
Monday, May 2, 2011
PyBDSM - Python Blob Detection and Source Measurement
•
Background map of rms and mean
•
Forms islands
•
Thresholds by FDR or hard
•
Does initial guess of gaussians by active + passive deblending
•
Fits multiple gaussians reluctantly
•
Groups them into sources
•
Spectral index, RM, poln properties of compact and extended sources
•
PSF variation across the image
•
Shapelet decomposition
•
Extended sources through wavelets
Monday, May 2, 2011
PyBDSM - some issues
•
It is sometimes super slow : when decomposing really large islands, >35-40 beams.
Working on a method to break-up islands reliably, without messing with thresholds.
•
Crashes at standard places
Should be fixed now
•
Wavelet decomposition, psf variation, shapelets modules not available
They now compile on compute nodes, available at /home/rafferty. (not to be used blindly)
•
Source models for A-team sources not good enough
It will never be good enough. Find some other method
Monday, May 2, 2011
PyBDSM - some issues
Some islands in small scale
wavelet images of Cyg A P
band image
Monday, May 2, 2011
PyBDSM - some more issues
•
Guessing the boxsize for determining the background rms is crucial. Dont assume the default
value will always make sense
•
Small scale wavelet transform images will have emission which is difficult to characterise.
Affects both images with high threshold for extraction and making source models for A-team
•
Monday, May 2, 2011
PyBDSM - source models for bright extended sources
• Run normal source extraction
• Compute residual image after subtraction
gaussians
• Run wavelet transform on residual image
• Fit gaussians to every wavelet image, without
assuming the beam
• Output all gaussians in bbs format
• Group gaussians in successive wavelet scales
to build a pyramidal source
Monday, May 2, 2011
PyBDSM - source models for calibration
Vir A input image
PyBDSM multi-scale (180) gaussian model
Virgo A (a good model)
Monday, May 2, 2011
PyBDSM - source models for calibration
Abell 2255
(a good model)
Monday, May 2, 2011
PyBDSM - source models for calibration
Hydra A(not so good)
Monday, May 2, 2011
PyBDSM - source models for calibration
•
•
For Cygnus A, it runs forever. Will be solved when large islands are broken up properly
•
Mainly due to
The calibrated image made with the PyBDSM model is always worse than that with the CC model,
obviously (see Reinout’s images)
(1) Unable to fit the complex small scale structure solely with gaussians
(2) Negatives in the large scale wavelet images
•
•
For characterising extended sources in catalogues, the current method is sufficient
For calibration of source like Cyg A, clean components will always do better, but will take much
slower (30000 CC versus few 100 gaussians)
•
Another option, when BBS allows it, is to use shapelets for the small scale wavelet image and
gaussians for large scale wavelets - that should work just fine, and have a minimum number of
components
Monday, May 2, 2011
PyBDSM - input parameters
•
•
•
•
•
•
•
•
•
•
•
•
•
BASIC STUFF
fits_name
= Option(None, String(), doc="input FITS file name")
use_pyrap
= Bool(False, doc="Use Pyrap, if available, to read in image file instead of pyfits")
beam
= Option(None, Tuple(Float(), Float(), Float()), doc="Override beam shape from file header: (bmaj, bmin, bpa)")
median frequency and scales with frequency for channels.")
THRESHOLDS
thresh
= Enum(None, "hard", "fdr", doc="hard/fdr thresholds. None => calculate inside program")
kappa_clip = Float(3, doc="kappa for clipped mean, rms")
thresh_isl = Float(3, doc="pixel threshold in sigma, for the island boundary")
thresh_pix = Float(5, doc="pixel threshold in sigma, for island peak")
fdr_alpha
= Float(0.05, doc="Alpha for FDR algorithm for thresholds")
fdr_ratio
= Float(0.1, doc="For thresh==None; if #false_pix/#source_pix < fdr_ratio thresh='hard' else 'fdr'")
RMSIMAGE
rms_map = Option(None, Bool(), doc="True => use rms map; False => constant rms; " \
"None => calculate inside program whether to use or not")
mean_map = String('default', doc="default => calc whether to use or not, 'zero' => 0, \
'const' => clipped mean, 'map' => use map")
rms_box = Option(None, Tuple(Int(), Int()), doc="box size, step size for rms/mean map calculation")
rms_value = Option(None, Float(), doc="Specify value of constant rms to use when rms_map=False, \
else take clipped_rms value")
Monday, May 2, 2011
PyBDSM - input parameters
PREPROCESSING
•
•
•
•
bmpersrc_th
= Option(None, Float(), doc="Theoretical estimate of number of beams " \
"per source. None => calculate inside program")
spline_rank = Enum(3,1, doc="rank of the interpolating function for rms/mean map")
minpix_isl = Int(4, doc="minimal number of pixels with emission per island")
check_outsideuniv = Bool(False, doc="Check for pixels outside the universe (takes time)")
GAUSSIAN FITTING AND FLAGGING
•
•
•
•
•
•
•
•
ini_gausfit
= String('fbdsm', doc = "Initial guess for gaussians, 'default', 'fbdsm', or 'nobeam'. See manual for details")
flag_smallsrc
= Bool(True, doc = "Flag sources smaller than the flag_minarea times beam area")
flag_minsnr
= Float(0.9, doc = "Flag gaussian if peak less than flag_minsnr times thresh_pix times local rms")
flag_maxsnr
= Float(1.5, doc = "Flag gaussian if peak greater than flag_maxsnr times max value in island")
flag_maxsize_isl = Float(1.0, doc = "Flag gaussian if x, y bounding box around sigma-contour is factor times \
island bbox")
flag_bordersize
= Int(0, doc = "Flag gaussian if centre is outside border-flag_bordersize pixels, default=0")
flag_maxsize_bm = Float(10.0, doc = "Flag gaussian if area greater than flag_maxsizebm times beam area")
flag_minsize_bm = Float(0.5, doc = "Flag gaussian if flag_smallsrc=True and area smaller than \
flag_minsize_bm times beam area")
Monday, May 2, 2011
PyBDSM - input parameters
WAVELETS
•
•
•
•
atrous_do
atrous_jmax
atrous_lpf
atrous_bdsm_do
•
•
•
SHAPELETS
shapelet_do
= Bool(False, doc="Decompose island into shapelets (True/False)")
shapelet_basis
= Enum("cartesian", "polar", doc="basis set for shapelet decomposition.")
shapelet_fitmode = Enum("fit", None, doc="Calculate shapelet coeff.s by fitting (fit) or integrating (None)")
•
•
•
•
CHANNEL COLLAPSE
collapse_mode = Enum('average', 'single', doc="Average channels or take single channel to perform source detection on")
collapse_ch0
= Int(0, doc="Number of the channel for source extraction, if collapse_mode='single'")
collapse_av
= List(None, doc="List of channels to average if collapse_mode='average'; None=all")
collapse_wt
= Enum('unity', 'rms', doc="Average channels with weights=1 or 1/rms_clip^2 if collapse_mode='average'")
Monday, May 2, 2011
= Bool(False, doc = "Decompose gaussian residual image into a-trous wavelets")
= Int(0, doc = 'J_max, max wavelength order, 0 => determine within')
= String('b3', doc = 'Low pass filter, either b3 or tr for B3 spline or Triangle')
= Bool(True, doc = "Perform source extraction on each wvavelet scale")
•
SPECTRAL INDEX
beam_spectrum = Option(None, List(Tuple(Float(), Float(), Float())), doc = "synthesized beam per channel; None=>all equal")
beam_sp_derive = Bool(False, doc = "If true and beam_spectrum is None, then assume header beam is for \
spectralindex_do = Bool(False, doc="Calculate spectral indices for multi-channel image")
spin_nchan0
= Int(40, doc='rough number of channels to average cube to, before attempting to fit spectral index.')
spin_flux
= String('total', doc = "'total' or 'peak'; determines which flux to compute spectral indices for")
flagchan_rms
= Bool(True, doc = "Flag channels before (averaging and) extracting spectral index, if their rms if \
more than 5 (clipped) sigma outside the median rms over all channel, but only if <= 10% of channels")
specind_Msrc_exclude1 = Float(0.06, doc = "Exclude gaussians less than this factor*max(gaussian peaks) inside that M source \
while fitting for spectral index for each gaussian")
specind_Msrc_exclude2 = Float(3, doc = "Exclude gaussians whose average SNR over channels is less than this")
specind_kappa = Float(7.5, doc = "Kappa for deciding Case I, II, III for spectral index")
specind_minchan = Int(6, doc = "Min number of channels to average to, for a given source, before deciding to sum \
over pixels if still below specin_kappa*noise")
specind_dolog = Bool(False, doc = "Fit flux vs freq in log space or in real space")
•
•
•
•
•
•
•
•
•
•
•
•
•
•
PSF VARIATION
psf_vary_do = Bool(False, doc = "Estimate psf field from image")
psf_generators = String('calibrators', doc = "calibrator/field; default=calibrator")
psf_nsig = Float(3.0, doc = "Kappa for clipping within each bin; default=3.0")
psf_over = Int(2, doc = "Factor of nyquist sample for binning bmaj etc vs snr; default=2")
psf_kappa2 = Float(2.0, doc = "Kappa for clipping for analytic fit; default=2.0")
psf_snrcut = Float(10.0, doc = "Minimum snr for statistics; default=10.0")
psf_snrtop = Float(0.15, doc = "Fraction of SNR>snrcut as primary generators; default=0.10")
psf_snrbot = Float(0.20, doc = "Fraction of SNR>snrcut as all generators; default=0.30")
psf_snrcutstack = Float(15.0, doc = "Unresolved sources with higher SNR taken for stacked psfs; default=15.0")
psf_gencode = String('list', doc = "List/file, default=list, primary gens from gaussian list or file")
psf_primarygen = String('', doc = "Filename for primary gens if gencode=file, x,y,snr")
psf_itess_method = Int(0, doc = "0=normal, 1=0+round, 2=LogSNR, 3=SqrtLogSNR ; default=0")
psf_tess_sc = String('s', doc = "(s)imple/(c)omplicated - normal or approximate (fuzzy); default=s")
psf_tess_fuzzy = Float(0.05, doc = "Fraction of overlap for fuzzy tesselation; default=0.05")
•
•
•
•
•
•
•
•
•
•
Monday, May 2, 2011
•
•
•
•
OUTPUT, BOOK-KEEPING
savefits_rmsim = Bool(True, doc="Save background rms image as fits file")
savefits_meanim = Bool(True, doc="Save background mean image as fits file")
savefits_normim = Bool(True, doc="Save normalised image as fits file")
savefits_rankim = Bool(False, doc="Save island rank image as fits file")
•
•
•
•
•
•
•
•
•
•
print_timing = Bool(True, doc="print basic timing information")
verbose_fitting= Bool(False, doc="print out extra information during fitting")
debug_figs_1 = Bool(False, doc='Plot gaussian fit images for each source for each channel for spectralindex')
debug_figs_2 = Bool(False, doc='Plot gaussian fit parameter plots for each source as a fn of channel')
debug_figs_3 = Bool(False, doc='Plot flagged channels before and after averaging for spectralindex for image')
debug_figs_4 = Bool(False, doc='Plot figs for Case I')
debug_figs_5 = Bool(False, doc='Plot for spectral index for M-sources')
debug_figs_6 = Bool(False, doc='Plot images for inigaus_nobeam')
debug_figs_7 = Bool(False, doc='Plot images for psf_vary')
debug_figs_8 = Bool(True, doc='Plotim for each wavelet plane if decomposed into gaussians')
•
bbs_patches = Enum('single', 'none', 'seperate', doc="For BBS sky_in file, whether there are no patches, all gaussians in one \
patch, or each island gaussian in its own patch")
opdir_overwrite= String('overwrite', doc="overwrite/append : deletes existing files or appends a new directory")
solnname
= Option(None, String(), doc="Name of the run, to be appended to the name of the output directory")
indir
= Option(None, String(), doc="Directory of input FITS files")
output_fbdsm = Bool(False, doc="write out fBDSM format output files or not, for use in Anaamika")
fbdsm_scratch = String('/Users/mohan/anaamika/image/', doc="scratch directory for storing fBDSM files")
polarisation_do = Bool(False, doc="Find polarization properties (True/False)")
rm_do
= Bool(False, doc="Find rotation measure properties (True/False)")
•
•
•
•
•
•
•
Monday, May 2, 2011