Motivation Statistical Results for Least Squares Newton algorithm Large Scale Problems Stopping the EM Algorithm Statistically Edge Detection for PSF Estimation Towards Solution of Large Scale Image Restoration and Reconstruction Problems Rosemary Renaut Joint work with Anne Gelb, Aditya Viswanathan, Hongbin Guo, Doug Cochran,Youzuo Lin, Arizona State University Jodi Mead, Boise State University November 4, 2009 National Science Foundation: Division of Computational Mathematics 1 / 49 Motivation Statistical Results for Least Squares Newton algorithm Large Scale Problems Stopping the EM Algorithm Statistically Edge Detection for PSF Estimation Outline 1 Motivation Quick Review 2 Statistical Results for Least Squares Summary of LS Statistical Results Implications of Statistical Results for Regularized Least Squares 3 Newton algorithm Algorithm with LSQR (Paige and Saunders) Results 4 Large Scale Problems Application in Image Reconstruction and Restoration 5 Stopping the EM Algorithm Statistically 6 Edge Detection for PSF Estimation National Science Foundation: Division of Computational Mathematics 2 / 49 Motivation Statistical Results for Least Squares Newton algorithm Large Scale Problems Stopping the EM Algorithm Statistically Edge Detection for PSF Estimation Signal/Image Restoration: Integral Model of Signal Degradation b(t) = R K(t, s)x(s)ds K(t, s) describes blur of the signal. Convolutional model: invariant K(t, s) = K(t − s) is Point Spread Function (PSF). Typically sampling includes noise e(t), model is Z b(t) = K(t − s)x(s)ds + e(t) Discrete model: given discrete samples b, find samples x of x Let A discretize K, assume known, model is given by b = Ax + e. Naı̈vely invert the system to find x! National Science Foundation: Division of Computational Mathematics 3 / 49 Motivation Statistical Results for Least Squares Newton algorithm Large Scale Problems Stopping the EM Algorithm Statistically Edge Detection for PSF Estimation Example 1-D Original and Blurred Noisy Signal Original signal x. Blurred and noisy signal b Gaussian PSF. National Science Foundation: Division of Computational Mathematics 4 / 49 Motivation Statistical Results for Least Squares Newton algorithm Large Scale Problems Stopping the EM Algorithm Statistically Edge Detection for PSF Estimation The Solution: Regularization is needed Naı̈ve Solution A Regularized Solution National Science Foundation: Division of Computational Mathematics 5 / 49 Motivation Statistical Results for Least Squares Newton algorithm Large Scale Problems Stopping the EM Algorithm Statistically Edge Detection for PSF Estimation Least Squares for Ax = b: A Quick Review Background Consider discrete systems: A ∈ Rm×n , b ∈ Rm , x ∈ Rn Ax = b + e, Classical Approach Linear Least Squares (A full rank) xLS = arg min ||Ax − b||22 x Difficulty xLS sensitive to changes in right hand side b when A is ill-conditioned. For convolutional models system is ill-posed. National Science Foundation: Division of Computational Mathematics 6 / 49 Motivation Statistical Results for Least Squares Newton algorithm Large Scale Problems Stopping the EM Algorithm Statistically Edge Detection for PSF Estimation Introduce Regularization to Find Acceptable Solution Weighted Fidelity with Regularization • Regularize xRLS (λ) = arg min{kb − Axk2Wb + λ2 R(x)}, x Weighting matrix Wb • R(x) is a regularization term • λ is a regularization parameter which is unknown. Solution xRLS (λ) depends on λ. depends on regularization operator R depends on the weighting matrix Wb National Science Foundation: Division of Computational Mathematics 7 / 49 Motivation Statistical Results for Least Squares Newton algorithm Large Scale Problems Stopping the EM Algorithm Statistically Edge Detection for PSF Estimation The Weighting Matrix: Some Assumptions for Multiple Data Measurements Given multiple measurements of data b: Usually error in b, e is an m−vector of random measurement errors with mean 0 and positive definite covariance matrix Cb = E(eeT ). For uncorrelated heteroskedastic measurements Cb is diagonal matrix of standard deviations of the errors. (Colored noise) For white noise Cb = σ 2 I. Weighting by Wb = Cb −1 in data fit term, theoretically, ẽ = Wb 1/2 e are uncorrelated. Difficulty if Wb increases ill-conditioning of A! For images find Wb from the image data National Science Foundation: Division of Computational Mathematics 8 / 49 Motivation Statistical Results for Least Squares Newton algorithm Large Scale Problems Stopping the EM Algorithm Statistically Edge Detection for PSF Estimation Formulation: Generalized Tikhonov Regularization With Weighting Use R(x) = kD(x − x0 )k2 x̂ = argmin J(x) = argmin{kAx − bk2Wb + λ2 kD(x − x0 )k2 }. (1) D is a suitable operator, often derivative approximation. Assume N (A) ∩ N (D) = {0} x0 is a reference solution, often x0 = 0, might need to be average solution. Having found λ, the posterior inverse covariance matrix is W̃x = AT Wb A + λ2 I Posterior information can give some confidence on parameter estimates. National Science Foundation: Division of Computational Mathematics 9 / 49 Motivation Statistical Results for Least Squares Newton algorithm Large Scale Problems Stopping the EM Algorithm Statistically Edge Detection for PSF Estimation Choice of λ is crucial: an example with D = I. National Science Foundation: Division of Computational Mathematics 10 / 49 Motivation Statistical Results for Least Squares Newton algorithm Large Scale Problems Stopping the EM Algorithm Statistically Edge Detection for PSF Estimation Choice of λ is crucial: an example with D = I. National Science Foundation: Division of Computational Mathematics 10 / 49 Motivation Statistical Results for Least Squares Newton algorithm Large Scale Problems Stopping the EM Algorithm Statistically Edge Detection for PSF Estimation Choice of λ is crucial: an example with D = I. National Science Foundation: Division of Computational Mathematics 10 / 49 Motivation Statistical Results for Least Squares Newton algorithm Large Scale Problems Stopping the EM Algorithm Statistically Edge Detection for PSF Estimation Choice of λ is crucial: an example with D = I. National Science Foundation: Division of Computational Mathematics 10 / 49 Motivation Statistical Results for Least Squares Newton algorithm Large Scale Problems Stopping the EM Algorithm Statistically Edge Detection for PSF Estimation Choice of λ is crucial: an example with D = I. National Science Foundation: Division of Computational Mathematics 10 / 49 Motivation Statistical Results for Least Squares Newton algorithm Large Scale Problems Stopping the EM Algorithm Statistically Edge Detection for PSF Estimation Choice of λ is crucial: an example with D = I. National Science Foundation: Division of Computational Mathematics 10 / 49 Motivation Statistical Results for Least Squares Newton algorithm Large Scale Problems Stopping the EM Algorithm Statistically Edge Detection for PSF Estimation Choice of λ is crucial: an example with D = I. National Science Foundation: Division of Computational Mathematics 10 / 49 Motivation Statistical Results for Least Squares Newton algorithm Large Scale Problems Stopping the EM Algorithm Statistically Edge Detection for PSF Estimation Choice of λ is crucial: an example with D = I. National Science Foundation: Division of Computational Mathematics 10 / 49 Motivation Statistical Results for Least Squares Newton algorithm Large Scale Problems Stopping the EM Algorithm Statistically Edge Detection for PSF Estimation Choice of λ is crucial: an example with D = I. National Science Foundation: Division of Computational Mathematics 10 / 49 Motivation Statistical Results for Least Squares Newton algorithm Large Scale Problems Stopping the EM Algorithm Statistically Edge Detection for PSF Estimation Choice of λ is crucial: an example with D = I. National Science Foundation: Division of Computational Mathematics 10 / 49 Motivation Statistical Results for Least Squares Newton algorithm Large Scale Problems Stopping the EM Algorithm Statistically Edge Detection for PSF Estimation Choice of λ is crucial: an example with D = I. National Science Foundation: Division of Computational Mathematics 10 / 49 Motivation Statistical Results for Least Squares Newton algorithm Large Scale Problems Stopping the EM Algorithm Statistically Edge Detection for PSF Estimation Choice of λ is crucial: an example with D = I. National Science Foundation: Division of Computational Mathematics 10 / 49 Motivation Statistical Results for Least Squares Newton algorithm Large Scale Problems Stopping the EM Algorithm Statistically Edge Detection for PSF Estimation Choice of λ is crucial: an example with D = I. National Science Foundation: Division of Computational Mathematics 10 / 49 Motivation Statistical Results for Least Squares Newton algorithm Large Scale Problems Stopping the EM Algorithm Statistically Edge Detection for PSF Estimation Choice of λ crucial: Different algorithms - different solutions. Discrepancy Principle Suppose noise is white: Cb = σb2 I. Find λ such that the regularized residual satisfies σb2 = 1 kb − Ax(λ)k22 . m Can be implemented by a Newton root finding algorithm. But discrepancy principle typically oversmooths. Others [Vog02] L-Curve Generalized Cross Validation (GCV) Unbiased Predictive Risk (UPRE) National Science Foundation: Division of Computational Mathematics 11 / 49 Motivation Statistical Results for Least Squares Newton algorithm Large Scale Problems Stopping the EM Algorithm Statistically Edge Detection for PSF Estimation Some standard approaches I: L-curve - Find the corner Let r(λ) = (A(λ) − A)b: Influence Matrix A(λ) = A(AT Wb A + λ2 DT D)−1 AT Plot log(kDxk), log(kr(λ)k) Find corner Trade off contributions. Expensive - requires range of λ. GSVD makes calculations efficient. Not statistically based No corner National Science Foundation: Division of Computational Mathematics 12 / 49 Motivation Statistical Results for Least Squares Newton algorithm Large Scale Problems Stopping the EM Algorithm Statistically Edge Detection for PSF Estimation Generalized Cross-Validation (GCV) Let A(λ) = A(AT Wb A + λ2 DT D)−1 AT Can pick Wb = I. Minimize GCV function kb − Ax(λ)k2Wb , [trace(Im − A(λ))]2 Multiple minima which estimates predictive risk. Expensive - requires range of λ. GSVD makes calculations efficient. Requires minimum Sometimes flat National Science Foundation: Division of Computational Mathematics 13 / 49 Motivation Statistical Results for Least Squares Newton algorithm Large Scale Problems Stopping the EM Algorithm Statistically Edge Detection for PSF Estimation Unbiased Predictive Risk Estimation (UPRE) Minimize expected value of predictive risk: Minimize UPRE function kb − Ax(λ)k2Wb +2 trace(A(λ)) − m Expensive - requires range of λ. GSVD makes calculations efficient. Need estimate of trace Minimum needed National Science Foundation: Division of Computational Mathematics 14 / 49 Motivation Statistical Results for Least Squares Newton algorithm Large Scale Problems Stopping the EM Algorithm Statistically Edge Detection for PSF Estimation Background: Statistics of the Least Squares Problem Theorem (Rao73) Let r be the rank of A and for b ∼ N (Ax, σb2 I), (errors in measurements are normally distributed with mean 0 and covariance σb2 I), then J = min kAx − bk2 ∼ σb2 χ2 (m − r). x 2 J follows a χ distribution with m − r degrees of freedom: Basically the Discrepancy Principle Corollary (Weighted Least Squares) For b ∼ N (Ax, Cb ), Wb = Cb −1 then J = min kAx − bk2Wb ∼ χ2 (m − r). x National Science Foundation: Division of Computational Mathematics 15 / 49 Motivation Statistical Results for Least Squares Newton algorithm Large Scale Problems Stopping the EM Algorithm Statistically Edge Detection for PSF Estimation Extension: Statistics of the Regularized Least Squares Problem Thm: χ2 distribution of the regularized functional (Renaut/Mead 2008) (NOTE: Weighting Matrix on Regularization term.) x̂ = argmin JD (x) = argmin{kAx − bk2Wb + k(x − x0 )k2WD }, WD = DT Wx D. (2) Assume Wb and Wx are symmetric positive definite. Problem is uniquely solvable N (A) ∩ N (D) = {0}. Moore-Penrose generalized inverse of WD is CD Statistics: Errors in the right hand side e ∼ N (0, Cb ), and x0 is known so that (x − x0 ) = f ∼ N (0, CD ), x0 is the mean vector of the model parameters. Then JD (x̂(WD )) ∼ χ2 (m + p − n) National Science Foundation: Division of Computational Mathematics 16 / 49 Motivation Statistical Results for Least Squares Newton algorithm Large Scale Problems Stopping the EM Algorithm Statistically Edge Detection for PSF Estimation Significance of the χ2 result JD ∼ χ2 (m + p − n) For sufficiently large m̃ = m + p − n E(J(x(WD ))) = m + p − n Moreover m̃ − E(JJ T ) = 2(m + p − n) √ √ 2m̃zα/2 < J(x̂(WD )) < m̃ + 2m̃zα/2 . (3) 2 zα/2 is the relevant z-value for a χ -distribution with m̃ = m + p − n degrees National Science Foundation: Division of Computational Mathematics 17 / 49 Motivation Statistical Results for Least Squares Newton algorithm Large Scale Problems Stopping the EM Algorithm Statistically Edge Detection for PSF Estimation General Result [MR09b], [RHM09], [MR09a] The Cost Functional follows a χ2 Statistical Distribution If x0 is not the mean value, then we introduce a non-central χ2 distribution with centrality parameter c. If the problem is rank deficient the degrees of freedom are reduced. Suppose degrees of freedom m̃ and centrality parameter c then E(JD ) = m̃ + c T E(JD JD ) = 2(m̃) + 4c Suggests: Try to find WD so that E(J) = m̃ + c First find λ only. Find Wx = λ2 I National Science Foundation: Division of Computational Mathematics 18 / 49 Motivation Statistical Results for Least Squares Newton algorithm Large Scale Problems Stopping the EM Algorithm Statistically Edge Detection for PSF Estimation What do we need to apply the Theory? Requirements Covariance Cb on data parameters b (or on model parameters x!) A priori information x0 , mean x. But x (and hence x0 ) are not known. If not known use repeated data measurements calculate Cb and mean b. Hence estimate the centrality parameter E(b) = AE(x) implies b = Ax. Hence c = kck22 = kQ̃U T Wb 1/2 (b − Ax0 )k22 E(JD ) = E(kQ̃U T Wb 1/2 (b − Ax0 )k22 ) = m + p − n + kck22 Given the GSVD estimate the degrees of freedom m̃. Then we can use E(J) to find λ National Science Foundation: Division of Computational Mathematics 19 / 49 Motivation Statistical Results for Least Squares Newton algorithm Large Scale Problems Stopping the EM Algorithm Statistically Edge Detection for PSF Estimation Assume x0 is the mean (experimentalists know something about model parameters) DESIGNING THE ALGORITHM: I Recall: if Cb and Cx are good estimates of covariance |JD (x̂) − (m + p − n)| should be small. Thus, let m̃ = m + p − n then we want √ √ m̃ − 2m̃zα/2 < J(x(WD )) < m̃ + 2m̃zα/2 . zα/2 is the relevant z-value for a χ2 -distribution with m̃ degrees GOAL Find Wx to make (3) tight: Single Variable case find λ JD (x̂(λ)) ≈ m̃ National Science Foundation: Division of Computational Mathematics 20 / 49 Motivation Statistical Results for Least Squares Newton algorithm Large Scale Problems Stopping the EM Algorithm Statistically Edge Detection for PSF Estimation A Newton-line search Algorithm to find λ = 1/σ. (Basic algebra) Newton to Solve F (σ) = JD (σ) − m̃ = 0 We use σ = 1/λ, and y(σ (k) ) is the current solution for which x(σ (k) ) = y(σ (k) ) + x0 Then 2 ∂ J(σ) = − 3 kDy(σ)k2 < 0 ∂σ σ Hence we have a basic Newton Iteration σ (k+1) = σ (k) (1 + 1 σ (k) 2 ( ) (JD (σ (k) ) − m̃)). 2 kDyk National Science Foundation: Division of Computational Mathematics 21 / 49 Motivation Statistical Results for Least Squares Newton algorithm Large Scale Problems Stopping the EM Algorithm Statistically Edge Detection for PSF Estimation Practical Details of Algorithm: Large Scale problems Algorithm Initialization Convert generalized Tikhonov problem to standard form.( if L is not invertible you just need to know how to find Ax and AT x, and the null space of L) Use LSQR (Paige and Saunders) algorithm to find the bidiagonal matrix for the projected problem. Obtain a solution of the bidiagonal problem for given initial σ. Subsequent Steps Increase dimension of space if needed with reuse of existing bidiagonalization. May also use smaller size system if appropriate. Each σ calculation of algorithm reuses saved information from the Lancos bidiagonalization. National Science Foundation: Division of Computational Mathematics 22 / 49 Motivation Statistical Results for Least Squares Newton algorithm Large Scale Problems Stopping the EM Algorithm Statistically Edge Detection for PSF Estimation Illustrating the Results for Problem Size 512: Two Standard Test Problems Comparison for noise level 10%. On left D = I and on right D is first derivative Notice L-curve and χ2 -LSQR perform well. UPRE does not perform well. National Science Foundation: Division of Computational Mathematics 23 / 49 Motivation Statistical Results for Least Squares Newton algorithm Large Scale Problems Stopping the EM Algorithm Statistically Edge Detection for PSF Estimation Real Data: Seismic Signal Restoration The Data Set and Goal Real data set of 48 signals of length 3000. The point spread function is derived from the signals. Calculate the signal variance pointwise over all 48 signals. Goal: restore the signal x from Ax = b, where A is PSF matrix and b is given blurred signal. Method of Comparison- no exact solution known: use convergence with respect to downsampling. National Science Foundation: Division of Computational Mathematics 24 / 49 Motivation Statistical Results for Least Squares Newton algorithm Large Scale Problems Stopping the EM Algorithm Statistically Edge Detection for PSF Estimation Comparison High Resolution White noise Greater contrast with χ2 . UPRE is insufficiently regularized. L-curve severely undersmooths (not shown). Parameters not consistent across resolutions National Science Foundation: Division of Computational Mathematics 25 / 49 Motivation Statistical Results for Least Squares Newton algorithm Large Scale Problems Stopping the EM Algorithm Statistically Edge Detection for PSF Estimation THE UPRE SOLUTION: x0 = 0 White Noise Regularization Parameters are consistent: σ = 0.01005 all resolutions National Science Foundation: Division of Computational Mathematics 26 / 49 Motivation Statistical Results for Least Squares Newton algorithm Large Scale Problems Stopping the EM Algorithm Statistically Edge Detection for PSF Estimation THE LSQR Hybrid SOLUTION: White Noise Regularization quite consistent resolution 2 to 100 σ = 0.0000029, .0000029, .0000029, .0000057, .0000057 National Science Foundation: Division of Computational Mathematics 27 / 49 Motivation Statistical Results for Least Squares Newton algorithm Large Scale Problems Stopping the EM Algorithm Statistically Edge Detection for PSF Estimation Illustrating the Deblurring Result: Problem Size 65536 Example taken from RESTORE TOOLS Nagy et al 2007-8: 15% Noise Computational Cost is Minimal: Projected Problem Size is 15, λ = .58 National Science Foundation: Division of Computational Mathematics 28 / 49 Motivation Statistical Results for Least Squares Newton algorithm Large Scale Problems Stopping the EM Algorithm Statistically Edge Detection for PSF Estimation Problem Grain noise 15% added : increasing subproblem size to validate against increasing subproblem size (a) Signal to noise ratio 10 log10 (1/e) relative error e (b) Regularization Parameter Against Problem Size National Science Foundation: Division of Computational Mathematics 29 / 49 Motivation Statistical Results for Least Squares Newton algorithm Large Scale Problems Stopping the EM Algorithm Statistically Edge Detection for PSF Estimation Illustrating the progress of the Newton algorithm post LSQR National Science Foundation: Division of Computational Mathematics 30 / 49 Motivation Statistical Results for Least Squares Newton algorithm Large Scale Problems Stopping the EM Algorithm Statistically Edge Detection for PSF Estimation Illustrating the progress of the Newton algorithm with LSQR National Science Foundation: Division of Computational Mathematics 31 / 49 Motivation Statistical Results for Least Squares Newton algorithm Large Scale Problems Stopping the EM Algorithm Statistically Edge Detection for PSF Estimation Problem Grain noise 15% added for increasing subproblem size Figure: Signal to noise ratio 10 log10 (1/e) relative error e National Science Foundation: Division of Computational Mathematics 32 / 49 Motivation Statistical Results for Least Squares Newton algorithm Large Scale Problems Stopping the EM Algorithm Statistically Edge Detection for PSF Estimation An Alternative Direction For Large Scale Problems : Domain Decomposition [Ren98] Domain decomposition of x into several domains: x = (xT1 , xT2 , ..., xTp )T . Corresponding to different splitting of image x, kernel operator A is split A = (A1 , A2 , ..., Ap ). eg: Different Splitting Schemes National Science Foundation: Division of Computational Mathematics 33 / 49 Motivation Statistical Results for Least Squares Newton algorithm Large Scale Problems Stopping the EM Algorithm Statistically Edge Detection for PSF Estimation Formulation - Regularized Least Squares [LRG09] The linear system Ax ≈ b is replaced with the split systems X Ai yi ≈ bi (x), bi (x) = b − Aj xj = b − Ax + Ai xi . j6=i Locally solve Ax ≈ b min kAi yi − bi (x)k2 , yi ∈<ni 1 ≤ i ≤ p. If the problem is ill-posed we have the regularized problem ˘ ¯ min k Ax − b k22 +λ2 kDxk22 . f Similarly, we will have splitting on operator assuming local regularization «„ « „ «« „ « „„ A2 Ap A A1 . = ··· λ2 D2 λp Dp λ1 D1 DΛ Solve iteratively using novel updates for changing right hand sides, [Saa87], [CW97] Update Scheme - Global solution update from local solutions at step k to step k + 1 x(k+1) = p X (k+1) τi (k+1) (xlocal )i , i=1 (k+1) where (xlocal )i (k) (k) (k+1) T = ((x1 )T , . . . , (xi−1 )T , (yi (k) (k) ) , (xi+1 )T , . . . , (xp )T )T National Science Foundation: Division of Computational Mathematics 34 / 49 Motivation Statistical Results for Least Squares Newton algorithm Large Scale Problems Stopping the EM Algorithm Statistically Edge Detection for PSF Estimation Feasibility of the Approach 1-D Phillips, size 1024, noise level 6% Regularization .25 (a) No Splitting Rel Error .0525 (b) 4 Sub Problems Rel Error .0499 National Science Foundation: Division of Computational Mathematics 35 / 49 Motivation Statistical Results for Least Squares Newton algorithm Large Scale Problems Stopping the EM Algorithm Statistically Edge Detection for PSF Estimation 2-D PET Reconstruction, Size 64 × 64, Noise Level 1.5% Regularization .2 (c) No Splitting SNR 11.73DB (d) 4 Sub Problems SNR 12.24DB National Science Foundation: Division of Computational Mathematics 36 / 49 Motivation Statistical Results for Least Squares Newton algorithm Large Scale Problems Stopping the EM Algorithm Statistically Edge Detection for PSF Estimation A New Stopping Rule for the EM Method [GR09]) Quick Rationale PET Well-known that ML-EM method converges to overly noisy solutions, iterative methods have to stop before convergence [SV82], [HHL92], [HL94]. P Detected counts in tube i are Poisson with mean bi = j aij xj . Hence basic relationship is b ≈ Ax, A is projection matrix, b are counts and x is the density to be reconstructed. EM iteration: x(k+1) = (AT (b./(Ax(k) ))). ∗ x(k) . (e) True (f) k = 95 National Science Foundation: Division of Computational Mathematics (g) k = 500 37 / 49 Motivation Statistical Results for Least Squares Newton algorithm Large Scale Problems Stopping the EM Algorithm Statistically Edge Detection for PSF Estimation A New Estimate of the Stopping Level Algorithm: For k until converged for m tubes Calculate step of EM, x(k) . Update tube means b(k) = Ax(k) . Bin all tubes to have b(k) > 20. √ Calculate y = (b − b(k) )./ b(k) , then y ∼ N (0, 1) Calculate mean ȳ and sample standard deviation s for yi , i = 1 : m. √ Calculate α = m − 1ȳ/s, and pt (α), α ∼ t(m − 1) (t-student density with m − 1 degrees of freedom). p Calculate β = (m − 1)s2 and pN (β), β ∼ N (m − 1, 2(m − 1)) (Gaussian density mean m − 1). Calculate likelihood of sampling α and β from the two distributions: l(k) = pt (α)pN (β). When l(k) is maximum, l(k) < l(k−1) . STOP Solution is x(k−1) . National Science Foundation: Division of Computational Mathematics 38 / 49 Motivation Statistical Results for Least Squares Newton algorithm Large Scale Problems Stopping the EM Algorithm Statistically Edge Detection for PSF Estimation Simulations: Validation Table: The best and the predicted stopping step for 11 simulations Best Pred 95 96 90 88 89 94 90 89 90 89 95 95 90 100 92 100 89 95 94 93 91 94 Figure: l(k) = pt (α)pN (β) for 500 steps. Maximized at k = 96. National Science Foundation: Division of Computational Mathematics 39 / 49 Motivation Statistical Results for Least Squares Newton algorithm Large Scale Problems Stopping the EM Algorithm Statistically Edge Detection for PSF Estimation Extension for Mammogram Denoising: Early Ideas The Model Assume blurring of the mammogram by PSF kernel K and measured image is b. Deconvolve in the Fourier domain and invert to give noisy estimate of optical density d. √ Each entry of d is a linear combination of x-ray energy with Poisson noise, and d is close to normally distributed [ANS48]. To find true optical density x denoise the deconvolved d. Use total variation min x m X √ √ ( di − xi )2 + λ2 kxkT V , s. t. x ≥ 0. i=1 Given knowledge of variance in noise in x-ray automatically select λ using statistical estimation approach. National Science Foundation: Division of Computational Mathematics 40 / 49 Motivation Statistical Results for Least Squares Newton algorithm Large Scale Problems Stopping the EM Algorithm Statistically Edge Detection for PSF Estimation Trial Experiment: Use data set from UoFlorida (DDSM) cancer case 0001, left breast with CC scanning angle (a) Original Image (b) Restored Image Figure: Total yellow (calcification) reduced by deblurring. Rectangle at bottom rhs indicates deblurring National Science Foundation: Division of Computational Mathematics 41 / 49 Motivation Statistical Results for Least Squares Newton algorithm Large Scale Problems Stopping the EM Algorithm Statistically Edge Detection for PSF Estimation PSF Estimation in Blurring Problems using Edge Detection( Cochran, Gelb, Viswanathan, Renaut, Stefan) Given the blurring model (PSF convolution operator K) and x ∈ L2 (−π, π) piecewise-smooth. We estimate the psf starting with 2N + 1 blurred Fourier coefficients b̂(j), j = −N, ..., N . b=K ∗x+e Principle: Apply a linear edge detector, denote by T . We shall assume that the edge detector can be written as a convolution with an appropriate kernel T ∗ (K ∗ x + e) = (K ∗ x + e) ∗ T =x∗K ∗T +e∗T = (x ∗ T ) ∗ K + e ∗ T ≈ [x] ∗ K + ẽ Here [x](s) is a jump function. For a jump discontinuity in a function the jump function at any point s only depends on the values of x at s+ and s− . [x](s) := x(s+ ) − x(s− ) Hence, we observe shifted and scaled replicates of the psf. National Science Foundation: Division of Computational Mathematics 42 / 49 Motivation Statistical Results for Least Squares Newton algorithm Large Scale Problems Stopping the EM Algorithm Statistically Edge Detection for PSF Estimation Example (No Noise) Function 2 1.5 1.5 1 1 h(x) f(x) 0.5 0 0.5 −0.5 −1 0 −1.5 −2 −3 −2 −1 0 x 1 2 −0.5 3 −3 (a) True function −1 0 x 1 2 3 2 3 (b) Motion Blur PSF Blurred Function 1.5 −2 2 1.5 1 1 0.5 f(x) g(x) 0.5 0 0 −0.5 −0.5 −1 Function −1 −1.5 Blur after edge detection True blur (normalized) −1.5 −3 −2 −1 0 x 1 (c) Blurred Function 2 3 −2 −3 −2 −1 0 x 1 (d) After applying edge detection Figure: Function subjected to motion blur, N = 128 National Science Foundation: Division of Computational Mathematics 43 / 49 Motivation Statistical Results for Least Squares Newton algorithm Large Scale Problems Stopping the EM Algorithm Statistically Edge Detection for PSF Estimation 2 2 1.5 1.5 1 1 0.5 0.5 f(x) f(x) Representative Examples : Gaussian PSF 0 −0.5 0 −0.5 −1 −1 Function Function Blurred, noisy function −1.5 Blurred, noisy function −1.5 Blur after edge detection Blur after edge detection True blur (normalized) −2 −3 −2 −1 True blur (normalized) 0 x 1 2 (a) Noisy blur estimation 3 −2 −3 −2 −1 0 x 1 2 3 (b) After low-pass filtering Figure: Function subjected to Gaussian blur, N = 128 Complex noise distribution on Fourier coefficients – ê ∼ N (0, 1.5 ) (2N +1)2 Second picture subjected to low-pass (Gaussian) filtering It is conceivable that parameter estimation for a Gaussian PSF can take into account the effect of Gaussian filtering National Science Foundation: Division of Computational Mathematics 44 / 49 Motivation Statistical Results for Least Squares Newton algorithm Large Scale Problems Stopping the EM Algorithm Statistically Edge Detection for PSF Estimation Representative Examples: Motion Blur 2 2 1.5 1.5 1 1 0.5 0.5 Function Blurred, noisy function f(x) f(x) Blur after edge detection 0 −0.5 True blur (normalized) 0 −0.5 −1 −1 Function Blurred, noisy function −1.5 −1.5 Blur after edge detection True blur (normalized) −2 −3 −2 −1 0 x 1 2 (a) Noisy blur estimation 3 −2 −3 −2 −1 0 x 1 2 3 (b) After TV denoising Figure: Function subjected to Motion blur, N = 128 Cannot perform conventional low-pass filtering since blur is piecewise-smooth We compute the noisy blur estimate for Fourier expansion of blurred jump SN [b] ≈ [x] ∗ K + ẽ Denoising problem formulation min x k x − SN [b] k22 + λ2 k Dx k1 . National Science Foundation: Division of Computational Mathematics 45 / 49 Motivation Statistical Results for Least Squares Newton algorithm Large Scale Problems Stopping the EM Algorithm Statistically Edge Detection for PSF Estimation Future Work Combining Approaches Extend the parameter selection methods to the domain decomposition problems for large scale. Use efficient schemes for large scale problems - eg right hand side updates Extend to edge detection approaches Use tensor product of the PSF for extension to 2D - is it feasible Use parameter estimation techniques for the 2D problem Further development of statistical techniques for estimating acceptable solutions. National Science Foundation: Division of Computational Mathematics 46 / 49 Motivation Statistical Results for Least Squares Newton algorithm Large Scale Problems Stopping the EM Algorithm Statistically Edge Detection for PSF Estimation Bibliography I F. J. ANSCOMBE. The transformation of poisson, binomial and negative-binomial data. Biometrika, 35:246–254, 1948. T. F. Chan and W. L. Wan. Analysis of projection methods for solving linear systems with multiple right-hand sides. 1997. H. Guo and R. A. Renaut. Revisiting stopping rules for iterative methods used in emission tomography: Analysis and developments. Physics of Medicine and Biology, submitted, 2009. H. M. Hudson, B. F. Hutton, and R. Larkin. Accelerated EM reconstruction using ordered subsets. J. Nucl. Med., 33:960, 1992. H. M. Hudson and R. Larkin. Accelerated imaging reconstruction using orded subsets of projection data. IEEE Trans. Med. Imag., 13(4):601–609, 1994. National Science Foundation: Division of Computational Mathematics 47 / 49 Motivation Statistical Results for Least Squares Newton algorithm Large Scale Problems Stopping the EM Algorithm Statistically Edge Detection for PSF Estimation Bibliography II Y. Lin, R. A. Renaut, and H. Guo. Multisplitting for regularized least squares. in prep., 2009. J. Mead and R. A. Renaut. Least squares problems with inequality constraints as quadratic constraints. Linear Algebra and its Applications, 2009. J. Mead and R. A. Renaut. A Newton root-finding algorithm for estimating the regularization parameter for solving ill-conditioned least squares problems. Inverse Problems, 25, 2009. R. A. Renaut. A parallel multisplitting solution of the least squares problem. BIT, 1998. R. A Renaut, I. Hnetynkova, and J. Mead. Regularization parameter estimation for large scale Tikhonov regularization using a priori information. Computational Statistics and Data Analysis, 54(1), 2009. National Science Foundation: Division of Computational Mathematics 48 / 49 Motivation Statistical Results for Least Squares Newton algorithm Large Scale Problems Stopping the EM Algorithm Statistically Edge Detection for PSF Estimation Bibliography III Y. Saad. On the Lanczos method for solving symmetric linear systems with several right-hand sides. 1987. L. A. Shepp and Y. Vardi. Maximum likelihood reconstruction for emission tomography. IEEE Trans. Med. Imag., MI-1(2):113–122, Oct. 1982. Curtis R. Vogel. Computational Methods for Inverse Problems. Society for Industrial and Applied Mathematics, Philadelphia, PA, USA, 2002. National Science Foundation: Division of Computational Mathematics 49 / 49 Motivation Statistical Results for Least Squares Newton algorithm Large Scale Problems Stopping the EM Algorithm Statistically Edge Detection for PSF Estimation Future Work Other Results and Future Work Software Package! Diagonal Weighting Schemes Edge preserving regularization - Total Variation Better handling of Colored Noise. Residual Periodogram for large scale. National Science Foundation: Division of Computational Mathematics 50 / 49 Motivation Statistical Results for Least Squares Newton algorithm Large Scale Problems Stopping the EM Algorithm Statistically Edge Detection for PSF Estimation Algorithm Using the GSVD GSVD Use GSVD of [Wb 1/2 A, D] For γi the generalized singular values, and s = U T Wb 1/2 r m̃ = m − n + p s̃i = si /(γi2 σx2 + 1), i = 1, . . . , p, Find root of F (σx ) = p X ( i=1 ti = s̃i γi . 1 γi2 σx2 + 1 )s2i + m X s2i − m̃ = 0 i=n+1 Equivalently: solve F = 0, where F (σx ) = sT s̃ − m̃ and F 0 (σx ) = −2σx ktk22 . National Science Foundation: Division of Computational Mathematics 51 / 49 Motivation Statistical Results for Least Squares Newton algorithm Large Scale Problems Stopping the EM Algorithm Statistically Edge Detection for PSF Estimation Practical Details of Algorithm Find the parameter Step 1: Bracket the root by logarithmic search on σ to handle the asymptotes: yields sigmamax and sigmamin Step 2: Calculate step, with steepness controlled by tolD. Let t = Dy/σ (k) , where y is the current update, then step = 1 1 ( )2 (JD (σ (k) ) − m̃) 2 max {ktk, tolD} Step 3: Introduce line search α(k) in Newton sigmanew = σ (k) (1 + α(k) step) National Science Foundation: Division of Computational Mathematics 52 / 49 Motivation Statistical Results for Least Squares Newton algorithm Large Scale Problems Stopping the EM Algorithm Statistically Edge Detection for PSF Estimation Key Aspects of the Proof I: The Functional J Algebraic Simplifications: Rewrite functional as quadratic form Regularized solution given in terms of regularization matrix R(WD ) x̂ R(WD ) = x0 + (AT Wb A + DT Wx D)−1 AT Wb r, = x0 + R(WD )Wb = x0 + y(WD ). = T 1/2 (4) r = b − Ax0 r, (5) T −1 (A Wb A + D Wx D) T A Wb 1/2 (6) Functional is given in terms of influence matrix A(WD ) A(WD ) = Wb 1/2 AR(WD ) JD (x̂) = rT Wb 1/2 (Im − A(WD ))Wb 1/2 r, = T (7) let r̃ = Wb 1/2 r r̃ (Im − A(WD ))r̃. A Quadratic Form National Science Foundation: Division of Computational Mathematics (8) (9) 53 / 49 Motivation Statistical Results for Least Squares Newton algorithm Large Scale Problems Stopping the EM Algorithm Statistically Edge Detection for PSF Estimation Key Aspects of the Proof II : Properties of a Quadratic Form χ2 distribution of Quadratic Forms xT P x for normal variables (Fisher-Cochran Theorem) Components xi are independent normal variables xi ∼ N (0, 1), i = 1 : n. A necessary and sufficient condition that xT P x has a central χ2 distribution is that P is idempotent, P 2 = P . In which case the degrees of freedom of χ2 is rank(P ) =trace(P ) = n. . When the means of xi are µi 6= 0, xT P x has a non-central χ2 distribution, with non-centrality parameter c = µT P µ A χ2 random variable with n degrees of freedom and centrality parameter c has mean n + c and variance 2(n + 2c). National Science Foundation: Division of Computational Mathematics 54 / 49 Motivation Statistical Results for Least Squares Newton algorithm Large Scale Problems Stopping the EM Algorithm Statistically Edge Detection for PSF Estimation Key Aspects of the Proof III: Requires the GSVD Lemma Assume invertibility and m ≥ n ≥ p. There exist unitary matrices U ∈ Rm×m , V ∈ Rp×p , and a nonsingular matrix X ∈ Rn×n such that » – Υ A=U X T D = V [M, 0p×(n−p) ]X T , (10) 0(m−n)×n Υ = diag(υ1 , . . . , υp , 1, . . . , 1) ∈ Rn×n , 0 ≤ υ1 ≤ · · · ≤ υp ≤ 1, υi2 + µ2i = 1, M = diag(µ1 , . . . , µp ) ∈ Rp×p , 1 ≥ µ1 ≥ · · · ≥ µp > 0, i = 1, . . . p. (11) The Functional with the GSVD Let Q̃ = diag(µ1 , . . . , µp , 0n−p , Im−n ) then J = r̃T (Im − A(WD ))r̃ = kQ̃U T r̃k22 = kkk22 National Science Foundation: Division of Computational Mathematics 55 / 49 Motivation Statistical Results for Least Squares Newton algorithm Large Scale Problems Stopping the EM Algorithm Statistically Edge Detection for PSF Estimation Proof IV: Statistical Distribution of the Weighted Residual Covariance Structure Errors in b are e ∼ N (0, Cb ). Now b depends on x, b = Ax hence we can show b ∼ N (Ax0 , Cb + ACD AT ) (x0 is mean of x) Residual r = b − Ax ∼ N (0, Cb + ACD AT ). r̃ = Wb 1/2 r ∼ N (0, I + ÃCD ÃT ), Ã = Wb 1/2 A. Use the GSVD I + ÃCD ÃT = U Q−2 U T , Q = diag(µ1 , . . . , µp , In−p , Im−n ) Now k = QU r̃ then k ∼ N (0, QU (U Q−2 U T )U Q) ∼ N (0, Im ) T T But J = kQ̃U T r̃k2 = kk̃k2 , where k̃ is the vector k excluding components p + 1 : n. Thus JD ∼ χ2 (m + p − n). National Science Foundation: Division of Computational Mathematics 56 / 49 Motivation Statistical Results for Least Squares Newton algorithm Large Scale Problems Stopping the EM Algorithm Statistically Edge Detection for PSF Estimation When mean of the parameters is not known, or x0 = 0 is not the mean Corollary: non-central χ2 distribution of the regularized functional Recall x̂ = argmin JD (x) = argmin{kAx − bk2Wb + k(x − x0 )k2WD }, WD = DT Wx D. Assume all assumptions as before, but x 6= x0 is the mean vector of the model parameters. Let c = kck22 = kQ̃U T Wb 1/2 A(x − x0 )k22 Then JD ∼ χ2 (m + p − n, c) The functional at optimum follows a non central χ2 distribution National Science Foundation: Division of Computational Mathematics 57 / 49 Motivation Statistical Results for Least Squares Newton algorithm Large Scale Problems Stopping the EM Algorithm Statistically Edge Detection for PSF Estimation A further result when A is not of full column rank The Rank Deficient Solution Suppose A is not full column rank. Then the filtered solution can be written in terms of the GSVD xFILT (λ) = p X p n n X X X fi γi2 s x̃ + s x̃ = s x̃ + si x̃i . i i i i i i + λ2 ) υi i=p+1 i=1 i=p+1 υi (γi2 i=p+1−r Here fi = 0, i = 1 : p − r, fi = γi2 /(γi2 + λ2 ), i = p − r + 1 : p. This yields J(xFILT (λ)) ∼ χ2 (m − n + r, c) notice degrees of freedom are reduced. National Science Foundation: Division of Computational Mathematics 58 / 49

© Copyright 2023 Paperzz