### Project Description

Over the last decade, a number of Computational Imaging (CI) systems have been proposed for tasks such as motion deblurring, defocus deblurring and multispectral imaging. These techniques increase the amount of light reaching the sensor via multiplexing and then undo the deleterious effects of multiplexing by appropriate reconstruction algorithms. Given the widespread appeal and the considerable enthusiasm generated by these techniques, a detailed performance analysis of the benefits conferred by this approach is important.

Unfortunately, a detailed analysis of CI has proven to be a challenging problem because performance depends equally on three components: (1) the optical multiplexing, (2) the noise characteristics of the sensor, and (3) the reconstruction algorithm which typically uses signal priors. A few recent papers [1,2] have performed analysis taking multiplexing and noise characteristics into account. However, analysis of CI systems under state-of-the-art reconstruction algorithms, most of which exploit signal prior models, has proven to be unwieldy. We present a comprehensive analysis framework incorporating all three components.

In order to perform this analysis, we model the signal priors using a Gaussian Mixture Model (GMM). A GMM prior confers two unique characteristics. Firstly, GMM satisfies the universal approximation property which says that any prior density function can be approximated to any fidelity using a GMM with appropriate number of mixtures. Secondly, a GMM prior lends itself to analytical tractability allowing us to derive simple expressions for the ‘minimum mean square error’ (MMSE) which we use as a metric to characterize the performance of CI systems. We use our framework to analyze several previously proposed CI techniques (focal sweep [3,4], flutter shutter [5], parabolic exposure [6], etc.), giving conclusive answer to the question: ‘How much performance gain is due to use of a signal prior and how much is due to multiplexing?’ Our analysis also clearly shows that multiplexing provides significant performance gains above and beyond the gains obtained due to use of signal priors.

### Publications

**"A Framework for Analysis of Computational Imaging Systems" **

K. Mitra, O. Cossairt, A. Veeraraghavan

IEEE Pattern Analyis and Machine Intelligence (PAMI), 2014

[PDF]

**"Performance Bounds for Computational Imaging" **

O. Cossairt, M. Gupta, K. Mitra, A. Veeraraghavan

Imaging and Applied Optics Technical Papers, OSA, 2013

[PDF]

**"Performance Limits for Computational Photography" **

O. Cossairt, K. Mitra, A. Veeraraghavan

International Workshop on Advanced Optical Imaging and Metrology, Springer, 2013

[PDF]

**"To Denoise or Deblur: Parameter Optimization for Imaging Systems" **

K. Mitra, O. Cossairt, A. Veeraraghavan

SPIE Electronic Imaging Conference, Jan. 2014

[PDF]

### Presentations

**“Performance Bounds for Computational Imaging,” **

O. Cossairt

Computational Optical Sensing and Imaging Conference, June 2013.

[PDF]

**“When Does Computational Imaging Improve Performance?,” **

O. Cossairt

CVPR Workshop on Computational Cameras and Displays, June 2013.

[PDF]

**“Compressive Imaging,” **

A. Veeraraghavan

CVPR Workshop on Computational Cameras and Displays, June 2013.

**“Performance Limits for Computational Photography,” **

O. Cossairt

Fringe Conference, September 2013.

### Images

**Image formation and noise model:**

Follow the convention adopted by Cossairt et al. [1], we define a conventional camera as an impulse imaging system which measures the desired signal directly (e.g. without blur). CI performance is then compared against the impulse imaging system. Noise is related to the lighting level, scene properties and sensor characteristics. To calculate the photon noise in our experiments, we assume an average scene reflectivity of R=0.5 and sensor quantum efficiency of q=0.5, aperture setting of F/11 and exposure time of t=6 milliseconds. We choose three different example cameras that span the a wide range of consumer imaging devices: 1) a high end SLR camera, 2) a machine vision camera (MVC) and 3) a smartphone camera (SPC). For each of these example camera types, we choose parameters that are typical in the marketplace today: sensor pixel size: δSLR=8 μm for the SLR camera, δMVC=2.5 μm for the MVC, and δSPC=1 μm for the SPC. We also assume a sensor read noise of σr=4e- which is typical for today’s CMOS sensors. The figure shows the relation between light levels and average signal levels for the different camera specifications.

**Image Simulations for Focal Sweep:**

Subplots (a) and (b) show the simulation results obtained by focal sweep and impulse imaging for low (J/σr2=0.2) and high (J/σr2=20) photon to read noise ratios. For the low photon to read noise ratio case, application of our GMM prior increases SNR by around 14dB for both focal sweep and impulse imaging. Multiplexing increases SNR by about 8 dB regardless of the use of prior. For the high photon to read noise ratio case, the SNR gains due to both prior and multiplexing decrease.

**Performance Analysis for Extended Depth Of Field (EDOF) Cameras:**

Here we plot the SNR gain of various EDOF systems at different photon to read noise ratios (J/σr2). In the extended x-axis, we also show the effective illumination levels (in lux) required to produce the given J/σr2 for the three camera specifications: SLR, MVC and SPC. The EDOF systems that we consider are: cubic phase wavefront coding [9], focal sweep camera [3.4], and the coded aperture designs by Zhou et al. [7] and Levin et al. [8]. Signal priors are used to improve performance for both CI and impulse cameras. Wavefront coding achieves a peak SNR gain of 8.8 dB and an average SNR gain of about 7 dB.

**Performance Analysis for Motion Deblurring Cameras:**

In this figure, we study the performance of motion invariant [6], flutter shutter [5] and impulse cameras when image priors are taken into account. Subplot (a) shows the analytic SNR gain (in dB) vs. photon to read noise ratio J/σr2 for the two motion deblurring systems. In the extended x-axis, we also plot the corresponding light levels (in lux) for the three different camera specifications: SLR, MVC and SPC. The motion invariant camera achieves a peak SNR gain of 7.3 dB and an average SNR gain of about 4.5 dB. Subplots (b-c) show the corresponding simulation results. At a low photon to read noise ratio of J/σr2=0.2, motion invariant imaging performs 7.4 dB better than impulse imaging. At the high photon to read noise ratio of J/σr2=20, it is only 1.2 dB better.

**Optimal exposure setting for motion deblurring:**

Here we compute the optimal exposure time for a conventional camera with signal priors taken into account. We first fix the exposure setting of the impulse imaging in such a way that the motion blur is less than a pixel. We then analytically compute the expected SNR gain of different exposure settings (PSF kernel lengths) with respect to the impulse imaging system (of PSF kernel length 1) at various light levels, see subplot (a). For light levels less than 150 lux capturing the image with a larger exposure and then deblurring is a better option, whereas, for light levels greater than 150 lux we should always capture impulse image and then denoise. Subplot (b) shows the optimal blur PSF length at different light levels. At light level of 1 lux, the optimal PSF is 23, whereas for light levels greater than or equal to 150 lux the optimal is 1, i.e., the impulse image setting. Subplots (c-e) show the simulated results with different PSF kernel lengths at a few lighting levels.

**Optimal aperture setting for defocus deblurring:**

Here we compute the optimal aperture setting for a conventional camera with signal priors taken into account. We fix the aperture size of the impulse imaging system so that the defocus blur is less than a pixel. We then analytically compute the SNR gain of different aperture settings (PSF kernel size) with respect to impulse imaging system of PSF kernel size 1×1 pixels for various light levels, see subplot (a). For light levels less than 400 lux capturing the image with a larger aperture and then deblurring is a better option, whereas, for light levels greater than 400 lux we should capture impulse image and then denoise. In subplot (b) we show the optimal blur PSF size at different light levels. At light level of 1 lux, the optimal PSF is 9×9 pixels, whereas for light levels greater than 400 lux the optimal is 1×1 pixels, i.e., the impulse image setting. Subplots (c-d) show the simulated results with different PSF size at a few lighting levels.

### Acknowledgements

Kaushik Mitra and Ashok Veeraraghavan acknowledge support through NSF Grants NSF-IIS: 1116718, NSF-CCF:1117939 and a Samsung GRO grant.

### References

[1] O. Cossairt, M. Gupta, and S.K. Nayar, *When Does Computational Imaging Improve Performance?,* IEEE Transactions on Image Processing, 2012.

[2] N. Ratner, Y. Schechner, and F. Goldberg. *Optimal multiplexed sensing: bounds, conditions and a graph theory link. *Optics Express, 2007.

[3] G. Hausler. *A method to increase the depth of focus by two step image processing.* Optics Communications, 1972.

[4] H. Nagahara, S. Kuthirummal, C. Zhou, and S. Nayar. *Flexible Depth of Field Photography.* In ECCV, 2008.

[5] R. Raskar, A. Agrawal, and J. Tumblin. *Coded exposure photography: motion deblurring using fluttered shutter.*In SIGGRAPH, 2006.

[6] A. Levin, P. Sand, T. Cho, F. Durand, and W. Freeman. *Motion-invariant photography.* In SIGGRAPH, 2008.

[7] C. Zhou and S. Nayar. *What are good apertures for defocus deblurring?* In ICCP, 2009.

[8] A. Levin, R. Fergus, F. Durand, and W. T. Freeman. *Image and depth from a conventional camera with a coded aperture.* In SIGGRAPH, 2007.

[9] E. R. Dowski and T. W. Cathey. *Extended depth of field through wave-front coding.* Applied Optics, 1995.