In this work, we propose using camera arrays coupled with coherent illumination as an effective method of improving spatial resolution in long distance images by a factor of ten and beyond. Recent advances in ptychography have demonstrated that one can image beyond the diffraction limit of the objective lens in a microscope. We demonstrate a similar imaging system to image beyond the diffraction limit in long range imaging. We emulate a camera array with a single camera attached to an X-Y translation stage. We show that an appropriate phase retrieval based reconstruction algorithm can be used to effectively recover the lost high resolution details from the multiple low resolution acquired images. We analyze the effects of noise, required degree of image overlap, and the effect of increasing synthetic aperture size on the reconstructed image quality. We show that coherent camera arrays have the potential to greatly improve imaging performance. Our simulations show resolution gains of 10x and more are achievable. Furthermore, experimental results from our proof-of-concept systems show resolution gains of 4x—7x for real scenes. Finally, we introduce and analyze in simulation a new strategy to capture macroscopic Fourier Ptychography images in a single snapshot using a camera array.
"Toward Long Distance, Sub-diffraction Imaging Using Coherent Camera Arrays"
J. Holloway, M.S. Asif, M.K. Sharma, N. Matsuda, R. Horstmeyer, O. Cossairt, and A. Veeraraghavan
IEEE Transactions on Computational Imaging, 2016.
Download code and data:
MATLAB code to reproduce results presented in the paper is available on GitHub. Simulation data and experimental data for the USAF target can be found in the repository. Experimental data for the Dasani label and fingerprint were too large to upload to GitHub and have been hosted separately. All three experimental datasets may be downloaded as a zip file [257 MB].
Using active illumination to overcome the diffraction limit in long range imaging.
Diffraction blur is the primary cause of resolution loss in long distance imaging. Consider this illustration of imaging a human-sized object 1 km away. (a) A conventional camera using passive illumination with a fixed aperture size of 12.5 mm induces a diffraction spot size of 50 mm on objects 1 km away, destroying relevant image features. (b) Using Fourier ptychography, an array of cameras using coherent illumination creates a synthetic aperture 10 times larger than (a) resulting in a diffraction spot size of 5 mm for scenes 1 km away. Phase retrieval algorithms recover the high resolution image.
Diffraction blur limits in long distance imaging.
(a) Diffraction blur spot size as a function of distance for common and exotic imaging systems. Consumer cameras (shaded in the pink region) are designed for close to medium range imaging but cannot resolve small features over great distances. Professional lenses (green region) offer improved resolution but are bulky and expensive. Identifying faces 1 kilometer away is possible only with the most advanced super telephoto lenses, which are extremely rare and cost upwards of $2 million US dollars. Obtaining even finer resolution (blue region) is beyond the capabilities of modern manufacturing; the only hope for obtaining such resolution is to use computational techniques such as Fourier ptychography. (b) Affordable camera systems (solid marks) are lightweight but cannot resolve fine details over great distances as well as professional lenses (striped marks) which are heavy and expensive. We propose using a camera array with affordable lenses and active lighting to achieve and surpass the capabilities of professional lenses. Note: These plots only account for diffraction blur and do not consider other factors limiting resolution.
Sampling the Fourier domain at the aperture plane
For a given high resolution target (a), the corresponding Fourier transform is formed at the aperture plane of the camera (b). The lens aperture acts as a bandpass filter, only allowing a subset of the Fourier information to pass to the sensor. A larger aperture may be synthesized by scanning the aperture over the Fourier domain and recording multiple images. (c) The full sampling mosaic acquired by scanning the aperture. The dynamic range has been compressed by scaling the image intensities on a log scale. (d) Larger detail images shown of four camera positions, including the center. Image intensities have been scaled linearly, note that only high frequency edge information is present in the three extreme aperture locations. Please zoom in to see details.
Block diagram of the image recovery algorithm
Constraints on the image domain magnitude and Fourier domain support are enforced in an alternating manner until convergence or a maximum iteration limit is met.
Recovering high frequency information using a Fourier ptychography
(a) We simulate imaging a 64x64 mm resolution target 50 meters away using sensor with a pixel pitch of 2 μm. The width of a bar in group 20 is 2.5 mm. (b) The target is observed using a lens with a focal length of 800 mm and an aperture of 18 mm. The aperture is scanned over a 21x21 grid (61% overlap) creating a synthetic aperture of $160$ mm. The output of phase retrieval is a high resolution Fourier representation of the target. The recovered image is shown in (d) and the recovered Fourier magnitude (log scale) is shown in (e). The plot in (c) shows the contrast of the groups in the intensity images of the recovered resolution chart (purple solid line) and the observed center image (blue dashed line). Whereas the central image can only resolve elements which have a width of 12 pixels before contrast drops below 20%, using Fourier ptychography we are able to recover features which are only 2 pixels wide. Detail images of groups 2, 5, and 6, for the ground truth, observed, and recovered images are shown in (f).
Effect of varying overlap between adjacent images
Holding the synthetic aperture size constant, we vary the amount of overlap between adjacent images. As the amount of overlap increases, reconstruction quality improves. When the amount of overlap is less than 50%, we are unable to faithfully recover the high resolution image, as shown in the RMSE plot in the top left. The contrast plots of the intensity images for four selected overlap amounts (0%, 41%, 50%, and 75\%) shows that insufficient overlap drastically reduces the resolution of the system. Recovered images show the increase in image reconstruction quality as the amount of overlap increases.
Varying synthetic aperture ratio (SAR)
For a fixed overlap of 61%, we vary the size of the synthetic aperture by adding cameras to array. As seen in the contrast plot of image intensities, as well as the reconstructed intensities below, the resolution of the recovered images increase as the SAR increases from 1 (the observed center image) to 11.8. We are able to recover group 2 with an SAR of 5.64, which requires 169 images.
Noise robustness of the coherent camera array
We repeat the varying SAR experiment with the Lena image (assuming the scene is 650 meters away), while varying the amount of added Gaussian noise. The signal-to-noise ratio (SNR) of the input images is set to be 10, 20, and 30 dB. As shown in the RMSE plot, we are able to recover the high resolution image even with noisy input images. The observed center images (SAR=1) are blurry and noisy, with a high RMS error. As the SAR increases, the resolution improves, the noise is suppressed, and RMSE decreases. A cropped portion of select images are shown to highlight the performance of our approach. Note: all other simulation experiments are conducted with an input SNR of 30 dB.
Simulation of imaging a fingerprint at 30 meters
We simulate image capture with an inexpensive 1200 mm lens, 75 mm aperture diameter (f/16), and a pixel pitch of 2 μm. (a) Using passive illumination the resulting diffraction blur (530 μm) removes details necessary to identify the fingerprint. (b) Imaging with a coherent camera array (61% overlap) that captures 81 images to create a synthetic aperture of 300 mm reduces diffraction blur to 130 μm, and leads to faithful recovery of minutiae. (c) Using a $150,000 lens with a 215 mm aperture diameter (f/5.6), the diffraction blur reduces to 190 μm, which is roughly comparable to our final reconstruction in (b). (d) Detail views of the three imaging systems. In this simulation diffraction is the only source of blur.
Simulation of capturing a face 1000 meters away
We simulate image capture with an inexpensive 1200 mm lens, 75 mm aperture diameter (f/16), and a pixel pitch of 2 μm. (a) Directly using the inexpensive f/16 lens in passive illumination results in a diffraction blur spot size of 17.8 mm on the face, obliterating detail necessary to recognize the subject. (b) Using Fourier ptychography (61% overlap, 81 images) to achieve a synthetic aperture of 300 mm the diffraction spot size is reduced to 4.4 mm. (c) Using the $150,000 215 mm (f/5.6) lens yields a diffraction spot size of 6.2 mm, 50% larger than the diffraction realized using Fourier ptychography. Detail views of the three systems are shown in (d). In this simulation diffraction is the only source of blur.
Overview of hardware configuration for data acquisition (side view)
From left to right: A helium neon laser passed through a spatial filter acts as the coherent illumination source. A focusing lens forms the Fourier transform of the transmissive object at the aperture plane of the camera's lens. The aperture acts as a bandpass filter of the Fourier transform and the signal undergoes an inverse Fourier transform as it is focused onto the camera's sensor. The camera (Point Grey Blackfly (BFLY-PGE-50A2M-CS) is mounted on a motorized 2D stage to capture overlapping images.
Experimental result: Resolving a fingerprint 4x beyond the diffraction limit
Using the experimental setup, we use a 75 mm focal length lens to acquire images of a fingerprint ~1.5 m away from the camera. A fingerprint was pressed on a glass slide and fingerprint powder was used to make the ridges opaque. (a) Stopping down the aperture to a diameter of 2.34 mm induces a diffraction spot size of 49 μm on the sensor (~20 pixels) and 990 μm on the object. (b) We record a grid of 17x17 images with an overlap of 81%, resulting in a SAR which is 4 times larger than the lens aperture. After running phase retrieval, we recover a high resolution magnitude image and the phase of the objects. (c) For comparison, we open the aperture to a diameter of 41 mm to reduce the diffraction blur to the size of a pixel. (d) Comparing zoomed in regions of the observed, recovered, and comparison images shows that our framework is able to recover details which are completely lost in a diffraction limited system.
Experimental result: Resolving 4x beyond the diffraction limit for a diffuse water bottle label
Using the same parameters as in the fingerprint capture we image a diffuse water bottle label ~1.5 m away from the camera. The diffuse nature of the water bottle label results in laser speckle. In the observed center image (a), diffraction blur and laser speckle render the text illegible. Using Fourier ptychography (b) we are able to reduce the effect of speckle and remove diffraction revealing the text. In the comparison image (c) the text is clearly legible. Detail views in (d) show the improvement in image resolution when using Fourier ptychography.
Experimental result: Recovering a USAF target with varying SAR
We capture a USAF resolution target 1.5 meters away from the camera. For this experiment, we use a slightly different lens and image sensor (focal length = 75 mm, pixel pitch = 3.75 μm). (a) The camera is stopped down to 2.5 mm which induces a 930 μm blur on the resolution chart, limiting resolution to 1.26 line pairs per millimeter (lp/mm). (b) We record a 23x23 grid of images with 72% overlap, resulting in a SAR of 7.16. Following phase retrieval, we are able to resolve features as small as 8.98 lp/mm, a 7.12x improvement in resolution. (c) We show the effect of varying the SAR on resolution. Using a subset of the captured images we vary the SAR from 2.12 up to 7.16. The gains in improvement closely track the SAR.
An illustration of ptychography with multiplexed illumination and a camera array
(a) Multiple coherent sources illuminate a distant scene. (Note: different colors are used to distinguish light sources; however, we assume each source operates in the same wavelength.) (b) Reflected light undergoes Fraunhofer diffraction forming spatially shifted Fourier transforms at the aperture plane of the camera array. (c) A single camera aperture (e.g. center camera) sums the intensities from each diffraction pattern (top), recording a single image (bottom).
Simulation of recovering a resolution target using multiplexed illumination patterns
We repeat the conditions of the simulation resolution target experiment using a 7x7 camera array. Each camera has an 800 mm lens with an aperture (front lens diameter) of 25 mm. Cameras are placed such that their apertures abut, though don't overlap, in the Fourier plane. Illumination sources provide 66% overlap among neighboring apertures in the Fourier plane. Nmux, in subcaptions denote the number of active illumination sources per image and T denotes the number of images captured with different illumination patterns. Recovered images show that increasing Nmux and T improves the quality of the reconstructed images.
This work was supported in part by NSF grants IIS-1116718, CCF-1117939, CCF-1527501, NSF CAREER grant IIS-1453192, ONR grant 1(GG010550)//N00014-14-1-0741, and a Northwestern University McCormick Catalyst grant.