Computational multifocal microscopy (CMFM) setup (a) and 3D reconstruction pipeline (d-f).


Project Description

Despite recent advances, high performance single-shot 3D microscopy remains an elusive task. By introducing designed diffractive optical elements (DOEs), one is capable of converting a microscope into a 3D “kaleidoscope”, in which case the snapshot image consists of an array of tiles and each tile focuses on different depths. However, the acquired multifocal microscopic (MFM) image suffers from multiple sources of degradation, which prevents MFM from further applications. We propose a unifying computational framework which simplifies the imaging system and achieves 3D reconstruction via computation. Our optical configuration omits optical elements for correcting chromatic aberrations and redesigns the multifocal grating to enlarge the tracking area. Our proposed setup features only one single grating in addition to a regular microscope. The aberration correction, along with Poisson and background denoising, are incorporated in our deconvolution-based fully-automated algorithm, which requires no empirical parameter-tuning. In experiments, we achieve the spatial resolutions of 0.35um (lateral) and 0.5um (axial), which are comparable to the resolution that can be achieved with confocal deconvolution microscopy. We demonstrate a 3D video of moving bacteria recorded at 25 frames per second using our proposed computational multifocal microscopy technique.



"Computational multifocal microscopy"
Kuan He, Zihao Wang, Xiang Huang, Xiaolei Wang, Seunghwan Yoo, Pablo Ruiz, Itay Gdor, Alan Selewa, Nicola J. Ferrier, Norbert Scherer, Mark Hereld, Aggelos K. Katsaggelos, and Oliver Cossairt
Biomedical Optics Express 9, (2018)


3D PSF of CMFM system.

A z-stack 3D PSFs of CMFM, measured from a 200nm fluorescent bead. Top row: CMFM lateral PSFs imaged under five different axial positions (columns). each PSF consists of one focused image (outlined by a green box) and eight out-of-focus version images of the bead. xy and xz PSFs (second and third rows) and corresponding OTFs (bottom two rows) of five differently focused tiles (columns). The focal shift property of CMFM can be observed from xz PSFs (third row), verifying that CMFM is capable of capturing a focal stack instantaneously. Although the central tile's PSF CA-free (first column), the off-axis tiles' PSFs suffer from directional CA (second to last columns) due to geometry. The lateral spatial frequencies that are lost by CA are shown in xy OTFs (fourth row).}

Chromatic aberration (CA) blur is computationally compensated.

CA blur vs defocus blur. An in-focus tile with CA blur (red) and a defocus blur (blue) are highlighted in (a), whose PSFs and OTFs are shown in (b). (c) plots a comparison of linecuts indicated by blue and magenta lines in (b). For reference, a linecut in CA-free central tile's OTF (shown in first column and fourth row of Fig. 1) is also plotted (red). A reconstruction comparison is shown in the right panel. (d) Object image. (e) Observation image (for visualization purpose, each tile image is cropped). (f) Reconstruction using only in-focus PSF. (g) Reconstruction using all the PSFs.}

Enlarging lateral tracking space

(a) Conventional MFM design uses a large tile spacing. (b) We propose to use a smaller tile spacing, so as to achieve a small lateral FOV that can be tracked over a large area for MFM tracking applications.

Simulation shows enlarged lateral tracking space of new MFM design.

Simulations showing the capability of the proposed MFM of achieving larger lateral tracking space than conventional MFM does. (a) The synthetic 3D ground truth of an ellipsoid (left) and its xz slice (right). The center of the ellipsoid is 35.8um away from the center of the detector in x direction. MFM measurements (e-f) and corresponding reconstructions (b-c) by different design methods. (d) 1D axial profile comparison between ground truth (red) and reconstructions (black and blue). It is clear that the ellipsoid is reconstructed poorly from conventional MFM method while our design provides a good reconstruction. Signal loss as a function of lateral position of the tracked object for conventional (g) and our designed MFM (h). Similar to vignetting effect, the signal falls off when approaching the edges. Our proposed design alleviates peripheral signal loss and achieves an enlarged lateral tracking area.

Two experimental snapshot MFM raw images

(a) Experiment 1: snapshot captured 2D MFM image of multiple static periplasms by using an MFG with 9 focal planes under exposure time of 0.5s. (b) Experiment 2: a frame from an MFM video of a moving bacterium captured at 25 fps by using an MFG with 25 focal planes. The raw MFM video is shown in Visualization 1.

CMFM 3D reconstruction

Proposed computational 3D reconstruction of CMFM image in comparison with confocal deconvolution results. (a) Confocal raw data and (b) its deconvolution results. (c) CMFM raw data and (d) its computational reconstruction results. In (a), confocal scan is taken with a dual spinning disk confocal microscope (Model: CSU-W1) with the total acquisition time of 20s, while in (c), the CMFM raw data is captured in a single exposure of 0.5s. The lateral resolution of the proposed computational reconstruction is about 0.35um (second row of d) and the axial resolution is about 0.5um (fourth row of d), which are comparable with those achievable with confocal deconvolution microscopy (second and fourth rows of b).

CMFM 3D video reconstruction

Experimental 3D reconstructions of a movable bacterium. A raw MFM video (shown in Visualization 1) was captured at 25fps as the bacterial moves in 3D space. The computational 3D reconstruction was performed for each video frame. Five out of sixty frames reconstruction is shown in (a-e). (f) 3D trajectory of the bacterium by computing and tracking its center of mass for each frame reconstruction. The colorbar indicate the frame index over time. The complete 3D video reconstruction from the first frame to the last frame is shown in Visualization 2.


Biological Systems Science Division, Office of Biological and Environmental Research, Office of Science, U.S. Dept. of Energy, under Contract DE-AC02-06CH11357; NSF CAREER grant IIS-1453192; ONR award N00014-15-1-2735.


    Leave a Reply