Figure above: Depth with ToF camera (a) and Depth with compressive reconstruction (b-d).

Project Description

Three-dimensional imaging using Time-of-flight (ToF) sensors is rapidly gaining widespread adoption in many applications due to their cost effectiveness, simplicity, and compact size. However, the current generation of ToF cameras suffers from low spatial resolution due to physical fabrication limitations. In this paper, we propose CS-ToF, an imaging architecture to achieve high spatial resolution ToF imaging via optical multiplexing and compressive sensing. Our approach is based on the observation that, while depth is non-linearly related to ToF pixel measurements, a phasor representation of captured images results in a linear image formation model. We utilize this property to develop a CS-based technique that is used to recover high resolution 3D images. Based on the proposed architecture, we developed a prototype 1-megapixel compressive ToF camera that achieves as much as 4 times improvement in spatial resolution and 3 times improvement for natural scenes. We believe that our proposed CS-ToF architecture provides a simple and low-cost solution to improve the spatial resolution of ToF and related sensors.


"CS-ToF: High-resolution compressive time-of-flight imaging"
Fengqiang Li, Huaijin Chen, Adithya Pediredla, Chiakai Yeh, Kuan He, Ashok Veeraraghavan, and Oliver Cossairt
Optics Express, 25(25) 31096-31110, 2017



CS-ToF architecture

The light from the laser diode hits the object and is reflected and imaged on the DMD. Then, the DMD-modulated image is re-imaged at the ToF sensor plane via a relay lens. The laser diode, DMD, and ToF camera are controlled and synchronized by a computer.

ToF depth imaging (assume single depth)

The computer sends out two signals: m(t) to control the laser diode and r(t) as reference to ToF sensor. The reflection from object is collected by ToF pixels, and then correlates with the reference signal to generate the camera’s output.

CS-ToF prototype

CS-ToF prototype system with components of the system are highlighted.

System Calibration

Calibration is performed by displaying an array of impulses on the DMD and measuring the sensor response for each individual impulse in the array. The response is then placed in the corresponding location in C. We traverse every DMD-sensor
pixel pair to complete the matrix C.

Pixel scanning of a resolution target

(a) shows original low-resolution ToF measurement of the resolution chart target. (b) shows the pixel-wise scanning for the resolution
target. Color boxes mark corresponding insets.

Real-world experiment setups

(a) Conceptual diagram of the resolution target
experiment. (b) Photo of the resolution targets used. (c) Conceptual diagram of the natural
scene experiment. (d) Photo of the natural scene.

Intensity reconstruction of resolution charts

(a), (b), (c), and (d) show the original
LR ToF intensity image, HR CS-ToF reconstruction results with no compression, 0:6 and
0:25 compression ratios, respectively. Fine patterns on resolution chart and the center of
Siemens Star are shown in the insets. Ground truth intensity of the insets, taken with a
12-MP camera, are displayed on the left.

Phase reconstruction for a natural scene

(a), (b), (c), and (d) show LR ToF phase
image and HR CS-ToF reconstruction phase images using no compression, 0:6 and 0:25
compression ratios, respectively. Colorbars show the depth information with unit of meter. A portion of the far resolution chart and the white board behind is also shown in the insets (a1-d1) with their corresponding photograph (marked with green box) in the left. Red arrows point out the boundary of two planes at different depths in (a1) and the corresponding photograph. Leaves and their branches on ”toy tree" are shown in the insets (a2-d2) with their corresponding photograph (marked with red box) in the left. Close-up images of (a2-d2) are further shown in (a3-d3).

Intensity reconstruction for a natural scene

(a), (b), (c), and (d) show LR ToF
intensity image and HR CS-ToF reconstruction intensity image using no compression, 0:6
and 0:25 compression ratios, respectively. Fine patterns on the toy tree and the metal star
are shown in the insets ((a1-d1, a2-d2)) with their corresponding photographs on the left
(marked with the green box, and the red box). Note the screw on the metal star (marked with
the red dashed circle) and the tip of the metal star (marked with the red arrow).

Scene projected on DMD plane with white field illumination

(a). The scene on DMD with ToF camera placed at the back focal plane of the relay lens. (c). The same scene on DMD with ToF camera slightly defocused. Color boxes represent insets in (a) and (c).

The quantification of depth accuracy for CS-ToF

(a). The photograph of the 3D
scene for the simulation experiments. (b). The ground truth depth for the 3D scene. (c). The
bicubic interpolation of LR ToF measurement depth with 25 dB Gaussian noise added in
the system. (d), (e) show the HR CS-ToF depth iamges with 0.6 and 0.2 compression ratios,
respectively. (25 dB Gaussian noise is added in the measurements). (f) shows the depth
values along the red lines in (b-e) with 30dB SNR Gaussian noise in the measurements. (g)
shows the depth values on the same pixels with (f) with 25dB SNR Gaussian noise added. (h) shows the depth values on the same pixels with (f) with 20dB SNR Gaussian noise added.


National Science Foundation (NSF) (CAREER IIS-1453192, CAREER IIS-1652633, and CCF-1527501);

Office of Naval Research (ONR) (N00014-15-1-2735);

Defense Advanced Research Projects Agency (DARPA) (REVEAL HR0011-16-C-0028);

Texas Instruments (TI) Graduate Student Fellowship.

    Leave a Reply