A focal surface display augments a conventional head-mounted display (HMD) with a spatially programmable focusing element placed between the eyepiece and the underlying color display. This design depicts sharp imagery and near-correct retinal blur, across an extended depth of focus, by shaping one or more virtual images to conform to the scene geometry. Our rendering algorithm uses an input focal stack and depth map to produce a set of phase
functions (displayed on a phase SLM) and RGB images (displayed on an OLED). Shown from left to right are: a prototype using a liquid crystal on silicon (LCOS) phase modulator, a target scene, the optimized focal surfaces and color images displayed across three frames, and the resulting focal stack, sampled at a near and far focus of 4.0 and 0.0 diopters, respectively.

Focal surface displays generalize the concept of manipulating the optical focus of each pixel on an HMD. Configurations (d,e) augment a fixed-focus HMD (a) with a programmable phase modulator placed between the eyepiece and display. (b) Varifocal HMDs use a globally addressed tunable lens. (c) Multifocal displays may use a high-speed tunable lens and display to create multiple focal planes. (f) In contrast, certain light field HMD concepts fall at the other end of this spectrum, using a finely structured phase modulator (a microlens array) placed near the display. (d) In this paper, we consider designs
existing between these extremes in which a phase modulator locally adjusts the focus to follow the virtual geometry, generalizing varifocal and multifocal concepts. (e) Similar to multifocal displays, multiple focal surfaces can be synthesized with high-speed phase modulators and displays.

Accommodation-supporting displays are assessed relative to optical and perceptual criteria. †Resolution is listed according to theoretical upper bounds (e.g., diffraction limits). ‡Field of view, eye box dimensions, and image quality depend on implementation choices: listed values correspond to the performance of prototypes in the cited publications, being indicative of current display technology limitations. Note that “moderate” resolution, field of view (FOV), eye box width, and depth of focus (DOF) are defined, respectively, as: 10–20 cycles per degree (cpd), 40–80 degrees, 5–10 mm, and 1–3 diopters. Excursions above or below these ranges are shaded green or red, respectively. Form factors are not compared, as most concepts are currently embodied by early stage prototypes not optimized for size or weight.

A focal surface display is created by placing a phase-modulation
element between an eyepiece and a display screen. This phase element and the eyepiece work in concert as a spatially programmable compound lens, varying the apparent virtual image distance across the viewer’s field of view.

A focal surface decomposition is presented above for a simple scene, containing: a background fronto-parallel plane at 1.0 diopters, a foreground fronto-parallel plane at 4.0 diopters, and a slanted plane spanning 2.0 to 4.0 diopters. (a) A single image from the target focal stack. (b) The target depth map. (c) A two-surface decomposition is compared to the target depth map for a profile taken along the middle row of the target imagery. (d,e) The color
images associated with each focal surface are shown, using the linear blending method of Akeley et al. [2004]. Note that this decomposition produces clearly delineated “foreground” and “background” components. (f,g) The color images associated with each focal surface, using the optimized blending algorithm
presented in Section 3.3. Note that, similar to Narain et al. [2015], the resulting color images present high spatial frequencies closer to the focal surface near where they occur in the target scene. We emphasize that, as with other multifocal displays reviewed in Section 2, time-multiplexed focal surface display reduces brightness, due to color image components being presented for a briefer duration than that occurring with a fixed-focus display mode.

Focal surface displays achieve lower depth map approximation errors, using less time multiplexing, than prior multifocal methods. As a result, such displays can support higher resolution image content (see Section 3.2). The upper row visualizes the optimized focal surfaces ranging from 0.0 to 5.0 diopters, abbreviated “D”. The lower row depicts the resulting depth map approximation errors in diopters. For a fixed focus design, the virtual image is positioned at
0.5 D. Following Narain et al. [2015], the fixed multifocal display employs four planes evenly spaced from 0.2 D to 2.0 D. The adaptive multifocal display and the focal surface display are optimized using k-means clustering, following Wu et al. [2016], and the methods in Sections 3.1 and 3.2 to position planes across a 5.0 D span, respectively. Focal surface displays show significantly fewer depth errors, with errors decreasing as more surfaces are used.

Focal surface displays represent natural scene depths with few image components. Box plots compare the depth map errors using
the denoted methods with the Middlebury 2014 dataset [Scharstein et al. 2014]. The bottom and top of the whiskers indicate the 5th and 95th percentiles, respectively. The bottom, middle, and top of the boxes represent the 1st quartile, the median, and the 3rd quartile, respectively. Focal surface displays produce fewer depth errors, especially when fewer planes are used.

We repeat the assessment with the database of rendered scenes described in Section 3.2. Note that the trends are repeated, but, due to the larger depth ranges in this database, additional virtual image surfaces are required with prior fixed and adaptive multifocal displays

Focal surface displays depict near-correct retinal blur with fewer virtual image surfaces than prior multifocal architectures. Following Figures 5–7, focal surface displays produce virtual images that more closely align with the scene geometry. As a result, sharply focused imagery can be obtained throughout the scene, reducing focusing errors occurring with prior fixed and adaptive multifocal displays. In this figure, we quantitatively assess the focal stack reproduction
error following the method of Narain et al. [2015]: the lower row depicts the maximum per-pixel probability of detecting a difference between the target and reconstructed focal stacks, as quantified using the HDR-VDP-2 metric [Mantiuk et al. 2011b]. The corresponding quality predictor of the mean opinion score
(MOS) is listed along the bottom. Note that focal surface displays achieve similar fidelity as prior adaptive multifocal displays, although with fewer virtual image surfaces.

The accommodation range of a focal surface display depends critically on the SLM placement. Here we denote, via the labeled plot contours, the virtual image distance zv achieved with an SLM, when used to represent a lens of focal length fp and positioned a distance zp from the eyepiece. Red lines indicate focal lengths beyond the dynamic range of the SLM. Note that these numbers correspond with the prototype described in Section 5.1.

Our binocular focal surface display prototype incorporates commodity optical and mechanical components, as well as 3D-printed support brackets. (a) The prototype is mounted to an optical breadboard to support the comparatively large LCOS driver electronics. A chin rest is used to position the viewer within the eye box. (b) A cutaway of of the prototype exposes the arrangement of the optical components.

Our prototype focal surface display achieves high resolution with near-correct retinal blur. Photographs of the prototype are shown in the first three columns, as taken by focusing the camera at the indicated distances. The last two columns depict the corresponding optimization outputs, including the phase functions and the color images. Note that optimized blending is applied with three time-multiplexed focal surfaces. The phase functions are wrapped
assuming a wavelength of 532 nm. Note that the results for the lower scene employ linear blending, following Akeley et al. [2004]. See the supplementary video for the full focal stack results.

To minimize time multiplexing, focal surface displays should operate
in a field simultaneous color mode. Following Section 6.1.2, artifacts due to axial chromatic aberration (ACA) may appear in this mode. (a) A target focal stack image. (b,c) Simulations comparing field simultaneous and field sequential modes, using the geometric optics model from Section 3. (d,e) Corresponding experimental results. Note that the contrast of experimental results differs from simulations due to stray light and misalignments that cannot be predicted without more accurate wave optics modeling and calibration, respectively.

The measured modulation transfer function (MTF) of our prototype
confirms high resolution is achieved. Following Section 5.3, the MTF was measured as the system varies focus from 0.0 to 4.0 diopters. Contrast loss is expected as the SLM synthesizes shorter focal lengths, due to the increased stray light from phase quantization and phase resets (see Section 5.3).

The measured axial chromatic aberration (ACA) of our prototype is
less than that of the typical human eye [Fernandez et al. 2013], confirming that focal cues are correctly rendered with field simultaneous color presentation, in spite of polychromatic illumination. †The SLM optical power was optimized, following Equation 5, for λ = 532 nm. ‡ACA is reported as the apparent optical distance in diopters, measured relative to the green channel. Focal distances are measured using a varifocal camera and a depth from-focus metric (i.e., maximizing contrast for a high-frequency pattern).