Compact single-shot metalens depth sensors inspired by eyes of jumping spiders

We model the image $I\left(x,y\right)$ formed on a photosensor as the convolution of the camera point spread function (PSF) with the magnified, all-in-focus object pattern as it would be observed with a pinhole camera. The camera PSF is the image captured on the photosensor when the object is a point light source. The width of the PSF depends on the optics and the distance Z between the object and the lens. For an ideal thin-lens camera, depicted in Fig. 2A, the PSF width $\sigma$ is related to object distance $Z$ by the thin-lens equation:where $Z_s$ is the distance between the lens and the photosensor, $\mathrm{\Sigma }$ is the radius of the entrance pupil, and $Z_f$ is the in-focus distance—i.e., the distance for which the PSF width $\sigma$ is equal to zero. (SI Appendix, section 1.1) On the right side of Eq. 1, all quantities but the object distance Z are known quantities determined by the optical system. Thus, for a calibrated camera, determining the PSF width $\sigma$ is equivalent to measuring object distance $Z$.

The PSF width $\sigma$ determines the amount of image blur. An object appears sharp when its distance $Z$ is equal to the in-focus distance $Z_f$ because then the PSF width $\sigma$ is zero (under ray optics approximation). Conversely, when the object distance $Z$ deviates from $Z_f$, the PSF width, $\sigma$, is nonzero, and the image is blurry. Recovering the PSF width (and thus depth) from a single blurry image is ill-posed without prior information about the underlying object pattern. However, when a second image of the same scene is captured with a different amount of blur, the width $\sigma$ can be determined directly from the contrast change between the 2 images. One way to understand this is to assume that the PSFs can be approximated as Gaussian functions that depend on the PSF width $\sigma$:where $\left(x,y\right)$ is the pixel position on the photosensor. Gaussian functions have the property that partial derivatives with respect to width $\sigma$ and location $\left(x,y\right)$ satisfyand because the defocused image $I\left(x,y\right)$ is the convolution of the PSF and the all-in-focus object pattern (which does not depend on $\sigma$), the same relationship between derivatives applies to the captured image:Eq. 4 indicates that $\sigma$ (and thus depth Z through Eq. 1) can be determined directly from the spatial Laplacian of the image ${\nabla }^{2}I\left(x,y\right)$ and the differential change of intensity with respect to varying PSF width $\frac{\partial I\left(x,y\right)}{\partial \sigma }$ (18, 19). The latter can be estimated via a finite difference, i.e., $\frac{\partial I\left(x,y\right)}{\partial \sigma }\approx \frac{\delta I\left(x,y\right)}{\delta \sigma }$, where $\delta I\left(x,y\right)$ is the change of image intensity induced by a small, known variation of the PSF width ($\delta \sigma$). According to Eq. 1, since in general no control can be made over the object distance $Z$, the only way to change the PSF width $\sigma$ when shooting an object is to vary the parameters of the optical system, such as the sensor distance $Z_s$ or the in-focus distance $Z_f$.

A jumping spider’s principal eye can use its transparent layered retinae to simultaneously measure the (minimally) 2 images that are required to compute the finite difference $\delta I\left(x,y\right)$ (Fig. 1A) because these retinae effectively capture images with different sensor distances $Z_s$. In contrast, we design a metalens that generates 2 images ($I_{+}\left(x,y\right)$, $I_{-}\left(x,y\right)$) side by side (Fig. 1B), with the images being equivalent to images that are captured with different in-focus distances ($Z_{f_{+}}$, $Z_{f_{-}}$) through the same pupil. We design the in-focus distances so that the difference in blur between the images, $\delta \sigma ={\sigma }_{+}-{\sigma }_{-}=\mathrm{\Sigma }{Z}_s\left(\frac{1}{Z_{f_{+}}}-\frac{1}{Z_{f_{-}}}\right)$, is small and approximately differential. From these 2 images (Fig. 2 C, Left), we compute the per-pixel difference $\delta I\left(x,y\right)=I_{+}\left(x,y\right)-I_{-}\left(x,y\right)$ and the image Laplacian ${\nabla }^{2}I\left(x,y\right)$. The latter is obtained by convolving the averaged image $I\left(x,y\right)=\frac{1}{2}\left(I_{+}\left(x,y\right)+I_{-}\left(x,y\right)\right)$ with a Laplacian filter, denoted ${\nabla }^2\left(x,y\right)$. To reduce the effects of sensor noise and optical nonidealities like vignetting, $\delta I\left(x,y\right)$ and ${\nabla }^{2}I\left(x,y\right)$ are spatially convolved with a purposefully designed linear filter $F\left(x,y\right)$. To the lowest order, the filter $F\left(x,y\right)$ is similar to a Gaussian filter that averages over neighboring pixels (SI Appendix, sections 1.3 and 2.2). The filtered results $F\left(x,y\right)*\delta I\left(x,y\right)$ and $F\left(x,y\right)*{\nabla }^{2}I\left(x,y\right)$ are shown in Fig. 2 C, Center. Finally, we combine Eqs. 1 and 4 to calculate the depth $Z$ at each pixel $\left(x,y\right)$:with $\alpha =\frac{1}{2}\left(\frac{1}{Z_{f_{+}}}+\frac{1}{Z_{f_{-}}}\right)$ and $\beta =-{\left(\mathrm{\Sigma }{Z}_s\delta \sigma \right)}^{-1}$ being constants that are determined by the optics. The correctness of Eq. 5 follows from the fact that Eq. 4 still holds when values $\delta I\left(x,y\right)$ and ${\nabla }^{2}I\left(x,y\right)$ are replaced by their filtered versions $F\left(x,y\right)*\delta I\left(x,y\right)$ and $F\left(x,y\right)*{\nabla }^{2}I\left(x,y\right)$.

In practice, even with filtering, random noise in the captured images ($I_{+}\left(x,y\right)$, $I_{-}\left(x,y\right)$) results in errors in the measured depth $Z\left(x,y\right)$. The error can be quantified in terms of the SD of the measured depth at each pixel, which can be approximated by the measurable quantitywith constants ${\gamma }_1,{\gamma }_2,{\gamma }_3$ that are determined by the optics (SI Appendix, section 1.2). This measurable quantity $s_Z\left(x,y\right)$ can serve as an indicator of the reliability of the measured depth $Z\left(x,y\right)$ at each pixel $\left(x,y\right)$. For convenience, we normalize the values of $s_Z\left(x,y\right)$ to the range $\left(\mathrm{0,1}\right)$ and define this normalized value as the confidence $C\left(x,y\right)$. A higher confidence value $C$ at pixel location $\left(x,y\right)$ indicates a smaller value of $s_Z$ and a more accurate depth measurement $Z$ (SI Appendix, section 1.2). Physically, the confidence $C\left(x,y\right)$ characterizes the expected accuracy of the measurement at each pixel $\left(x,y\right)$: A larger confidence value $C\left(x,y\right)$ at a pixel indicates a statistically smaller error in the depth measurement.

Since Eq. 5 calculates depth using simple, local calculations, it fails in regions of the images that have uniform intensity and thus no measurable contrast for $\delta I\left(x,y\right)$ and ${\nabla }^{2}I\left(x,y\right)$. To automatically identify the locations of these failures, we use the confidence score $C\left(x,y\right)$ as a criterion, and we report depth only at pixels $\left(x,y\right)$ whose confidence is above a certain threshold. The choice of confidence threshold affects the depth resolution, which we define as the smallest depth difference that can be resolved within a certain confidence range. In this paper, with a confidence threshold of 0.5, we achieve a depth resolution of about $5\%$ of the object distance over the distance range $\left[0.3m,0.4m\right]$ (see Fig. 4B).

The complete sequence of calculations for the depth map and confidence map is depicted in Fig. 2C. For visualization, the depth map is thresholded by confidence to show only the depth at pixels where the latter is greater than 0.5.

The metalens is designed to incorporate phase profiles of 2 off-axis lenses with different in-focus distances on a shared aperture. For each off-axis lens, the required phase profile is an offset convex shape determined by in-focus distances $\left(Z_{f_{+}},Z_{f_{-}}\right)$, sensor distance $Z_s$, and transverse displacement of the image center $\pm D$ (Fig. 2B):Here, $\left(x,y\right)$ indicates location on the metalens. The overall phase profile is achieved by spatially interleaving the 2, ${\varphi }_{+}\left(x,y\right)$ and ${\varphi }_{-}\left(x,y\right)$, on the metalens at a subwavelength scale. The design specifications are in SI Appendix, section 2.1.

The required phase profile can be wrapped to $\left[\mathrm{0,2}\pi \right]$ (i.e., modulo $2\pi$) without changing its functionality. Therefore, the key requirement for precise wavefront shaping is to control locally the phase between 0 and $2\pi$. Here, we use titanium dioxide (${\text{TiO}}_2$) nanopillars as the building blocks for local phase control. Physically, the nanopillars function as truncated waveguides and impart a phase shift to the transmitted light. By varying the pillar width, one can tune their effective refractive index, and thus the phase shift. Fig. 3A shows that by changing the pillar width ($W$) from 90 to 190 nm, one can achieve 0 to $2\pi$ phase coverage, while maintaining a high transmission efficiency. The nanopillars have a uniform height ($H$) of 600 nm and can be fabricated with a single-step lithography. The center-to-center distance ($U$) between the neighboring nanopillars is 230 nm, smaller than half the operating wavelength. This allows us to spatially interleave different phase profiles at a subwavelength scale, which is essential to eliminating unwanted higher-order diffractions.

The metalens is fabricated with a technique demonstrated by Devlin et al. (34). Fig. 3 B–D show the scanning electron microscope (SEM) images of a fabricated sample. The location of the imaged area on the metalens is in SI Appendix, Fig. S6. The phase wrapping introduces a discontinuity at locations where the phase profile equals an integer number of $2\pi$—i.e., the “zone” boundaries. This corresponds to an abrupt change of nanopillar arrangement, as shown in Fig. 3 B–D. The phase profiles of 2 off-axis lenses have different zone spacing and orientation, corresponding to the 2 nearly vertical boundaries and the diagonal boundary, respectively (Fig. 3B and SI Appendix, Fig. S6). The spatial multiplexing scheme is illustrated explicitly in Fig. 3C, with nanopillars belonging to different focusing profiles highlighted in different colors.

We built a prototype metalens depth sensor by coupling the metalens with off-the-shelf components. The sensor’s current size, including mechanical components such as optical mounts, is 4 $\times$ 4 $\times$ 10 cm, but since the metalens is only 3 mm in diameter, the overall size of the assembled sensor could be reduced substantially with a purpose-built photosensor and housing. We paired a 10-nm bandpass filter with the metalens, which is designed for monochromatic operation at 532 nm. A rectangular aperture was placed in front of the metalens to limit the field of view and prevent the 2 images from overlapping. The blur change between the 2 images can be seen in Fig. 4A, which shows PSFs for each of the 2 images [$I_{+}\left(x,y\right)$, $I_{-}\left(x,y\right)$] that were measured by using a green light-emitting diode (LED) mated to a 10-$\mathrm{\mu }$m-diameter pinhole and placed at different depths Z along the optical axis. The PSFs are more disc-like than Gaussian, and they are asymmetric due to off-axis chromatic aberration. (SI Appendix, Fig. S8 shows that this asymmetry disappears under monochromatic laser illumination.)

To suppress the effects of noise and imaging artifacts in images ($I_{+}$, $I_{-}$), and to increase the number of high-confidence pixels in the output maps, we computed 9 separate depth and confidence maps using 9 instances of Eqs. 5 and 6 that have distinct and complementary spatial filters $F_i$, and then we fused these 9 “channels” into 1. We also designed a calibration procedure that tuned the parameters simultaneously, using back-propagation and gradient descent (SI Appendix, section 3). In addition to being user-friendly, this end-to-end calibration has the effect of adapting the computation to the shapes of the metalens PSFs, which differ substantially from Gaussians.

To analyze the depth accuracy, we measured the depths of test objects at a series of known distances and compared them with the true object distances. The test objects were textured planes oriented parallel to the lens plane. At each object distance, the mean deviation of depth, ${\text{mean}}_{x,y}|Z\left(x,y\right)-{\text{mean}}_{x,y}Z\left(x,y\right)|$, was computed by using pixels $\left(x,y\right)$ that have confidence values greater than a threshold. Fig. 4B shows the measured depth for different confidence thresholds as a function of object distance. For a confidence threshold of 0.5, the measured depth was accurate to within a mean deviation below or around $5\%$ of the true depth, over a range of true object distances between 0.3 and 0.4 m. Beyond this range, the measured depth defaulted to the extreme depth value that the system can predict, as indicated by plateaus on the left and right ends. This indicated that the 2 images were so blurry that there was an insufficient contrast difference between them.

Fig. 5 shows depth maps for a variety of scenes. Because it uses a single shot, the metalens depth sensor can measure objects that move, such as the fruit flies and water stream in Fig. 5 A and B. It can also measure the depth of translucent entities, such as the candle flames of Fig. 5C, that cannot typically be measured by using active sensors like Lidar and time-of-flight. Fig. 5D shows a slanted plane with printed text, where the blur change between the 2 images is particularly apparent. In general, the sensor reports a larger number of depth measurements near regions with edges and texture, whereas regions with uniform intensity and low contrast are typically discarded as having low confidence values. Note that the blur differences between $I_{+}$ and $I_{-}$ are visually apparent in Fig. 5 A, B, and D, but that the system still succeeds in Fig. 5C, where the differences in blur are hard to discern.

For scenes other than Fig. 5C, we used green LED light sources, and the overall transmission efficiency of the metalens plus the bandpass filter was around $15\%$. For sunlight illumination, the bandpass filter transmitted around $4\%$ of the visible light of the solar spectrum. The absolute irradiance that supports the function of the sensor varied based on the sensitivity of the photosensor that was used and can be estimated from specifications including absolute sensitivity threshold, dynamic range, etc. For our experimental setup, the irradiance at the aperture was estimated to be between 0.3 and 0.5 W/m² within the working bandwidth to support the function of the sensor.

The sensor generates depth and confidence maps of 400$\times$400 pixels at more than 100 frames per second using a combined central processing unit and graphics processing unit (Intel i5 8500k and NVIDIA TITAN V). It could be accelerated substantially by optimizing the code and/or the hardware because the calculations are spatially localized and few in number. Producing the depth and confidence values at each output pixel required 637 FLOPs and involved only the 25 $\times$ 25 surrounding pixels. For context, an efficient implementation of a binocular stereo algorithm requires about 7,000 FLOPs per output pixel (37), and a system-on-chip implementation of the well-known Lucas–Kanade optical flow algorithm (with spatial dependence similar to that of our sensor) requires over 2,500 FLOPs per pixel (38).

The metalens depth sensor inherits some of the limitations that exist in the vision system of jumping spiders, such as a limited spectral bandwidth and a limited field of view. However, these limits are not fundamental, and they can be alleviated by more sophisticated metalens designs. The spectral bandwidth can be expanded by using achromatic metalenses (35, 39, 40), which also improve light efficiency. The field of view can be improved, for example, by using metalens nanopillars that are sensitive to polarization to induce 2 differently focused images that are superimposed on the sensor plane with orthogonal polarizations (36) and then transducing the 2 images with a spatially multiplexed, polarization-sensitive sensor array. This would effectively trade spatial resolution and light efficiency for an increase in field of view.

The proposed computational algorithm produces a dense field of depth estimates that are each associated with a confidence value. The confidence is essential for the users of the depth sensor to remove unreliable predictions. It also uses a multiscale filtering approach to handle image textures at different spatial frequency and takes advantage of the confidence to merge all different spatial scales together, compared to previous methods (20, 22) that only use filters at a single, predetermined spatial scale. The proposed algorithm does not incorporate inference-based methods such as Markov random fields (MRFs) or conditional random fields (CRFs) that could exploit longer-range coherence between depth values across the field of view. Instead, the depth and confidence estimations at each pixel are only based on information of its spatial neighborhood. The advantages of this design choice are flexibility and generality. For tasks that require high speeds, the output can be used as-is, with simple thresholding of confidence values. For tasks that require higher accuracy and fewer holes in the depth map, the current output can be fed into an MRF/CRF (or any other spatial regularizer) that is appropriate for that task. Moreover, because the pipeline is end-to-end differentiable, its parameters can be fine-tuned in conjunction with MRF/CRF parameters to optimize performance on the specific task.

By combining cutting-edge nanotechnology and computer vision algorithms, this work introduces a passive snapshot depth sensor that mimics some of the capabilities of a jumping spider. The sensor’s small volume, weight, and computation (i.e., power) bring depth-sensing capabilities closer to being feasible on insect-scale platforms, such as microrobots, ingestible devices, far-flung sensor networks, and small wearable devices. Combinations of nanophotonics and efficient computation that are different from the ones in this paper might lead to other forms of compact visual sensors, and this is an area that remains relatively unexplored (41).

This project was supported by Air Force Office of Scientific Research Multidisciplinary University Research Initiative Grants FA9550-14-1-0389 and FA9550-16-1-0156; and NSF Award IIS-1718012. Y.-W.H. and C.-W.Q. are supported by the National Research Foundation, Prime Minister’s Office, Singapore under Competitive Research Program Award NRF-CRP15-2015-03. E.A. is supported by NSF Graduate Research Fellowship DGE1144152. Metalens fabrication was performed at Harvard’s Center for Nanoscale Systems, supported by NSF Grant 1541959. Q.G. and Z.S. thank Mohammadreza Khorasaninejad for helpful discussions. Z.S. thanks Zhehao Dai for helpful comments and discussions.