跳到正文
Apple Machine Learning Research·· 2 天前AI 评分58

苹果发布 Depth Pro 2:兼顾锐利与时间一致性的单目深度估计

Depth Pro 2: Sharp and Temporally Consistent Metric Monocular Depth

AI 导读

苹果研究团队提出 Depth Pro 2,可从普通相机拍摄的图像和视频中以原生分辨率估计每个像素的米制深度。该方法针对以往单目深度估计中多帧方法偏向时间一致性、单帧方法偏向边界锐度的取舍,训练一个可同时处理图像与视频的统一网络,使深度图在物体边缘锐利且帧间稳定。团队同时发布了一组在度量精度、锐度、时间一致性和运行时间上构成新 Pareto 前沿的模型,并公开代码与权重。

正文 · 原文

AuthorsPeiyun Hu, Marko Mihajlovic, Lyndon Duong, Lahav Lipson, Wei Dong, Amaël Delaunoy, Stephan R. Richter, Vladlen Koltun

We present Depth Pro 2, an approach to estimating dense metric depth for images and video at full resolution. By analyzing prior methods for monocular depth estimation, we find that they commonly trade off temporal consistency and boundary sharpness: multi-frame methods favor consistency, single-frame methods favor sharpness. To overcome this trade-off, we train a single, scalable network that processes both images and video and predicts depth maps at native resolution. Depth Pro 2 produces depth maps that are metrically accurate, sharp, and temporally consistent all at once, outperforming prior work in all of these dimensions. We release a family of models that define a new Pareto frontier in terms of metric accuracy, sharpness, temporal consistency, and runtime.

We release code and weights here.

Depth Pro 2 estimates the depth of every pixel, in meters, from images taken with an ordinary camera. One network handles a single photo, a whole video, or a live camera feed, at the input’s full resolution. Its depth is sharp at object edges and steady from frame to frame.

Related readings and updates.

We present SHARP, an approach to photorealistic view synthesis from a single image. Given a single photograph, SHARP regresses the parameters of a 3D Gaussian representation of the depicted scene. This is done in less than a second on a standard GPU via a single feedforward pass through a neural network. The 3D Gaussian representation produced by SHARP can then be rendered in real time, yielding high-resolution photorealistic images for nearby…

Read more

We present a foundation model for zero-shot metric monocular depth estimation. Our model, Depth Pro, synthesizes high-resolution depth maps with unparalleled sharpness and high-frequency details. The predictions are metric, with absolute scale, without relying on the availability of metadata such as camera intrinsics. And the model is fast, producing a 2.25-megapixel depth map in 0.3 seconds on a standard GPU. These characteristics are enabled by…

Read more

来源:Apple Machine Learning Research · machinelearning.apple.com