06f1fd69a6
4.9 KiB
This model was published in HF papers on 2024-12-18 and contributed to Hugging Face Transformers on 2025-03-21.
Prompt Depth Anything
Overview
The Prompt Depth Anything model was introduced in Prompting Depth Anything for 4K Resolution Accurate Metric Depth Estimation by Haotong Lin, Sida Peng, Jingxiao Chen, Songyou Peng, Jiaming Sun, Minghuan Liu, Hujun Bao, Jiashi Feng, Xiaowei Zhou, Bingyi Kang.
The abstract from the paper is as follows:
Prompts play a critical role in unleashing the power of language and vision foundation models for specific tasks. For the first time, we introduce prompting into depth foundation models, creating a new paradigm for metric depth estimation termed Prompt Depth Anything. Specifically, we use a low-cost LiDAR as the prompt to guide the Depth Anything model for accurate metric depth output, achieving up to 4K resolution. Our approach centers on a concise prompt fusion design that integrates the LiDAR at multiple scales within the depth decoder. To address training challenges posed by limited datasets containing both LiDAR depth and precise GT depth, we propose a scalable data pipeline that includes synthetic data LiDAR simulation and real data pseudo GT depth generation. Our approach sets new state-of-the-arts on the ARKitScenes and ScanNet++ datasets and benefits downstream applications, including 3D reconstruction and generalized robotic grasping.
Prompt Depth Anything overview. Taken from the original paper.
Usage example
The Transformers library allows you to use the model with just a few lines of code:
import requests
import torch
from PIL import Image
from transformers import AutoImageProcessor, AutoModelForDepthEstimation
url = "https://github.com/DepthAnything/PromptDA/blob/main/assets/example_images/image.jpg?raw=true"
image = Image.open(requests.get(url, stream=True).raw)
image_processor = AutoImageProcessor.from_pretrained("depth-anything/prompt-depth-anything-vits-hf")
model = AutoModelForDepthEstimation.from_pretrained("depth-anything/prompt-depth-anything-vits-hf", device_map="auto")
prompt_depth_url = "https://github.com/DepthAnything/PromptDA/blob/main/assets/example_images/arkit_depth.png?raw=true"
prompt_depth = Image.open(requests.get(prompt_depth_url, stream=True).raw)
# the prompt depth can be None, and the model will output a monocular relative depth.
# prepare image for the model
inputs = image_processor(images=image, return_tensors="pt", prompt_depth=prompt_depth).to(model.device)
with torch.no_grad():
outputs = model(**inputs)
# interpolate to original size
post_processed_output = image_processor.post_process_depth_estimation(
outputs,
target_sizes=[(image.height, image.width)],
)
# visualize the prediction
predicted_depth = post_processed_output[0]["predicted_depth"]
depth = predicted_depth * 1000
depth = depth.detach().cpu().numpy()
depth = Image.fromarray(depth.astype("uint16")) # mm
Resources
A list of official Hugging Face and community (indicated by 🌎) resources to help you get started with Prompt Depth Anything.
If you are interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.
PromptDepthAnythingConfig
autodoc PromptDepthAnythingConfig
PromptDepthAnythingForDepthEstimation
autodoc PromptDepthAnythingForDepthEstimation - forward
PromptDepthAnythingImageProcessor
autodoc PromptDepthAnythingImageProcessor - preprocess - post_process_depth_estimation
PromptDepthAnythingImageProcessorPil
autodoc PromptDepthAnythingImageProcessorPil - preprocess - post_process_depth_estimation