Introducing 3DiffTection, an innovative approach to 3D object detection from single images, leveraging the power of a state-of-the-art 3D-aware diffusion model. The process of annotating extensive image datasets for 3D object detection is not only resource-intensive but also time-consuming. While large image diffusion models have proven effective as feature extractors for 2D perception tasks, adapting them directly to 3D tasks presents challenges, such as misalignment with target data. To address these issues, our method employs two specialized tuning strategies: geometric and semantic.
In the geometric tuning phase, we enhance the diffusion model through refinement on a view synthesis task, introducing a novel epipolar warp operator. This task is chosen for its dual significance—it requires 3D awareness and relies solely on posed image data, which is readily available from sources such as videos. The refinement ensures that the model is better aligned with the requirements of 3D object detection.
For semantic refinement, we further train the model on target data using box supervision. This phase enhances the model's ability to recognize and understand specific object features in the context of 3D detection. Both geometric and semantic tuning phases employ a ControlNet, preserving the original feature capabilities and ensuring the integrity of the model throughout the refinement process.
In the final step, we leverage these refined capabilities to perform a test-time prediction ensemble across multiple virtual viewpoints. This comprehensive methodology results in the derivation of 3D-aware features specifically tailored for 3D object detection. Our approach excels in identifying cross-view point correspondences, offering a cutting-edge solution to the challenges posed by 3D detection from single images.
