Image Disclaimer: This image was not created using Zero123 models.
Stable Diffusion has broadened its technological horizon into 3D object generation with the unveiling of Stable Zero123, marking a pivotal advancement in the field as highlighted by the researchers. This innovation stands as a notable enhancement over prior models such as Zero1-to-3 and Zero123-XL, with its superiority largely attributed to refined training datasets and the introduction of elevation conditioning. The methodology adopted involves the meticulous selection and more lifelike rendering of high-quality 3D objects from Objaverse, leading to significant progress in the model's ability to generate novel views of objects, thereby demonstrating a deep understanding of their appearance from multiple angles.
Leveraging the Stable Diffusion 1.5 framework, Stable Zero123 is designed to consume the same amount of VRAM for producing a single new view. However, the generation of 3D objects necessitates greater computational resources, with a recommendation of 24GB of VRAM. The model is made available for non-commercial and research purposes, with a commercial version, Stable Zero123C, accessible to Stability AI members.
The researchers have achieved a significant increase in training efficiency—up to 40 times faster than Zero123-XL—through the strategic incorporation of elevation conditioning during both training and inference phases, complemented by a pre-computed dataset and an enhanced dataloader. These improvements not only elevate the quality of the generated 3D objects but also contribute to a more expedient and efficient training process.
In an effort to promote open research within the realm of 3D object generation, the open-source code of threestudio has been updated to support Stable Zero123 and its predecessor, Zero123. This enhancement facilitates the creation of textured 3D meshes via Score Distillation Sampling (SDS), optimized through the Stable Zero123 model. The methodology is versatile, allowing for text-to-3D object generation by initially creating an image with SDXL, followed by the use of Stable Zero123.
Stable Zero123's licensing framework is dual-faceted, catering to both non-commercial research and commercial applications. The non-commercial variant incorporates CC-BY-NC 3D objects, while Stable Zero123C, designated for commercial use, is trained exclusively on CC-BY and CC0 3D objects and requires an active Stability AI membership for commercial deployment. Both models have been tested internally, showcasing similar levels of prediction quality, which speaks to the robustness and adaptability of Stable Zero123 across various applications.
We will delve into the foundational paper upon which Stable Zero123 is built, acknowledging that it originates from a different set of authors. To clarify any confusion, it's important to note that the paper specifically dedicated to Stable Zero123 has yet to be released. However, our exploration will center on the core concepts and methodologies outlined in the Zero123 paper. This initial groundwork has been further developed in subsequent research, heralding significant improvements and breakthroughs in the model's functionality and performance. Such progress is anticipated to redefine excellence in the domain of 3D object generation technology.
Imagine you have a picture of a toy car, and you wish you could see that car from a different angle, but all you have is this one picture. Traditionally, to achieve this, artists or 3D modelers would have to spend hours, even days, creating a new image or a 3D model from that single photo. What the researchers behind Zero123 have done is developed a smart computer program that can do this automatically, in much less time.
Zero123 is like a highly intelligent artist that has studied millions of images and learned how objects look from various angles. So, when you show it a picture of the toy car, it uses its knowledge to imagine and create a new picture of the car from a different viewpoint. This is revolutionary because it can understand and manipulate images in 3D space, even though it only sees a 2D picture, just like how our brain can imagine the other sides of an object by looking at it from one angle.
Moreover, Zero123 can do this for complex objects and scenes, not just simple items. Whether it's a toy car, a piece of intricate machinery, or even a scene from a storybook, Zero123 attempts to recreate these from new perspectives, making it a powerful tool for artists, designers, and anyone interested in visual creativity.
The paper discusses how they achieved this, the challenges they faced, such as ensuring the newly created images look realistic and detailed, and how their method compares to other techniques. They also talk about the future possibilities this opens up, like creating 3D animations from single photos or helping in designing products and artworks by providing a quick way to see them from different angles.
In essence, Zero123 is like having a magic sketchbook that brings your 2D images to life in 3D, making what was once a time-consuming task both fast and fun.
Ruoshi Liu, Rundi Wu, Basile Van Hoorick, Pavel Tokmakov, Sergey Zakharov, Carl Vondrick. (2023). Zero-1-to-3: Zero-shot One Image to 3D Object
🔗 https://arxiv.org/abs/2303.11328
The study introduces Zero-1-to-3, a novel framework designed to alter the camera viewpoint of an object from merely a single RGB image. To achieve novel view synthesis in this under-constrained scenario, the researchers harness geometric priors learned by large-scale diffusion models about natural images. Their conditional diffusion model is trained using a synthetic dataset to master the manipulation of the relative camera viewpoint, enabling the generation of new images of the same object under a specified camera transformation. Remarkably, despite its training on a synthetic dataset, the model exhibits a robust zero-shot generalization capability to both out-of-distribution datasets and in-the-wild images, including impressionist paintings. The researchers' viewpoint-conditioned diffusion methodology is also applicable to the task of 3D reconstruction from a single image. Through qualitative and quantitative experiments, the authors demonstrate that their approach significantly surpasses existing models in single-view 3D reconstruction and novel view synthesis, benefiting from the extensive pre-training on Internet-scale data.
The study delves into the innate human ability to perceive the 3D shape and appearance of objects from merely a single viewpoint. This capability plays a pivotal role not only in practical tasks such as object manipulation and environmental navigation but also in the realm of visual creativity, including painting. Interestingly, this skill transcends basic geometric assumptions, allowing for the visualization of complex objects that challenge or even defy real-world physical and geometric norms. Such an advanced level of perception is grounded in an extensive array of visual experiences accumulated over one's lifetime.
Contrastingly, contemporary techniques for 3D image reconstruction tend to be restricted by their reliance on detailed 3D annotations, like CAD models, or on category-specific priors, thus confining their utility to a limited set of circumstances. Although recent efforts aim to transcend these limitations by employing large and diverse datasets for open-world 3D reconstruction, they often still depend on some geometric information for training, such as stereo views or camera positions. This approach falls short of leveraging the full potential of the massive datasets that train state-of-the-art diffusion models, which, while adept at capturing semantic nuances, have yet to be fully explored for their geometric comprehension.
The researchers set out to investigate the capacity of large-scale diffusion models, such as Stable Diffusion, to manipulate camera viewpoints for the purpose of creating novel views and reconstructing 3D shapes from a single RGB image. Despite the inherent limitations posed by these tasks' under-constrained nature, the extensive training datasets these models have been exposed to—comprising over 5 billion images—provide them with a nuanced representation of the natural image distribution across a multitude of objects and perspectives. Through fine-tuning, the study develops mechanisms to adjust camera rotation and translation, facilitating the generation of images from new, specified viewpoints. Examples of these advancements highlight the models' capabilities in reinterpreting images from various perspectives.
This research makes two primary contributions: it unveils the depth of 3D geometric understanding embedded within diffusion models trained solely on 2D images, and it showcases leading-edge results in both novel view synthesis and zero-shot 3D reconstruction of objects from single images. Subsequent sections will outline the methodology adopted to teach diffusion models about camera extrinsics and present a collection of both quantitative and qualitative experiments to assess the effectiveness of zero-shot view synthesis and 3D reconstruction of geometry and appearance from a singular image perspective.
3D Generative Models: The fusion of advanced generative image architectures with expansive image-text datasets has enabled the synthesis of highly detailed scenes and objects, with diffusion models emerging as particularly adept at scaling image generation through a denoising process. Transitioning these capabilities to the 3D realm traditionally demands substantial volumes of costly annotated 3D data. Recent strategies, however, have pivoted towards adapting pre-trained 2D diffusion models for 3D applications, sidestepping the need for explicit 3D ground truth data. Neural Radiance Fields (NeRFs) have been recognized for their precise scene encoding abilities. While NeRFs are commonly applied to reconstruct specific scenes from multiple posed images, innovations like DreamFields have repurposed NeRFs into versatile components within 3D generative systems, with subsequent studies leveraging a 2D diffusion model's distillation loss to generate text-driven 3D objects and scenes.
The authors’ research takes a novel path in novel-view synthesis by treating it as a viewpoint-conditioned image-to-image translation challenge, utilizing diffusion models. This approach is further enriched by integrating 3D distillation for single-image 3D shape reconstruction. Previous initiatives have explored similar frameworks but lacked in demonstrating zero-shot generalization. Competing methodologies have introduced image-to-3D generation techniques informed by language-guided priors and textual inversion, yet this study distinguishes itself by mastering viewpoint control via a synthetic dataset and achieving zero-shot generalization for images captured in natural settings.
Single-view Object Reconstruction: The endeavor to reconstruct 3D objects from a singular viewpoint presents considerable difficulties, necessitating robust priors. A segment of this research domain has focused on developing priors through the aggregation of 3D primitives, such as meshes, voxels, or point clouds, complemented by image encoders for contextual conditioning. These approaches, however, are limited by the diversity of their 3D data sources and often exhibit subpar generalization due to their reliance on global conditioning. Furthermore, they necessitate an extra pose estimation phase to align the estimated shape with the input image. Alternatively, models that condition on local image features have shown promise in scene reconstruction with better cross-domain generalization, though their applicability tends to be restricted to reconstructing views within close proximity. More recently, the introduction of MCC has marked a significant leap by learning a universal representation for 3D reconstruction from RGB-D views, trained on a comprehensive dataset of object-centric videos.
In contrast to these methods, the study showcases the feasibility of deriving intricate geometric details directly from a pre-trained Stable Diffusion model, thereby eliminating the requirement for explicit depth data. This advancement opens new avenues for 3D reconstruction, leveraging the inherent capabilities of diffusion models to understand and interpret 3D geometry without reliance on traditional data sources.
Given a single RGB image:
of an object, the objective of the research is to synthesize an image of the object from a different camera viewpoint. The variables:
and
Represent the relative camera rotation and translation, respectively, for the desired new viewpoint. The goal is to develop a model, denoted as f, which is capable of generating a new image that reflects the specified camera transformation, as represented by the equation:
Here, x_hat_R,T symbolizes the synthesized image. The ambition is for the estimated x_hat_R,T to exhibit a high degree of perceptual resemblance to the true, yet unseen, novel view x_R,T.
The process of synthesizing novel views from a monocular RGB image is highly under-constrained. To tackle this, the approach will leverage large diffusion models such as Stable Diffusion, which demonstrate remarkable zero-shot capabilities in generating diverse images from textual descriptions. Owing to the breadth of their training data, these pre-trained diffusion models are considered leading representations for the distribution of natural images in the current landscape.
However, the creation of model f presents two significant challenges. Firstly, even though large-scale generative models are trained across a broad array of objects and viewpoints, they do not inherently encode correspondences between these viewpoints. Secondly, generative models often exhibit biases toward certain viewpoints, as a consequence of the predominance of such perspectives on the Internet. For instance, as depicted in Figure 2 of the study, Stable Diffusion tends to produce images of chairs facing forward in canonical stances. These issues significantly impede the extraction of 3D knowledge from large-scale diffusion models.
The challenge with diffusion models, despite their training on vast internet-scale datasets, is that they inherently lack the ability to control the viewpoints represented within their learned distribution of natural images. By introducing a mechanism to manage the camera extrinsics associated with a photo, the potential to perform novel view synthesis is unlocked.
To achieve this, the authors utilize a dataset comprising pairs of images and their corresponding camera extrinsics {x, x_(R,T), R, T}. As illustrated in Figure 3, the approach involves fine-tuning a pre-trained diffusion model to gain mastery over camera parameters while preserving the integrity of the original representation. In alignment with established practices, a latent diffusion architecture is employed, consisting of an encoder E, a denoiser U-Net ϵθ, and a decoder D. At any diffusion time step t sampled from a range of [1, 1000], the embedding c(x, R, T) represents the input view and the relative camera extrinsics. The model is then fine-tuned by optimizing the following objective:
Once ϵθ is trained, the inference model f can generate an image through iterative denoising from a Gaussian noise image, conditioned on c(x, R, T).
The key finding of the paper is that through such fine-tuning, pre-trained diffusion models acquire a versatile mechanism for camera viewpoint manipulation that applies beyond the objects present in the fine-tuning dataset. Essentially, these controls can be "bolted on" to the diffusion model, enabling it to maintain its capability to generate photorealistic images, now with the added benefit of viewpoint control. This compositional feature introduces zero-shot capabilities into the model, whereby the enhanced model can synthesize new views for object classes that are devoid of 3D data and were absent from the fine-tuning process.
For the task of 3D reconstruction from a single image, both low-level perceptual information (such as depth, shading, and texture) and high-level understanding (including object type, function, and structure) are vital. To address this, the study implements a dual conditioning mechanism. The first component involves generating a "posed CLIP" embedding, c(x, R, T), by combining a CLIP embedding of the input image with the relative camera extrinsics (R, T). This embedding is then utilized to apply cross-attention within the denoising U-Net, infusing the process with high-level semantic insights of the input image.
Concurrently, on a second pathway, the input image is channel-concatenated with the image currently undergoing the denoising process. This technique helps preserve the identity and intricate details of the object that is being synthesized. To further refine this approach and enable the application of classifier-free guidance, a method introduced in prior research is adopted. This involves intermittently setting both the input image and the posed CLIP embedding to a null vector, which occurs randomly. During inference, this conditional information is scaled, enhancing the model's ability to generate the desired outputs with precision.
In numerous contexts, the mere synthesis of novel views does not suffice—comprehensive 3D reconstructions that encapsulate an object's appearance and geometry are often the goal. To this end, the study adopts the Score Jacobian Chaining (SJC) technique from a recently open-sourced framework. SJC utilizes the priors learned by text-to-image diffusion models to optimize a 3D representation. Nonetheless, given the inherently stochastic nature of diffusion models, the gradient updates involved in this process are notably random. A pivotal strategy derived from DreamFusion is to increase the classifier-free guidance value significantly beyond the standard setting. This adjustment reduces sample diversity but enhances the accuracy of the reconstruction.
As depicted in Figure 4 and aligning with the SJC methodology, the researchers randomly select viewpoints for volumetric rendering. The rendered images are then perturbed with Gaussian noise:
and subsequently denoised by applying the U-Net ϵθ which is conditioned on the input image x, the posed CLIP embedding c(x, R, T), and the timestep t. This process aims to estimate the score that guides the image towards an accurate, noise-free input x_π:
Here,
represents the PAAS score as introduced by previous research. This methodology is instrumental in achieving a more precise and faithful 3D reconstruction by effectively leveraging the predictive strength of diffusion models to simulate the intricate details of an object's structure.
🧐 This section is not in the paper. To enhance our understanding of the paper, let's dissect the concept of Score Jacobian Chaining (SJC) and its role in 3D reconstruction.
Background
- Diffusion Models: These are a group of generative models that excel at creating high-quality images by iteratively refining noise into coherent visuals through a denoising process.
- 3D Generation Challenge: Transitioning diffusion models from 2D to 3D is a complex task due to the cost of annotating 3D data. Although Neural Radiance Fields (NeRFs) offer a compelling method for 3D scene encoding, they don't inherently capitalize on the strengths of pre-trained 2D diffusion models.
Core Idea of SJC
SJC ingeniously adapts pre-trained 2D diffusion models to facilitate 3D scene generation, operating in the following manner:
- Scores as Guidance: In diffusion models, 'scores' refer to the gradients of the data distribution. These scores guide the model in transitioning from a noisy state towards a representation resembling real-world data.
- From 2D Scores to 3D Insights: SJC takes points from a 3D representation, projects them onto multiple virtual cameras to create 2D images, and then computes the 2D diffusion score for these images.
- Jacobian Integration: By applying the chain rule, SJC integrates these 2D scores with the Jacobian, which maps how changes in the input (the 3D points) affect the output (the 2D images). This produces an estimated 3D score.
- NeRF Optimization: The estimated 3D score is used to optimize a NeRF, which models the 3D scene. The NeRF is thus fine-tuned based on the information provided by the 2D diffusion model, marrying the two realms.
Benefits
- Leveraging 2D Prowess: By reutilizing existing 2D diffusion models, which are already well-tuned and optimized, we can avoid the resource-intensive process of training new 3D models from scratch.
- Efficiency Gains: SJC offers a more resource-efficient pathway to 3D model training, achieving faster training and inference times relative to building 3D diffusion models from the ground up.
Challenges
- Handling Novel Data: The 2D diffusion model must adapt to the rendered images from the SJC process, which may differ from the natural images it was trained on, potentially presenting out-of-distribution challenges.
- Balancing Act: While SJC provides computational benefits, there's a trade-off; the resulting 3D models might not reach the accuracy levels of those trained with dedicated 3D diffusion processes.
We're now better equipped to appreciate the nuanced interplay between 2D and 3D modeling presented in the paper.
The researchers have chosen the Objaverse dataset for the fine-tuning phase, which is a recently unveiled large-scale, open-source collection comprising over 800,000 3D models crafted by more than 100,000 artists. Unlike ShapeNet, Objaverse does not categorize its contents with explicit class labels but offers a vast array of high-quality 3D models. These models are not only diverse but also feature rich geometries, intricate details, and material properties.
For each model within the dataset, the study generates 12 camera extrinsic matrices that focus on the model's center, utilizing a ray-tracing engine to render 12 different viewpoints. During training, this setup allows for the selection of two views per object to establish an image pair (x, x_{R,T}). The relative viewpoint transformation (R, T), which is crucial for mapping the perspectives, is readily calculable from the pair of extrinsic matrices. This methodical data preparation is pivotal for training the model to understand and manipulate camera viewpoints accurately.
The performance of the model is evaluated on the tasks of zero-shot novel view synthesis and 3D reconstruction. It is important to note, as confirmed by the authors of the Objaverse dataset, that the datasets and images employed in this paper are external to the Objaverse collection, qualifying the results as zero-shot. The study quantitatively pits the model against the current leading techniques on synthetic objects and scenes, across a spectrum of complexities. Additionally, qualitative outcomes are presented using an array of in-the-wild images, including everyday photographs to artwork.
The paper outlines two interrelated tasks using single-view RGB images as inputs, approached from a zero-shot perspective:
Novel View Synthesis: This longstanding challenge in computer vision necessitates that a model implicitly learns an object's depth, texture, and shape from a single view, leveraging prior knowledge due to the extreme constraint of limited input information. While recent methods have focused on optimizing implicit neural fields with CLIP consistency objectives from random views, this study adopts an orthogonal strategy. By reversing the typical order of 3D reconstruction and novel view synthesis, the model retains the object's identity within the input image, allowing probabilistic generative models to account for aleatoric uncertainty in self-occlusion scenarios and effectively harnessing the semantic and geometric priors learned by large diffusion models.
3D Reconstruction: The model also adapts to a stochastic 3D reconstruction framework, such as SJC or DreamFusion, to forge a most probable 3D representation. This is parameterized as a voxel radiance field and further refined into a mesh using the marching cubes technique. Applying the view-conditioned diffusion model to 3D reconstruction offers a feasible avenue for translating the rich 2D appearance priors learned by the diffusion model into 3D geometry.
In keeping with the nature of the proposed method, comparisons are drawn exclusively against techniques that are zero-shot and utilize single-view RGB images as input.
For novel view synthesis, benchmarks include DietNeRF, which employs CLIP-driven regularization across viewpoints, and Image Variations (IV), a Stable Diffusion model fine-tuned on image conditioning rather than text prompts. Additionally, SJC has been adapted into SJC-I, converting the text-conditioned diffusion model to an image-conditioned one.
In the domain of 3D reconstruction, the baselines are two leading single-view algorithms: Multiview Compressive Coding (MCC), which leverages neural fields to complete RGB-D observations into a 3D representation, and Point-E, a diffusion model for colorized point clouds. MCC's training occurs on CO3Dv2, while Point-E benefits from training on OpenAI's extensive internal 3D dataset. Comparisons are also made against SJC-I.
Given that MCC requires depth data, the MiDaS system is employed for depth estimation, converting relative disparity maps into absolute pseudo-metric depth maps using standard scale and shift values that are uniformly applicable across the test set.
For evaluating the tasks of novel view synthesis and 3D reconstruction, the model is tested against Google Scanned Objects (GSO) and RTMV datasets, which feature high-quality scans of household items and complex scenes, respectively. Ground truth 3D models from these datasets are the reference standard for assessing 3D reconstruction quality.
Novel View Synthesis: The method and its baselines are rigorously quantified using metrics such as PSNR, SSIM, LPIPS, and FID, which collectively measure various aspects of image similarity.
3D Reconstruction: The evaluation is based on Chamfer Distance and volumetric Intersection over Union (IoU) to measure the accuracy of the reconstructed 3D models.
Numerical results are detailed in the study's tables, while Figures 5 and 6 illustrate that, especially on the GSO, the proposed method generates images with a high degree of photorealism and alignment with the ground truth. This high-fidelity synthesis extends to RTMV scenes, which are more complex and divergent from the Objaverse dataset. Among the baselines, Point-E stands out for its zero-shot generalizability, though its lower resolution limits its utility for this task.
Diversity Across Samples: The task's under-constrained nature makes diffusion models particularly suitable due to their capability to account for the inherent uncertainty. The partial views in 2D input images leave ample room for interpretation, and Figure 8 showcases the diversity and quality of images sampled from novel viewpoints.
Tables 3 and 4 present the numerical data, and Figure 9 qualitatively depicts that the method achieves high-fidelity 3D meshes. MCC offers good surface estimation from visible angles but struggles with unseen areas. SJC-I frequently falls short in reconstructing meaningful geometries. Point-E, while demonstrating impressive generalization, yields sparse point clouds that may result in incomplete surface reconstructions. The proposed method combines view-conditioned diffusion model priors with NeRF-style representation, leading to advancements in both Chamfer Distance and volumetric IoU as shown in the results.
The capability of the model extends beyond in-the-wild images; it also shows proficiency with images generated from text-to-image models like Dall-E-2. As demonstrated in Figure 10, the model capably generates novel views that retain the objects' identities, suggesting its potential utility in text-to-3D generation applications. The study suggests that this method could offer valuable contributions to the field, especially in the context of creative and design-focused tasks.
In this work, the authors have advanced from individual objects to complex scenes. Initially trained on a dataset of single objects against plain backgrounds, their method has shown promising generalization to scenes containing multiple objects within the RTMV dataset. However, there is a noticeable decline in quality when compared to in-distribution samples from the GSO dataset, highlighting that generalization to scenes with intricate backgrounds is a significant challenge that remains to be addressed.
Furthermore, the study paves the way from static scenes to dynamic videos. The ability to comprehend the geometry of dynamic scenes from a single viewpoint could lead to innovative research directions, such as improved understanding of occlusions and dynamic object manipulation. While recent methods have begun to explore diffusion-based video generation, extending these to incorporate a 3D perspective is crucial for unlocking new opportunities.
The researchers have also showcased a novel framework that harnesses the 3D knowledge embedded within Stable Diffusion. This model, renowned for its generative capabilities in natural images, also encapsulates implicit knowledge about aspects such as lighting, shading, and texture. Future research could exploit these mechanisms to perform traditional graphics tasks, such as scene relighting, opening up new possibilities for the application of generative models in graphics.
Stable Zero123 is a refined and advanced based on the foundational concepts of this paper, and it is anticipated to further elevate the capabilities of Stable Diffusion in the domain of 3D object generation.
As someone deeply immersed in the world of drawing and 3D art, and having spent considerable time exploring Blender's capabilities—from modeling and rigging to sculpting—I've experienced firsthand the enlightening yet incredibly demanding process of creating 3D models. It's a labor of love that borders on the meticulousness of manual labor.
Similarly, drawing, particularly complex organic forms like human figures or animals, demands a commitment to developing a fine-tuned skill set. Mastering the intricacies of perspective and lighting, not to mention the daunting concept of foreshortening, is a journey that spans years.
The prospect of generating 3D models from single images is nothing short of revolutionary, a fantastical leap that could democratize the fields of art and 3D modeling. It has the potential to transform the creative landscape, seamlessly connecting the expertise of seasoned artists with the enthusiasm of hobbyists in an instant. The implications for artistry and professional endeavors are profound and multifaceted. Drawing parallels to historical resistance against technological advancements, some may adopt a Luddite stance towards this tidal wave of change, yet the momentum of progress is undeniable and, arguably, irresistible.
Personally, I have welcomed this transformative shift, integrating AI into my 2D and 3D creative process. The relief from the burden of repetitive and labor-intensive tasks is liberating. The anticipation of what lies ahead for the intersection of AI and art is palpable.
We stand on the brink of a new frontier, and my enthusiasm for the unfolding possibilities is boundless. Let's watch with eager eyes what the next chapter of this era brings.
As a concluding thought, it's essential to recognize a profound insight gleaned from these deep dives of ours: the capabilities that once were thought to be exclusively within the realm of human expertise are now increasingly within reach of AI tools. And not just within reach—AI is poised to perform these tasks with an efficiency and depth of understanding that surpass human ability in ways we're only beginning to comprehend. Remember this as we continue to witness and participate in the evolution of AI's role in creativity and beyond.