TexGaussian: Generating High-quality PBR Material via Octree-based 3D Gaussian Splatting

Bojun Xiong1*, Jialun Liu2*, Jiakui Hu3, Chenming Wu2, Jinbo Wu2,
Xing Liu2, Chen Zhao2, Errui Ding2, Zhouhui Lian1†
1Wangxuan Institute of Computer Technology, Peking University
2Baidu VIS
3Institute of Medical Technology, Peking University
*Equal contribution †Corresponding author
Paper arXiv Code (Coming Soon!)
Interpolate start reference image

Our proposed TexGaussian is capable of generating high-quality PBR material given the input 3D mesh based on the corresponding textual descriptions. The generated results are naturally compatible with modern graphical engines for photo-realistic rendering under different environment maps.

Abstract

Physically Based Rendering (PBR) materials play a crucial role in modern graphics, enabling photorealistic rendering across diverse environment maps. Developing an effective and efficient algorithm that is capable of automatically generating high-quality PBR materials rather than RGB texture for 3D meshes can significantly streamline the 3D content creation. Most existing methods leverage pre-trained 2D diffusion models for multi-view image synthesis, which often leads to severe inconsistency between the generated textures and input 3D meshes. This paper presents TexGaussian, a novel method that uses octant-aligned 3D Gaussian Splatting for rapid PBR material generation. Specifically, we place each 3D Gaussian on the finest leaf node of the octree built from the input 3D mesh to render the multiview images not only for the albedo map but also for roughness and metallic. Moreover, our model is trained in a regression manner instead of diffusion denoising, capable of generating the PBR material for a 3D mesh in a single feed-forward process. Extensive experiments on publicly available benchmarks demonstrate that our method synthesizes more visually pleasing PBR materials and runs faster than previous methods in both unconditional and text-conditional scenarios, which exhibit better consistency with the given geometry.

Method

Interpolate start reference image

An overview of our PBR material generation framework. (a) We propose octant-aligned 3D Gaussian Splatting, which positions a 3D Gaussian at the center of each finest leaf node of the constructed octree. Additional channels are added at the end of the Gaussian parameters to model PBR material. (b) We use the 3D U-Net built upon octree-based convolutional networks to predict the Gaussian parameters. Our octree-based 3D U-Net is trained by minimizing the difference on 2D raster images and 3D Gaussian parameters. (c) We bake the multi-view rendered images to the UV space of the input 3D model to realize physically based rendering under new illumination environments.



More results

Interpolate start reference image

RGB texture generative results on ShapeNet.


Interpolate start reference image

Our text-conditioned PBR material generation results compared with three state-of-the-art text-conditioned PBR material synthesis methods: Fantasia3D, FlashTex, and DreamMat.


Interpolate start reference image

Diverse material generation. Our method can generate different materials with different text prompts on the same mesh.


Interpolate start reference image

More generative results of our method on different input 3D models and text prompts.

Video

Dynamic rendering effects of the input 3D objects paired with their generative PBR materials across different environment maps.

BibTeX

@article{xiong2024texgaussian,
      title={TexGaussian: Generating High-quality PBR Material via Octree-based 3D Gaussian Splatting},
      author={Bojun Xiong and Jialun Liu and Jiakui Hu and Chenming Wu and Jinbo Wu and Xing Liu and Chen Zhao and Errui Ding and Zhouhui Lian},
      year={2024},
      eprint={2411.19654},
}