MeshXL: Neural Coordinate Field for Generative 3D Foundation Models

Sijin Chen1,2, Xin Chen2, Anqi Pang2, Xianfang Zeng2, Wei Cheng2, Yijun Fu2, Fukun Yin1,2, Yanru Wang2, Zhibin Wang2, Chi Zhang2, Jingyi Yu3, Gang Yu2, Bin Fu2, Tao Chen1
1Fudan University, 2Tencent PCG, 3ShanghaiTech University
arXiv Code Video

MeshXL is a family of generative pre-trained foundation models for 3D mesh generation. With the Neural Coordinate Field representation, the generation of unstructured 3D mesh data can be seaminglessly addressed by modern LLM methods.

Abstract

The polygon mesh representation of 3D data exhibits great flexibility, fast rendering speed, and storage efficiency, which is widely preferred in various applications. However, given its unstructured graph representation, the direct generation of high-fidelity 3D meshes is challenging. Fortunately, with a pre-defined ordering strategy, 3D meshes can be represented as sequences, and the generation process can be seamlessly treated as an auto-regressive problem. In this paper, we validate the Neural Coordinate Field (NeurCF), an explicit coordinate representation with implicit neural embeddings, is a simple-yet-effective representation for large-scale sequential mesh modeling. After that, we present MeshXL, a family of generative pre-trained auto-regressive models, which addresses the process of 3D mesh generation with modern large language model approaches. Extensive experiments show that MeshXL is able to generate high-quality 3D meshes, and can also serve as foundation models for various down-stream applications.

Scaling Up Generative Mesh Pre-training

After combining 3D data from ShapeNet, 3D-FUTURE, Objaverse, and Objaverse-XL, we obtain a total of over The textures for the meshes are generated by Paint3D. 3D object meshes for the generative mesh pre-training. We analyze the scaling effects of model parameters. We train all the models from scratch on 150 billion tokens, and observe that the performance grows with model sizes.

Unconditional 3D mesh generation

MeshXL generates high-quality and diverse 3D meshes.

objaverse

We show the wireframe of selected generated 3D objects below.

Below showcases some of the generated 3D meshes. Feel free to drag your mouse for a closer look.

Category-specific 3D mesh generation

shapenet

MeshXL can serve as a 3D foundation model that can be fine-tuned to produce high-quality 3D meshes on a specific category.

Texturing generated 3D meshes

Below shows the raw model, the textured model, and the UV map of the generated 3D objects. The textures are generated by Paint3D.

texturing

Feel free to drag your mouse for a closer look.

BibTeX

If you find our project helpful, please kindly consider citing our paper. 

@misc{chen2024meshxl,
    title={MeshXL: Neural Coordinate Field for Generative 3D Foundation Models}, 
    author={Sijin Chen and Xin Chen and Anqi Pang and Xianfang Zeng and Wei Cheng and Yijun Fu and Fukun Yin and Yanru Wang and Zhibin Wang and Chi Zhang and Jingyi Yu and Gang Yu and Bin Fu and Tao Chen},
    year={2024},
    eprint={2405.20853},
    archivePrefix={arXiv},
    primaryClass={cs.CV}
}