InstructP2P: Learning to Edit 3D Point Clouds with Text Instructions

作者：Jiale Xu, Xintao Wang, Yannan Cao, Weihao Cheng, Ying Shan, Shenghua Gao · 发表于：arXiv.org · 年份：2023 · DOI：10.48550/arXiv.2306.07154 · 被引用次数：16 · 研究领域：Computer Science

Enhancing AI systems to perform tasks following human instructions can significantly boost productivity. In this paper, we present InstructP2P, an end-to-end framework for 3D shape editing on point clouds, guided by high-level textual instructions. InstructP2P extends the capabilities of existing methods by synergizing the strengths of a text-conditioned point cloud diffusion model, Point-E, and powerful language models, enabling color and geometry editing using language instructions. To train InstructP2P, we introduce a new shape editing dataset, constructed by integrating a shape segmentation dataset, off-the-shelf shape programs, and diverse edit instructions generated by a large language model, ChatGPT. Our proposed method allows for editing both color and geometry of specific regions in a single forward pass, while leaving other regions unaffected. In our experiments, InstructP2P shows generalization capabilities, adapting to novel shape categories and instructions, despite being trained on a limited amount of data.