Scientists at Carnegie Mellon University have developed a neural network called LegoGPT, capable of generating LEGO assembly diagrams from text descriptions. The system produces an image of the model (.png), step-by-step instructions (.txt), and a file compatible with CAD programs (.ldr).
The neural network was trained on a dataset of 47,000 LEGO models created from 28,000 3D objects sourced from ShapeNetCore. Each model was tested for structural robustness using Gurobi, and the descriptions were generated using GPT-4o.
LegoGPT currently supports 21 object categories, including furniture, vehicles, and musical instruments, but does not support categories outside this range. The project uses a retrained version of Llama-3.2-1B-Instruct and is available as open source on GitHub and Hugging Face.
In the future, LegoGPT could streamline the creation of assembly instructions for designers and enhance integration with CAD software. Similar neural networks could also be applied in education, design, and industrial prototyping.