Deep learning in 3D with Facebook AI’s new tool PyTorch3D
PyTorch3D is the latest deep learning tool by Facebook AI. The open source tool is designed to integrate with PyTorch to make 3D deep learning easier. Along with it, the codebase of the 3D shape prediction method Mesh R-CNN, which was built with the help of PyTorch3D, has been released as well.
Facebook AI is having a busy week—after the data visualization tool HiPlot, this week’s second deep learning tool PyTorch3D has been released. Developed by the Facebook AI Research Computer Vision Team a while back, it is now available open source on GitHub.
In PyTorch3D, Facebook AI sees “the potential for building systems that make high-quality 3D predictions without relying on time-intensive, manual 3D annotations,” as stated in the announcement blog post.
Let’s see what features the deep learning tool for 3D shape prediction offers.
PyTorch3D is designed to integrate with deep learning methods for 3D data prediction and manipulation. For example, its operators are implemented using PyTorch tensors and it can utilize GPUs.
It comes with a differentiable mesh renderer and allows you to store and manipulate data meshes. Different operations can be performed on the meshes, namely sampling, loss functions, projective transformations and graph convolution.
The concept of triangle meshes is visualized in a video on the Facebook AI blog:
On GitHub, you can find tutorials for four different PyTorch3D use cases, ranging from deforming a sphere mesh into a dolphin to rendering textured meshes:
Mesh R-CNN, announced on the Facebook AI blog last October, is a method for predicting 3D shapes that was built with the help of PyTorch3D. Along with the open sourcing of PyTorch3D, Mesh R-CNN’s code is now available on GitHub as well.