Skip to content

🧬 3D Mesh

DocArray supports many different modalities including 3D Mesh. This section will show you how to load and handle 3D data using DocArray.

A 3D mesh is the structural build of a 3D model consisting of polygons. Most 3D meshes are created via professional software packages, such as commercial suites like Unity, or the open-source Blender.

Note

This feature requires trimesh. You can install all necessary dependencies via:

pip install "docarray[mesh]"

Vertices and Faces representation

A 3D mesh can be represented by its vertices and faces:

  • Vertices are points in a 3D space, represented as a tensor of shape (n_points, 3).
  • Faces are triangular surfaces that are defined by three points in 3D space, corresponding to the three vertices of a triangle. They can be represented as a tensor of shape (n_faces, 3). Each number in that tensor refers to an index of a vertex in the tensor of vertices.

Load vertices and faces

First, let's define our class MyMesh3D, which extends BaseDoc and provides attributes to store our 3D data:

  • The mesh_url attribute of type Mesh3DUrl.
  • The optional tensors attribute, of type VerticesAndFaces
  • The VerticesAndFaces class has the attributes vertices and faces, both of type AnyTensor. This especially comes in handy later when we want to display our 3D mesh.

Tip

Check out our predefined Mesh3D to get started and play around with our 3D features.

But for now, let's create a MyMesh3D instance with a URL to a remote .obj file:

from typing import Optional

from docarray import BaseDoc
from docarray.documents.mesh.vertices_and_faces import VerticesAndFaces
from docarray.typing import Mesh3DUrl


class MyMesh3D(BaseDoc):
    mesh_url: Mesh3DUrl
    tensors: Optional[VerticesAndFaces]


doc = MyMesh3D(mesh_url="https://people.sc.fsu.edu/~jburkardt/data/obj/al.obj")

To load the vertices and faces information, you can call .load() on the Mesh3DUrl instance. This will return a VerticesAndFaces object:

doc.tensors = doc.mesh_url.load()
doc.summary()
Output 
📄 MyMesh3D : 9d8c26f ...
╭─────────────────────┬────────────────────────────────────────────────────────╮
│ Attribute           │ Value                                                  │
├─────────────────────┼────────────────────────────────────────────────────────┤
│ mesh_url: Mesh3DUrl │ https://people.sc.fsu.edu/~jburkardt/data/obj/al.o ... │
│                     │ (length: 52)                                           │
╰─────────────────────┴────────────────────────────────────────────────────────╯
└── 🔶 tensors: VerticesAndFaces
    └── 📄 VerticesAndFaces : 8cae4c4 ...
        ╭───────────────────┬──────────────────────────────────────────────────╮
        │ Attribute         │ Value                                            │
        ├───────────────────┼──────────────────────────────────────────────────┤
        │ vertices: NdArray │ NdArray of shape (3980, 3), dtype: float64       │
        │ faces: NdArray    │ NdArray of shape (7152, 3), dtype: int64         │
        ╰───────────────────┴──────────────────────────────────────────────────╯

Display 3D mesh in notebook

You can display your 3D mesh interactively from its URL as well as a VerticesAndFaces instance, by calling .display() on either one. The latter will always display without color, whereas the display from the URL will show with color if this information is included in the file content.

doc.mesh_url.display()

Point cloud representation

A point cloud is a representation of a 3D mesh. It is made by repeatedly and uniformly sampling points within the surface of the 3D body. Compared to the mesh representation, the point cloud is a fixed size ndarray and hence easier for deep learning algorithms to handle.

Load point cloud

Tip

Check out our predefined PointCloud3D to get started and play around with our 3D features.

In DocArray, loading a point cloud from a PointCloud3DUrl instance will return a PointsAndColors instance. Such an object has a points attribute containing the information about the points in 3D space as well as an optional colors attribute.

First, let's define our class MyPointCloud, which extends BaseDoc and provides attributes to store the point cloud information:

from typing import Optional

from docarray import BaseDoc
from docarray.documents.point_cloud.points_and_colors import PointsAndColors
from docarray.typing import PointCloud3DUrl


class MyPointCloud(BaseDoc):
    url: PointCloud3DUrl
    tensors: Optional[PointsAndColors]


doc = MyPointCloud(url="https://people.sc.fsu.edu/~jburkardt/data/obj/al.obj")

Next, we can load a point cloud of size samples by simply calling .load() on the PointCloud3DUrl instance:

doc.tensors = doc.url.load(samples=1000)
doc.summary()
Output 
📄 MyPointCloud : a63374d ...
╭──────────────────────────────────┬───────────────────────────────────────────╮
│ Attribute                        │ Value                                     │
├──────────────────────────────────┼───────────────────────────────────────────┤
│ url: PointCloud3DUrl             │ https://people.sc.fsu.edu/~jburkardt/dat… │
│                                  │ ... (length: 52)                          │
╰──────────────────────────────────┴───────────────────────────────────────────╯
└── 🔶 tensors: PointsAndColors
    └── 📄 PointsAndColors : 70ae175 ...
        ╭─────────────────┬────────────────────────────────────────────────────╮
        │ Attribute       │ Value                                              │
        ├─────────────────┼────────────────────────────────────────────────────┤
        │ points: NdArray │ NdArray of shape (1000, 3), dtype: float64         │
        ╰─────────────────┴────────────────────────────────────────────────────╯

Display 3D point cloud in notebook

You can display your point cloud and interact with it from its URL as well as from a PointsAndColors instance. The first will always display without color, whereas the display from PointsAndColors will show with color if .colors is not None.

doc.url.display()