Skip to content

🧬 3D Mesh

DocArray supports many different modalities including 3D Mesh. This section will show you how to load and handle 3D data using DocArray.

A 3D mesh is the structural build of a 3D model consisting of polygons. Most 3D meshes are created via professional software packages, such as commercial suites like Unity, or the open-source Blender.

Note

This feature requires trimesh. You can install all necessary dependencies via:

pip install "docarray[mesh]"

Vertices and Faces representation

A 3D mesh can be represented by its vertices and faces:

  • Vertices are points in a 3D space, represented as a tensor of shape (n_points, 3).
  • Faces are triangular surfaces that are defined by three points in 3D space, corresponding to the three vertices of a triangle. They can be represented as a tensor of shape (n_faces, 3). Each number in that tensor refers to an index of a vertex in the tensor of vertices.

Load vertices and faces

First, let's define our class MyMesh3D, which extends BaseDoc and provides attributes to store our 3D data:

  • The mesh_url attribute of type Mesh3DUrl.
  • The optional tensors attribute, of type VerticesAndFaces
  • The VerticesAndFaces class has the attributes vertices and faces, both of type AnyTensor. This especially comes in handy later when we want to display our 3D mesh.

Tip

Check out our predefined Mesh3D to get started and play around with our 3D features.

But for now, let's create a MyMesh3D instance with a URL to a remote .obj file:

from typing import Optional

from docarray import BaseDoc
from docarray.documents.mesh.vertices_and_faces import VerticesAndFaces
from docarray.typing import Mesh3DUrl


class MyMesh3D(BaseDoc):
    mesh_url: Mesh3DUrl
    tensors: Optional[VerticesAndFaces] = None


doc = MyMesh3D(mesh_url="https://people.sc.fsu.edu/~jburkardt/data/obj/al.obj")

To load the vertices and faces information, you can call .load() on the Mesh3DUrl instance. This will return a VerticesAndFaces object:

doc.tensors = doc.mesh_url.load()
doc.summary()
Output 
📄 MyMesh3D : 9d8c26f ...
╭─────────────────────┬────────────────────────────────────────────────────────╮
│ Attribute           │ Value                                                  │
├─────────────────────┼────────────────────────────────────────────────────────┤
│ mesh_url: Mesh3DUrl │ https://people.sc.fsu.edu/~jburkardt/data/obj/al.o ... │
│                     │ (length: 52)                                           │
╰─────────────────────┴────────────────────────────────────────────────────────╯
└── 🔶 tensors: VerticesAndFaces
    └── 📄 VerticesAndFaces : 8cae4c4 ...
        ╭───────────────────┬──────────────────────────────────────────────────╮
        │ Attribute         │ Value                                            │
        ├───────────────────┼──────────────────────────────────────────────────┤
        │ vertices: NdArray │ NdArray of shape (3980, 3), dtype: float64       │
        │ faces: NdArray    │ NdArray of shape (7152, 3), dtype: int64         │
        ╰───────────────────┴──────────────────────────────────────────────────╯

Display 3D mesh in notebook

You can display your 3D mesh interactively from its URL as well as a VerticesAndFaces instance, by calling .display() on either one. The latter will always display without color, whereas the display from the URL will show with color if this information is included in the file content.

doc.mesh_url.display()

Point cloud representation

A point cloud is a representation of a 3D mesh. It is made by repeatedly and uniformly sampling points within the surface of the 3D body. Compared to the mesh representation, the point cloud is a fixed size ndarray and hence easier for deep learning algorithms to handle.

Load point cloud

Tip

Check out our predefined PointCloud3D to get started and play around with our 3D features.

In DocArray, loading a point cloud from a PointCloud3DUrl instance will return a PointsAndColors instance. Such an object has a points attribute containing the information about the points in 3D space as well as an optional colors attribute.

First, let's define our class MyPointCloud, which extends BaseDoc and provides attributes to store the point cloud information:

from typing import Optional

from docarray import BaseDoc
from docarray.documents.point_cloud.points_and_colors import PointsAndColors
from docarray.typing import PointCloud3DUrl


class MyPointCloud(BaseDoc):
    url: PointCloud3DUrl
    tensors: Optional[PointsAndColors] = None


doc = MyPointCloud(url="https://people.sc.fsu.edu/~jburkardt/data/obj/al.obj")

Next, we can load a point cloud of size samples by simply calling .load() on the PointCloud3DUrl instance:

doc.tensors = doc.url.load(samples=1000)
doc.summary()
Output 
📄 MyPointCloud : a63374d ...
╭──────────────────────────────────┬───────────────────────────────────────────╮
│ Attribute                        │ Value                                     │
├──────────────────────────────────┼───────────────────────────────────────────┤
│ url: PointCloud3DUrl             │ https://people.sc.fsu.edu/~jburkardt/dat… │
│                                  │ ... (length: 52)                          │
╰──────────────────────────────────┴───────────────────────────────────────────╯
└── 🔶 tensors: PointsAndColors
    └── 📄 PointsAndColors : 70ae175 ...
        ╭─────────────────┬────────────────────────────────────────────────────╮
        │ Attribute       │ Value                                              │
        ├─────────────────┼────────────────────────────────────────────────────┤
        │ points: NdArray │ NdArray of shape (1000, 3), dtype: float64         │
        ╰─────────────────┴────────────────────────────────────────────────────╯

Display 3D point cloud in notebook

You can display your point cloud and interact with it from its URL as well as from a PointsAndColors instance. The first will always display without color, whereas the display from PointsAndColors will show with color if .colors is not None.

doc.url.display()

Getting started - Predefined documents

To get started and play around with the 3D modalities, DocArray provides the predefined documents Mesh3D and PointCloud3D, which includes all of the previously mentioned functionalities.

Mesh3D

The Mesh3D class provides a Mesh3DUrl field and VerticesAndFaces field.

class Mesh3D(BaseDoc):
    url: Optional[Mesh3DUrl] = None
    tensors: Optional[VerticesAndFaces] = None
    embedding: Optional[AnyEmbedding] = None
    bytes_: Optional[bytes] = None

PointCloud3D

class PointCloud3D(BaseDoc):
    url: Optional[PointCloud3DUrl] = None
    tensors: Optional[PointsAndColors] = None
    embedding: Optional[AnyEmbedding] = None
    bytes_: Optional[bytes] = None

You can use them directly, extend or compose them:

from docarray import BaseDoc
from docarray.documents import Mesh3D, PointCloud3D


class My3DObject(BaseDoc):
    title: str
    mesh: Mesh3D
    pc: PointCloud3D


obj_file = 'https://people.sc.fsu.edu/~jburkardt/data/obj/al.obj'

doc = My3DObject(
    title='My first 3D object!',
    mesh=Mesh3D(url=obj_file),
    pc=PointCloud3D(url=obj_file),
)

doc.mesh.tensors = doc.mesh.url.load()
doc.pc.tensors = doc.pc.url.load(samples=100)