Let’s use the powerful vectorization capabilities of NumPy to change between 2D spherical images and 3D point clouds

Introduction

By harnessing the duality between 3D point clouds and 2D spherical coloured images, we will synergize robust 3D reconstruction algorithms with powerful 2D object detection techniques.

To concurrently capture spherical coloured images and point clouds, specialized devices akin to depth sensors or LiDAR scanners are commonly equipped with spherical cameras.

Let’s establish a precise definition of the objects we’re talking about:

• it provides a direct representation of the spatial layout of the scene
• each point is defined by its 3D spatial coordinates (x, y, z)
• additional attributes akin to color, intensity, or other properties might be related to each point

• It provides a panoramic view of the scene, capturing a large field of view in all directions around a central viewpoint. As an example, it may very well be provided by two back-to-back fisheye cameras (See my previous Medium article Understanding 360 images 🌎)
• Each pixel is defined by its 2D coordinates (latitude φ, longitude θ) and refers to a particular viewing direction, like on the earth’s surface.

Hidden potential of 2D algorithms in 3D realm

Object detection algorithms are frequently more robust in 2D than in 3D. Hence, the flexibility to transition to a 2D representation for efficient detection and segmentation and return to the 3D domain with the identified objects holds tremendous promise and excitement.

With this in mind, Florent Poux, Ph.D., has recently showcased a sensible maneuver: using Meta’s Segment Anything Model (SAM) for 3D point cloud segmentation.

Within the upcoming sections, we are going to define essential functions to facilitate the bidirectional transformation of knowledge between the 2D and 3D domains.

Spherical coordinates

This section will convert the worldwide cartesian coordinates (x,y,z) to local spherical coordinates (φ,θ,r).

Angles φ and θ correspond, respectively, to the rows and columns of the 2D spherical image, while the radius r refers back to the distance from which a degree is observed, i.e., its depth.

: For the sake of simplicity, we are going to consider that the spherical coordinates are situated on the origin of the reference frame. Nonetheless, a translation/rotation might be applied to the purpose cloud before converting the coordinates to render it from any desired pose (See my previous article Converting camera poses from OpenCV to OpenGL might be easy).

I strongly advise you to have a look at my Understanding 360 images 🌎 article, which explains the spherical coordinate system. The image and equation below comprehensively summarize all of the essential elements required.

Python code

To start with, we use the above equation to convert from cartesian to spherical coordinates. The input, xyz, is a floating-point numpy array of shape (N,3) containing the purpose cloud’s vertices.

We then convert the (φ,θ) angles to actual coordinates in pixels within the spherical image. The input parameters’ height and width correspond to the form of the spherical image and are used to scale the coordinates.

Because the height of the image represents 180°, i.e., from top to bottom, and the image’s width represents 360°, a spherical image at all times has a facet ratio of 2, i.e., width=2*height.

Combining the 2 previously defined methods makes it possible to project each 3D point of the purpose cloud onto the 2D spherical image.

Python code

This section goals at determining the vertex colours of a degree cloud by leveraging the colour information present in a spherical image.

As is common with numpy, we must meticulously plan our approach for sampling our arrays to make sure optimal performance. Below are some examples of concepts that may result in slow outcomes when executed:

• Iterate over the (u,v) coordinates using a Python for-loop
• Store for every pixel of the spherical image an inventory containing the ids of the vertices projected onto this very pixel

A quick approach uses array indexing or array slicing on the image using the (u,v) coordinates as indices.

Please note that this code doesn’t handle occlusions and color all vertices. This may very well be easily fixed by generating a binary mask that indicates the visibility of every vertex, thus stopping it from being coloured when not obligatory. The next section will explain the technique of generating a spherical depth map. A visibility mask might be created by sampling the depth map using (u,v) coordinates and determines whether the r values are inside a suitable range relative to those depths.

: Reasonably than rounding the (u, v) coordinates, we will preserve them as floating-point values and employ image interpolation for a more accurate color inference. Nonetheless, plain array indexing isn’t any longer applicable on this scenario. To deal with this, we could pre-enlarge the spherical image using interpolation or use an interpolation scheme akin to scipy.interpolate.RectBivariateSplineor scipy.interpolate.RegularGridInterpolator

Example

In this instance, we colorize a degree cloud using a spherical image with overlaid coloured segmentation masks. The ceiling is drawn in orange, the ground in green, and the door in blue.

For this point cloud of 250,000 points and its corresponding spherical image of shape 5.000×10.000 pixels, it takes about 15ms to run.

(It took about 750ms for 10 million points).

Python code

This section goals at rendering a spherical image by leveraging the colour information present within the vertices of the purpose cloud.

It’s interesting to notice that this problem might be solved in the identical way as above for the coloring of the purpose cloud. This time, array indexing is used to set the colour on the pixels corresponding to the projection of the vertices.

Nonetheless, several vertices might find yourself falling on the identical pixel. On this case, occlusion have to be considered to find out which color should appear within the foreground. We could compare the depths of all of the vertices inside a pixel and choose the closest one. But it surely’s smarter to sort all of the vertices by their depth in descending order and let the array indexing does the job for us. Indeed, the last occurrence of a pixel index during array indexing will ultimately determine its final value. Any previous occurrences can have been overwritten.

Likewise, we will acquire the depth map with none additional expense.

Example

In this instance, we render a spherical coloured image and depthmap of a coloured point cloud.

For this point cloud of 250,000 points, it takes about 30ms to generate the 2 images below:

There are particular sampling artifacts present within the rendered images. That is primarily resulting from the sparsity of the output image, as each vertex contributes to only a single pixel. The difficulty becomes more pronounced as we increase the image resolution. To deal with this, potential solutions may very well be to diminish the resolution, apply inpainting to fill the holes, or utilize point splatting inside a dedicated point cloud renderer.

I hope you enjoyed reading this text and that it gave you more insights on coloring or rendering point clouds!

See more of my code on GitHub.