Overview

In this guide, we'll walk through the steps to upload your 3D and sensor fusion data to Nucleus.

1. Create Dataset for LidarScenes
2. Schematize and upload remote pointcloud JSONs
3. Grant scale access to your data.
4. Upload remote camera + schematize camera intrinsics
5. Create pointcloud and camera image DatasetItems
6. Collect the above into Frames
7. Collect the above into LidarScenes
8. Upload LidarScenes to Dataset

Nucleus supports LidarScenes, which are sequences of Frames for each timestep of capture. Each Frame is a collection of sensor data, namely a single lidar pointcloud with any number of associated camera images.

Dataset

We'll begin by creating a new Dataset to which to upload our 3D data. In order to create a Dataset to which you can upload LidarScenes you need to set the is_scene flag to True during creation.

from nucleus import NucleusClient

client = NucleusClient(YOUR_API_KEY)

dataset = client.create_dataset(YOUR_DATASET_NAME, is_scene=True)

Preparing Your Data

Nucleus expects a remote JSON for pointclouds and a remote image file. For camera images, Nucleus also expects a metadata field camera_params, which gives details about the extrinsics and intrinsics of your camera sensor. This information is used to create integrated 2D visualizations such as cuboid projections and FOV highlighting.

Please refer to our API documentation for more specifics.

Granting Scale Access to Your Data

If you are using non-public cloud storage for your images, then you need to make sure your remote data is accessible to Scale by following this guide.

Lidar Pointclouds

Lidar pointclouds must follow a specific JSON schema for uploads into Nucleus. This schema is very similar to that of Frame payloads for Scale annotation projects. However, Nucleus payloads only pertain to the pointcloud and should not have camera information.

Pointcloud JSONs can have five properties:

The most important property to have is points. Meanwhile, device_position and device_heading are helpful for calibrating a static offset of the sensor relative to the captured points. Providing device_gps_pose and timestamp is optional, but may prove helpful when querying your data in Nucleus.

{
    "points": [
        {
            "x": 100.0,
            "y": 200.0,
            "z": 300.0,
            "i": 0.7
        },
        {
            "x": 400.0,
            "y": 500.0,
            "z": 600.0,
            "i": 0.5
        },
        {
            "x": 700.0,
            "y": 800.0,
            "z": 900.0,
            "i": 0.3
        }
    ],
    "device_position": {
        "x": 0.0,
        "y": 0.0,
        "z": 0.0
    },
    "device_heading": {
        "x": 0.0,
        "y": 0.0,
        "z": 0.0,
        "w": 0.0
    },
    "device_gps_pose": {
      	"lat": 90.0,
      	"lon": 180.0,
        "bearing": 360.0
    },
    "timestamp": 9001
}

Once you've constructed a JSON for each of your pointclouds, you can upload them to your favorite remote file system or cloud storage provider. Nucleus expects each DatasetItem to be initialized with the URL of a pointcloud JSON -- we'll show how to construct these in a later section.

📘

Privacy Mode

As an enterprise customer, you can use Nucleus without ever transferring data onto Scale servers! Read more on Privacy Mode here.

Camera Images

Nucleus processes each camera image as a single DatasetItem, even if there are multiple camera images associated with the same Frame (e.g. multiple angles captured at the same timestep).

Intrinsics and extrinsics for each camera, such as focal length or camera position and heading, should be organized into a Python dictionary as in the below example. These dictionaries should use the same schema as Scale annotation CameraImages, excluding the image_url property (schema defined here).

camera_params = {
    "position": {
        "x": -0.6,
        "y": -0.1,
        "z": 1.7
    },
    "heading": { # Hamiltonian quaternion
        "x": 0.5,
        "y": 0.5,
        "z": -0.5,
        "w": -0.5
    },
    "cx": 123.0, # principal point x value
    "cy": 456.0, # principal point y value
    "fx": 900.0, # focal length in x direction (pixels)
    "fy": 901.0, # focal length in x direction (pixels)
    # other optional camera intrinsics keys:
    # - timestamp
    # - scale_factor
    # - priority
    # - camera_index
    # - camera_model
    # - skew
    # - k1, k2, k3, k4 (radial distortion coeffs)
    # - p1, p2 (tangential distortion coeffs)
    # - xi (reference frame offset)
}

All camera images should also be uploaded to a remote file system from which Scale can access URLs to each image.

As for camera instrinsics like the dictionary above, we recommend serializing each camera's intrinsics to JSON or a similar key-value format. These files should be organized such that it's easy to link them back to their associated camera images. More on how we'll use these in the next section.

Constructing Lidar Scenes

Now that your data is formatted properly, and you have granted access to your data to scale, we can start using the Nucleus Python client to construct LidarScene objects for upload via API.

There are three main steps in this process:

1. Create pointcloud and camera image DatasetItems
2. Create Frames from 1.
3. CreateLidarScenes from 2.

DatasetItem

Pointclouds

Since we've already packaged much of the necessary pointcloud information into the JSON blobs, creating pointcloud DatasetItems is very straightforward.

from nucleus import DatasetItem

pointcloud = nucleus.DatasetItem(
    pointcloud_location="s3://your-bucket-name/001/lidar/00.json",
    reference_id="scene-1-pointcloud-0",
    metadata={ "is_raining": False }
)

Camera Images

Camera intrinsics such as focal length or camera position and heading must be packaged as a dictionary and passed into image metadata using the camera_params key. As mentioned, this dictionary must follow the schema defined here, excluding image_url.

import json
import boto3

# remote camera_params JSON
s3 = boto3.client("s3")
res = s3.get_object(Bucket="your-bucket-name", Key="001/front-camera/intrinsics.json")
front_camera_params = json.loads(res["Body"].read())

frontcam_image = DatasetItem(
    image_location="s3://your-bucket-name/001/front-camera/00.jpeg",
    reference_id="frontcam-scene-1-image-0",
    metadata={
        # arbitrary metadata
        "is_raining": False,

      	# camera paramaters/intrinsics
      	# MUST USE KEY `camera_params`
        "camera_params": front_camera_params
    }
)

# local camera_params JSON
with open("local/path/to/001/rear-camera/intrinsics.json", "r") as f:
  rear_camera_params = json.loads(f.read())

rearcam_image = DatasetItem(
    image_location="s3://your-bucket-name/001/rear-camera/00.jpeg",
    reference_id="rearcam-scene-1-image-0",
    metadata={
        "is_raining": False,
        "camera_params": rear_camera_params
    }
)

Frame

Next, you'll want to group your newly created pointcloud and camera image DatasetItems into Frames.

A Frame corresponds to a single timestep of sensor capture. It must contain exactly one pointcloud, and any number of camera images. Sensor names may be passed in as arbitrary keyword arguments, mapped to their respective DatasetItems.

from nucleus import Frame

frame = Frame(
  	lidar=pointcloud,
  	front_camera=frontcam_image,
  	rear_camera=rearcam_image
)

LidarScene

After creating several Frames for each timestep of capture, you can string them together into a LidarScene.

from nucleus import (DatasetItem, Frame, LidarScene)

scene = LidarScene(
  	reference_id="scene_by_list",
  	frames=[frame0, frame1, frame2]
)

You can also make changes to an existing LidarScene. The add_frame method allows you to add or update frames in the sequence by index.

# scene.frames: [frame0, frame1, frame2]

# add a new Frame
scene.add_frame(
  	frame=frame3,
  	index=3 # add to end of sequence
)

# overwrite existing Frame
scene.add_frame(
  	frame=frame0_new
  	index=0,
  	update=True # default is False, which will ignore collisions
)

# scene.frames: [frame0_new, frame1, frame2, frame3]

Similarly, you can add pointclouds and camera images to specific frames of a scene by index.

# scene.frames[0].items: {
#			'lidar': pointcloud0,
#			'front_camera': frontcam_image0,
#			'rear_camera': rearcam_image0
# }

# add a new camera image to frame 0
scene.add_item(
  	sensor_name='left_camera',
  	item=leftcam_image0,
  	index=0
)

# overwrite existing pointcloud in frame 0
scene.add_item(
  	sensor_name='lidar',
  	item=pointcloud0_new,
  	index=0
)

# scene.frames[0].items: {
#			'lidar': pointcloud0_new,
#			'front_camera': frontcam_image0,
#			'rear_camera': rearcam_image0,
#			'left_camera': leftcam_image0
# }

🚧

A single scene can have at most 300 items (pointclouds or images). Support for larger scenes is under active development! In the meantime, we recommend splitting up your larger scenes such that each one has <300 items.

Uploading to Nucleus

We'll upload to the Dataset created earlier in this guide. You can always retrieve a Dataset by its dataset ID, which are always prefixed with ds_. You can list all of your datasets' IDs using NucleusClient.list_datasets(), or extract one from the Nucleus dashboard's URL upon clicking into the Dataset.

from nucleus import NucleusClient

client = NucleusClient(YOUR_API_KEY)

dataset = client.get_dataset(YOUR_DATASET_ID)

With your scenes and dataset ready, you can now upload to Nucleus!

# after creating or retrieving a Dataset
job = dataset.append(
  	items=[scene0, scene1, scene2, ...],
  	update=True,
  	asynchronous=True # required for 3D uploads
)

# async jobs will run in the background, poll using:
job.status()

By setting the update flag to True, your upload will overwrite any existing scene- or item-level metadata for any collisions on reference_id.

🚧

A note on update=True

Nucleus expects scenes to maintain the same structure in terms of frame sequence, and items per frame.

Intended scene/frame updates with a different structure will be ignored.