Geometric (Box, Polygon, Line, Keypoints, Cuboid) Predictions

In this guide, we'll use the Python SDK to construct geometric predictions. Each geometric
prediction object in Nucleus has five components:

Label of the prediction in the class taxonomy, e.g. car or pedestrian
Geometry to define the position, size, etc. of the box / polygon / line / cuboid
Reference ID of the item to which to apply the prediction
Confidence of the model for the prediction
Metadata, optional key-value pairs pertaining to the prediction, e.g. vehicle_color = blue
Annotation ID, an optional unique (per dataset item) key to refer to the prediction
Class PDF, an optional mapping from each class to its corresponding value in the model probability distribution

All geometric prediction types (box, polygon, line, cuboid) share the same structure aside from the geometry. Below, we'll explore how to construct the prediction objects, including geometries, for each type:

`BoxPrediction`

A 2D bounding box based on a given XY position, width, and height.

For more info, see our Python SDK Reference for BoxPredictions.

from nucleus import BoxPrediction

box_gt = BoxPrediction(
    label="car",
    x=4,        # px between left edge of the box and left border of the image
    y=6,        # px between top edge of the box and top border of the image
    width=12,   # width in px
    height=18,  # height in px
    reference_id="image_1",
    confidence=0.7	# confidence of the prediction
    class_pdf={"car": 0.7, "bus": 0.2, "person": 0.1}, # probability distribution
    annotation_id="image_1_car_box_1",
    metadata={"vehicle_color": "ruby"}
)

`PolygonPrediction`

A 2D polygon constructed by connecting 3+ vertices.

For more info, see our Python SDK Reference for PolygonPredictions.

from nucleus import Point, PolygonPrediction

polygon_gt = PolygonPrediction(
    label="bus",
    vertices=[	# list of vertices as Point objects
        Point(x=100, y=100),
        Point(150, 200),
        Point(200, 100),
    ],
    reference_id="image_2",
    confidence=0.8, # confidence of the prediction
    class_pdf={"car": 0.15, "bus": 0.8, "person": 0.05}, # probability distribution
    annotation_id="image_2_bus_polygon_1",
    metadata={"vehicle_color": "sapphire"}
)

`LinePrediction`

A 2D line constructed by connecting 2+ vertices (an unclosed polygon).

For more info, see our Python SDK Reference for LinePredictions.

from nucleus import Point, LinePrediction

line_gt = LinePrediction(
    label="lane",
    vertices=[	# list of vertices as Point objects
        Point(x=100, y=100),
        Point(200, 300),
        Point(300, 200),
    ],
    reference_id="image_1",
    confidence=0.7, # confidence of the prediction
    class_pdf={"lane": 0.7, "median": 0.2, "crosswalk": 0.1}, # probability distribution
    annotation_id="image_1_lane_line_1",
    metadata={"position": "shoulder"},
)

`KeypointsPrediction`

A collection of 2D points that are named and connected via a domain-specific skeleton.

For more info, see our Python SDK Reference for KeypointsPrediction.

from nucleus import Keypoint, KeypointsPrediction

keypoint_prediction = KeypointsPrediction(
    label="face",
    keypoints=[
        Keypoint(100, 100, visible=True),
        Keypoint(200, 300, visible=False),
        Keypoint(visible=False),
    ],
    names=["left_eye", "right_eye", "nose"],
    skeleton=[[0, 2], [1, 2]],
    reference_id="image_1",
    confidence=0.8, # confidence of the prediction
    annotation_id="image_1_face_1",
    metadata={"visibility": "partial"},
)

`CuboidPrediction`

A 3D cuboid based on a given position, dimensions, and yaw. Cannot be applied to 2D images, only 3D pointcloud DatasetItems.

For more info, see our Python SDK Reference for CuboidPredictions.

from nucleus import Point3D, CuboidPrediction

cuboid_gt = CuboidPrediction(
    label="car",
    position=Point3D(100, 100, 10), # center of the cuboid
    dimensions=Point3D(x=5, y=10, z=5),	# length (x), width (y), height (z)
    yaw=0, # rotation in rad about the Z axis
    reference_id="pointcloud_1",
    confidence=0.8, # confidence of the prediction
    class_pdf={"car": 0.8, "bus": 0.1, "person": 0.1}, # probability distribution
    annotation_id="pointcloud_1_car_cuboid_1",
    metadata={"vehicle_color": "emerald"}
)