Category Predictions

In this guide, we'll use the Python SDK to construct category predictions for image classification tasks. A category prediction object in Nucleus has four components:

• Label of the prediction in the class taxonomy, e.g. shirt or dress
• Reference ID, of the item to which to apply the prediction
• Name of the taxonomy, optional but recommended
• Confidence, 0-1 indicating the confidence of the prediction
• Class PDF, a optional complete class probability distribution on this prediction. Each value should be between 0 and 1 (inclusive), and sum up to 1 as a complete distribution. This can be useful for computing entropy to surface places where the model is most uncertain
• Metadata, optional key-value pairs pertaining to the prediction, e.g. color = blue

Nucleus currently supports single-label categorization natively, and can support multi-label classification with a workaround.

For more information, check out our Python SDK reference for CategoryPredictions.

Taxonomies

To begin, we recommend manually creating and adding your own taxonomies to your datasets. This allows you to apply more than one category prediction to a single image, where each prediction pertains to a different taxonomy.

By default, if taxonomy_name is not provided in the CategoryPrediction object, Nucleus will create a "default taxonomy" automatically based on the set of all unique labels in the uploaded prediction payload. If you continue to upload more CategoryPredictions, new labels will automatically be added to the default taxonomy.

You can create a taxonomy and associate it with a dataset using Dataset.add_taxonomy:

import nucleus
client = nucleus.NucleusClient("YOUR_SCALE_API_KEY")
dataset = client.get_dataset("YOUR_DATASET_ID")

response = dataset.add_taxonomy(
    taxonomy_name="clothing_type",
    taxonomy_type="category",
    labels=["shirt", "trousers", "dress"],
    update=False
)

Once added, you can then use labels from and reference this taxonomy in any CategoryPrediction by taxonomy_name.

By setting update=True, you can add new labels to an existing taxonomy (e.g. from labels=["shirt", "trousers", "dress"] to labels=["shirt", "trousers", "dress", "jacket"]). We do not yet support changing or deleting existing labels, but this is coming soon! In the meantime, we recommend deleting the taxonomy and recreating it from scratch with the updated/removed labels.

Single Category Predictions

For single-label categorization tasks, simply create one CategoryPrediction per image, per taxonomy. For instance, if you have 2 taxonomies clothing_type and designer and 100 images, you would create 200 predictions as follows:

from nucleus import CategoryPrediction

predictions = []
predictions.append(CategoryPrediction(
    label="shirt",
    reference_id="image_1", 
    taxonomy_name="clothing_type",
    confidence=0.8,
    class_pdf={"shirt": 0.8, "trousers": 0.1, "dress": 0.1},
    metadata={"color": "off-white"}
))
predictions.append(CategoryPrediction(
    label="Virgil Abloh"
    reference_id="image_1",
    taxonomy_name="designer",
    confidence=0.9
))

Multi-Category Predictions

Currently, the best way to add multi-category predictions or predictions to an item is to use multiple taxonomies. Rather than a single taxonomy with N classes, you can add N binary taxonomies each with two classes: True or False.

For example, consider a multi-category model predicting classes: shirt, trousers, and dress. You would first add three True/False taxonomies for each of the three classes. To add a CategoryPrediction for an item of both the shirt and dress class, you would simply add True predictions for the shirt and dress taxonomies, and a False prediction for the trousers taxonomy.

Unfortunately, the confidence and class PDFs will not be calibrated across taxonomies; one way to get around this is to attach actual PDF values or multiplicative weights as metadata.

from nucleus import CategoryPrediction, NucleusClient
client = nucleus.NucleusClient("YOUR_SCALE_API_KEY")
dataset = client.get_dataset("YOUR_DATASET_ID")

# add binary taxonomies
for label in ["shirt", "trousers", "dress"]:
    dataset.add_taxonomy(
        taxonomy_name=label,
        taxonomy_type="category",
        labels=["True", "False"]
    )

# create CategoryPredictions
predictions = []
predictions.append(CategoryPrediction(
    label="True",
    reference_id="image_1", 
    taxonomy_name="shirt",
    metadata={"pdf_value": 0.5}
))
predictions.append(CategoryPrediction(
    label="False",
    reference_id="image_1", 
    taxonomy_name="trousers",
    metadata={"pdf_value": 0.4}
))
predictions.append(CategoryPrediction(
    label="True",
    reference_id="image_1", 
    taxonomy_name="dress",
    metadata={"pdf_value": 0.1}
))

Note this is a temporary workaround for native multi-category support in Nucleus, which is coming soon!

📘

Automatic Calculation of Evaluation Metrics

Unlike with other annotation types, you don't need to call the calculate model metrics endpoint to update matches against ground truth annotations and calculate various metrics such as IOU for categorization. Sorting by metrics, as well as the evaluation plots and metrics present in the Insights page will be automatically refreshed when you upload new category annotations and/or predictions.

This is because by default you can only have a single category ground truth annotation per dataset item, and a single category prediction per dataset item per model, making matching category ground truth annotations and predictions trivial (other task types require user input to calibrate results).

Scene Category Predictions

Nucleus supports uploading and visualizing category predictions at the scene level. A SceneCategoryPrediction object contains a subset of those for regular category predictions: Label, Reference ID, Taxonomy Name, Confidence, and Metadata.

To upload predictions for a scene (a video or a point cloud), begin by uploading all taxonomies present in your dataset as demonstrated earlier in this section.

Then, create a SceneCategoryPrediction for a scene with a given reference_id the same way you would for a CategoryPrediction.

from nucleus import SceneCategoryPrediction

predictions = []
predictions.append(SceneCategoryPrediction(
    label="skiing",
    reference_id="scene_1",
    taxonomy_name="action",
    confidence=0.2,
    metadata={ "weather": "snowy" },
))
predictions.append(SceneCategoryPrediction(
    label="playing table tennis",
    reference_id="scene_2",
    taxonomy_name="action",
    confidence=0.4,
    metadata={ "weather": "sunny" },
))