How to Annotate Multispectral Images for Computer Vision Models

Introduction

When working with multispectral images, annotators need to consider information from multiple spectral bands. This can be a challenging task, as it requires annotating each spectral band separately, which can be time-consuming and prone to errors. One of the most effective ways to handle multispectral images is to label them as groups, which allows us to consider images in context, leading to more accurate and consistent annotations.

Neural Networks can be used to speed up the labeling process. Supervisely has many ready-to-use models in the Labeling Tool. You can use them directly, or if you prefer, you can train your own models for your specific task to make the labeling process even more efficient, e.g. in this blog post we show how to train a custom interactive segmentation model for agricultural images.

The Supervisely platform offers several benefits to make labeling multispectral images quick and efficient, including:

• Easy import with a simple file structure
• Conveniently grouped display: images from a group are displayed together, providing a cohesive view.
• View synchronization: the ability to synchronize the view, including panning and zooming, across all images in the group.
• Mirrored annotation: annotations made on one image are automatically applied to all the images within the group in real-time

In this post, we demonstrate how the Supervisely Labeling Tool streamlines the annotation of multispectral images. You'll discover the process of importing multispectral images into the Supervisely platform and explore the benefits of using the Labeling Tool.

Video tutorial

In this video tutorial, you will learn how to import multispectral images and label them in Supervisely Labeling Tool using grouped display. Here's what we'll cover:

1. Importing multispectral images into Supervisely

2. Exploring the multi-view display functionality in the Image Labeling Tool.

3. Manually annotating multispectral images.

4. Speeding up the labeling process with AI-assistance using the Supervisely Smart Tool

What do multispectral images mean?

Multispectral images are images captured from multiple spectral bands, such as the visible, near-infrared, and infrared bands. These images are captured using specialized sensors or cameras, and they provide a wealth of information that can be used to train computer vision models for a variety of applications, including agriculture, medicine, forestry, and environmental monitoring. Since multispectral images contain different information about the same scene, annotators need to consider all spectral bands when labeling objects and features.

Example of 7-channel Landsat ETM multispectral image

Multispectral Imaging in Various Industries

Multispectral images find utility in various industries, such as:

• Agriculture: monitoring crop health, detecting disease, and optimizing crop yield
• Medical imaging: identifying tumors and other abnormalities
• Depth estimation: estimating the depth of objects in a scene
• Environmental monitoring: tracking changes in the environment, such as deforestation and water pollution
• Damage detection: identifying damage to different objects, such as cars and buildings
• Biometrics: identifying people based on their unique characteristics, such as palm veins

Below, we have outlined some examples along with accompanying Multispectral image samples.

1. Environmental monitoring

Multispectral images can be used to observe the environment. Multispectral image [1] from Multispectral and panchromatic Landsat ETM images over the Venice region in Italy contains information from the visible, near-infrared, and short-wave infrared bands, allowing us to see the vegetation, water, and soil in the scene. This information can be used to monitor changes in the environment, such as deforestation and water pollution.

[1] Environmental monitoring with multispectral images.

2. Crop monitoring

Multispectral images are valuable for wheat ear detection. Image [2] from Wheat Ear Detection in RGB and Thermal Images Using Deep Neural Networks includes an RGB photo and a thermal image, providing visibility to the wheat ears. This data is instrumental for monitoring crop health and detecting diseases.

[2] Wheat ear detection with multispectral thermal images

3. Leaf disease detection

Multispectral thermal images aid in the detection of plant diseases. In image [3] from Engineering & validation of a cost effective plant stress and disease sensor, comprising an RGB photo and a thermal image, the actual disease is not visible on the RGB photo but is discernible on the thermal image. This capability can greatly assist farmers in identifying diseases promptly and taking preventive measures.

[3] Plant disease detection with multispectral images

4. Depths estimation

Multispectral images are capable of estimating the depth of objects in a scene. Image [4] from Drilling Down on Depth Sensing and Deep Learning illustrates a basic pair of images in 3D graphics for demonstration purposes; however, in real-world scenarios, this capability can be immensely beneficial for various applications, including autonomous driving.

[4] Multispectral images for depths estimation

5. UV biometric authentication

UV images of palms in image [5] from Contact-Free Multispectral Identity Verification System Using Palm Veins and Deep Neural Network are used for contactless authentication using palm vein recognition showcasing a prime example of multispectral images in biometric applications.

[5] UV pictures of palm veins

Why Annotating Multispectral Images Can Be Tricky?

Lack of Labeling Tools. When dealing with Multispectral Images, a common and significant challenge is finding effective annotation solutions. The options available are limited, making the task quite daunting. Supervisely provides more than just an annotation tool; it offers an entire ecosystem of tools with collaborative features, particularly valuable for team-based projects or extensive data annotation. Check out our blog posts on how to effectively perform annotation at scale using Labeling Jobs, Labeling Queues and Labeling Consensus approaches.

Labor-intensive Nature of Annotating Multispectral Images. Labeling Multispectral Images is time consuming. Imagine that you have to annotate hundreds of multispectral images, when each image contains 4 spectral bands. That's 400 images to annotate! And if you have to annotate each spectral band separately, 400 images become 1600, does it sound like an efficient process? Not really. That's a lot of work, and it can be very tedious and error-prone. But what if there was a way to annotate all these images simultaneously? That would be a huge time-saver, and it would make the annotation process much more efficient. So, now it's time to talk about the advantages of annotating multispectral images as Groups in Supervisely:

• Object and details visibility: some objects are visible only on one spectral band, while on others they are not visible at all. The grouped display allows you to see all the spectral bands at once, and the most important thing is that you can annotate all of them simultaneously even if on some bands the object is not visible, or it's visible only partially.
• Better Context Understanding: Multispectral images contain information from different spectral channels, allowing for a more comprehensive understanding of object and scene characteristics.
• Cross-Validation of Labels: Annotators can verify and confirm labels by considering information from various spectral channels, leading to more accurate object identification.
• Pattern and Regularity Detection: Identifying common patterns and regularities within spectral data. This can aid in pinpointing characteristic object features and developing more precise identification methods.
• Reduced Noise Influence: Annotators can better recognize and filter out random noise that might be present in spectral data. Processing image groups enhances confidence in labels and classification.
• Dynamic Consideration: If you have sequential multispectral images, it can help in annotating objects and features that change over time by more accurate tracking of object evolution.
• Processing Optimization: It can optimize data processing, for instance, in analyzing agricultural fields or environmental changes within a specific area.

How to work with Multispectral Images in Supervisely

Supervisely not only supports multispectral images but also provides a convenient workflow, from import using the Application or Python SDK to annotation with the Labeling Tool, enabling the viewing and annotation of multispectral images as groups. Follow the brief guide below to import and annotate multispectral images in Supervisely.

Step 1. Prepare Images for Import

To annotate your multispectral images in Supervisely, start by preparing and importing them. The platform offers a simple import function, making it easy to bring in your data and set it up for annotation.

Just organize the images from one scene into different directories. Put images to be uploaded as-is into the images directory and images to be split into separate channels into the split directory. When getting your images ready for import, remember to use the suggested above file structure. For more detailed information about file structure refer to the README of the Import Multispectral Images application to ensure proper file organization.

Example of the file structure:

📦 archive
 ┣ 📂 group_name_1
 ┃ ┣ 📂 split
 ┃ ┃ ┗ 🏞️ demo1.png
 ┣ 📂 group_name_2
 ┃ ┣ 📂 images
 ┃ ┃ ┣ 🏞️ demo4-rgb.png
 ┃ ┃ ┗ 🏞️ demo4-thermal.png
 ┃ ┣ 📂 split
 ┃ ┃ ┗ 🏞️ demo4-thermal copy.png
 ┣ 📂 group_name_3
 ┃ ┣ 📂 images
 ┃ ┃ ┣ 🏞️ demo8-mri1.png
 ┃ ┃ ┣ 🏞️ demo8-mri2.png
 ┃ ┃ ┗ 🏞️ demo8-rgb.png

Step 2. Import Images

After preparing your images for import, you can easily bring them into the Supervisely platform. Follow these steps to do so:

Supervisely App

Import Multispectral Images

Import multispectral images as channels or as separate images.

Run in Supervisely GitHub

1. Run the Import Multispectral Images application.

2. Drag and drop the archive with your Project into the application or upload it into the Team Files.

3. Click the Run button to start the import process.

Running the Import Multispectral Images application

Step 3. Explore the Advantages of the Supervisely Labeling Tool

After importing your images, you can open the project in the Labeling Tool to start the annotation process. As you can see, the images are displayed as a group, and the view is synchronized across all images. Whenever you annotate an object on one image, the annotation is automatically applied to all the images in the group. This allows you to annotate all images in the group simultaneously, which is a huge time-saver.

Here's a brief list of features in the Supervisely Labeling Tool for annotating multispectral images:

1. Intuitive and easy-to-use interface.

2. Support of different tools such as polygons, bounding boxes, masks, etc.

3. Easy-to-use and highly customizable Smart Tool for AI-assisted annotation.

4. Support of custom applications, which can be run right in the Labeling Tool.

5. Collaborative features, such as Labeling Jobs and Labeling Queues.

You may also wonder if it's possible to speed up the labeling process even more. And the answer is yes, it is possible. You can use different Neural Networks right in the Labeling Tool and it will also work with all images in the group. It's a great way to speed up the labeling process, especially if you have a lot of images to annotate. As an example, you may check out this blog post about the Segment Anything in High Quality (SamHQ) model to learn more about it.

If you need to work with images individually, you can disable (and then re-enable) the group view at any time. To do this, go to the Settings section of the Labeling Tool, and use the corresponding toggle switches:

• Group Images mode for enabling or disabling the grouped display
• Group Images sync mode for enabling or disabling the synchronized view (pan and zoom) and annotation synchronization

Group Images mode and Group Images sync mode toggle switches of Multispectral satellite image

Automate workflow with Python SDK

You can also automate the process of working with multispectral images using Supervisely Python SDK.

pip install supervisely

You can learn more about it in our Developer Portal, but here we'll just show how you can upload your multispectral images with just a few lines of code.

# Setting multispectral settings for the project.
api.project.set_multispectral_settings(project.id)

# Preparing images for upload.
image_name = "demo7.png"
images = ["demo_data/demo7-rgb.png", "demo_data/demo7-thermal.png"]

# Reading thermal image and extracting its channels as 2d numpy arrays.
image = cv2.imread(images[1])
channels = [image[:, :, i] for i in range(image.shape[2])]

# Uploading images.
image_infos = api.image.upload_multispectral(dataset.id, image_name, channels, images)

In the example above we uploaded two images as they are and also split a thermal image into separate channels. And after the images are uploaded, there's so much you can do with them using our Python SDK! You can find a set of Python SDK tutorials to work with images on our Developer Portal.

Summary

In this article, we have learned how to import multispectral images into the Supervisely Platform and explored the advantages of the Labeling Tool, including its synchronized viewing and annotation capabilities. With Supervisely, annotating multispectral images as groups becomes a breeze, and you can tackle this task quickly and efficiently.