Run a labeling team of any size
with quality per person.
Hand out labeling tasks to your own people and outside vendors, check their work with consensus, review and quality checks, and measure every annotator and reviewer on one Labeling Performance dashboard.
Hand out the work, and keep everyone busy.
Split a dataset into labeling tasks, or put it in a queue that people pull from. Each task shows its progress as the work goes.
The next item goes to whoever is free
A labeling queue hands out one item at a time, so fast annotators simply label more and nobody waits on a fixed share.
Hundreds of people on one dataset
Tasks never overlap, so a team of any size labels the same dataset in parallel, each person in the labeling tool for the data.
Vendors see only their work
Invite an outside team as labelers: they open only the tasks assigned to them, with no data exported and uploaded back.
Consensus settles disagreements before they reach the dataset.
Turn on consensus in an image or video queue and two to four annotators label each item independently. The platform scores how far they agree and pre-fills a result, and a reviewer settles the rest.
Independent versions
Annotators see only their own labels, so every item gets two to four versions made without looking at each other.
Agreement per annotator
Queue stats score each annotator on objects found, geometry, classes and tags, plus frames on video, with the least agreeing people first.
Only resolved labels ship
A reviewer takes one annotator's version, ignores a difference that does not matter or reopens it, and the resolved labels go to a separate final dataset.
Catch labeling mistakes before they reach training.
Every labeling task and queue has a review step, and quality checks go further on a random sample, so problems surface while the work is still going.
Review on your rules
Decide whether reviewers may edit labels themselves, and whether rejected items go back to their author or to someone else.
Quality checks, object by object
Reviewers judge the geometry, class and tags of every object in a random sample, and the check reports accuracy for each.
Error regions for what was missed
Reviewers outline an area with missing labels and note the class and how many objects are missing.
Labeling Performance shows speed and quality for every person.
One dashboard for the whole labeling operation: how much each person labeled and reviewed, how fast, and how much of it passed review. Managers plan with it, and annotators see their own numbers.
Labeling speed
Objects per hour of active work and the average time per object. Idle time is left out.
Acceptance rate
The share of items reviewers accepted, for the team, a project, a labeling task or one person.
Review time
Active time reviewers spend in total and per item, to balance their workload.
Time in the labeling tool
Total time in the labeling tool next to the time spent creating and editing labels.
Quality checks
Acceptance rate and review time of quality checks, apart from the regular review.
Members table
Objects and tags created, items submitted, accepted and rejected, and time for each person, with team totals.
Slice it any way
Filter by period, data type, workspace, project, labeling task or person, and see each figure against a longer period.
Share the report
Export the dashboard to PDF or Excel.
Annotators see their own
Annotators open the same page with only their own numbers, so they follow their speed and acceptance rate themselves.
Here’s why teams trust us.
“A very high product quality.”
BMW Group uses Supervisely to annotate manufacturing images from production lines in their world-wide plants for enhancing quality inspections using deep learning. Supervisely is integrated into the BMW Group AI Platform in order to empower computer vision based AI use cases.
BMW Group · Manufacturing quality inspection“Immensely grateful for their pivotal role in our work”
Working with Supervisely has significantly enhanced our capability to develop AI models for lung CT scans. What sets Supervisely apart is its exceptional support team who are really responsive and adapt to our unique needs with a range of apps and helper files.
“Helps us to do data scientist work better and faster”
We use Supervisely since 2019. The key advantage of this tool is that Supervisely provides a complete data treatment pipeline. An important advantage is that a Supervisely instance can be deployed autonomously on a Client infrastructure, and distributed on different servers.
“A valuable part of our computer vision operations”
Supervisely is a valuable part of our computer vision operations, enabling our teams to efficiently work with camera and LiDAR data across sensor fusion projects. We use Supervisely for its extensive feature set and its ability to adapt to our changing requirements.
“Extremely satisfied with Supervisely and would highly recommend it”
The review tools have been invaluable in ensuring the quality and accuracy. The Python SDK has also been incredibly helpful in automating and streamlining our workflow. The support team on Slack has been extremely helpful and responsive.
“Supervisely excels at that and much more”
We’ve been exceedingly impressed with the customer support, addition of new features, and the flexibility of the publicly available SDK/API. The Supervisely team has also been fast to respond to support questions, and has shown a lot of openness when given feedback on potential improvements.
Your team next?
Talk to a computer vision engineer about your data, your models and your infrastructure.
Book a demoFrequently asked questions.
How do I assign labeling tasks to annotators in Supervisely?
Create a labeling task (Labeling Jobs in the app) with the data, the classes and tags to use, the annotators and a reviewer, or put the data in a labeling queue that hands the next item to whoever is free. Annotators label in the browser and submit their work for review. See labeling tasks in the docs.
Can outside annotators work on our data without access to the rest of our projects?
Yes. Invite them as labelers: they see only the tasks assigned to them, and the data never has to be exported and uploaded back. A labeling member, limited to the labeler and reviewer roles, costs €35 per user, and labeling hours at €0.65 per hour suit one-off projects; see pricing.
How does consensus labeling measure agreement between annotators?
In a queue with consensus, two to four annotators label each image or video independently. The platform matches their objects and scores agreement on objects, geometry, classes and tags, plus frames on video, per annotator and for the whole queue. A reviewer settles each discrepancy, and only resolved labels go to the final dataset.
What does Labeling Performance measure?
Labeling speed in objects per hour, acceptance rate, review time, time in the labeling tool and quality-check results, for the team, a project, a labeling task or one person, with export to PDF or Excel. Annotators see their own numbers. It is part of every plan, including the free Community edition. See Labeling Performance in the docs.
How do quality checks differ from the regular review?
The regular review accepts or rejects whole items in a labeling task. A quality check takes a random sample and has reviewers judge the geometry, class and tags of every object separately and mark missing labels with error regions, so it reports accuracy per object, not only per item. See labeling quality control in the docs.