Tutorials & How-Tos

Whether you're developing precision agriculture systems to detect crop diseases, creating AI-powered tools for early lung cancer detection from CT scans, or building theft detection systems for convenience stores, the success of your AI project hinges on one crucial element: high-quality annotated data. Even the most sophisticated AI models are only as good as the data they're trained on.‍“OK, but how do I make sure the data we get from our in-house annotation team or data labeling agency is actually good?”, you ask. And we answer: data labeling specifications.‍What are data labeling specifications? And, why does your project need them?Data labeling specifications (or annotation specifications) are documentation that provides clear instructions and guidelines for annotators on how to annotate or label data. Depending on the project, these guidelines may include class definitions, detailed descriptions of labeling rules, examples of edge cases, and visual references such as annotated images or diagrams.Labeling specifications serve several critical purposes:‍Ensure all annotators follow the same standardsMaintain consistency across large datasetsEnable quality controlHelp achieve the required accuracy for model trainingServe as a reference document for both the client and annotation teamThe lack of well-thought-out specifications leads to all sorts of issues for all stakeholders involved—clients, labeling service providers, annotation teams, and ultimately, the end users of the data:#1 Inconsistent annotation resultsPoor specifications result in inconsistent annotation outcomes, as annotators are left to make assumptions and interpret tasks as they see fit. For example, if the guidelines don't specify how to handle occluded objects (e.g., a pedestrian behind a car), one annotator might use a bounding box while another uses a polygon. These inconsistencies can make the dataset unusable for model training, and often require its complete re-annotation. Source: https://cocodataset.org#2 Wasted time and moneyInconsistent annotation results inevitably trigger a costly cycle of revisions and rework, with each iteration requiring additional time from annotators, reviewers, and project managers. The result? Blown budgets and missed deadlines that could have been avoided with clear specifications from the start.‍#3 Frustrated annotation teamNothing kills team morale faster than having to redo work that's already been done. When annotators spend hours labeling data only to learn that the requirements weren't clear or complete, it's more than just frustrating—it's demoralizing. Productivity drops, attention to detail suffers, and the entire project enters a downward spiral. ‍#4 Project management overheadUnclear specifications turn project managers into full-time firefighters. Instead of focusing on strategic tasks, they're stuck in an endless cycle of retraining annotators, clarifying instructions, and double-checking work. Every vague requirement creates a ripple effect of questions, corrections, and additional reviews. This translates into more management hours, higher costs, and project managers who can't focus on what really matters—delivering quality results on time. ‍So, what makes a good specification?A well-crafted specification is like a detailed roadmap—it guides annotators to their destination without leaving room for wrong turns. Based on our experience working with hundreds of clients, here's what separates great specifications from the rest:‍Project Context. Don't just tell annotators what to do—help them understand why they're doing it. Whether your AI will be scanning crops for disease or monitoring store security, this context helps annotators make better decisions when they encounter tricky cases.Comprehensive Class Definitions. Think of this as your annotation dictionary. Every object class should be clearly defined, along with its key characteristics. For instance, what exactly counts as a "ripe tomato" in your agricultural dataset? What specific visual indicators should annotators look for?Clear Annotation Rules. Spell out exactly how you want things labeled. Should that partially visible car be marked with a bounding box or a polygon? How precise should segmentation masks be? Leave no room for guesswork.Edge Case Playbook. Every dataset has its tricky cases. Maybe it's a car hidden behind a tree or a disease symptom that's barely visible. Document these scenarios and provide clear instructions on how to handle them consistently.Red Flags and Common Pitfalls. Show annotators what not to do. By highlighting common mistakes upfront, you can prevent errors before they happen and save countless hours of revision time.Visual Examples (That Actually Help). A picture is worth a thousand words. This is true for labeling specs too. Include plenty of annotated examples showing both perfect and poor annotations. These real-world references are often more valuable than written descriptions alone.When you nail your specifications, the benefits cascade throughout your entire project:‍Every annotator follows the same playbook, delivering uniform results that your AI models can actually learn from. No more dealing with a mishmash of annotation styles that confuse your training process.Clear instructions mean fewer mistakes and less back-and-forth. Your team can work confidently and efficiently, keeping your project timeline on track.Every round of corrections burns through your budget. With crystal-clear specifications, you slash the need for revisions and keep costs under control. Plus, modern annotation platforms like CVAT come with built-in specification support, making it even easier for your team to stay on track.Now, let's put it to the test and see how good vs. bad labeling specs play out with a real-world dataset.‍“Good vs. Bad” Labeling Specifications: A Head-to-Head Test‍Source: https://cocodataset.orgThe setupAn image of a parking lot with different cars, road signs, people, trees, and fences. .Two annotators.Two different specs.The specs‍The first annotator was given very basic instructions: ‍Annotate the road, signs, people and vehicles using masks. Transportation must additionally be annotated with boxes.‍That's it. No quality guidelines, no examples, no nothing.The second annotator got a bit more lucky, and received a few more details:Annotate only the driveway and exclude the sidewalk from the annotation.Annotate signs together with their posts.Use only a mask, not a bounding box, for vehicles with less than 50% visibility.The results‍‍The results are quite descriptive. Without extra clarification, the first annotation is less accurate, missing some attributes such as signposts, and incorrectly labeling the sidewalk as part of the street. The second annotation is 100% accurate.‍A super-simple example, but when applied to a real use case, leaving out extra details can lead to thousands of inconsistent annotations, missed deadlines, unhappy annotators, and, worst of all, AI models that fail to perform reliably in production.‍Build better AI with better specifications Creating thorough labeling specs takes time and effort, but it's an investment that pays off many times over through consistent results, faster delivery, and significant cost savings. ‍To help you get started, we've created a comprehensive data labeling specification template based on our experience with hundreds of successful annotation projects. It covers all the essential elements we discussed and includes practical examples you can adapt for your specific needs. ‍Free Data Labeling Specs Template‍Download our free template and set your AI project up for success from day one. ‍{{labeling-specs-banner="/blog-banners"}}

In today's fast-paced digital environment, the efficiency of team collaboration can make or break project success, especially when it comes to complex tasks like data annotation, team management and workflow setup. ‍For businesses and research teams that depend on precise image and video annotation, Computer Vision Annotation Tool (CVAT) offers a powerful solution to improve team productivity and accuracy. One of the standout features of CVAT.ai is the Organization feature, designed specifically for teamwork. ‍Here’s a practical guide on how to use Organizations for image annotations, structured around a common use-case scenario.‍BackgroundImagine you are responsible for a data annotation project that requires organizing and labeling large volumes of images or videos. You have a team ready to do the work, and your goal is to ensure that everyone operates efficiently and cohesively to deliver high-quality results. To help achieve this, you've chosen CVAT.ai as your preferred tool.‍Or you are a student leading a research project, working with your peers on a similar task of data annotation. This project involves organizing and labeling images or videos for academic research purposes. Your main objective is to make sure that your classmates understand the tasks clearly, ensuring that everyone is on the same page, which will allow you to annotate the dataset effectively and proceed with your research. For this, CVAT.ai is your tool of choice.‍This article will guide you through using CVAT.ai effectively with your team to ensure the best possible results. From setting up your project to managing tasks and collaborating. Whether you're annotating data for commercial use or an academic study, these guidelines will help you and your team succeed in your efforts.‍Step 1: Setting Up Your Organization in CVAT.ai‍Setting up CVAT.ai for optimal team collaboration involves a number of necessary steps: from registering in CVAT.ai to subscribing to the Team plan. Below is a guide outlining each action you need to take for a successful start:‍1. Create an Account and Log In: Begin by going to the CVAT.ai website and creating an account. Once you've registered, log in with your credentials.‍2. Create an Organization: In CVAT.ai, an Organization acts as a central hub where all projects, team members, and tasks can be managed under a single umbrella. Once logged in, create an Organization.‍3. Switch to the Organization Account: After creating your Organization, switch from your individual account to your newly created organization account. Switching to the Organization account is mandatory for the next step, where you need to subscribe to a Team plan if you want to collaborate and annotate without any limits.‍4. Subscribe to the Team Plan: To lift all the limitations of the Free plan and start working on the project with your team you need to subscribe. Before subscribing, check if you need to add any additional information to your invoices. Also, note that you, as an organization owner, are also part of the team. So, if you have three annotators working, you’ll need to pay for 4 seats (3 annotators + 1 organization owner (you!)).Now all done and you are ready to invite team members and start working on the project. Let’s move on to the next step and invite team members for collaboration. ‍Step 2: Adding Team Members‍Once your organization within CVAT.ai is established, the next step is to add your team members, ensuring that each participant has the appropriate access and tools needed to annotate. Here’s how to manage this process smoothly:1. Invite members: Go to Organization > Settings you will see an Organization page with list of members and an Invite member button. Click on it to proceed. A dialogue box will appear where you can enter the email addresses of the people you want to add—these could be your annotators, reviewers, and any supervisory staff.‍‍2. Assigning Roles and Responsibilities: As you invite each member, you’ll have the option to assign specific roles. Assigning roles is crucial for establishing a clear hierarchy and division of responsibilities within the team. Depending on their role, users will have access only to the functionalities necessary to perform their specific tasks.Once you’ve added members and assigned roles, you can create project add tasks and assign jobs to annotators.‍Step 3: Creating a Project and Uploading DataOnce your organization in CVAT is up and running and team members have accepted invitations to join, you'll need to create projects, add tasks, and assign jobs to the annotators. Here's how you can proceed:‍1. Creating Projects: In CVAT.ai, projects serve as broad categories that organize related tasks under a specific theme or goal. Any labels or specifications added at the project level will automatically apply to all tasks and jobs within that project, ensuring consistency and saving time. ‍To create a project, go to the Projects section within your organization’s dashboard, and click + to create a new Project.. You'll be prompted to enter your project details such as the project name, description, and so on. ‍2. Adding Tasks to Projects: Tasks are the specific assignments that annotators work on within a project. Each task involves annotating a particular set of images or videos according to predefined guidelines and objectives.‍To add a task to a project in CVAT.ai, first navigate to the project page. Then, click on + > Create a new task. Have your dataset ready, as you will need to upload it for the task to be successfully created. When you create a task, CVAT.ai automatically generates jobs within that task. You can divide a single task into several jobs, allowing multiple annotators to work on different parts of the task simultaneously. ‍3. Specifications for Annotators: Clear specifications with guidelines for annotators help maintain consistency across annotations, which is crucial for training machine learning models. They also ensure that all team members are aligned with the project's standards, which helps in achieving high-quality outputs. You can easily create specifications within CVAT and add them at the Project or Task level, so all annotators can be on the same page. ‍4. Quality Assurance and QA: In CVAT.ai, you can ensure the quality of annotations through two methods: by creating a specific job known as a Honeypot for automatic QA, or by assigning a dedicated worker for manual QA. If you opt for the Honeypot, it's important to create this job before beginning the annotation process.‍Step 4: Assigning Jobs/Tasks to Annotators and Annotation. Once tasks are created and specifications set, assign them to individual annotators or, on the later stage, to reviewers. To do this, click on the Task you will see a list of jobs, all of them having an assignee field. ‍‍Click on it and select the name of the annotators and the Job’s stage. And with that, you're all set!‍Step 5: Configure Webhooks‍This step is optional, but we recommend setting up webhooks for a seamless workflow.Webhooks are a powerful tool within CVAT that allow for real-time notifications and automated reactions to specific events within the platform. By configuring webhooks, you can set up CVAT to send instant alerts or perform automated tasks whenever certain actions occur within your projects.Step 6: Annotation‍After you assign the jobs, annotators will see them and proceed with the annotation of the images or videos. This step is critical as it involves the direct application of data labeling based on the project guidelines.Note, that after annotation is done, annotators need to save the work and change the job state to completed.‍Step 7: Quality Assurance and Issue ResolutionAfter annotations are completed, it is essential to verify the quality of the work before acceptance. In CVAT.ai, you can do this in two ways: with automatic or manual quality assurance options.‍1. Honeypot for Automatic QA: If you have set up a Honeypot (also known as the Ground Truth job), allow some time for the CVAT platform to accumulate data. This setup helps in checking the accuracy and quality of the annotations made by comparing them with pre-validated 'ground truth' data.‍2. Assign Jobs to Validators for Manual Validation: You can manually validate annotations by assigning jobs to validators. Input the validator’s name in the 'Assignee' field and change the 'Job stage' to 'Validation'. Validators will review the assigned jobs and report any issues found. In CVAT.ai validators can easily report any discrepancies or errors in the annotations.Once all reports are available, the annotators can review and address any identified issues. ‍3. Correction of Issues: Review the issues reported by validators. If there is a need for further improvements, reassign the jobs to either the original annotator. Once annotators receive the reports, they can review and address any identified issues. Validators may also correct issues directly. This dual role of validating and correcting improves the quality control process, ensuring more accurate outcomes in the annotation project. However, the best process ultimately depends on your preference.‍Step 8: Analytics and PerformanceWhat follows is the annotating and quality assurance process. To streamline these stages, CVAT.ai offers analytics tools that help monitor the progress and performance of your team. These analytics provide valuable insights into task completion rates, annotator performance, and can highlight areas that may require additional attention or adjustment. ‍Step 9: Export DataOnce the annotation and validation stages are complete, and all quality checks are satisfied, export the annotated data. This data is now ready for use in machine learning models or for any other required purpose.‍ConclusionCVAT.ai Organizations were designed for team collaboration on annotation projects, making it easier to handle complex tasks. By using described steps, businesses can improve the data annotation processes, which in turn helps speed up the development of dependable and effective machine learning models. ‍Not a CVAT.ai user? Click through and sign up here‍Do not want to miss updates and news? Have any questions? Join our community:‍Facebook‍DiscordLinkedInGitterGitHub

Annotating data for machine learning is notoriously time-consuming, and striving for precision doesn't make it any easier. The industry—be it retail, automotive, medical imaging, or any other—doesn't change this fundamental need for quick, accurate data annotation.‍Now, let's explore a scenario (which might not be so hypothetical) where your ML model requires datasets annotated with masks. CVAT.ai has tool for it, but there's an even cooler feature that streamlines the process: annotation actions, particularly the shape converter.‍With this feature, you can initially use whatever annotation shape suits you best or is easiest for you—let's say polygons, for instance. This approach saves time, especially if you're more familiar with a specific tool or leveraging automatic annotation. Once you're done, you can easily convert all your annotations from masks to polygons with just a few clicks, ensuring both speed and accuracy in your work. You can also filter out and delete shapes that you do not need anymore.‍To see how this works in action, check out our latest video:‍‍The video covers the following topicsYou can use shapes converter in the retail sector. For example, ensuring that all products on shelves are monitored accurately for restocking requires uniform annotations. CVAT.ai allows quick conversion of different shapes to standardize data input, making it easier for models to learn and predict.In the automotive industry, getting the details right when marking street signs, pedestrians, and vehicles is crucial. CVAT.ai's shape converter assists in creating precisely annotated sets for these needs, enhancing the training of autonomous systems to navigate and understand the real world with greater accuracy.In the medical field, where the accurate analysis of diagnostic images can be a matter of life and death, CVAT.ai shape conversion tool allows flexible, precise annotations. This is vital for developing medical tools that can accurately diagnose conditions from medical imagery, enhancing research outcomes and patient care.ConclusionCVAT Annotations Actions simplify and improves the annotation process, making it faster and more efficient to prepare datasets for machine learning across various industries. This not only saves time but also improves the quality of data, leading to more reliable and effective AI models.‍Happy annotating!Not a CVAT.ai user? Click through and sign up here‍Do not want to miss updates and news? Have any questions? Join our community:‍Facebook‍DiscordLinkedInGitterGitHub

In the realm of digital annotation, not all projects are created equal. Some are straightforward, while others present intricate challenges – like the annotation of crowded images with overlapping objects. Our latest video tutorial delves into this complex task, demonstrating effective strategies using CVAT's advanced features.‍‍‍We tackled two types of tasks: one with still images and another with video conten to demonstrate CVAT's capabilities in managing complex scenarios. This groundwork allows us to showcase how CVAT's functionalities can be leveraged to bring order and clarity to crowded scenes.‍In the image annotation task, we encountered a scenario with numerous overlapping rectangles, posing the challenge of differentiating between objects. By adjusting settings like 'color by instance' and 'Selected opacity,' we were able to enhance visibility and distinguish each object with ease.‍Transitioning to video annotation, we explored how to efficiently track objects across multiple frames, even amidst crowded scenes. Key features like the 'Switch hidden property' and filtering options were used to maintain focus on specific objects, simplifying the tracking process.‍While these tips might seem straightforward, their impact on the accuracy and speed of annotation is significant. We invite you to watch our tutorial to see these strategies in action and experience the efficiency of CVAT firsthand.‍Happy annotating!Not a CVAT.ai user? Click through and sign up here‍Do not want to miss updates and news? Have any questions? Join our community:‍Facebook‍DiscordLinkedInGitterGitHub

IntroductionImplementing annotation crowdsourcing right can be a challenging task, especially for Computer Vision problems. While crowdsourcing offers huge benefits, such as unlimited scaling and low overhead, it also has fundamental challenges, such as low resulting annotation quality, difficult validation, and overwhelming assignment management. Now, we’d like to introduce our solution for annotation at scale, based on CVAT.ai and HUMAN Protocol integration. Read on to learn how you can use it for your datasets and for annotation.‍Overview‍The solution is an online platform based on HUMAN Protocol services, which uses CVAT for data annotation and Web3 technologies for payments in crypto currencies.‍On the platform, we bring together 2 key user roles: ‍Requesters - individuals who need their data annotated. AI model delevelopers, researchers, and AI challenge organizers all fall into this category.Annotators, workers - people who want to earn money by annotating data. This category includes people looking for a side hustle, freelancers, and professional data annotators.‍For Requesters, the platform provides automatic data annotation assignment preparation and management, automatic annotation validation and merging, and payments based on the annotation quality. Together these features create a place where you can just ask for an annotated dataset - describe the task, provide the data, quality requirements, and reward details - and get the data annotated with the specified quality after some time. No more manual control and management is needed. Keep in mind that currently the platform supports only several Computer Vision task types, but it can be extended in future.‍For Annotators, the platform is a place where they can start earning money in just several clicks. All they need to do is register, select a task, and ask for an assignment. Once they get an assignment, they can learn annotation requirements by reading the attached annotation guide, and start annotating - using one of the most popular open-source annotation tools in the Computer Vision community. Each assignment is small enough to be completed in a matter of minutes, allowing flexible participation for the workers. ‍Bounty payments are based on cryptocurrency; so annotators are required to have a crypto wallet to get the money. For requesters, however, there is also an option to pay with a bank card. Funds for payments are reserved at the time of task creation and are disbursed automatically after the entire dataset has been annotated and validated.‍Quick startIf you want to get your dataset annotated‍Before initiating a new annotation task, please be aware of the following platform requirements:‍A crypto wallet and a configured browser extension for wallet accessAn Amazon S3 data bucket configured for public accessA dataset with images in common formats (.jpg, .png, …), with at least 2 imagesSupport for only two task types: bounding box and single point annotations‍Examples:‍‍‍‍‍Only the MS COCO Object Detection / Keypoint Detection dataset format is supported, which is one of the most widely used formats in the industry. Note that points are encoded as 1-keypoint skeletons in the COCO Keypoint Detection formatAnnotations for ground truth (validation) images should be in the COCO .json file format‍Please keep in mind that simpler tasks tend to be easier and quicker to annotate. If your annotations require complex work, it can be a good idea to split the whole task into simpler subtasks, and annotate them separately for better quality and efficiency.‍Now, let’s go over the steps required to start an annotation task.‍ 1) Create a crypto wallet using one of the corresponding applications (e.g. MetaMask)‍Example for MetaMask in Chrome:‍ 1. Install the browser extension. 2. Open the extension page in the browser, create a password. 3. Select the Polygon Mainnet network or add it from the list. 4. Use an existing ETH-based wallet or click Add an account in MetaMask. 5. Rename the account to “Job Requester” or any other name you like. 6. Now, you'll have a wallet (ETH-based) in the network. It should show 0 MATIC on the balance. 7. Transfer some MATIC from another ETH-based wallet or buy (e.g. $5). Make sure to buy in the Polygon Mainnet network. 8. Click Import Tokens. Add the HMT token: Id: 0xc748B2A084F8eFc47E086ccdDD9b7e67aEb571BF Name: HMT‍ 9. Now, it should also show 0 HMT on the balance. 10. Convert some MATIC into HMT‍Now that you can perform payments in cryptocurrencies, it’s time to prepare your data and create an annotation task. ‍ 2) An AWS S3 bucket with public access is required. Сreate an AWS S3 bucket and upload your images as separate files into the bucket. Make sure to note the link to the directory containing the files within the bucket. You can obtain this URL by clicking Copy URL (the directory URL is needed).‍Example: 3) Select a small validation subset from the original images (e.g., 3%, 30 images from 1000), annotate it, and upload annotations in the COCO format to the bucket. You will only need a .json file. You can prepare such a dataset using CVAT or other tools. Remember to note the URL to this file as well.‍Example: ‍‍‍‍‍‍‍‍‍‍Using 0.1-5% of the images is recommended, depending on the dataset size. It is recommended to select just random images from the whole dataset. These images will appear randomly during the annotation process to check the annotation quality. The annotations produced by workers for these images are not included in the resulting annotated dataset; instead, the ground truth annotations are used.‍Please note that only 1-keypoint skeletons are supported for the COCO Keypoints format.‍ 4) Proceed to the platform and register a new account, if you don’t have one. 5) Click Create a Job.‍ 6) Select the payment method you prefer. You can pay via a crypto wallet or with a bank card. 7) Once the payment method is selected, you’ll see the job configuration page. Please select the CVAT job type and the Polygon Mainnet network.‍‍Let’s consider the fields one by one:Type of job: Bounding Boxes / PointsDescription: brief task description in 1-2 sentencesLabels: the list of label names (classes, categories) to be used during annotationData URL: a link to the bucket directory with dataset imagesGround truth URL: a link to the bucket file in .json format with GT annotations in COCO formatUser guide URL: a link to the full task description document with public access. Such documents describe annotation rules for the workers. You can share this document via the same S3 bucket, Google Docs or other similar servicesAccuracy target: the required accuracy target for annotations. The typical value range is 50-95% (keep in mind that normal human image classification accuracy is ~94%, and values from the 75-93 range are the most applicable. The metric used is IoU). 8) At the next step, you’ll need to enter the bounty details for workers. The resulting value is the price for the whole dataset. The bounties for the annotators will be calculated automatically based on the number of images annotated.‍Here you can choose to pay in crypto currency or with a bank card. The money will be reserved until the whole dataset is annotated.‍ 9) Once you’re ready, click Pay now.‍ 10) If everything is set up correctly, after the next several minutes, the job will appear in the job list. ‍In the list, you can check the current annotation status and details of the jobs you have created. Now, the annotators will look for assignments and begin their work. The process is fully autonomous for you, so you will only need to wait for the process to finish. It can take some time, depending on the current market prices, available worker pool, and other conditions. The workers will be able to join the task and get an assignment via this platform link.‍If, at some point, you decide to cancel a job, here you can find a button for this. The money you reserved for the work will get back to you.‍‍‍ 11) Once the data is annotated, the final annotations will be validated and merged automatically, and the result will be available by the URL you receive in the job details section.‍If you want to earn money with image annotation and video annotation‍1. Create a crypto wallet (check the explanation in the requester part of this guide). 2. Go to the annotator platform and register. You’ll need to pass a mandatory KYC procedure to finish the registration process, this is required by the applicable laws. 3. Currently, we require users to explicitly request participation in CVAT labeling. Please email us at app@humanprotocol.org to express your interest.4. On the platform, open the list of available CVAT annotation tasks:‍‍You will see the list of open annotation tasks, and find assignment details such as bounty per assignment, brief task description, the size (i.e. the number of images) to be annotated, and the annotation type (bounding boxes. points, etc.):‍5. To join a task, press the button on the right side of the task entry:‍‍Once it is done, switch to the “My Jobs” tab, and you should see your assignment for this task:‍6. To start annotation, press the “Go to job” button in the “Job URL” column‍You will be navigated to the CVAT job, where you can draw annotations.‍7. Make sure to check the job requirements by clicking on the Guide button:‍‍‍‍‍‍8. Draw annotations as required in the task, and move between the job frames:‍‍‍9. When all the job frames are finished, click Save.10. Switch the job state to Completed.Implementation details‍In this section, we’ll discuss several implementation details to give you a clearer understanding of how everything works together. At the heart of the project lie three key components:‍A tool that allows annotators to create high-quality annotationsA platform that manages assignments, tasks, and dataA contract that specifies the job, acceptance criteria, and bounty‍All these components contribute to solving the crowdsourcing platform challenges, improving the overall efficiency of the system, and providing an effective solution, when combined. Now, let’s look closer at the specific problems and solutions we implemented in the platform.‍Task creation and assignment management. The platform handles the datasets and worker assignments in an automatic way, which is measurable, consistent, and reliable. The platform splits the dataset given into small chunks (the assignments), allowing to annotate data efficiently in parallel. Each annotator gets a fixed amount of work in an assignment, and each assignment can be validated with no human interaction. As there is no human factor in the assignment creation or management, the process of getting an assignment is quick and is protected from occasional errors. ‍Assignment validation. Once an assignment is annotated, it goes to validation. There are several validation strategies used in the industry, including consensus scoring, honey pots/ground truth, model validation, etc. In the current implementation, we decided to use ground truth-based scoring. With this approach, the task requester is required to provide a small number of ground truth annotations during task creation. Each assignment includes several images from the validation set so we can always check the quality of the annotations in the assignment. Then, we extrapolate the resulting quality for the whole assignment. If the assignment quality is below the required level, the annotations are discarded, and the assignment is sent for reannotation.‍‍Payments. When the task is created, the requester configures the bounty and the money is reserved from their wallet. The platform relies on Smart Contracts in blockchain to guarantee fair payments and to allow cancellation of tasks. Each contract gets a clear definition of the work required to receive the bounty, which can be checked automatically. The workers are rewarded for each assignment completed, after the whole dataset quality is accepted during validation.‍‍‍‍‍Not a CVAT.ai user? Click through and sign up here‍Do not want to miss updates and news? Have any questions? Join our community:‍Facebook‍DiscordLinkedInGitterGitHub‍‍

Data annotation is a critical step in the development of machine learning models. However, manual annotation can be time-consuming specifically for big datasets. What if you could automate this process and therefore make it faster? And with any model? ‍In CVAT.ai it is possible with CVAT CLI. But before diving into the technical details, let’s set up the basic understanding. ‍The CVAT CLI leverages Software Development Kit (CVAT SDK) to auto-annotate, or pre-annotate your dataset, allowing you to focus more on model development and less on data preparation. ‍The SDK enables you to incorporate functionalities from a variety of machine learning libraries. Including torchvision, but you can use others. The SDK provides you with a range of options for automated annotation, also known as Auto-Annotation (AA) functions.‍What are AA Functions?‍Auto-Annotation, or AA, functions, are Python objects designed to perform specific annotation tasks. These functions translate your raw data into annotations. ‍A typical AA function generally includes the following components: Code to load the machine learning model.A specification outlining the types of annotations that can be generated.Code to transform CVAT data into a format the machine learning model understands.Code to run the model to obtain predictions.Code to convert predicted annotations back into a CVAT-friendly format.‍The CVAT SDK is built on a layered architecture comprising several parts:‍The Interface: Defines the protocol that any AA function must implement.The Driver: Manages the execution of AA functions and performs the actual annotation on the CVAT dataset. Predefined AA Functions: Includes a set of predefined functions.‍This is just a glimpse; the following article will walk you through the steps and specifics to get you started on your automated annotation journey.‍There are two ways to auto annotate using the CVAT CLI:‍Annotating with predefined Auto-Annotation Functions in CVAT SDK.Annotating with your own Auto-Annotation Function‍Before starting the annotation‍Before starting the annotation process let’s set up a task in CVAT Cloud. In this case it is a simple dataset with animals and labels “cat” and “dog”.‍CVAT screen with image for annotation‍For both cases, first we need to create an environment where we could run the function. Let's begin by installing a few Python packages on the local machine. Please note that commands might vary for different operating systems. For the sake of this article, all commands that we use are for Windows.‍Run the following command:‍python -m venv venv‍When the virtual environment is ready, you will need to activate it:‍.\venv\Scripts\Activate.ps1‍The next step is to install the CVAT.ai CLI. Execute the command and wait for the installation to complete.‍pip install cvat-sdk[pytorch] cvat-cli‍To allow CVAT CLI access to CVAT, you'll need to store your CVAT password in the PASS environment variable. We'll utilize the Read-Host command here to prevent the password from being displayed.‍$ENV:PASS = Read-Host -MaskInput‍Enter your CVAT password and hit Enter. Now you are ready to run the automatic annotation.‍Easy Guide to Using Predefined Auto-Annotation Functions in CVAT SDK ‍You can auto-annotate with its two functions that utilize models from the torchvision library.The CVAT SDK includes two predefined AA functions. Each function is implemented as a module to allow usage through the CLI auto-annotate command. ‍After you’ve installed Python and environment is ready, run an Automatic Annotation from CLI we will use the following command:‍cvat-cli auto-annotate "<task ID>" --function-module cvat_sdk.auto_annotation.functions.torchvision_detection \ -p model_name=str:"<model name>" ...‍Let’s come back to the task that was created earlier. To run the function you will need a host, task ID, and username. For the model name check the torchvision documentation. In the example below we’ll use fcos_resnet50_fpn.‍The score_thresh=float:0.7 parameter is used to specify the threshold for object detection confidence scores. In this case, it's setting the confidence score threshold to 0.7, meaning that only object detections with a confidence score greater than or equal to 0.7 will be included in the results of the auto-annotation process. Objects with lower confidence scores will be filtered out. ‍CVAT screen showing where to get all parameters‍‍With these elements added to the command, you will get the following result:‍cvat-cli --server-host app.cvat.ai --auth mk auto-annotate 274373 --function-module cvat_sdk.autocvat_sdk.auto_annotation.functions.torchvision_detection -p model_name=str:fcos_resnet50_fpn -p score_thresh=float:0.7 --allow-unmatched-labels ‍Where app.cvat.ai is the host, 274373 is the task ID, and mk is the username.‍By default, the CLI will check that every label that the function can output exists in the task. In this case, our task only has "cat" and "dog" labels, while the function can output 80 labels in total, --allow-unmatched-labels tells the CLI to ignore all labels that don't exist in the task.‍It's a good practice to start with a clean state. So if there are any annotations that were done before, you can add –-clear-existing option the command, that will clear all existing annotations. ‍The annotation will start. Wait until it’s over, then go back to the task. You might need to refresh the page for annotations to be visible.‍CVAT annotated image‍It's time to check the quality. Go through the dataset to ensure that the annotations meet your requirements.‍How to Auto-Annotate Your Dataset with Model of Choice and the Command Line Interface‍The second method is when you use the auto-annotation feature not with predefined functions but with any model of your choice. In this guide, we'll walk through using YOLO v8 for auto-annotation via the Command Line Interface (CLI). Here is the task that will be annotated:‍CVAT with image to be annotated‍When the environment is ready, you can run a model function. Something like this:‍import PIL.Image from ultralytics import YOLO ​ import cvat_sdk.auto_annotation as cvataa import cvat_sdk.models as models ​ _model = YOLO("yolov8n.pt") ​ spec = cvataa.DetectionFunctionSpec( labels=[cvataa.label_spec(name, id) for id, name in _model.names.items()], ) ​ def _yolo_to_cvat(results): for result in results: for box, label in zip(result.boxes.xyxy, result.boxes.cls): yield cvataa.rectangle(int(label.item()), [p.item() for p in box]) ​ def detect(context, image): return list(_yolo_to_cvat(_model.predict(source=image, verbose=False))) open_in_new MORE content_copy COPY @cvataicode at thiscodeWorks.com‍To move to the next step, you'll need to install the Ultralytics library, which houses the YOLO models. To do it, execute the following command. Wait for the installation to finish.‍pip install ultralytics‍It's a good practice to start with a clean slate. For this purpose, the –-clear-existing option is added to the command, which will clear all existing annotations. ‍Note that you’ll need to specify the path to the file implementing the function. ‍You can also exclude labels that you don't need.‍Here’s how you'd run the command in the CLI:‍cvat-cli --server-host app.cvat.ai --auth mk auto-annotate 274373 --function-file .\yolo8.py --allow-unmatched-labels –-clear-existing‍Press Enter and wait for the auto-annotation by YOLO8 to be accomplished.‍Once the auto-annotation is complete, it's time to check the quality. Go through the dataset to ensure that the annotations meet your requirements.‍CVAT annotated image‍There you have it! Now you know how to use any model, including YOLO v8, to auto-annotate your dataset via the CLI. Using auto-annotate can save you a tremendous amount of time and help you achieve consistent annotation across your datasets. If you have more questions, please see Auto Annotation documentation.‍And check the video to see the full process:‍‍Remember to like, share, and subscribe for more updates!Happy Annotating!‍‍Not a CVAT.ai user? Click through and sign up here‍Do not want to miss updates and news? Have any questions? Join our community:‍Facebook‍DiscordLinkedInGitterGitHub‍‍

Ensuring top-quality annotations in Computer Vision Annotation Tool (CVAT) is simpler than you think. Whether you're a project owner, an annotator, or a QA specialist, our platform makes the process seamless.‍Watch the tutorial and read on to discover how to navigate this critical aspect of machine learning and video annotation.‍‍‍‍Setting Up Your Project‍The first step is initializing your annotation project. After creating a Project and adding a Task, you assign Jobs to your Annotators. These jobs contain images for annotation. In our demonstration video, we've intentionally introduced errors for educational purposes—such as labeling "dogs" as "cats".‍Switching Roles for Quality Assurance (QA)‍When the annotator has completed their tasks, it's time for Quality Assurance. To show how this works, we'll switch back to the Project owner's account to initiate the QA process.Assigning a QA specialist to review the annotations is a breeze. Just invite the person to your project and assign them to the specific job. Then change the status of the Job to "Validation".‍Review and Issue Tracking‍The person assigned as QA will log in and have access to the QA interface which has been designed specifically for issues reporting and tracking. It lacks the typical annotation tools but includes an "Issue tracker" icon.‍QA will go through each annotation to identify errors. Once found, QA creates an issue and submits it. CVAT also provides predefined issues for common errors, saving time and ensuring consistency.‍Navigating and Resolving Issues‍After the QA specialist completes their review, we’ll go back to the annotator’s account and interface to see how the reported issues look. The annotator can easily navigate through the list of issues and correct the errors. After all is done, the annotator saves the work, making the annotations complete and ready for future use. And that’s it!‍Happy Annotating!Not a CVAT.ai user? Click through and sign up hereDo not want to miss updates and news? Have any questions? Join our community:‍Facebook‍DiscordLinkedInGitterGitHub‍‍

In the fast-paced world of data annotation, striking the perfect balance between speed and accuracy is essential. With complex datasets and strict deadlines becoming the norm, annotation professionals are constantly seeking solutions to streamline their workflow without compromising the quality. ‍This is where the power of Layers in CVAT.ai comes into play.‍Understanding the Challenge‍Imagine having to annotate a dataset that features intricate objects or multiple subjects in each image. Traditional annotation methods might force you to choose between speed and accuracy – a decision that can have significant implications on the overall quality of your work.‍Introducing Layers ‍Layers in CVAT.ai improve the way you approach annotation tasks. Whether you're dealing with multi-object images, complex scenes, or projects with strict timelines.‍By allowing you to separate objects or subjects into distinct layers, CVAT.ai lets you focus on annotating individual elements without the clutter of overlapping annotations. This focused approach translates into increased efficiency as you no longer need to be worried about gaps between annotated objects and you also reduce the number of objects to be annotated overall.‍Want to know how to do it? Check out our latest video!‍‍Share your opinion and stay tuned!‍Happy annotating!‍Do not want to miss updates and news? Have any questions? Join our community:‍Facebook‍DiscordLinkedInGitterGitHub‍‍

Every day, the vast world of artificial intelligence (AI) becomes increasingly interconnected with our lives. In essence, the backbone of AI technology is data. More specifically, for AI to understand and interpret the visual world around us, we need data in the form of images. These images must be labeled or annotated, turning them from raw pixels into a language that AI can understand. This process, called image annotation, is an integral part of AI development.However, image annotation is not simply a case of attaching labels to pictures. It requires meticulous work to ensure the data is labeled correctly and consistently. This is where the importance of image annotation quality comes into play.‍Imagine you have a dataset full of labeled images. But how do we know the labels are accurate? How can we be sure that these labels are reliable enough to train AI models? To answer these questions, we need to assess the quality of our data. Fortunately, in CVAT, an image labeling tool specifically designed for such tasks, there's a simple way to do this using a method known as the 'Honeypot'.‍‍The Magic of CVAT Honeypot ‍n the world of image annotation, quality is king. That's where CVAT and its Honeypot method come in. The Honeypot method is all about comparing actual annotations with a 'ground truth', or known correct annotations. This ground truth is set up in a unique job within CVAT.‍Worried about double-annotating an entire dataset for the ground truth? Fret not. Just a fraction of images, say 5-15%, is enough to give an estimate of the overall quality. The size of the 'ground truth' job is flexible, you can set it as a specific number or a percentage of frames .‍These 'ground truth' jobs are different from regular jobs. They don’t mingle with your main dataset, whether it's exporting, importing, or automatic annotation. And if you ever need to tweak parameters, you can delete the ground truth job and create a fresh one.‍Once your ground truth job is complete, annotate the dataset and let CVAT crunch the data. Once processed, all the information will appear on the Task Analytics page which is dedicated to showing annotation quality results. There you'll find your task's quality score, including the average annotation quality, the number of conflicts and issues, and a per-job breakdown. For a closer look, you can always download the detailed report for the task or for each job.‍And if you need to customize quality score requirements? CVAT's got you covered. You can set parameters, for example what counts as a 'bad' overlap and others. Once set, these will be applied in the next quality check. So, there you have it. Ensuring high-quality annotations is a breeze with CVAT and the Honeypot feature.‍Check the video about this new feature:‍‍‍Thank you for choosing CVAT!‍Stay tuned and follow the news here:‍Facebook‍DiscordLinkedInGitterGitHub‍‍

Animal classification is the process of categorizing different species of animals based on their physical and biological characteristics. ‍Here, when we say physical biological charastetrisitcs, we mean:1. Symmetry: radial like a starfish or bilateral like a butterfly?2. Body plan: does it have a backbone? Is it covered in fur or scales?3. Reproduction: does it lay eggs or is it some kind of internal fertilization?4. Metabolism: is it a herbivore or carnivore? Or something else?And many more. ‍By studying these characteristics, scientists can better understand the evolutionary relationships between different species and how they have adapted to different environments over time. They can aslo see the changes in the behavior and appearance of the animals by checking the data from different time periods. Based on this information, scientists make conclusions and provide recommendations for ecological improvements that can benefit both endangered and non-endangered species alike.‍So animal classification is something really important. And challenging. As it requires both: collection and processing of Information. For this very reason it is also very time consuming and costly: the animal kingdom is big and so is the volumes of the collected data. ‍Image annotation can help with animal classification by providing a way to analyze large amounts of visual data quickly and efficiently. ‍The procedure is straightforward: assign labels, such as bird, starfish, bear, or zebra. When necessary, add attributes like the presence of a backbone, radial or bilateral symmetry, or even the gender of the subject. Once completed, export the annotated dataset and apply the machine learning model to it. This will classify animals based on the provided labels, resulting in a quicker and more accurate animal classification procedure.To make the process of adding classification labels easier, ecologists use different tools and one of them is CVAT (Computer Vision Annotation Tool).‍ Here is the short video describing the whole process step-by-step:‍We are waiting for your feedback here:DiscordLinkedInGitterGitHub‍You can find more information at our YouTube channel

IntroductionCVAT is a visual data annotation tool. Using it, you can take a set of images and mark them up with annotations that either classify each image as a whole, or locate specific objects on the image.But let’s suppose you’ve already done that. What now?Datasets are, of course, not annotated just for the fun of it. The eventual goal is to use them in machine learning, for either training an ML model, or evaluating its performance. And in order to do that, you have to get the annotations out of CVAT and into your machine learning framework.Previously, the only way to do that was the following:1. Export your CVAT project or task in one of the several dataset formats supported by CVAT.2. Write code to read the annotations in the selected format and convert them into data structures suitable for your ML framework.‍This approach is certainly workable, but it does have several drawbacks:The third-party dataset formats supported by CVAT cannot necessarily represent all information that CVAT datasets may contain. Therefore, some information can be lost when annotations are exported in such formats. For example, CVAT supports ellipse-shaped objects, while the COCO format does not. So when a dataset is exported into the COCO format, ellipses are converted into masks, and information about the shape is lost.Even when a format can store the necessary information, it may not be convenient to deal with. For example, in the COCO format, annotations are saved as JSON files. While it is easy to load a generic JSON file, data loaded in this way will not have static type information, so features like code completion and type checking will not be available.Dataset exporting can be a lengthy process, because the server has to convert all annotations (and images, if requested) into the new format. If the server is busy with other tasks, you may end up waiting a long time.If the dataset is updated on the server, you have to remember to re-export it. Otherwise, your ML pipeline will operate on stale data.All of these problems stem from one fundamental source: the use of an intermediate representation. If we could somehow use data directly from the server, they would be eliminated.So, in CVAT SDK 2.3.0, we introduced a new feature that will, for some use cases, implement exactly that. This feature is the cvat_sdk.pytorch module, also informally known as the PyTorch adapter. The functionality in this module allows you to directly use a CVAT project or task as a PyTorch-compatible dataset.Let’s play with it and see how it works.SetupFirst, let’s create a Python environment and install CVAT SDK. To use the PyTorch adapter, we’ll install the SDK with the pytorch extra, which pulls PyTorch and torchvision as dependencies. We won’t be using GPUs, so we’ll get the CPU-only build of PyTorch to save download time.‍$ python3 -mvenv ./venv $ ./venv/bin/pip install -U pip $ ./venv/bin/pip install 'cvat_sdk[pytorch]' \ --extra-index-url=https://download.pytorch.org/whl/cpu $ . ./venv/bin/activate‍Now we will need a dataset. Normally, you would use the PyTorch adapter with your own annotated dataset that you already have in CVAT, but for demonstration purposes we’ll use a small public dataset instead.‍To follow along, you will need an account on the public CVAT instance, app.cvat.ai. If you have access to a private CVAT instance, you can use that instead. Save your CVAT credentials in environment variables so CVAT SDK can authenticate itself:‍$ export CVAT_HOST=app.cvat.ai $ export CVAT_USER='<your username>' CVAT_PASS $ read -rs CVAT_PASS <enter your password and hit Enter>‍The dataset we’ll be using is the Flowers Dataset available in the Harvard Dataverse Repository. This dataset is in an ad-hoc format, so we won’t be able to directly import it into CVAT. Instead, we’ll upload it using a custom script. We won’t need the entire dataset for this demonstration, so the script will also reduce it to a small fraction.‍Get that script from our blog repository and run it:‍$ python3 upload-flowers.py‍The script will create tasks for the train, test and validation subsets, and print their IDs. If you open the Tasks page, you will see that the tasks have indeed been created:‍‍And if you open any of these tasks and click the “Job #XXXX” link near the bottom, you will see that each image has a single annotation associated with it: a tag representing the type of the flower.‍‍Interactive usage‍Note: the code snippets from this section are also available as a Jupyter Notebook.We’re now ready to try the PyTorch adapter. Let’s start Python and create a CVAT API client:‍$ python3 Python 3.10.6 (main, Nov 14 2022, 16:10:14) [GCC 11.3.0] on linux Type "help", "copyright", "credits" or "license" for more information. >>> import logging, os >>> from cvat_sdk import * >>> # configure logging to see what the SDK >>> # is doing behind the scenes >>> logging.basicConfig(level=logging.INFO, format='%(levelname)s - %(message)s') >>> client = make_client(os.getenv('CVAT_HOST'), credentials=( os.getenv('CVAT_USER'), os.getenv('CVAT_PASS')))‍Now let’s create a dataset object corresponding to our training set. To follow along, you will need to substitute the task ID in the first line with the ID of the Flowers-train task that was printed when you ran the upload-flowers.py script.‍>>> TRAIN_TASK_ID = 77708 >>> from cvat_sdk.pytorch import * >>> train_set = TaskVisionDataset(client, TRAIN_TASK_ID) INFO - Fetching task 77708... INFO - Task 77708 is not yet cached or the cache is corrupted INFO - Downloading data metadata... INFO - Downloaded data metadata INFO - Downloading chunks... INFO - Downloading chunk #0... INFO - Downloading chunk #1... INFO - Downloading chunk #2... INFO - Downloading chunk #3... INFO - Downloading chunk #4... INFO - All chunks downloaded INFO - Downloading annotations... INFO - Downloaded annotations‍As you can see from the log, the SDK has downloaded the data and annotations for our task from the server. All subsequent operations on train_set will not involve network access.‍But what is train_set, anyway? Examining it will reveal that it is a PyTorch Dataset object. Therefore we can query the number of samples in it and index it to retrieve individual samples.‍>>> import torch.utils.data >>> isinstance(train_set, torch.utils.data.Dataset) True >>> len(train_set) 354 >>> train_set[0] ( <PIL.JpegImagePlugin.JpegImageFile image mode=RGB size=320x263 at 0x7F4CFE7E52D0>, Target( annotations=FrameAnnotations(tags=[ { 'attributes': [], 'frame': 0, 'group': None, 'id': 426655, 'label_id': 431494, 'source': 'manual' } ], shapes=[]), label_id_to_index=mappingproxy({431492: 0, 431493: 1, 431494: 2, 431495: 3, 431496: 4}) ) )‍The sample format is broadly compatible with that used by torchvision datasets. Each sample is a tuple of two elements:The first element is a PIL.Image object.The second element is a cvat_sdk.pytorch.Target object representing the annotations corresponding to the image, as well as some associated data.The annotations in the Target object are instances of LabeledImage and LabeledShape classes from the CVAT SDK, which are direct representations of CVAT’s own data structures. This means that any properties you can set on annotations in CVAT — such as attributes & group IDs — are available for use in your code.In this case, though, we don’t need all this flexibility. After all, the only information contained in the original dataset is a single class label for each image. To serve such simple scenarios, CVAT SDK provides a couple of transforms that reduce the target part of the sample to a simpler data structure. For this scenario (image classification with one tag per image), the transform is called ExtractSingleLabelIndex. Let’s recreate the dataset with this transform applied:>>> train_set = TaskVisionDataset(client, TRAIN_TASK_ID, target_transform=ExtractSingleLabelIndex()) INFO - Fetching task 77708... INFO - Loaded data metadata from cache INFO - Downloading chunks... INFO - All chunks downloaded INFO - Loaded annotations from cache‍Note that the task data was not redownloaded again, as it had already been cached. The SDK only made one query to the CVAT server, in order to see if the task had changed.Here’s what the sample targets look like with the transform configured:>>> for i in range(3): print(train_set[i]) ... (<PIL.JpegImagePlugin.JpegImageFile image mode=RGB size=320x263 at 0x7F4CFE7E5720>, tensor(2)) (<PIL.JpegImagePlugin.JpegImageFile image mode=RGB size=320x213 at 0x7F4CFE7E56C0>, tensor(0)) (<PIL.JpegImagePlugin.JpegImageFile image mode=RGB size=500x330 at 0x7F4CFE7E5720>, tensor(4))‍Each target is now simply a 0-dimensional PyTorch tensor containing the label index. These indices are automatically assigned by the SDK. You can also use these indices without applying the transform; they are provided by the label_id_to_index field on the Target objects.‍‍ExtractSingleLabelIndex requires each sample to have a single tag. If a sample fails this requirement, the transform will raise an exception when that sample is retrieved. ‍Our dataset is now almost ready to be used for model training, except that we’ll also need to transform the image, as PyTorch cannot directly accept a PIL image as input. torchvision supplies a variety of transforms to convert and postprocess images, which can be applied using the transform argument. For example:‍>>> import torchvision.transforms as transforms >>> train_set = TaskVisionDataset(client, TRAIN_TASK_ID, transform=transforms.ToTensor(), target_transform=ExtractSingleLabelIndex()) INFO - Fetching task 77708... INFO - Loaded data metadata from cache INFO - Downloading chunks... INFO - All chunks downloaded INFO - Loaded annotations from cache >>> train_set[0] (tensor([[[0.5294, 0.5412, 0.5569, ..., 0.6000, 0.6118, 0.5804], [0.5255, 0.5373, 0.5529, ..., 0.6000, 0.6118, 0.5804], [0.5216, 0.5333, 0.5529, ..., 0.6000, 0.6078, 0.5725], [...snipped...] [0.1020, 0.1020, 0.1020, ..., 0.4980, 0.4980, 0.4980]]]), tensor(2))‍Full model training & evaluation exampleEquipped with the functionality that we just covered, we can now plug a CVAT dataset into a PyTorch training/evaluation pipeline and have it work the same way it would with any other dataset implementation.‍Programming an entire training pipeline in interactive mode is a bit cumbersome, so instead we published two example scripts that showcase using a CVAT dataset as part of a simple ML pipeline. You can get these scripts from our blog repository.‍The first script trains a neural network (specifically ResNet-34, provided by torchvision) on our sample dataset (or any other dataset with a single tag per image). You run it by passing the training task ID as an argument:‍$ python3 train-resnet.py 77708 2023-02-03 16:55:17,268 - INFO - Starting... 2023-02-03 16:55:18,623 - INFO - Created the client 2023-02-03 16:55:18,623 - INFO - Fetching task 77708... 2023-02-03 16:55:18,867 - INFO - Loaded data metadata from cache 2023-02-03 16:55:18,867 - INFO - Downloading chunks... 2023-02-03 16:55:18,869 - INFO - All chunks downloaded 2023-02-03 16:55:18,901 - INFO - Loaded annotations from cache 2023-02-03 16:55:19,103 - INFO - Created the training dataset 2023-02-03 16:55:19,104 - INFO - Created data loader 2023-02-03 16:55:20,407 - INFO - Started Training 2023-02-03 16:55:20,407 - INFO - Starting epoch #0... 2023-02-03 16:55:32,451 - INFO - Starting epoch #1... 2023-02-03 16:55:44,086 - INFO - Finished training‍It saves the resulting weights in a file named weights.pth. The evaluation script will read these weights back and evaluate the network on a validation subset—which you, again, specify via a CVAT task ID:‍$ # this script uses the torchmetrics library to calculate accuracy $ pip install torchmetrics $ python3 eval-resnet.py 77709 2023-02-03 16:58:32,745 - INFO - Starting... 2023-02-03 16:58:33,669 - INFO - Created the client 2023-02-03 16:58:33,669 - INFO - Fetching task 77709... 2023-02-03 16:58:33,887 - INFO - Task 77709 is not yet cached or the cache is corrupted 2023-02-03 16:58:33,889 - INFO - Downloading data metadata... 2023-02-03 16:58:34,107 - INFO - Downloaded data metadata 2023-02-03 16:58:34,108 - INFO - Downloading chunks... 2023-02-03 16:58:34,109 - INFO - Downloading chunk #0... 2023-02-03 16:58:34,873 - INFO - All chunks downloaded 2023-02-03 16:58:34,873 - INFO - Downloading annotations... 2023-02-03 16:58:35,166 - INFO - Downloaded annotations 2023-02-03 16:58:35,362 - INFO - Created the testing dataset 2023-02-03 16:58:35,362 - INFO - Created data loader 2023-02-03 16:58:35,749 - INFO - Started evaluation 2023-02-03 16:58:36,355 - INFO - Finished evaluation Accuracy of the network: 80.00%‍Since training involves randomness, you may end up seeing a slightly different accuracy number.‍Working with objectsNote: the code snippets from this section are also available as a Jupyter Notebook.‍The PyTorch adapter also contains a transform designed to simplify working with object detection datasets. First, let’s see how raw CVAT shapes are represented in the CVAT SDK.‍Open the Flowers-train task, click on the “Job #XXX” link, open frame #2, and draw rectangles around some sunflowers:‍‍Press “Save”. Now, restart Python and reinitialize the client:‍>>> import logging, os >>> from cvat_sdk import * >>> from cvat_sdk.pytorch import * >>> logging.basicConfig(level=logging.INFO, format='%(levelname)s - %(message)s') >>> client = make_client(os.getenv('CVAT_HOST'), credentials=( os.getenv('CVAT_USER'), os.getenv('CVAT_PASS'))) >>> TRAIN_TASK_ID = 77708‍Create the dataset again:‍>>> train_set = TaskVisionDataset(client, TRAIN_TASK_ID) INFO - Fetching task 77708... INFO - Task has been updated on the server since it was cached; purging the cache INFO - Downloading data metadata... INFO - Downloaded data metadata INFO - Downloading chunks... INFO - Downloading chunk #0... [...snipped...] INFO - All chunks downloaded INFO - Downloading annotations... INFO - Downloaded annotations‍Note that since we have changed the task on the server, the SDK has redownloaded it.‍Now let’s examine the frame that we modified:‍>>> train_set[2] ( <PIL.JpegImagePlugin.JpegImageFile image mode=RGB size=500x330 at 0x7F98AD088220>, Target( annotations=FrameAnnotations( tags=[ { 'attributes': [], 'frame': 2, 'group': None, 'id': 426657, 'label_id': 431496, 'source': 'manual' } ], shapes=[ { 'attributes': [], 'elements': [], 'frame': 2, 'group': 0, 'id': 41000665, 'label_id': 431496, 'occluded': False, 'outside': False, 'points': [ 170.1162758827213, 158.9655911445625, 349.43134126663244, 329.23956079483105 ], 'rotation': 0.0, 'source': 'manual', 'type': 'rectangle', 'z_order': 0 }, [...snipped...] ] ), label_id_to_index=mappingproxy({431492: 0, 431493: 1, 431494: 2, 431495: 3, 431496: 4}) ) )‍You can see the newly-added rectangles listed in the shapes field. As before, the values representing the rectangles contain all the properties that are settable via CVAT.‍Still, if you’d prefer to work with a simpler representation, there’s a transform for you: ExtractBoundingBoxes.‍>>> train_set = TaskVisionDataset(client, TRAIN_TASK_ID, target_transform=ExtractBoundingBoxes( include_shape_types=['rectangle'])) INFO - Fetching task 77708... INFO - Loaded data metadata from cache INFO - Downloading chunks... INFO - All chunks downloaded INFO - Loaded annotations from cache >>> train_set[2] ( <PIL.JpegImagePlugin.JpegImageFile image mode=RGB size=500x330 at 0x7F98C414B7F0>, { 'boxes': tensor([ [170.1163, 158.9656, 349.4313, 329.2396], [255.2533, 59.5135, 458.6779, 256.9108], [117.3765, 115.2670, 240.9382, 253.8971] ]), 'labels': tensor([4, 4, 4]) } )‍The output of this transform is a dictionary with keys named “boxes” and “labels”, and tensor values. The same format is accepted by torchvision’s object detection models in training mode, as well as the mAP metric in torchmetrics. So if you want to use those components with CVAT, you can do so without additional conversion.‍Closing remarksThe PyTorch adapter is still new, so it has some limitations. Most notably, it does not support track annotations and video-based datasets. Still, we hope that even in its early stages it can be useful to you.‍Meanwhile, we are working on extending the functionality of the adapter. The development version of CVAT SDK already features the following additions:‍A ProjectVisionDataset class that lets you combine multiple tasks in a CVAT project into a single dataset.Ability to control the cache location.Ability to disable network usage (provided that the dataset has already been cached).If you have suggestions for how the adapter may be improved, you’re welcome to create a feature request on CVAT's issue tracker.‍‍

Object detection is a field within Computer Vision that involves identifying and locating objects within an image. Advances in object detection algorithms have made it possible to detect objects in real-time as they are moving.‍There are a number of Object detection technologies available. One of the most popular ones to date is the YOLO object detector. Currently, YOLO is almost the gold standard algorithm for Object detection, owing to its high speed. As such, it finds widespread applications in a number of crucial areas like security and surveillance, traffic management, autonomous vehicles, as well as healthcare.In this article, we will learn how YOLO works and how you can use it to annotate images in CVAT automatically. ‍A brief history of YOLOEarly object detectors were mainly region-based. They used a 2-step process to detect objects. In the first step, these algorithms proposed regions of interest that are likely to contain objects. In the second step, they classified the images in these proposed regions.‍Some of the popular region-based algorithms include:R-CNNFast R-CNNFaster R-CNNRFCNMask R-CNN‍Region-based detectorsR-CNN (Regions with CNN features) was the first region-based object detector, proposed in the year 2014. This detector used a process of selective search to cluster similar pixels into regions and generate a set of region proposals. These regions were then fed into a Convolutional Neural Network (CNN) to generate a feature vector. The feature vector was then used to classify and put a bounding box around detected objects. Besides other limitations, this algorithm proved to be quite time-intensive.‍So, it was succeeded by the Fast R-CNN detector, which put the whole image through a CNN and used ROI pooling to extract the region proposals. The feature vectors thus generated were passed through several fully connected layers for classification and bounding box regression. Although this was faster than R-CNN, it was still not fast enough, as it still required selective search.‍The Faster R-CNN detector drastically speeded up the detection process by getting rid of the selective search approach and using a Region Proposal Network instead. This network used an ‘objectness score’ to produce a set of object proposals. The objectness score indicates how confident the network is that a given region contains an object. ‍Another approach used was the R-FCN detector, which used position-sensitive score maps. ‍All the above methods required 2 steps to detect objects in an image:‍Detect the object regionsClassify the objects in those regions‍This 2-step process made the object detection process quite slow. A more sophisticated approach was required if object detection was to be used in real-time applications. ‍Emergence of YOLOThe YOLO algorithm was first proposed by Joseph Redmond et al in 2015. In contrast to earlier Object Detection algorithms, YOLO does not use regions to find objects in a given image. Neither does it require multiple iterations over the same image. ‍It passes the entire image through a Convolutional Neural Network that simultaneously locates and classifies objects in one go. That is how the algorithm gets its name (You Only Look Once).‍This approach enables the algorithm to achieve substantially better results than other object detection algorithms. ‍How YOLO worksThe YOLO algorithm divides the image into an NxN grid of cells (typically it is 19X19). It then finds B bounding boxes in each cell of the grid. For each bounding box, the algorithm finds 3 things: The probability that it contains an objectThe offset values for the bounding box corresponding to that objectThe most likely class of the object‍After this, the algorithm selects only the bounding boxes that most certainly contain an object. ‍IoU (Intersection Over Union)The YOLO algorithm uses a measure called the IoU to determine how close the detected bounding box is to the actual one.The IoU is a measure of the overlap between two bounding boxes. During training, the YOLO algorithm computes the IoU between the bounding box predicted by the network and the ground truth (the bounding box that was pre-labeled for training). ‍It is calculated as follows:IoU = area of intersection of the overlapping boxes / area of union of all the overlapping boxes‍An IoU of 1 means that both bounding boxes completely overlap one another, whereas an IoU of 0 means that the two bounding boxes are completely distinct. A threshold for the IoU is fixed, and only those bounding boxes that have an IoU above the threshold value are retained, while others are ignored. This helps eliminate a lot of unnecessary bounding boxes so that you’re left with only the ones that best fit the object.‍Non-Maximum SuppressionDuring the testing phase, since there are a number of cells detecting the same object, it is possible to be left with several bounding boxes corresponding to the same object. YOLO takes care of this by using a technique called Non-maximum suppression.Non-max suppression involves first selecting the bounding box with the highest probability score and removing (suppressing) all other boxes that have a high overlap with this box. This again makes use of the IoU, this time between all the candidate bounding boxes and the one with the highest probability score.‍‍Bounding boxes that have a high IoU with the most probable bounding box are considered to be redundant and are thus removed. However, those with a low IoU are considered to perhaps belong to a different object of the same class and are thus retained.In this way, the YOLO algorithm selects the most appropriate bounding box for an object.‍The YOLO ArchitectureYOLO is essentially a CNN (Convolutional Neural Network). The YOLOv1 network consists of 24 convolutional layers, and 4 max-pooling layers, followed by 2 fully connected layers. The model resizes the input image to 448x488 before passing it through the CNN.‍ ‍The convolutional layers in the network alternate 1x1, followed by 3x3 reduction layers to reduce the feature space as the image goes deeper into the network. ‍The last convolutional layer uses a linear activation function, while all others use leaky ReLU for activation. ‍Limitations of YOLOThe YOLO algorithm has been a great leap in the field of object detection. Since it can process frames much faster than traditional object detection systems, it is ideal for real-time object detection and tracking. However, it does come with some limitations.‍The YOLO model struggles when there are small objects in the image. It also struggles when the objects are too close to one another. For example, if you have an image of a flock of birds, the model would not be able to detect them very accurately. ‍Popular YOLO VariationsTo overcome the limitations of YOLOv1, many new versions of the algorithm have been introduced over the years. ‍The YOLOv2 was introduced in 2016 by the same author (Joseph Redmond). It addressed the most important limitations of YOLOv1 - the localization accuracy and the detection of small clustered objects. The new model allowed the prediction of multiple bounding boxes (anchor boxes) per grid cell, so more than one object could now be detected in a single cell. Moreover, to improve accuracy the model used Batch Normalization in the convolutional layers. ‍YOLOv2 uses the Darknet-19 network, which consists of a total of 19 convolutional layers and 5 max-pooling layers.‍Following YOLOv2, the YOLO9000 was introduced. This model was trained on the COCO dataset (which is almost a superset of ImageNet), allowing it to detect more than 9000 image classes.‍When YOLOv3 came about, it brought with it an architectural novelty that made up for the limitations of both YOLO and YOLOv2. So much so that it is still the most popular of the YOLO versions to date. This model uses a much more complex network – the Darknet-53. It gets its name from the 53 convolutional layers that make up its architecture. The model itself consists of 106 layers, with feature maps extracted at 3 different layers. In this way, it allows the network to predict at 3 different scales. This means that the network is especially great at detecting smaller objects.‍Besides that, YOLOv3 uses logistic classifiers for each class, instead of a softmax function (used in the previous YOLO models). This allows the model to label multiple classes for a single object. For example, an object could be labeled as both a ‘man’ as well as a ‘person’.‍After YOLOv3, other authors introduced newer versions of YOLO. For example, Alexey Bochkovskiy introduced the YOLOv4 in 2020. This new version mainly increased the speed and accuracy of the model with new technologies like weighted residual connections, cross mini batch normalization, and more.‍Many other versions have come about following the YOLOv4, like the YOLOv5, YOLOACT, PP-YOLO, and more. The latest version to date is the YOLOv7. The paper for this model was released in July 2022 and is already quite popular. ‍According to the authors, the YOLOv7 could outperform most conventional object detectors, including YOLOR, YOLOX, and YOLOv5. In fact, the YOLOv7 is being hailed by its authors as the ‘New State-of-the-Art for Real-Time Object Detectors’.‍How you can use YOLO in CVAT / Integration of YOLO and CVATTo train any object detection model on image data, you need pre-annotated images (containing labeled bounding boxes). There are a number of tools available both online and offline to help you do this. One such tool is CVAT (Computer Vision Annotation Tool). ‍This is a free, open-source online tool that helps you label image data for computer vision algorithms. Using this tool, you can simply annotate your images and videos right from your browser.‍Here’s a quick tutorial on how to annotate objects in your image using CVAT.‍Using CVAT to Annotate ImagesLet’s say you have the following image and you want to put bounding boxes and labels on the two cars, dog, and pedestrians. ‍To do it, you need to go to cvat.ai, create an account and upload an image.‍When it comes to image uploading, the whole process includes several steps. First step is to set up a project and add task with the labels of choice (in this case: ‘pedestrian‘, ‘dog‘, and ‘car‘). ‍Second step is to upload one or more images you want to annotate and click ‘Submit and Open‘.‍Once everything is in place, you will see your task and all the details as a new job (with a new job number). The window below is the ‘Task dashboard’. Click on the job number link:‍‍ It will take you to the annotation interface:‍‍Now you can start annotating.‍How to Manually Annotate Objects in an ImageIn this example we show annotations with rectangles. To add a rectangular bounding box manually, you need to select a proper tool on the controls sidebar. Hover over the ‘Draw new Rectangle’, and from the drop-down list select the label you want to add to the annotated object. Click `Shape`.‍‍‍With a rectangle, you can annotate using either 2 or 4 points. Let’s say, you chose 2 points, then simply click on the top left corner and then the bottom right corner of the object, like this:‍‍CVAT will put the bounding boxes with specified labels around the objects. ‍This method is good when you do not have too many objects on the image. But if you have a lot of them, then the manual method can get quite tedious. For multiple objects cases CVAT has a more efficient tool to get the job done – the YOLO object detection.‍Using YOLO to Quickly Annotate Images in CVATCVAT incorporates YOLO object detection as a quick annotation tool. You can automate the annotation process by using the YOLO model instead of manually labeling each object. ‍Currently, two YOLO versions are available in CVAT: YOLO v3 and YOLO v5. In this example, we will use YOLO v3.To use the YOLO v3 object detector, on the controls sidebar hover over the AI Tools and go to Detectors tab. You will see a menu with a drop-down list of available models. From this drop-down, select ‘YOLO v3’.‍‍The next thing you need to do is the labels’ matching. This need is based on the fact that some models are trained on the datasets with a predefined list of labels. ‘YOLO v3’ is a model like that and to start annotating you need to give YOLO a hint - how its model’s labels are correlating with the ones you’ve added to CVAT. ‍For example, you want to label all people on the image and added a ‘pedestrian’ label in CVAT. The most fitting YOLO label for this type of object will be ‘person’. To start annotating, you need to match the YOLO label ‘person’ to the CVAT label ‘pedestrian’ in the Detectors menu.‍Luckily, for other objects there is no need to think twice, as YOLO has `dog` and `car` model labels:‍‍‍Once you’re done matching the labels, click ‘Annotate’. CVAT will use YOLO to annotate all the objects for which you have specified labels.‍‍After annotation is done, go ahead and save your task by clicking the Save button, or export your annotations in the .xml format from Menu > Export Job Dataset.‍Quickly Annotating Objects in VideosYou can use CVAT Automatic Annotation with YOLO detector to label objects in videos directly from your Task Dashboard with a few simple steps.‍First step is to find the task of the required video. Once you’ve identified it, hover over three dots to open the pop up menu.‍‍‍In the menu click on Automatic Annotation to open the dialog box, and from the drop down menu select ‘YOLO v3’ . ‍Second step is to check the labels matching, and adjust them to fit your needs and requirements (if needed).‍‍When all is set and ready, click ‘Annotate’ to start labeling objects in the video.‍It will take some time for automatic annotation to complete. The progress bar will show the status of the process. ‍‍When it is done, you will see a notification box along with a link to the task.‍‍Click on the link to open the task dashboard, and again on the job link to open the annotation interface.‍‍Where you will see the video with objects automatically labeled in every frame: ‍‍‍You can now go ahead and edit the annotations as needed if you find any false positives or negatives. ‍ConclusionYOLO is a specialized Convolutional Neural Network that detects objects in images and videos. It gets its name (You Only Look Once) from its technique of localizing and classifying objects in an image in just one forward pass over the network. ‍The YOLO algorithm presented a major improvement over the previous 2-stage object detection algorithms like R-CNN and Faster R-CNN in the inference speed. In an attempt to increase the speed and accuracy of object detection, numerous versions of YOLO have been introduced over the years. The latest version is the YOLOv7.‍Using YOLO on the CVAT platform, you can annotate images and videos within minutes, significantly reducing the amount of manual work that image annotations usually call for.‍We hope this tutorial helped you understand the concept and architecture of YOLO, and that you can now use it to detect and annotate objects in your own image data.‍