Annotation 101

In computer vision, machine learning, and spatial analysis, 3D point cloud annotation is often used to help convert raw 3D data into structured, meaningful information. In doing so, the transformation enables algorithms to recognize objects, environments, and spatial relationships and powers the use of real-world applications from autonomous navigation to industrial inspection.But what exactly is a point cloud, what applications use them, and what are the best ways to annotate them?This article explains what point clouds are, where they’re used, and how CVAT streamlines the annotation process, from raw scan to labeled dataset.‍What is a Point Cloud?A point cloud is a digital map of an object's surface, made up of individual points captured by a scanning system. These datasets form the foundation for most 3D computer vision tasks and are produced using methods such as LiDAR, photogrammetry, stereo vision, laser light or structured light-based systems. There are subtle variations between how each type of 3D scanner works, but fundamentally, they all use light to capture surface geometries, resulting in a 3D map.Depending on the size of the object, the number of points can range from just a few up to trillions of points. For example, a point cloud of a cube could be created from just 8 points (one for each corner), while the recent 3D scan of the Titanic wreck resulted in a 16 terabyte dataset comprising billions of points. The Titanic scan (made using photogrammetry) is so detailed that individual bolts on the ship and even a necklace are visible in the scan data.Titanic 3D scan (Source)Urban planning and planetary LiDAR scans are even bigger and can feature trillions of points in the point clouds.Once the raw scan data has been acquired by the scanning hardware of choice, it can be processed into a usable format. This is typically achieved by aligning, filtering, and converting the raw data into a structured point cloud for visualization, measurement, modeling…or annotation.‍Applications of Point Clouds Data in Computer VisionFrom autonomous vehicles and robotics to drones and geospatial analysis, point clouds enable machines to interpret real-world geometry. Annotated 3D data enables detection, reconstruction, and spatial reasoning across a broad swath of AI applications such as:Autonomous DrivingLiDAR-generated point clouds are used for 3D object detection, lane and road mapping, obstacle avoidance, and real-time vehicle localization.RoboticsRobots need point clouds to map their surroundings, stay clear of crashes, spot objects, and move around in dynamic environments.Augmented and Virtual Reality (AR/VR)Used to reconstruct real-world environments for immersive experiences, enabling realistic interaction with virtual objects.Industrial Inspection and Quality ControlCapture precise 3D models of manufactured parts to detect defects, verify tolerances, and ensure conformity with design specifications.Construction and ArchitectureUsed for as-built documentation, site surveys, clash detection, and the creation of accurate digital twins for buildings and infrastructure.Geospatial Analysis and MappingPoint clouds from aerial LiDAR or drones are used for terrain modeling, land classification, flood simulation, and urban planning.AI and Machine LearningAnnotated point clouds are used by researchers to train machine learning models for segmentation, object classification, and scene understanding in 3D.Each application relies on the precision and richness of point cloud data to bridge the gap between raw spatial input and actionable digital insight.‍Point Cloud Raw Data Point clouds are typically represented using Cartesian (XYZ) coordinates, but they aren't always captured that way at the source. Many 3D scanners (LiDAR systems in particular) initially collect data in spherical coordinates, recording each point’s distance from the scanner (r), horizontal angle (θ), and vertical angle (φ). In other cases, such as tunnel inspection or pipe mapping, scanners may use cylindrical coordinates. Range imaging systems often store depth as pixel intensity in a 2D grid. These native coordinate systems reflect the scanner’s internal geometry and sensing method, optimized for capturing specific environments. However, for consistency and compatibility (whether in CAD, simulation, or AI pipelines), these formats must be converted. Using trigonometric transformations, spherical and cylindrical data are recalculated into standard XYZ coordinates, where each point is defined by its position along three perpendicular axes. The XYZ format is universally supported by common point cloud file types like .ply, .pcd, .xyz, and .las, making it essential for downstream processing. So, while point clouds may originate in various coordinate systems, they are almost always converted into XYZ for storage, visualization, and further analysis.‍Readily Available 3D Datasets2D datasets are quite laborious to obtain manually, so you can imagine how much of a resource-intensive task gathering training data for 3D applications is.Thankfully, there is a wide range of publicly available point cloud datasets available. Many are free and open access, and some require the purchase of a license. The table below shows a selection of popular datasets available, which you may wish to use for your model training.<table class="table-class"> <tr> <th>Dataset</th> <th>Application Type</th> <th>Environment</th> <th>Open Access?</th> </tr> <tr> <td>KITTI Autonomous Driving, SLAM</td> <td>KITTI Outdoor (Urban)</td> <td>KITTI Yes</td> </tr> <tr> <td>nuScenes Autonomous Driving</td> <td>nuScenes Outdoor (Urban)</td> <td>nuScenes Yes</td> </tr> <tr> <td>Waymo Open Autonomous Driving</td> <td>Waymo Open Outdoor (Urban/Suburban)</td> <td>Waymo Open Yes (non-commercial use)</td> </tr> <tr> <td>ApolloScape 3D Scene Parsing</td> <td>ApolloScape Outdoor (Urban)</td> <td>ApolloScape Yes</td> </tr> <tr> <td>ScanNet 3D Reconstruction, Semantic Segmentation</td> <td>ScanNet Indoor</td> <td>ScanNet Yes</td> </tr> <tr> <td>ShapeNet Object Classification, Segmentation</td> <td>ShapeNet Object-Level</td> <td>ShapeNet Yes</td> </tr> <tr> <td>ObjectNet3D 2D-3D Alignment, Pose Estimation</td> <td>ObjectNet3D Object-Level</td> <td>ObjectNet3D Yes</td> </tr> <tr> <td>Toronto-3D Urban Scene Segmentation</td> <td>Toronto-3D Outdoor (Street-Level)</td> <td>Toronto-3D Yes</td> </tr> <tr> <td>DALES Aerial Mapping, Segmentation</td> <td>DALES Outdoor (Aerial/Suburban)</td> <td>DALES Yes</td> </tr> <tr> <td>NPM3D Mobile Mapping, Localization</td> <td>NPM3D Mixed (Indoor/Outdoor)</td> <td>NPM3D Yes (CC-BY-SA License)</td> </tr> </table>‍3D Point Cloud Annotation in CVATCVAT contains a variety of tools for annotating a range of data types, from static images to moving video. A later addition to the software saw the ability to annotate 3D scan data in the form of point clouds.This is particularly important for those who wish to train their models with three-dimensional data.Cuboids are used for point cloud annotation in CVATWhile the 2D image and video data comes with a large selection of annotation tools (such as skeleton, ellipse, square, mask), annotation of point cloud data in CVAT is done exclusively with cuboids.Cuboids provide a balance between simplicity and spatial context. Cuboids are:Easy to manipulate in 3D spaceSufficient for common tasks like object detection and trackingCompatible with widely used datasets like KITTI and formats used in frameworks like OpenPCDet, MMDetection3D, etc.‍Understanding Labeling Tasks and TechniquesBefore beginning the annotation workflow in CVAT, it helps to understand how labeling tasks are structured and what techniques can improve accuracy and consistency.‍What Is a Labeling Task in CVAT?In CVAT, a labeling task defines the scope of your annotation project. Each task includes:A name and descriptionA set of labels or object classes (e.g., “car,” “pedestrian,” “tree”)Optional attributes (e.g., “moving,” “occluded”)A dataset to be annotated (images, video frames, or 3D point clouds)For 3D point clouds, tasks support both static scans and sequential frames, allowing for temporal annotation (e.g., tracking objects across time). Each task can contain multiple jobs, which are the individual segments of data assigned to annotators. It allows us to make the annotation process parallel. ‍‍Defining Clear Labels and AttributesBefore uploading your dataset, define your label structure carefully according to your business or research requirements. Avoid vague labels, and keep class names consistent. For example, use “vehicle” consistently instead of alternating with “car” or “van.” Add attributes to capture additional information, such as:Object state: moving, stationary, partially_visiblePhysical characteristics: damaged, open, closedAttributes can be marked as mutable (changes over time) or immutable (stays constant), which helps simplify the annotation interface and improve training consistency.‍Techniques for Effective 3D AnnotationTo annotate efficiently:Use Track mode to maintain object IDs across framesPlace cuboids in the main 3D viewport and refine them in the Top, Side, and Front orthogonal views. Two projections are usually enough to place the cuboid correctly across all axes.Use contextual 2D images (if available) to support difficult annotationsApply interpolation for objects in motion across multiple framesFlag ambiguous or occluded annotations with appropriate attributesProper task setup and labeling discipline not only make the process smoother, they also ensure that the resulting dataset is accurate, structured, and ready for downstream AI training.‍Annotating Point Clouds in CVAT: OverviewCVAT’s 3D point cloud annotation workflow is straightforward. The user simply creates a task, loads the dataset, places and adjusts cuboids, and optionally propagates or interpolates the cuboids across frames. Here’s an abbreviated overview of the full process, from start to finish.Create a 3D annotation task.Open the task and explore the interface layout.Navigate the 3D scene using mouse and keyboard controls.Create and adjust cuboids for annotation.Copy and propagate annotations across frames.Interpolate cuboids between frames.Save, export, and integrate annotated data into your pipeline.‍Before training begins, it’s good practice to run a validation script to check for label inconsistencies, misaligned cuboids, or frame mismatches. Ensuring clean, well-structured annotation data is just as critical as the model architecture itself.For a more fleshed out tutorial on how to annotate 3D point clouds, head on over to our official guide.‍Challenges in 3D Point Cloud AnnotationAnnotating 3D point clouds comes with a distinct set of challenges (both technical and human) that can significantly affect the quality of your dataset.One of the biggest issues is occlusion. Since point clouds are generated from specific sensor perspectives, any surfaces not visible to the scanner (the back of an object or areas blocked by other objects, for example) simply don’t appear. This missing data can make it difficult to annotate complete geometries with confidence. This lack of visual information can make it difficult to fully interpret object boundaries, identify shapes, or distinguish between overlapping items. In dense or cluttered scenes, occlusion can lead to under-representation of key objects and introduce ambiguity during annotation.Example of an occluded and non-occluded 3D objects annotated in CVAT Point density is another problem. Objects close to the sensor may be richly detailed, while distant objects can appear sparse or fragmented. Low-density regions often result in uncertainty when drawing precise cuboids or estimating object boundaries.Add to that sensor noise, which can result from misfired points, ghosting from reflective surfaces, or jitter from moving elements in a scan, and the result is a lot of visual clutter that annotators must mentally filter out.Then there’s annotation fatigue. Unlike 2D image annotation, working in 3D often involves constant panning, zooming, and adjusting the scene from different angles. This level of interaction increases the mental load and can lead to inconsistency across sessions.To help mitigate this, CVAT allows the use of contextual 2D images alongside point clouds, displayed in separate windows within a 3D annotation task.‍Best Practices for High-Quality Point Cloud Data AnnotationGetting 3D point cloud annotation right isn’t complicated, but it does require discipline. The most common issues come from inconsistency and lack of structure, both of which are easy to avoid if you put the right systems in place from the start.Be consistent: Start with label consistency. Stick to a fixed label set. Don’t call something a “car” in one frame and a “vehicle” in another. CVAT’s label constructor locks this down, so use it. It stops annotators from improvising with naming conventions.Attribute properly: If you’re annotating objects with different states (like “open/closed” or “damaged”), don’t create separate labels. Add a mutable attribute. For fixed traits (like color or make), use immutable ones. It keeps the label space clean and keeps your training data flexible.Establish Annotation Guidelines: Whether you're working solo or with a team, define clear rules for edge cases, like how to handle partial occlusion, ambiguous shapes, or overlapping objects. A short internal guideline document can eliminate confusion and reduce rework later on.Quality assurance: Apply some basic QA. Do a second pass. Spot-check frames. Use annotation guidelines. If multiple annotators are involved, establish consensus rules for edge cases. You don’t need a formal pipeline, you just need to try to avoid leaving junk labels, floating cuboids, or inconsistent tags in the data.Don’t Over-Label: Last, but not least, it can be tempting to annotate every object in the scene, but not all data is equally useful. Focus on what your model actually needs to learn. Prioritize annotation quality over quantity, especially when resources are limited.Clean data is trainable data. Anything else just provides more work unnecessarily down the line.‍CVAT Labeling ServiceWhile point cloud data annotation with CVAT is relatively straightforward, not everybody has the luxury of time or other resources to commit to the data annotation process - it can be time-consuming after all, particularly when dealing with huge datasets.If you fit into this category and would rather outsource your data labeling needs, you will be pleased to know that CVAT offers our own services for such tasks.Our professional data annotation services offer expert annotation of your computer vision data at scale, regardless of if the data is point cloud, image- or video-based. Our team of experts ensures high-quality annotations and provides detailed QA reports so you can focus on your core experience and computer vision algorithms. ‍Why You Should Consider CVAT for Point Cloud Data Annotation3D point cloud annotation isn’t glamorous, but it’s a vital step in building reliable 3D perception systems. Whether you’re working on autonomous vehicles, machine learning, robotics, or spatial AI, well-structured annotations make the difference between a model that just runs and a model that performs.CVAT offers annotation tools such as cuboids, multi-view layouts, contextual 2D images, and export formats compatible with common frameworks like OpenPCDet, TensorFlow, and MMDetection3D. Getting high-quality annotations means doing the basics right: using consistent labels, applying attributes carefully, and maintaining coherence across frames. CVAT’s propagation and interpolation tools help speed that up while reducing manual error. And before you push your dataset into training, take the time to review and validate it, because annotation mistakes are a lot cheaper to fix before the model starts learning from them.In short, clean data leads to cleaner results. The effort you put into annotation shows up later in model accuracy, stability, and generalization. And CVAT gives you the foundation to build clean datasets. What you choose to do with the annotated data afterwards is down to your own ingenuity!To try CVAT for your own workflows, you can sign up for a free account here.

Picture a self-driving car avoiding incoming vehicles, turning at the right junction, and gently stopping at the traffic light. It waits patiently until the light turns green, and it gracefully rejoins the traffic. All without the driver lifting a finger. We know that’s artificial intelligence (AI) technology at its finest. Or, specifically, computer vision, a term that machine learning engineers are familiar with. Advanced computer vision systems can identify objects in ways like humans do. But how does it do so? The answer — potentially hundreds of hours of videos, carefully annotated so that the computer vision model can learn and perceive the vehicle’s surroundings just like humans would. Video annotation isn’t only helpful for innovating self-driving cars. Today, computer vision is at the heart of various applications, which underscores the importance of video annotation. In this guide, we’ll explore:What video annotation is and why it matters.Video annotation benefits and applications.Types of video annotation.Challenges and best practices when annotating videos.Let’s start. What is Video Annotation? Video annotation is the process of labeling or masking specific objects in videos based on their types or categories. A human annotator or labeler would highlight specific parts of the video frame and tag it with a label. The annotated video dataset then becomes the ground truth to train computer vision models, often through supervised learning.By teaching itself on each of the labels or masks, the machine learning algorithm becomes more adept at associating visual data to real-life objects as how humans see them. Video annotation is laborious, where human labelers patiently identify and classify multiple objects in frame after frame. Often, they’ll use automated video annotation software to accelerate the annotation process.‍Why does Video Annotation Matter? Startups and global enterprises are in a race to market state-of-the-art computer vision systems. By 2031, the computer vision market is predicted to hit US $72.66 billion. But to compete and thrive in this industry, relying on state-of-the-art computer vision models isn’t enough. By itself, a computer vision model cannot interpret objects from video data correctly. Like other machine learning algorithms, it needs to learn from datasets curated and annotated for a specific application. Let’s take a traffic monitoring system as an example. Without learning from the annotated dataset, the computer vision model can’t identify cars, pedestrians, and other objects the camera captured. Instead, the system sees only pixelated data, including contrasts, hues, and brightness for each frame that passes through. But that changes when you annotate the video.For example, you can place a bounding box on a car to teach the computer vision model to identify it as such. Likewise, you can train the model to identify pedestrians by drawing key points on the people. We’ll cover more of this later. But the point is —video annotation makes computer vision smarter by training it to interpret video data just like we would with what we see in real life. AI models operate with so-called "garbage" in principle. If you feed the model with low-quality datasets, it produces inaccurate results. Therefore, what’s equally critical is the dataset the model trains from, which calls for improved annotation quality.Video Annotation vs. Image Annotation Video annotation is a subset of data annotation, which also includes image annotation. Some people might draw similarities between both types of annotation. The common argument: video is made up of sequential frames of images. Just like how you can draw a bounding box on an image, you can do so on a still frame in a video. But that’s where the similarity ends. Video annotation is more suitable in certain use cases, especially those that require more contextual data like layers and movements. Caption: Annotating an image vs. video. That said, video annotation is also more complex. And that’s why annotators use automated data labeling tools like CVAT to assist their efforts. Meanwhile, image annotation is simpler, as annotation is limited to a static visual. So, how do you decide if video annotation is more suitable in your application? Simple. Consider the checklist below.InterpolationLabeling an image is straightforward. But what happens when you need to label dozens of sequential images? In that case, video annotation is better. Advanced video annotation lets you interpolate between frames. All you need to do is label the starting and ending images, and the tool will automatically annotate the rest.Motion detectionSome computer vision applications require temporal context or time-related information to learn more effectively. Image annotation cannot deliver such information, while video annotation can. For example, you can mark a tennis ball across its trajectory to train a motion detection system.Data accuracyAn image can only convey information limited to what was captured on a static visual plane. Video, on the other hand, has more depth, which is more helpful for AI applications that require advanced tracking, monitoring, or identification. For example, you can help a computer vision model identify a partially obscured object by masking it with a brush tool. Cost efficiency Annotating a handful of images seems effortless. But do so for dozens of them, and you’ll start feeling the strain of committing time, labor, and money. Instead, video annotation is often more cost-effective for complex applications. For example, you can annotate several keyframes and automatically label in–between frames to save time.How is Video Annotation Applied in Various Industries As computer vision evolves, so does adoption amongst industries. We share real-life applications where computer vision models, trained with annotated video, are making an impact.Autonomous vehiclesAt the heart of the vehicle is an AI-powered system that processes streams of video information in real life. The car can differentiate vehicles, pedestrians, buildings, and other objects accurately in real time. And this is only possible because the computer vision model that guides the vehicle was trained with annotated datasets. HealthcareDoctors, nurses, and medical staff benefit from imaging systems trained with annotated video datasets. Conventionally, they rely on manual observation to detect anomalies like polyps, cancer, or fractures. Now, they’re aided by computer vision to diagnose more accurately.AgricultureComputer vision technologies trained with properly annotated datasets can improve yield in agriculture. Farmers, challenged by monitoring acres of land and crops manually, leverage AI to optimize land usage, combat pests, fertilize plants, and more.Caption: Tracking harvester movement across farms. ManufacturingProduct defects, left unnoticed, can negatively impact manufacturers both financially and reputationally. Installing a visual-inspection system trained with annotated datasets allows for more precise quality checks. In addition, such systems also create a safer workspace by proactively detecting abnormal or unsafe situations.Security surveillanceAnother area where video annotation is sought after is security surveillance. CCTV cameras allow security officers to oversee people's movement in real time. However, they might need help in identifying suspicious behavior, especially when monitoring multiple feeds. With computer vision, untoward incidents can be prevented as the computer vision system picks up patterns it was trained to identify and promptly alerts the officers. Traffic managementTraffic rules violations, congestions, and accidents are concerns that governments want to resolve. With computer vision, the odds of doing so are greater. Upon training, the AI model can analyze traffic patterns, recognize license plates, and identify accidents from camera feeds. Disaster responseFirst responders need to make prompt and accurate decisions to save lives and property during large-scale emergencies. Computer vision technologies, coupled with aerial imagery, can help responders strategize rescue operations. For example, emergency teams send drones augmented with computer vision algorithms to locate victims affected by wildfires. Types of Video Annotation Video annotators use different techniques when labeling datasets. Depending on the project requirements, they might label an object with techniques like bounding boxes. Or if they need to train the model to capture pose or movement, they’ll use keypoint annotation.A skilled annotator knows how to use and combine various techniques according to the labeling requirement. Bounding boxesA bounding box is the simplest type of annotation you can make on a video. The annotator would draw a rectangle over an object, which is then tagged with a label. It’s suitable when you need to classify an object and aren’t concerned about separating background elements. For example, you can draw a rectangular box over a dog and tag it as an animal.‍Bounding box on a moving vehiclePolygonsLike bounding boxes, polygons enclose an object in a video frame. However, you can remove unwanted background information by drawing the polygon according to the object’s outline. Usually, we use polygons to label complex, irregular objects.Caption: Polygon annotation of a car.PolylinesPolylines are sequences of continuous lines drawn over multiple points. They are helpful when you’re annotating straight-line objects across frames, such as roads, railways, and pathways.Caption: Polyline annotation for railway.EllipsesEllipses annotations are oval-shaped and drawn across objects with similar geometrical outlines. For example, you can use ellipses when annotating eyes, balls, or bowls.Caption: Ellipses annotation for a tennis ball.Keypoints & skeletonsSome video annotation projects require pose estimation and motion tracking. That’s where keypoint and skeleton annotation come in handy. Keypoints are tags assigned to specific parts of the object. For example, you assign keypoints to body joints and facial features. Then, the machine learning algorithm could track how they move relative to each other. On top of that, you can join various keypoints to form skeletons, which helps track body movement more precisely. Skeleton annotation for tracking a horse’s movementCuboidsCuboids allow computer vision models to annotate 3D objects with a rather uniform structure, such as furniture, buildings, or vehicles. You can add spatial information, such as orientation, size, and position in cuboids, to train computer vision models.3D object animationBased on cuboids, this type of annotation further expands on the property of the labeled object, enabling depth perception and volume estimation. For example, annotators use 3D object animation when training a traffic surveillance system to identify vehicles.Automated annotationEven with a diverse range of annotation tools, labeling dozens of hours of video can be painstaking. Instead of manually tagging objects, you can automatically label them with a video annotation tool like CVAT. Once configured, our software automatically finds objects that you want to label in the video and tags them accordingly. Then, you review them to ensure they’re accurate and make changes if necessary. Learn more about how automated annotation works in CVAT.How to do Video Annotation for Computer Vision? Whether you want to train an autonomous vehicle or identify human faces, you start by labeling the datasets. If you’ve never done any, follow these steps that experienced video annotators use.Step 1 - Define your annotation requirementEvery annotation project is different. Know what you need to label in the video and consider the complexity of doing so. For example, categorizing people in public areas is relatively simple. But tracking individual movements requires more effort. From there, decide on which data annotation tool to use. Step 2 - Choose the right video annotation toolNot all data annotation tools are suitable for video annotation tasks. Some lack advanced features, such as automatic or semantic annotation, that help you save labeling time. Besides annotation features, pay attention to project management tools, user-friendly interface, and data security when choosing data labeling software.Step 3 - Upload video dataNext, prepare the video that you want to annotate. In some cases, you might need to resize, denoise, or extract certain frames so that you can improve the video quality. After that, import the video file to the annotation software.Step 4 - Annotate the video datasetCreate a class for the object you want to label. Then, use appropriate tools, such as bounding boxes, polygons, or skeletons, to label the objects. You can identify keyframes, tag objects in them, and interpolate those in between. This allows you to annotate faster without going through every single frame.Step 5 - Review the annotation Mistakes might happen during annotation, even with automated labeling tools. So, it’s important to check the annotated frames thoroughly before using the dataset to train the computer vision model. Look for mistakenly annotated objects, missing labels, and other inconsistencies.Key Challenges in Video Annotation Video annotation is key to enabling state-of-the-art computer vision applications. But creating accurate and consistent datasets remains challenging, even for experienced annotators. If you’re starting a video annotation project, be mindful of these challenges.Labeling inconsistencyHuman labelers play a vital role in video annotation, regardless of the tools you use. Therefore, annotation results are subject to individual interpretations. For example, one annotator may classify a dog as a Poodle, while the other may label it a Toy Poodle. Both are similar but not the same as far as machine learning algorithms are concerned. So, to avoid misinterpretations, provide your team with clear labeling specs. You can read more about it here. Inadequate training Before they annotate, labelers must receive proper training to ensure they’re familiar with the video annotation process, tools, and expectations. Otherwise, you risk compromising the outcome with inaccurate labeling, reworks, and costly delays.Immense datasetsVideo data are larger than their textual and image counterparts. So, the time and effort spent on annotating video frames might take up considerable resources that not all companies can spare.Explore the advanced data annotation course we offer to upkill your annotators.Data security and privacyVideo annotation requires collecting, storing, and processing large volumes of videos, some of which might contain sensitive information. You need ways to secure datasets throughout the entire labeling pipeline and comply with data privacy laws. Project timelineTime to market is another concern that puts additional pressure on annotators. By itself, video annotation is a laborious process. Plus, if they use manual tools, delays might happen as they’ll need to spend time addressing labeling issues. We know that video labeling can be very tedious, even if you’re equipped with the right tool. That’s why we help companies save time and costs with professional video annotation services.Best Practices when Annotating Videos Don’t be discouraged by the hurdles that might complicate video annotation. By taking precautions and smarter approaches, you can improve annotation quality without committing excessive resources.Here’s how.Automate when you canDon’t hesitate to automate the labeling process. Sure, automatic annotation is not perfect. You’ll likely need to review all the frames to ensure they’re correctly labeled. But don’t forget, automatic automation saves tremendous time that you can better spend on strategizing the computer vision project.‍‍If you use CVAT, you can take automated labeling further with SAM-powered annotation. We integrate SAM 2, or Segment Anything Model 2, with our data labeling software to enable instant segmentation and automated tracking of complex objects. Prioritize video qualityWe know that annotators have little or no control over the video they annotate. But on your part, try to ensure the recordings are high quality to start with. Also, the annotation software you use matters, as some might unknowingly degrade the video quality. Keep labels and datasets organizedVideo annotation can get out of hand quickly if you don’t stick to an organized annotation workflow. Overlapping classes, misplaced datasets, and other confusion can limit your video annotator’s productivity. Thankfully, they can be addressed if you’re using a user-friendly data annotation tool. Interpolate sequences with keyframesYou don’t need to label every single frame in a video. Instead, you can assign keyframes in between predictable sequences and interpolate them. Trust us; this will save you lots of time.Set up a feedback systemAnnotators need feedback from domain experts and machine learning engineers to know if they’re labeling correctly. Likewise, any updates in labeling requirements must be communicated to the entire team. Usually, good data annotation software is equipped with a feedback mechanism that streamlines communication. ‍Annotation feedback in CVATImport shorter videosLong videos clog up bandwidth if you’re uploading them to an online annotation tool. If you don’t want to spend hours waiting for the video to load, break it into smaller ones. Preferably, keep the videos below the 1-minute mark.ConclusionVideo annotation is key to creating accurate computer vision models. Compared to image annotation, video labeling provides more details and is more practical in most real-world applications. That said, video annotation isn’t without challenges. If you want to annotate video, you need to address concerns like dataset volume, labeling consistency, and annotator expertise.‍Hopefully, you found useful tips in this guide to improve your annotation quality and reduce time to market. Remember, a data annotation tool equipped with automated features helps a long way in video annotation. Explore CVAT and annotate video more effectively now.

If an artificial intelligence model was an engine, then its training data would be its fuel. And like an engine of an automobile, the quality of its functionality is largely dependent on the quality of what is fed into it. To paraphrase a well-known computer science phrase, if you put garbage in, you get garbage out.The key to ensuring that your models are providing accurate outputs lies in the training data, and data annotation is vital in providing the structure and context to datasets that allow AI models to learn effectively. Without well-structured context, those datasets are just noise.But annotating huge datasets, whether images or videos, can be a never-ending (and tedious) task, and the size of these tasks are set to grow significantly as AI models demand larger datasets. Thankfully, there are a range of tools designed to make image and video data annotation an efficient process. In this article, we will examine what is image annotation, who is using it, how to use it, and how CVAT might be a good choice for your organization’s computer vision or machine learning projects.What is Image Annotation?Image annotation (or image data labeling) is the process of adding labels and tags to image datasets for training computer vision models. Doing so provides context for the machine learning model to understand, and make predictions.With image data annotation software, the annotations generally come in the form of a shape such as a bounding box, polygon, or segmentation mask, along with a textual tag, or label. The geometric shape helps to visually and spatially define the object of interest in the image, while the textual tags help the AI model to identify and classify the object(s) in the image. Example of an image with bounding box annotations.‍Because in computer vision, identification and classification are key processes that help machines interpret and understand visual data, and image annotation is required for achieving this goal.Image annotation can be a huge undertaking in terms of the amount of time and resources needed. With datasets ranging in size from a few thousand images, to several million images, it’s important to determine the best strategy for both acquiring datasets, and for annotating them. Such strategies can involve usage of public versus proprietary datasets, and the choice to use in-house annotation versus professional annotation services. We will examine these strategies in more depth later.‍Where and What Image Annotation Is Used For?Autonomous vehicles, medical imaging, facial recognition, and satellite imagery, all use computer vision and artificial intelligence to perform tasks such as detection, classification and segmentation. All of these industrial applications make use of labeled datasets, and image annotation plays a significant role in the transformation from a dataset (for example, a collection of drone imagery), to a labelled dataset, that provides context for the AI model to use.Data annotation is mostly used for training machine learning models. And by training, we mean that we are teaching a computer vision model to identify objects in various kind of images. This is analogous to how a child learns by pointing at things, and calling them out by name. In short, image annotation is providing ground truth labels for the computer vision model. Image annotation is also used for supervised learning, in which the model learns from annotated examples in the form of input-output pairs. In this case, the annotated data assists the model in understanding how an object should be identified.Finally, image annotation can also be used with performance evaluation to assess the accuracy of a model’s learning. The accuracy can be tested by comparing the model’s output to the annotated data (ground truth).As mentioned in the introduction, quality training data for these systems is largely dependent on quality image annotation. A proper image annotation process can help to improve model accuracy/data consistency, reduce time, costs, and biases, and lead to generally more efficient model training.‍How Are Companies Doing Image Annotation?Any company or organization looking to develop their own computer vision AI models will require high-quality image datasets. These datasets tend to be quite huge, so consideration must be made as to how the datasets are obtained in the first place.This begs the question, should an entity opt for proprietary or open datasets? Let’s examine both cases in more detail.Creating Your Own Proprietary Image DatasetsThe benefits of creating a proprietary dataset include having complete control over the subject matter in the images, as well as the quality of the image. Many open datasets lack the selectivity of a particular subject matter, meaning that they might not be well suited for specialized models.Imagine that you wanted to train an AI to detect a particular type of defect on a manufacturing process developed in-house. By definition of the process’ proprietary nature, it would be close to impossible to find an open dataset with the images needed for training. The datasets, in this case, would necessarily need to be produced in house.A Google Street view car, capturing real world images as it drives around. Source: Wikimedia Commons‍The downside of creating such a dataset, is that the task requires vast resources in terms of image data collection (taking photos), data cleaning and preprocessing, annotation and labelling, quality control, storage and management. It is a time-consuming, and costly exercise.Some examples of companies making use of their own proprietary datasets include Tesla and Google. Tesla collects footage from their own vehicle sensors and makes use of this image data to train its self-driving AI feature (also known as FSD). Similarly, Google uses images gleaned from their own image assets for training Google Lens and Street View AI.Using Open DatasetsOne alternative (often used by smaller companies) is to use open image datasets, which reduce costs and speed up development of models. Many such datasets tend to be created by universities and government institutes, and are often freely available for research use and non-commercial use. Some are available for commercial use, but they are dependent on license conditions, and may require some fee to be paid.The downside of using open datasets is that they may lack specialization for specific tasks, as per the manufacturing example in the section on proprietary datasets.Panoptic segmentation datasets from COCO. Source: https://cocodataset.org.Think of proprietary and open datasets as bespoke tailored clothing versus clothing bought “ready to wear” from a shopping mall. With the tailored garment, you get to choose the material, the fit, the exact color, and any other features that you desire, but the customization comes at a premium price. With ready to wear clothing, you are restricted to whatever is on the shelf in terms of size, style and color, but you save a lot of money compared to the bespoke option. The table below shows several open datasets that can all be imported into CVAT image annotation tool for data labeling.{{image-annotation-table-1="/blog-banners"}}To summarize, your organization’s choice of proprietary or open datasets will depend largely on the resources available and the level of specialization needed for your training data.Proprietary datasets allow a high level of specialization, at the cost of time and financial resources, while open datasets allow faster development at a more cost-effective price point, but may take a penalty when it comes to specialization.Some companies might benefit from a hybrid approach, where open data is used for initial training, and then switch to real data to fine tune their own models.Each method has its own merits and trade-offs, and should be well considered before embarking on the task of developing an AI model.‍What Are Different Types of Image Annotation Tasks?So, you’ve decided exactly where your dataset is coming from, and are ready to begin the process of adding context to the data for training your model. This is where the image annotation phase (and image annotation software like CVAT) comes into play.At its core, the process of image annotation involves highlighting the item of interest in an image or video data, and adding context via text-based notes to the item in question. The type of annotation would depend on the intended use of the data.For example, if you wanted to train a model to recognize the presence of a cat in an image (image classification), you would upload image data consisting of cats in various scenes. You could then instruct your in-house or third-party image annotation services team to sort through the images, and add a text description indicating if a cat is present in the image, or not.More advanced tasks (such as “detection”) would require a bounding box to be drawn around the cat in the image, with various other descriptions (such as color or breed) added as tags.CVAT provides a variety of different image annotation tools for all of these tasks. Such tools include cuboid annotation (for objects with depth or volume), attribute annotation, tag annotation, and a plethora of different shape annotation tools for 2D objects.Image Annotation TasksThe image annotation process requires applying various labels to images (or video) in order to add structure and context to datasets.Image annotation tasks can generally be divided into three different categories, which are classification, detection, and segmentation. CVAT has a number of drawing tools which are aimed at each category. Before we delve into the drawing tools, let’s take a look at the three categories in more detail. Image ClassificationImage classification is the most basic of image annotation categories. It involves applying a label (or labels) to a singular image and simply helps the AI model to identify if such an object is present. With the image classification method, the object location is not specified, only its presence in the image. The label will then aid the computer vision model in identifying similar objects throughout the whole dataset. As an example, your team might be training an AI to recognize images of cats. With the image classification method, each image could be labelled as “cat” (if present) or “no cat” (if not present). Additional tags could be added to classify each image by breed, or by color. But the classification model would not be able to identify where exactly the cat is located within the image. With image classification, it is not necessary to use the shape drawing notation tools, as the labels/tags are applied to the entire image.To indicate where the object is located in the image, you need to use a detection model.‍Object DetectionDetection expands upon classification by adding a localization element. Detection not only identifies the presence of an object in an image, but adds spatial information, which helps to identify the object’s location in the image.Such tasks require the use of drawing tools (such as a rectangle/bounding box, polygon, or ellipse) to be added during notation to highlight where the object of interest is positioned. These drawing tools help the AI model understand both the object’s presence and its position in the image.Additionally, if there are multiple objects in the image, the detection model can specify how many there are, and more advanced models can even assign a confidence score, which indicates the likelihood that the identification is correct. Finally, more advanced models can also detect interactions and relationships between multiple models.So going back to our cat example, a detection model could identify that there are two cats, it could classify them according to breed (with a confidence score), and then infer one cat’s position in relation to another cat.Image segmentationSegmentation annotation is the most advanced of the three categories, and divides an image into discrete areas, providing pixel-level accuracy. There are four main subcategories of segmentation in computer vision, which are Semantic, Instance and Panoptic Segmentation types.Semantic SegmentationWith semantic segmentation, each pixel of the object of interest is assigned a class label. Whereas the detection model will use a bounding box to assign a general class and location within the image, defining the object at a pixel level allows the model to detect the shape with more precision.For example, an image might have a cat drinking out of a bowl while another cat sleeps nearby. During the annotation process, the annotator could use a brush tool to paint the pixels of both of the cats, or use a polygon tool. All the pixels within the masks would be classed as “cat”. Similarly, the bowl could also be annotated, with all the bowl pixels labeled as “bowl”.With semantic segmentation, the model does not distinguish between multiple objects of the same class. Both cats, despite being in their own discrete regions, would simply be classified as “cat”. To distinguish multiple instances of the same class, you need to use instance segmentation. Instance SegmentationAn instance segmentation model also uses masks to assign pixel-level classification to objects, but unlike semantic segmentation, it can identify different instances of the same class. For example, it could distinguish between two cats in an image, each with a different label. During the annotation process, the annotator would create a mask around each cat, showing the exact shape and boundaries of each cat. Unlike detection, which only provides a bounding box, instance segmentation gives a detailed pixel-by-pixel representation of each cat. Panoptic SegmentationThe final category is panoptic segmentation, which combines the benefits of both instance segmentation and semantic segmentation to create a more complete understanding of an image. In this approach, annotators categorize both background elements (such as a wall, or carpet) as well as countable objects such as people, cars, or cats.In this case, if there was an image of three cats lounging on a patterned rug, using the panoptic segmentation method, we would treat the rug as a single background element, applying one uniform label to it. Each cat would be identified individually, with separate segmentation masks, distinguishing them even if they are curled up together or partially overlapping (occluded). This method gives AI a more complete understanding of a scene, allowing it to recognize both the setting and the objects within it.‍Annotation Category SummaryBefore we take a more in-depth look at the various drawing tools, let’s just summarize the annotation categories in terms of their function, along with some non-feline related applications.{{image-annotation-table-2="/blog-banners"}}‍Types of Image Annotation TechniquesAn effective image annotation software should be capable of annotating objects that are both static (shape mode) and objects in motion, across multiple frames (frame mode). And it should be able to use any kind of common image file, such as JPEG, PNG, BMP, GIF, PPM and TIFF.To make shape annotation tasks a cinch, CVAT allows users to annotate with rectangles, polygons, polylines, ellipses, cuboids, skeletons, and with a brush tool.While the various shape annotation tools can be used interchangeably in many situations, each tool works optimally for specific types of task.RectanglesAnnotating with rectangles is one of the easiest methods of image annotation. Also known as a “bounding box”, this shape is best suited for the detection of uncomplicated objects such as doors on a building, street furniture, packing boxes, animals, and faces. They can even be used for notation of people both static and in motion. This is particularly useful for surveillance or tracking projects, although if pose estimation is required, more detailed annotations such as skeleton or polygons could be a better option.‍When multiple objects obstruct each other, the "Occluded" label can be used.Overall, notation with rectangles is an easy and computationally efficient method well-suited for quick object detection of a broad range of subjects. If you want a quick way to identify the general presence and location of the object, then notation with rectangles is a great place to start.Image annotation with rectangles is incredibly straightforward. In CVAT, simply select the rectangle icon in the controls sidebar, choose a label, and select a Drawing Method (2-point or 4-point). Click Shape (or Frame, if annotating video) to enter drawing mode.2-Point Method: Click two opposite corners (top-left and bottom-right) to create a rectangle.4-Point Method: Click the top, bottom, left, and right-most points of the object for a more precise fit. The rectangle is completed automatically after the fourth click.Users can adjust the boundaries of the resulting rectangle using the mouse, and rotate it to best fit the object of interest. Polygons Offering a higher level of precision than rectangles, annotating with polygons is better suited for objects with irregular shapes requiring a more accurate boundary delineation.Drawing a polygon allows a much higher level of detail, as it can closely follow the curves and shape of an object, making it well suited for tasks that require pixel-level analysis. Polygons can also be used for creating masks for semantic segmentation, instance segmentation and panoptic segmentation.Polygon annotation can be used for the detection of objects such as geographical features on satellite images, tumors in medical imagery, types of plants in plant identification, and pretty much anything where an object’s shape is too complex to be captured by a rectangle.If a rectangular notation is best suited for broad object detection tasks, polygons are more optimally used for tasks such as image localization, segmentation, or detailed recognition. To put it another way, while a rectangle annotation is fine for detecting faces, polygons are better for detecting facial features such as mouths, eyes, and noses.Like the rectangle annotation function, drawing polygons is uncomplicated. To draw a polygon in CVAT, locate the polygon option in the controls sidebar, and choose a label. Click Shape to enter drawing mode, then the polygon can be drawn with either of two methods. With the first method, the user can simply use the mouse to draw dots around the outline of the object. With the second method, the user can hold the shift key down, and trace the object with the mouse as a continuous contour. Dots will appear around the object automatically. You can see an example of this in the graphic below.‍Manual drawing of a polygon annotation‍Once the polygon is completed (with either method), the user can adjust the polygon by clicking on the dots, and dragging them until they are happy with the result.‍EllipsesAnnotating with ellipses is the method most useful for the detection of round objects, either elliptical, circular or spherical. If you want to quickly annotate objects such as wheels, various fruits, or even the eyes on a face, then ellipses are the perfect shape for the task.Applications where you might wish to use elliptical annotations include cell detection in medical imaging, pupils in eye tracking, astronomical objects, circular craters in geospatial mapping, or egg monitoring in a hatchery.‍‍In CVAT, ellipses are created in much the same way as rectangles. Simply specify two opposite points, and the ellipse will be inscribed in an imaginary rectangle. And like the rectangular notations, ellipses can also be rotated about a point. You can see how easy it is to annotate with ellipses in the video above.‍PolylinesThe previously mentioned notation types have focused on objects with enclosed regions. Polylines also allow for the notation of elongated, thin enclosed shapes, but also permit the notation of non-enclosed linear, continuous objects.To that end, polyline notation is the most optimal choice for objects with long boundaries and contours that do not need to be fully enclosed, such as railways lines or roads.It is also extremely handy when it comes to tasks requiring path-based analysis, such as object tracking, and for connecting key points in pose estimation tasks. Specific examples of applications using polylines include footpaths and rail lines in aerial mapping, general linear infrastructure inspection, text lines and paragraphs in OCR, animal and human skeletons in pose estimation, and moving objects in video sequences.‍A polyline on a continuous road marking‍To sum it up, polylines are at their most useful when the goal is to track, detect, or measure linear features.Drawing polylines in CVAT is similar to drawing a polygon. Simply select the polyline tool from the control panel, select the shape (or track), and set the number of points needed for the polyline. The drawing will complete when the specified number of points has been reached. Also, like the polygon tool, there are two ways in which a polyline can be drawn - it can be drawn with dots, or it can be traced along manually by holding down the shift key.‍Brush ToolThe brush tool is a free-form tool that allows the manual painting of objects, and the creation of masks. Masking is particularly useful for annotating singular objects that may appear split in two, such as a vehicle with a human standing in front of it. You can see an example of this in the graphic below.Example of image annotation with a brush tool‍The brush tool in CVAT features various modes such as brush shape selection, erase pixels, and polygon-to-mask. Polygon-to-mask mode enables quick conversion of polygon selections into masks. Annotations can be saved and modified via the Brush Tool menu, enhancing efficiency in detailed image segmentation tasks.Annotating with the brush tool is ideal for applications that require a high level of precision, such as medical imaging, object detection, or autonomous driving.‍SkeletonsAnnotating with skeletons is the best option when dealing with tasks requiring the analysis of complex and consistent structures, such as human figures. It’s also a little more involved than the other annotation processes we have looked at in this article, which is why we have saved it until the end!‍Example of a skeleton annotation‍A Skeleton consists of multiple points (also referred to as elements), which may be connected by edges. Each point functions as an individual object, with its own unique attributes and properties such as color, occlusion, and visibility.Skeleton annotations can be used for both static and moving images, although they are used in different ways for each type. When using skeleton notation with static images, they are best used when analyzing a single pose, whereas in video, they can be used for more dynamic applications (such as tracking movement over time).Other specific applications of skeleton-based annotations include gait analysis, workplace ergonomics assessments, gesture recognition for sign language, crime scene analysis, and avatar posture recognition in AR/VR environments.Out of the various other methods we have looked at for notating static images in this article, notating with skeletons is generally the most complex. However, the whole process of annotating with skeletons is made much more user-friendly with CVAT.If you wish to annotate with skeletons with CVAT, then the process is summarized as follows:There are two main methods of annotating with skeletons. The first is to do it manually, and the second is to load a skeleton from a model. The Skeleton Configurator allows users to add, move, and connect points, upload/download skeletons in .SVG format, and configure labels, attributes, and colors for each point.To use the Skeleton Configurator, set up a Skeleton task in the configurator, and click Setup Skeleton to enable manual creation. To create the skeleton, simply add points and edges in the drawing area, configure the required attributes, upload the files, and submit the task. ‍AI-Assisted Image Data AnnotationAs seen in the previous section, annotating with various shapes is a straightforward experience. But these tasks can be made easier still, thanks to various automation features.AI-assisted image annotation makes use of pre-trained ML models for the detection, classification, and segmentation of objects within image datasets. CVAT can use pre-installed models, and can also integrate with Hugging Face and Roboflow for cloud-hosted instances. For organizations using a self-hosted setup, custom models can be used with Nuclio. AI models in CVAT, such as YOLOv3, YOLOv7, RetinaNet, and OpenVINO-based models, provide accurate object detection, facial recognition, and text detection. CVAT’s automated shape annotations and labeling features can significantly accelerate the complex image annotation process, potentially improving speed by up to 10 times. These features leverage various machine learning algorithms for tasks like object detection, semantic segmentation, and tracking. Automatic Labeling using pre-trained deep learning models (e.g., OpenVINO, TensorFlow, PyTorch).Semi-Automatic Annotations (e.g., interactive segmentation).Automatic Mask Generation: AI models can generate segmentation masks for complex objects.Smart Polygon Tool: Automatically refines polygon shapes around detected objects.Pre-Trained Object Detectors: Detects and labels objects using AI models like YOLO, Mask R-CNN, or Faster R-CNN.We will do a deep dive into the automation side of image annotation in another post - we just thought we would draw your attention to its existence, just in case you wanted to know how AI itself can be used to make the model training process even more efficient.‍Easy Annotation & Labeling of Images with CVAT Annotation SoftwareAs you have seen in this article, there are numerous techniques in image annotation specific for a range of different computer vision projects and use cases. The good news is that CVAT offers all the aforementioned tools in a handy and easily accessible solution.So whether your team is training a computer vision model, engaging in supervised learning, or conducting a performance evaluation, then the CVAT platform can help with all of your data annotation needs.CVAT takes away the headaches of creating annotated datasets with its innovative and user-friendly approach to annotation and task allocation. With its image annotation tool , your organization can upload datasets of visual assets, break the sets down into smaller chunks, and distribute them to team members anywhere on Earth. Once the team members receive their tasks, they are able to use the intuitive image annotation engine to quickly add context to both image and video datasets.CVAT also integrates seamlessly with HUMAN Protocol’s innovative task distribution and compensation system, creating a seamless, efficient workflow for crowdsourcing annotations. And, if you don't have enough resources to do annotation in-house, CVAT's professional annotation services team is available to provide high-quality, expertly labeled datasets, ensuring your machine learning models receive the precise training data they need.So, to summarize - CVAT's image annotation platform can be used for any visual object, whether it's flat or three-dimensional, static or dynamic. And the drawing tools are fundamentally the same for whichever scenario. Naturally, there are more advanced features for power users, and if you would like to know more about those, you can learn more at this link.And if you haven’t yet got to grips with the basics of image annotation and would like to get started with the features in this article, you can try out the free SaaS version of CVAT right here. For those wanting to try the on-premise community version, you can find that over on Github.

Let’s start with an official explanation of the term and what’s behind it. Don’t worry if you don’t understand it yet; we will explain it further.‍‍Semantic segmentation is a computer vision technique that uses deep learning algorithms to assign class labels to pixels in an image. This process divides an image into different regions of interest, with each region classified into a specific category.‍Now, let’s break down this concept step by step with a simple example of use.‍Meet Alex, a young and enthusiastic urban planner. He has a big dream: to design smarter, more efficient cities. To understand what makes a city "smarter" or "more efficient," Alex needs to study how cities function. For example, he needs to distinguish between different types of land cover, such as buildings, roads, water bodies, and green spaces.‍ This helps him assess the amount of green space, evaluate vegetation health, and plan for the creation or preservation of parks and natural areas. Additionally, he can classify areas based on pedestrian usage, identify heavily used and underutilized spaces, and plan interventions to improve accessibility and safety, like adding benches, lighting, or pedestrian crossings.‍These are just a few ways Alex can use data to make informed decisions and design better cities.‍After understanding the goals, Alex starts thinking: Now what? How can he analyze a city? How can he make informed decisions about best practices and areas for improvement?‍So his journey beginsStep 1: Understanding Semantic Segmentation‍Semantic segmentation gives a computer the ability to see and understand images the same way humans do. Instead of just recognizing an entire image as a "cityscape" or "street view," it breaks down the image into tiny parts and labels each one. Every pixel in the image is assigned a category: this pixel is part of a road, that one is a building, and those over there are trees.‍‍By using this technique, Alex can automatically categorize and label every pixel in each image. He can then use the labeled dataset to train a machine learning algorithm, which can gather valuable insights on how a city works. The algorithms analyze the labeled data, identify common city practices based on the labels, and return actionable results. These insights can inform urban planning decisions, optimize traffic management, and enhance public space design.‍Step 2: Preparing the Data‍Alex learns that to teach a computer to understand images, it needs a lot of examples. So, he gathers a dataset of city images as in the example above and organizes them into a folder. This process is known as the data collection step, which involves several challenges.Note, that at this step Alex might have some challenges:‍It can be difficult to collect sufficient data, as privacy issues may arise when using images from certain sources. Additionally, finding the most useful data for training a deep learning model requires careful consideration. Alex also needs to filter out duplicated data to ensure the dataset's quality. With the data ready, the next step is to add labels to the objects in the images. ‍We will discuss these steps and their challenges in more detail in future articles.‍Step 2: Labeling the DataAlex uploads the folder with data into the Computer Vision Annotation Tool (CVAT.ai). Then he carefully labels each pixel in the images, categorizing elements like roads, buildings, and trees.Alex can do it manually, or from the cloud storages.For this task, Alex can use various tools: Polygons or Brush tool. Here is how it looks, when he adds buildings to one category, and pools to the other.‍‍And here is what is going on behind the curtains at this very moment:Semantic segmentation models create a detailed map of an input image by assigning a specific category to each pixel. This process results in a segmentation map where every pixel is color-coded according to its category, forming segmentation masks. ‍A segmentation mask highlights a distinct part of the image, setting it apart from other regions. To achieve this, semantic segmentation models use complex neural networks. These networks group related pixels together into segmentation masks and accurately identify the real-world category for each segment.‍For example, all the pixels that belong to the object “pool” now belong to the “pool” category, and all the pixels that belong to the object “building” are assigned to the “building” category.‍ One key point to understand is that semantic segmentation does not distinguish between instances of the same class; it only identifies the category of each pixel. This means that if there are two objects of the same category in your input image, the segmentation map will not differentiate between them as separate entities. To achieve that level of detail, instance segmentation models are used. These models can differentiate and label separate objects within the same category.Here is a video showing different types of the segmentation applied to the same image:‍Step 3: Training the Model with Annotated Data‍Once the annotation is complete, Alex exports the annotated dataset from CVAT.ai. He then feeds this labeled data into a deep learning model designed for semantic segmentation. Some examples are Cityscapes, PASCAL VOC and the very popular Yolo8. Models are usually evaluated with the Mean Intersection-Over-Union (Mean IoU) and Pixel Accuracy metrics.‍After selecting and training the model, Alex runs it on new, unseen images to test its performance. The model, now trained with Alex's labeled data, can automatically recognize every object in the images and provide detailed segmentation results.‍Here are some examples of how it may look:‍‍‍‍Step 4: Gathering Insights‍By analyzing the results from the model, Alex gathers valuable insights:‍Traffic Patterns: Improved traffic flow and reduced congestion by optimizing traffic light timings and road designs.Green Space Distribution: Identification of areas needing more green space and better urban planning for environmental health.Public Space Utilization: Enhanced public space planning to increase accessibility and usage.Infrastructure Development: Efficient monitoring of construction projects and better planning of new infrastructure.Urban Heat Islands: Implementation of cooling strategies to mitigate heat island effects.‍Now Alex can make informed decisions, because he has processed data on hands. Conclusion‍Thanks to semantic segmentation, Alex can transform raw images into valuable insights without spending countless hours analyzing each one manually. The technology not only saves time but also enhances the accuracy of Alex’s work, making the dream of designing smarter cities a reality. In the end, semantic segmentation turns complex visual data into actionable knowledge and helps create a better urban environment for everyone.‍ And Alex couldn’t be happier with the results.‍Not a CVAT.ai user? Click through and sign up here‍Do not want to miss updates and news? Have any questions? Join our community:‍Facebook‍DiscordLinkedInGitterGitHub