Feature Highlight

We’re excited to announce another special addition to our automatic and model-assisted labeling suite: Hugging Face Transformers.Hugging Face Transformers is an open-source Python library that provides ready-to-use implementations of modern machine learning models for natural language processing (NLP), computer vision, audio, and multimodal tasks.The library includes thousands of pre-trained models, including a broad selection of computer vision models that you can now connect to CVAT Online and CVAT Enterprise for automated data annotation.The current integration supports the following tasks:Image classificationObject detectionObject segmentationAll you need to do is pick a model you want to label your dataset with from Transformers library, connect it to CVAT via the agent, run the agent, and get fully labeled frames or even entire datasets, complete with the right shapes and attributes in a fraction of the time.Annotation possibilities unlockedJust like with Ultralytics YOLO and Segment Anything Model 2 integrations, this addition opens up multiple workflow optimization and automation opportunities for ML and AI teams.(1) Pre-label data using the right model for the taskConnect any supported Hugging Face Transformers model that matches your annotation goals—whether it’s a classifier, detector, or segmentation model—and run it directly in CVAT to pre-label your data. Each model can be triggered individually, enabling you to generate different types of annotations for the same dataset without scripts or external tools.(2) Label entire tasks in bulkWorking with a large dataset? Apply a model to an entire task in one step. Open the Actions menu and select Automatic annotation. CVAT will send the request to your agent and automatically annotate all frames across all jobs, reducing manual effort and repetitive work.(3) Share models across teams and projectsRegister a model once and make it instantly available across your organization in CVAT. Team members can use it in their own tasks with no local setup, ensuring consistent labeling workflows at scale.(4) Validate model performance on real dataEvaluate any custom or fine-tuned Hugging Face Transformers model directly on annotated datasets in CVAT. Compare model predictions with human labels side-by-side, identify mismatches, and spot edge cases—all within the same environment.How it worksStep 1. Register the functionCreate a native Python function that loads your Hugging Face model (e.g., ViT, DETR, or segmentation transformers) and defines how predictions are returned to CVAT. Register this function via the CLI.Note: The same function works for both CLI-based and agent-based annotation.Step 2. Start the agentLaunch an agent through the CLI. It connects to your CVAT instance, listens for annotation requests, runs your model, and returns predictions back to CVAT.Step 3. Create or select a task in CVATUpload your images or video and define the labels, depending on your evaluation needs and model output.Step 4. Choose the model in the UIOpen the AI Tools panel inside your job and select your registered Hugging Face model under the corresponding tab.Step 5. Run AI annotationCVAT sends the request to the agent, which performs inference and delivers predictions back in the form of annotation shapes tied to the correct label IDs.Get startedReady to enhance your annotation workflow with Hugging Face Transformers? Sign in to CVAT Online and try it out.For more information about Hugging Face Transformers, visit the official documentation.For more details on CVAT AI annotation agents, read our setup guide.

Note: ‍This is the first part of a three-step integration. At the moment, SAM 3 is available only for segmentation tasks in a visual-prompt mode (clicks/boxes), not via text prompts. Free tiers (Community and Online Free) get demo-mode access to SAM 3 suitable for evaluation, not high-volume labeling.For regular production labeling at scale, we recommend using SAM 3 from Online Solo, Team, or Enterprise editions where it’s part of the standard AI tools offering.‍We’re excited to announce the integration of Meta’s new Segment Anything Model 3 (SAM 3) for images and videos segmentation in CVAT.Since we introduced SAM in 2023, it has quickly become one of the most popular methods for interactive segmentation and tracking, helping teams label complex data much faster and with fewer clicks.SAM 3 takes this even further by introducing a completely new segmentation approach, and we couldn’t wait to bring it to CVAT as soon as it was publicly released. The current SAM 3 integration is already available across all editions of CVAT and continues our commitment to bringing state-of-the-art AI tools into your annotation workflows.What’s New in SAM 3?Released in November 2025, SAM 3 is not just a better version of SAM 2, it’s a new foundation model built for promptable concept segmentation. Unlike its predecessor, which segments (and tracks) specific objects you explicitly indicate with clicks, SAM 3 can detect, segment, and track all instances of a visual concept in images and videos using text prompts or exemplar/image prompts.Say you want to label all fish in an underwater video dataset. With SAM 2, you would typically have to initialize the object manually (for example, click or outline fish one by one), and then keep re-initializing or correcting the model whenever new fish appear, fish overlap, or the scene changes. In other words, the workflow is object-by-object and sometimes frame-by-frame. While tracking can propagate the masks or polygons to subsequent frames, every new object or missed instance still requires manual initialization.With SAM 3, the goal is closer to concept-first labeling: you provide a prompt that describes the concept (like “fish”) or show an example, and the model attempts to find and segment all matching instances across the video, and then track them as the video progresses.This shift toward concept-first segmentation opens up new possibilities for automated data annotation workflows. Instead of repeatedly initializing individual objects with clicks or boxes, annotators can focus on defining what needs to be labeled, while the model handles identifying matching instances. This can significantly reduce manual effort on large or visually complex datasets.Key Capabilities of SAM 3Concept-level promptsSAM 3 takes short text phrases, image exemplars, or both, and returns masks with identities for all matching objects, not just one instance per prompt.Unified image + video segmentation and trackingA single model handles detection, segmentation and tracking, reusing a shared perception encoder for both images and videos.Better open-vocabulary performanceOn the new SA-Co benchmark (“Segment Anything with Concepts”), SAM 3 reaches roughly 2× better performance than prior systems on promptable concept segmentation, while also improving SAM 2’s interactive segmentation quality.Massive concept coverageSAM 3 is trained on the SA-Co dataset with millions of images and videos and over 4M unique noun phrases, giving it wide coverage of long-tail concepts.Open-source releaseMeta provides code, weights and example notebooks for inference and fine tuning in the official SAM 3 GitHub repo.SAM 3 for Image Segmentation in CVATNote: This is the first part of a three-step integration. At the moment, SAM 3 is available only for segmentation tasks in a visual-prompt mode (clicks/boxes), not via text prompts. Free tiers (Community and Online Free) get demo-mode access to SAM 3 suitable for evaluation, not high-volume labeling.In the current integration, CVAT exposes the visual side of SAM 3 with point/box based interactive segmentation, because that’s what fits naturally into the existing AI tools UX and doesn’t force you to change your labeling pipelines overnight.Text prompts, open-vocabulary queries, and SAM 3’s native video tracking API are not wired into the UI yet, so it doesn’t behave as a full concept-search engine. Yet, now that it's in CVAT, it remains a very strong interactive segmentation tool compared to other deep learning models, including its predecessor. So, while our engineering team is working on adding the textual prompts annotation, our in-house labeling team decided to test drive SAM 3 labeling capabilities versus SAM 2 on real annotation tasks, and see in which use cases and scenarios each model performs best.SAM 3 vs. SAM 2 Head-to-Head TestTo understand how SAM 3 performs in real labeling workflows, our in-house labeling team compared it with SAM 2 on 18 images covering different object types, sizes, textures, colors, and scene complexity. Both models were tested using interactive visual prompts (points and boxes), with and without refinement.As expected, the results show that there is no single “better” model — each performs best in different scenarios.Where SAM 2 still performs betterSAM 2 tends to produce cleaner, more stable masks with fewer edge artifacts when:Working with simple to medium-complexity objectsObjects have clear, well-defined boundaries and stable shapesSmooth, clean edges are importantAnnotating people on complex backgroundsMinimal refinement and predictable behavior are requiredWhere SAM 3 shows advantagesSAM 3 starts to outperform SAM 2 in more challenging conditions, such as:Complex scenes with many objects, noise, or motion blurObjects where a fast initial shape is more important than perfect boundariesSmall, dense, or touching objects (for example, bacteria)Low-contrast imagery or objects with subtle visual cues, such as soft or ambiguous boundariesWhere both models perform similarlyIn many common cases, both models deliver comparable results:Simple, high-contrast objectsLarge numbers of similar objects annotated individually (for example, grains)Common objects such as flowers, berries, or sports balls when pixel-perfect accuracy is not requiredKey takeawayThere is no universal winner here. At least in the current integration setup. SAM 2 is more stable and predictable, especially around boundaries, while SAM 3 is more flexible and often better suited for complex scenes and hard-to-separate objects. In practice, the best results come from having both tools available and choosing based on the specific task.Get Started with SAM 3 in CVATTo try SAM 3 in CVAT:Create a segmentation task (images or video frames).Open a job in the CVAT Editor.In the right panel, go to AI tools → Interactors.Select Segment Anything Model 3.Use positive and negative clicks or boxes to generate a mask and accept the result.If needed, convert the mask to a polygon and refine it manually.SAM 3 is available in all CVAT editions. In Community and Online Free plans, it runs in demo mode for evaluation purposes.What’s NextThis post covers the first stage of the SAM 3 integration in CVAT: interactive image segmentation. Looking ahead, SAM 3 opens up several directions we’re actively working toward:Text-driven object discovery and pre-labelingMore advanced video object tracking built on SAM 3’s internal tracking capabilitiesWe’ll introduce these features incrementally and share updates as they become stable and ready for production annotation workflows.For now, we anchorage you to try SAM 3 in your next segmentation task and compare it with SAM 2 on your own data.Have questions or feedback? Please reach out via our Help Desk or open an issue on GitHub. Your input helps shape the next steps of the integration.

Point cloud annotation is fundamentally more demanding and complex than 2D labeling. Annotators have to work in sparse, multi-view environments, constantly switch perspectives, and maintain spatial context while dealing with small or partially occluded objects. Our labeling services team knows better than anyone that even minor issues in navigation, zoom behavior, or camera stability can significantly slow down the workflow.That’s why, in our recent release, we’ve focused on practical updates that remove everyday friction and make 3D annotation faster, smoother, and more predictable.#1 Quick object focus with double-clickFinding and focusing on objects in 3D space used to require manual camera rotation and careful positioning. Now, a simple double-click on any object automatically centers the camera on it. This seemingly small change dramatically speeds up annotation, especially when working with smaller objects that are difficult to locate manually. The feature works consistently across both 3D and 2D annotation tasks.#2 Consistent zoom behavior across modes Previously, zoom settings were inconsistent across different annotation modes, requiring constant readjustment. We've standardized the Attribute annotation zoom margin setting to work uniformly across all annotation modes, including 3D side view projections. Even better, zoom levels now persist when switching between objects—no more losing your carefully adjusted view every time you move to the next annotation.#3 Stable and configurable control pointsControl points are essential for precise annotation, but they were behaving erratically in 3D views. They would change size, become transparent, or scale incorrectly with camera distance. We've fixed this completely: the Control point size setting now works properly in 3D, and control points maintain consistent, readable size regardless of camera position or zoom level.#4 Camera stability across projections and view changesSwitching between the main view and side views used to reset the camera position, forcing annotators to re-navigate each time they wanted to check an object from different angles. The camera now maintains its position when moving between projections, allowing smooth multi-angle verification without repetitive navigation.#5 Improved UI readabilityWe've improved the contrast and readability of contextual menus and reduced transparency in areas where it was obscuring text. Small visual improvements like these reduce eye strain and cognitive load during long annotation sessions.#6 Improved zoom limits and touchpad supportWe've increased the maximum zoom level and completely reworked the zoom algorithm to work properly with trackpads. Navigation is now smoother and more predictable, particularly on laptops where trackpad gestures are the primary input method.These updates are available across all CVAT editions: CVAT Online, Enterprise, and Community. Improvements like these come directly from our hands-on work with real customer data and have immediate impact on annotation speed and quality, especially for complex workflows like 3D and point cloud annotation. If you work with point cloud data, we hope these updates will streamline your labeling workflow. Have feedback or suggestions? We'd love to hear from you! Contact our team via HelpDesk or submit an issue on GitHub.

We're excited to announce Personal Access Tokens (PATs), a new and improved authentication method for CVAT's API, SDK, and CLI. If you're building integrations, running automated scripts, or working with CVAT programmatically, this feature is designed to make your workflow more secure and convenient. Why Personal Access Tokens? Until now, authenticating with CVAT's API required using your username and password or legacy API keys. While functional, this approach had some limitations: Security risks: Sharing your password with multiple applications increases exposure if one gets compromised Limited control: No way to restrict what specific applications can do Manual management: Changing your password meant updating it everywhere Personal Access Tokens solve these problems by giving you better control over API access. What Are Personal Access Tokens? Think of PATs as specialized keys for your CVAT account. Instead of using your password in scripts or third-party tools, you create a unique token for each use case. Each token can have: Custom permissions: Choose between read-only or read/write access Expiration dates: Set tokens to automatically expire after a specific time Individual management: Revoke any token instantly without affecting others This means if you suspect a token has been compromised or simply don’t need a specific token anymore, you can revoke just that one token without disrupting your other integrations or changing your password. Key Benefits Enhanced Security: Use separate credentials for each application and eliminate the need to embed passwords in your code. Better Control: Configure exactly what each token can do. Need a script that only reads data? Create a read-only token. Easy Management: Create, edit, and revoke tokens anytime directly from your CVAT user settings. Automatic Cleanup: Unused tokens are automatically removed after a period of inactivity, reducing security risks from forgotten credentials. Getting Started Creating your first Personal Access Token is simple: Navigate to your Profile page Go to the "Security" section Click the "+" button to create a new token Configure the name, expiration date, and permissions Save and securely store the token value (it's only shown once!) Once you have your token, use it in your API requests with the Authorization header: import requests token = "your_token_value" response = requests.get( "https://app.cvat.ai/api/tasks", headers={"Authorization": f"Bearer {token}"} ) Or, if you prefer working with the CLI, set the token as an environment variable: export CVAT_ACCESS_TOKEN="your_token_value" && cvat-cli task ls Important Security Reminders Store tokens securely: Treat them like passwords Set expiration dates: Always configure tokens to expire Use minimal permissions: Grant only the access level needed Revoke immediately: If you suspect a token is compromised, revoke it right away Never share tokens: Each user and application should have its own token Learn More Personal Access Tokens are available now for all CVAT users across Community, Online, and Enterprise editions. We recommend migrating your existing integrations to use PATs for improved security and control. Ready to start using Personal Access Tokens? Check out our complete documentation for detailed instructions on creating, managing, and using PATs in your CVAT workflows. Have questions or feedback? Join the conversation on our Discord or open an issue on our GitHub repository.

Announcing the new Ultralytics YOLO support for automatic annotation via CVAT agents. Powerful computer vision libraries such as Ultralytics YOLO, Detectron2, and MMDetection have made it easier to train high-performing models for a wide variety of tasks. However, using these models for automated annotation often requires custom code, format conversions, and one-off integrations, especially when labeling workflows span multiple tasks. As a result, many teams fall back on manual labeling because they find automation too complex to adopt at scale. Ultralytics YOLO is one of the most widely used model families in the computer vision community. Until now, CVAT included a single built-in YOLO model for auto-annotation, but expanding beyond that required manual setup. That's why we're excited to announce our new integration with Ultralytics YOLO via the CVAT AI annotation agent. Introducing the new Ultralytics YOLO and CVAT integration With this new integration, you can use native Ultralytics models (YOLOv5, YOLOv8, YOLO11) and third-party YOLO models with Ultralytics compatibility (YOLOv7, YOLOv10, etc.) for automatic image or video annotation for a wide range of computer vision tasks, including: Classification Object detection Instance segmentation Oriented object detection Pose estimation Just pick a YOLO model you want to label your dataset with, connect it to CVAT via the agent, run the agent, and get fully labeled frames or even entire datasets, complete with the right shapes and attributes, and all, in a fraction of the time. Annotation possibilities unlocked This integration opens up multiple workflow optimization and automation opportunities for ML and AI teams. Here are just a few. Pre-label data using the right model for the task Connect the YOLO models that match your annotation goals and run them sequentially to pre-label your data. Each model can be triggered individually through the CVAT interface, allowing you to generate different types of labels for the same dataset without custom scripts or external tools. This works for any YOLO model, out-of-the-box or fine-tuned. Label entire tasks in bulk Working with a large dataset? You don’t have to annotate each frame manually. Apply a YOLO model to the entire task in one step. Just open the Actions menu in your task and select Automatic annotation. CVAT will send the job to the agent and automatically annotate all frames across all jobs in a task, saving you time and reducing repetitive work. Share models across teams and projects Register a model once via a native function and agent, and make it instantly available across your organization in CVAT. Team members can use it in their own tasks without any local setup. Validate model performance on real data Test your fine-tuned YOLO model directly on annotated datasets and compare its predictions side-by-side with human labels in CVAT. Spot mismatches, edge cases, or underperforming classes, all without leaving your annotation environment. How it works Here’s what a typical YOLO auto-annotation setup via agents looks like: Step 1. Write and register the function Start by implementing a native function–a Python script that loads your YOLO model (e.g., yolov8n, yolo11m-seg) and defines how predictions will be generated and returned to CVAT. Then register this function in CVAT using the CLI. Note: You can reuse the same native function both in CLI-based annotation and agent-based mode. Step 2. Start the agent Once the function is registered, launch an agent using the CLI command. This starts a local service that automatically connects to your account in CVAT Online or Enterprise, and listens for annotation requests from CVAT. The agent then runs the model (inside your function), generates predictions, and sends them back to CVAT. For more in-depth information about how to set up automated data annotation with a YOLO or any custom model using a CVAT AI agent, read this article. Step 3. Create or select a task in CVAT Log into your CVAT instance and create a new task (or select an existing one). Upload your images or video, and define the labels you want to annotate (e.g., "person", "car", "helmet"). Depending on your use case, you can define different types of labels such as bounding boxes, polygons, or skeletons to match the expected output from your model. Step 4. Choose the model in the UI Once your task and the job are created and the agent is running, go to the AI Tools panel inside your job. Select the Detector tab and the YOLO model you registered earlier. Step 5. Run AI annotation on selected frames After selecting the model, CVAT sends a request to the running agent. The agent runs the model and returns predictions in the form of shapes (e.g., boxes, polygons, or keypoints), each associated with a label ID. Get started now Ready to speed up your annotation workflow with YOLO? Sign in to your CVAT Online account and try it out yourself. For more information about Ultralytics YOLO models and the tasks they support, check the Ultralytics documentation page. For more information about CVAT AI annotation agents, visit Announcing CVAT AI Agents: A New (and Better) Way to Automate Data Annotation using Your Own Models

SAM2 Object Tracking Comes to CVAT Online Through AI Agent Integration Previously on this blog, we described the use of the Segment Anything Model 2 (SAM2) for quickly annotating videos by tracking shapes from an initial frame. However, this feature was limited to self-hosted CVAT Enterprise deployments. We have also covered using arbitrary AI models via agents and auto-annotation functions to annotate a CVAT task from scratch. Today we’ll talk about a new CVAT feature that combines the benefits of the two approaches: tracking support in auto-annotation (AA) functions. This enables each user of CVAT Online to make use of an arbitrary tracking AI model by writing a small wrapper (AA function) around it, and running a worker process (AI agent) on their hardware to handle requests. In addition, we have implemented a ready-to-use AA function based on SAM2, so that users who want to make use of that particular model can skip the first step and just run an agent. In this article we will explain how to use the SAM2-based AA function, as well as walk through some of the implementation details. Quick start Let’s get started. You will need: Installed Python (3.10 or a later version) and Git. An account at either CVAT Online or an instance of CVAT Enterprise version 2.42.0 or later. First, clone the CVAT source repository into some directory on your machine. We’ll call this directory <CVAT_DIR>: git clone https://github.com/cvat-ai/cvat.git <CVAT_DIR> Next, install the Python packages for CVAT CLI, SAM2 and Hugging Face Hub: pip install cvat-cli -r <CVAT_DIR>/ai-models/tracker/sam2/requirements.txt If you have issues with installing SAM2, note that the SAM2 install instructions contain solutions to some common problems. Next, register the SAM2 function with CVAT and run an agent for it: cvat-cli --server-host <CVAT_BASE_URL> --auth <USERNAME>:<PASSWORD> \ function create-native "SAM2" \ --function-file=<CVAT_DIR>/ai-models/tracker/sam2/func.py -p model_id=str:<MODEL_ID> cvat-cli --server-host <CVAT_BASE_URL> --auth <USERNAME>:<PASSWORD> \ function run-agent <FUNCTION_ID> \ --function-file=<CVAT_DIR>/ai-models/tracker/sam2/func.py -p model_id=str:<MODEL_ID> where: <CVAT_BASE_URL> is the URL of the CVAT instance you want to use (such as https://app.cvat.ai). <USERNAME> and <PASSWORD> are your CVAT credentials. <FUNCTION_ID> is the number output by the function create-native command. <MODEL_ID> is one of the SAM2 model IDs from Hugging Face Hub, such as facebook/sam2.1-hiera-tiny. Optionally: Add -p device=str:cuda to the second command to run the model on your NVIDIA GPU. By default, the model will run on the CPU. Add --org <ORG_SLUG> to both commands to share the function with your organization. <ORG_SLUG> must be the short name of the organization; it is the name displayed under your username when you switch to the organization in the CVAT UI. The last command should stay running, indicating that the agent is listening to annotation requests from the server. This completes the setup steps. Now you can try the function in action: Open the CVAT UI. Create a new CVAT task or open an existing one. The task must be created either from a video file or from a video-like sequence of images (all images having the same dimensions). Open one of the jobs from the task. Draw a mask or polygon shape around an object. Right-click the shape, open the action menu and choose “Run annotation action”. Choose “AI Tracker: SAM2” in the window that appears. Enter the number of the last frame that you want to track the object to and press Run. Wait for the annotation process to complete. Examine the subsequent frames. You should now see a mask/polygon drawn around the same object on every frame up to the one you selected in the previous step. Instead of selecting an individual shape, you can also track every mask & polygon on the current frame by opening the menu in the top left corner and selecting “Run actions”. Implementation Now let’s take a peek behind the curtains and see how the SAM2 tracking function works. This will be useful if you need to troubleshoot, or if you want to implement a tracking function of your own. Unfortunately, the source of the module is too long to explain in its entirety in this article, but we’ll cover the overall structure and key implementation features. First, let’s look at the top-level structure of func.py: @dataclasses.dataclass(frozen=True, kw_only=True) class _PreprocessedImage: ... @dataclasses.dataclass(kw_only=True) class _TrackingState: ... class _Sam2Tracker: ... create = _Sam2Tracker Since we wanted to support multiple model variants, as well as multiple devices, with a single implementation, we did not place the function’s required attributes directly in the module. Instead, we put them inside a class, _Sam2Tracker, which we want to be instantiated by the CLI with the parameters passed via the -p option. To tell the CLI which class to instantiate, we alias the name create to our class. There are also two auxiliary dataclasses, _PreprocessedImage and _TrackingState. These are not part of the tracking function interface, but an implementation detail. We will see their purpose later. Let’s now zoom in on _Sam2Tracker. __init__ and spec Similar to detection functions that we’ve covered before, in the constructor we load the underlying model (SAM2VideoPredictor). We also create the PyTorch device object and create an input transform. def __init__(self, model_id: str, device: str = "cpu") -> None: self._device = torch.device(device) self._predictor = SAM2VideoPredictor.from_pretrained(model_id, device=self._device) self._transform = torchvision.transforms.Compose([...]) Also similar to detection functions, our tracker must define a spec, although it has to be of type TrackingFunctionSpec: spec = cvataa.TrackingFunctionSpec(supported_shape_types=["mask", "polygon"]) In a tracking function, the spec describes which shape types the function is able to track. However, the other attributes of _Sam2Tracker are entirely unlike those of detection functions. On a high level, a tracking function must analyze an image with a shape on it, then predict the location of that shape on other images. However, to allow more efficient tracking of multiple shapes per image, as well as to enable interactive usage, this functionality is split across three methods. preprocess_image def preprocess_image( self, context: cvataa.TrackingFunctionContext, image: PIL.Image.Image ) -> _PreprocessedImage: image = image.convert("RGB") image_tensor = self._transform(image).unsqueeze(0).to(device=self._device) backbone_out = self._predictor.forward_image(image_tensor) ... return _PreprocessedImage( original_width=image.width, original_height=image.height, vision_feats=...(... backbone_out...), ..., ) This method is supposed to perform any processing that the function can do without knowing the details of the shape it’s tracking. In this way, the results can be reused for multiple shapes. In our case, the underlying model has a dedicated method for doing this, so we transform our input image, and pass it to this method. We then return all information we’ll need later as a new instance of our class _PreprocessedImage. The agent does not care what type of object is returned by preprocess_image - it just saves that object so it can pass it to the other methods. Speaking of which… init_tracking_state def init_tracking_state( self, context: cvataa.TrackingFunctionShapeContext, pp_image: _PreprocessedImage, shape: cvataa.TrackableShape, ) -> _TrackingState: mask = torch.from_numpy(self._shape_to_mask(pp_image, shape)) resized_mask = ...(... mask ...) current_out = self._call_predictor(pp_image=pp_image, mask_inputs=resized_mask, ...) return _TrackingState( frame_idx=0, predictor_outputs={"cond_frame_outputs": {0: current_out}, ...}, ) def _call_predictor(self, *, pp_image: _PreprocessedImage, frame_idx: int, **kwargs) -> dict: out = self._predictor.track_step( current_vision_feats=pp_image.vision_feats, frame_idx=frame_idx, ... **kwargs, ) return ...(... out ...) This method is supposed to analyze the shape on the initial frame. Here we convert the input shape to a mask tensor (for brevity we’ll omit the definition of _shape_to_mask here), and then pass it, alongside the preprocessed image, to the underlying model (via a small wrapping function). The method then encapsulates all information that will be needed to track the shape on subsequent frames in a new _TrackingState object and returns it. Much like preprocess_image, the agent doesn’t care what type of object the method returns, so the tracking function can choose the type in order to best suit its own needs. The agent will simply pass this object into our final method… track def track( self, context: cvataa.TrackingFunctionShapeContext, pp_image: _PreprocessedImage, state: _TrackingState ) -> cvataa.TrackableShape: state.frame_idx += 1 current_out = self._call_predictor( pp_image=pp_image, frame_idx=state.frame_idx, output_dict=state.predictor_outputs, ... ) non_cond_frame_outputs = state.predictor_outputs["non_cond_frame_outputs"] non_cond_frame_outputs[state.frame_idx] = current_out ... output_mask = ...(... current_out["pred_masks"] ...) if output_mask.any(): return self._mask_to_shape(context, output_mask.cpu()) else: return None This method is supposed to locate the shape being tracked on another frame. Here we pass data from the state object and the preprocessed image to the model and get a mask back. If the mask has any pixels set, we return it as a TrackableShape object. The _mask_to_shape method (whose definition we’ll omit) will convert the mask to a shape of the same type as the original shape passed to init_tracking_state. If the mask is all zeros, we presume that we lost track of the shape, and return None. The model also returns additional data that can be used to better track the shape on subsequent frames. track adds it to the tracking state, as can be seen with the non_cond_frame_outputs update. This way, future calls to track are able to make use of this data. Agent behavior Now that we’ve examined the purpose of each method, we can see how they all fit together by looking at the tracking process from the agent’s perspective. Let’s say an agent has loaded tracking function F, and a user makes a request for shape S0 from image I0 to be tracked to images I1, I2, and I3. In this case, the agent will make the following calls to the tracking function: STATE = F.init_tracking_state(SC, F.preprocess_image(C, I0), S0) S1 = F.track(SC, F.preprocess_image(C, I1), STATE) S2 = F.track(SC, F.preprocess_image(C, I2), STATE) S3 = F.track(SC, F.preprocess_image(C, I3), STATE) It will then return resulting shapes S1, S2, and S3 to CVAT. Here C and SC are context objects, created by the agent. For more information on these, please refer to the reference documentation. Limitations There are a few things to keep in mind when using tracking functions (SAM2 included). First, agents currently keep the tracking states in their memory. This means that: Only one agent can be run at a time for any given tracking function. If you run more than one agent for a function, users may see random failures as agents try to complete requests referencing some other agent’s tracking states. If the agent crashes or is shut down, all tracking states are destroyed. If this happens while a user is tracking a shape, the process will fail. Second, tracking functions can only be used via agents. There is no equivalent of the cvat-cli task auto-annotate command. Third, tracking functions may be used either via an annotation action (as was shown in the quick start), or via the AI Tools dialog (accessible via the sidebar). However, the latter method only works with tracking functions that support rectangles - other functions will not be selectable. Fourth, skeletons cannot currently be tracked. Conclusion Tracking with SAM2 saves significant time compared to manually annotating each frame. If you are a user of CVAT Online, this feature is now available to you - sign in and try it out! If there is another model you’d like to use for tracking, you can likely do that as well, as long as you implement the corresponding auto-annotation function. For more details on that, refer to the reference documentation: https://docs.cvat.ai/docs/api_sdk/sdk/auto-annotation/ https://docs.cvat.ai/docs/api_sdk/cli/#examples---functions For more information on other capabilities of AA functions and AI agents, see our previous articles on the topic: https://www.cvat.ai/resources/blog/an-introduction-to-automated-data-annotation-with-cvat-ai-cli https://www.cvat.ai/resources/blog/announcing-cvat-ai-agents https://www.cvat.ai/resources/blog/cvat-ai-agents-update

Whether it's a small university research or a large enterprise activity, project owners often face similar challenges. They need to maintain consistent quality, track team productivity effectively, and avoid extra costs — no matter what tools they use. CVAT addresses these challenges by providing clear, detailed, and easy-to-understand analytics that includes all the necessary metrics for annotation projects, tasks, and jobs. This allows managers to effortlessly monitor progress, pinpoint productivity bottlenecks, making the annotation workflows smoother and more efficient. But what exactly does CVAT Analytics offer, how do you access Analytics data, and how can you practically use it in your projects? In this article, you'll discover how CVAT Analytics helps to approach these challenges by providing practical tools and actionable insights. What is CVAT Analytics? CVAT Analytics provides insightful metrics for project managers and annotation teams to monitor and improve the annotation workflows. The following outlines the types of metrics that can be tracked: Working time: See exactly how much time annotators spend on tasks. Monitoring time allocation for job stages: Track how long each stage of annotation takes, helping identify slow stages. Quantifying total objects annotated: Keep accurate counts of annotated objects to evaluate productivity. Measuring annotation speeds: Monitor the pace at which annotations are completed and identify efficient annotators or potential issues. Identifying annotation activity and label usage: Gain insights into how labels are being used and annotation patterns. Accessing CVAT Analytics Analytics is available only to users with paid CVAT Online plans or with CVAT Enterprise. The level of access users have depends on their role within CVAT: Owners and Maintainers: Can access analytics for all projects, tasks, and annotation jobs across their workspace. For example, a project owner can review metrics of all team activities to estimate the overall productivity and resource allocation. Supervisors: Can access analytics data only for the projects, tasks, and jobs they have visibility over. For instance, a supervisor overseeing two specific tasks can see analytics for those tasks but not other unrelated projects. Workers: Have access only to analytics related to tasks and jobs assigned directly to them. For example, an annotator will see metrics for their assigned job, allowing them to track their own productivity and performance. Navigating CVAT Analytics To access analytics data in CVAT, navigate to the overview page where all your projects, tasks, or jobs are listed. Find the specific project, task, or job for which you need analytics data, click the three-dot menu next to it, and select "Analytics." When opening the Analytics page for the first time, no data will be displayed immediately. You'll need to click the "Request" button to load the data. This will gather analytics for the selected item and any associated tasks or jobs. Whenever updated analytics are needed, simply click the "Request report update" button to refresh the metrics. Understanding the Analytics Page The Analytics page in CVAT is divided into several tabs, each giving a different view of annotated data to help with tracking progress and improving performance. The Summary tab gives you a quick overview of key project metrics. By default, data is shown for the entire lifetime, but it can be filtered by specifying a UTC start and end date. You can also view the number of created and deleted objects, along with the overall difference in the Objects Diff section. Total Working Time displays the cumulative time spent across all events. Average Annotation Speed indicates the average number of objects annotated per hour. You’ll also see the total working time spent by all annotators, and the average annotation speed, calculated in the number of annotated objects per hour. Scroll down to view pie charts displaying annotation data for shapes and tracks, broken down by type and label. Hover over each segment to see a tooltip with additional information. The Annotations tab breaks down annotation statistics further, depending on the type of annotation. The Detection tab shows count by shapes, there you’ll find a number of objects labeled per category (polygons and masks). This is useful when you want to check whether the distribution of labels aligns with your dataset goals. In the Tracking tab, you’ll get data on how many keyframes, interpolated frames, and object tracks were annotated. Both views come with searchable, filterable tables that you can export annotation statistics or raw events if needed. The Events tab gives a deeper look into what happened and when. This is the most important tab. This tab allows you to track how, when, and by whom each child's job was changed. It shows how everything changed over time. The Total objects, Total images, Total working time, and Avg. annotation speed, which are recalculated automatically depending on the selected filters. You can see who the task and job were assigned to, as well as the annotation stage and current status. On all three tabs of the Analytics page, you can use the calendar to select the time period for which you want to view analytics. For example, you can see if a particular annotator spent extra time in a specific stage or made a excessive number of edits. This level of detail helps identify inconsistencies or inefficiencies in the workflow. Events are grouped based on the job's status and who performed the actions, making it easier to follow the history of work done over time. The Export Events button downloads raw, non-aggregated event data based on the selected dates for users who need custom analytics beyond what's shown on the Events tab. Each table allows users to customize visible columns. Note that not all columns are shown by default on the Events tab. Best Practices for Using CVAT Analytics To get the most out of CVAT Analytics, many teams apply a few simple habits that make a big difference in their workflow. These practices help ensure annotations are not only accurate but also completed efficiently. For instance, in a project focused on labeling traffic signs for autonomous vehicles, a small team of annotators works across multiple batches of city footage. The project manager downloads and reviews analytics reports every Friday. They look for patterns like a sudden drop in label volume or a spike in rejected tasks. Lets say, one of team members has consistently low object counts after a schema change.Weekly review helps the manager catch this early and leads to a quick clarification of the labeling rules. Without the analytics check, several batches could have gone out with missing data. In another scenario, a healthcare research group is annotating MRI scans with regions of interest. Different annotators handle different patient sets. Over time, the team notices that one annotator is completing far fewer images than the others. Analytics shows they spend significantly more time per image, because, as it turned out, they’re unsure how to label edge cases in a new category. With that insight, the team arranges a brief retraining session and updates their labeling guide. Productivity improves, and uncertainty drops across the board. Monitoring quality metrics can also prevent wasted effort on downstream tasks. In a project detecting damaged packages from warehouse photos, annotation speed and object counts are tracked closely. If an annotator suddenly doubles their speed but the object count per image drops, it may signal they’re rushing or misunderstanding instructions. This could be due to the fact that a new guideline wasn't fully communicated, and several batches had to be rechecked. Having access to annotation speed and density trends helped the lead catch the issue before model training began. How to Use CVAT Analytics: Step-by-Step Guide CVAT Analytics helps teams keep their annotation projects on track by showing clear, useful data. It makes it easier to spot problems early, check the quality of work, and make sure tasks are shared fairly among team members. Whether the project is small or large, using analytics regularly can save time and improve results Still have questions? Check out the Analytics documentation or watch a short video that explains everything in detail. Ready to explore the new analytics? Create a CVAT account to get started, or contact us to deploy CVAT on your own premises.

We're thrilled to introduce Immediate Job Feedback—a powerful new feature in CVAT that takes annotation quality to the next level. Immediate Job Feedback provides annotators with real-time quality insights right after completing a job, helping ensure every annotation meets your project quality standards.This new update streamlines workflows, boosts annotation accuracy, and gives you confidence that your labeling meets the expected quality—every single time.What Is Immediate Job Feedback?Immediate Job Feedback is a smart CVAT capability that delivers instant feedback on annotation quality after a job is completed. While annotators work, CVAT automatically evaluates their performance using its built-in quality control tools – specifically the Ground Truth and Honeypots (explained here).Once the task is finished, CVAT compares the results against a predefined quality threshold and immediately displays a pop-up summary to the annotator: the message clearly states whether the quality meets the required standard or needs improvement.How to Set It UpNote: ​​Immediate Job Feedback is available with a paid subscription to CVAT Online and with the Enterprise edition of CVAT On‑Premises.To enable it:Go to Actions → Quality Control → SettingsDefine the following parameters: Target metricTarget metric threshold (your quality bar)Max validations per job (the maximum number of job validations per assignee)Once configured, annotators will automatically receive a quality summary window after every completed job.Instant Rework for Better ResultsIf the annotation quality falls below your predefined threshold, annotators can immediately re-annotate the job to meet the necessary standards – no delays, no guesswork.Detailed instructions on setting up Immediate Job Feedback can be found in our documentation.Drive Accuracy, Speed, and Insight – Right NowImmediate Job Feedback brings CVAT one step closer to a smarter, more responsive annotation environment.Ready to integrate Immediate Job Feedback into your pipeline? Sign up or log in to your CVAT Online account to try Immediate Job Feedback today, or contact us to enable it on your self-hosted Enterprise instance.

Introducing Honeypots: Smarter Quality Checks, Smaller Validation SetsAt CVAT, we know how important quality is when it comes to labeling large volumes of data. A small mistake can have serious consequences, such as a misclassified traffic light impacting autonomous vehicle safety or mislabeled medical images impacting patient care. That's why validation is crucial to ensure annotations meet quality standards before training AI models.To help our customers maintain high labeling standards, CVAT already supports multiple validation approaches, including manual reviews and Ground Truth (GT) jobs. While both methods deliver excellent accuracy, they come with significant trade-offs: manual reviews require a lot of expert time, and GT jobs need carefully curated validation sets — making them expensive, time-consuming, and hard to scale for large volumes of data.That’s why we’re excited to introduce Honeypots — a powerful new addition to CVAT’s automated quality assurance workflow. Designed for scalability, honeypots make it possible to validate large datasets more efficiently and cost-effectively, especially when traditional methods become impractical.What are Honeypots?Honeypots are a smart way to monitor annotation quality without disrupting your team’s workflow. It works by randomly embedding extra validation frames—so-called “honeypots”—directly into your labeling tasks.With this validation mode, annotators don’t know which frames are being checked, so you can measure attentiveness, consistency, and accuracy in a completely natural and unobtrusive way. Why and When Use Honeypots?Quality assurance in data labeling traditionally requires significant resources — either extensive manual reviews or large validation sets. Honeypots offer a more scalable and efficient solution that maintains high standards while reducing overhead.Consider a medical imaging project with 10,000 images. Traditional validation might use 100 expert-annotated images (1% of the dataset) for quality checks—far from ideal when patient care is at stake. This is where Honeypots transform the validation process. Instead of using those 100 validated images just once, Honeypots let you embed them multiple times throughout your annotation pipeline, achieving 5-10% validation coverage without requiring additional expert input.By embedding a small, independent validation set randomly throughout your project, Honeypots' approach ensures that even the largest datasets stay under consistent quality control without requiring expensive and time-consuming Ground Truth validation for every batch. And, since Honeypots reuse the same validation set across multiple jobs, they scale automatically as your project grows, so you don’t need to generate new validation data for each task. This dramatically reduces both validation time and cost while maintaining high-quality standards.Combined with Immediate Feedback, this validation technique creates a highly reliable yet cost-effective validation system that scales effortlessly with your projects.How Honeypots WorkAs Honeypots are based on the idea of a validation set — a sample subset used to estimate the overall quality of a dataset—you’ll need to create a Honeypots validation set first when creating your task and select your validation frames. The whole validation set is available in a special Ground Truth job, which needs to be annotated separately.Note: It’s not possible to select new validation frames after the task is created, but it’s possible to exclude “bad” frames from validation. Honeypots are available only for image annotation tasks and aren’t supported for ordered sequences such as videos. Next, these validation frames are randomly mixed into regular annotation jobs. While your annotators work on the project, CVAT tracks how accurately annotators handle these ‘hidden’ honeypot frames. After the job is done, you can go to the analytics page to see all the errors and inconsistencies within the project. There, you get quality scores and error analyses for each job, the validation frame, and the whole task. No need for extra tools or manual comparison. Just label, review, and improve.For more detailed setup instructions, shape comparison notes, and analytics explanations, read our full guide.Try Automated QA and Honeypots TodayThe Honeypots QA mode is available out of the box in all CVAT versions, including the open-source edition. However, using it for quality analysis requires a paid subscription to CVAT Online or the Enterprise edition of CVAT On‑Premises.Ready to catch bad labels before they bite? Sign up or log in to your CVAT Online account to try honeypots today, or contact us to enable them on your self-hosted Enterprise instance.‍

We’re excited to announce one of our most requested features — Keyboard Shortcut Customization. This update puts you in full control of how you interact with CVAT, making your workflow more comfortable, personalized, and highly efficient.What’s NewWith this new update, you can:Customize a wide range of shortcuts: create your own button combinations both globally for the whole application and scope-limited for specific sections or workspaces.Never worry about setting up anything incorrectly: CVAT will automatically warn you about any detected shortcut conflict, helping you configure everything smoothly.Choose any button combination: customize to what feels right to you. Shortcuts can be any combination of modifiers (Ctrl, Shift, or Alt) and up to one non-modifier key (e.g., Ctrl+Shift+F1). Only a few browser-linked combinations are limited.Assign shortcuts for almost every action — from drawing and switching tools to managing playback and annotation, you have the freedom to set up shortcuts that fit your workflow.Why We Built ItUntil now, CVAT users worked with a fixed set of shortcuts. While functional, they weren’t always the most intuitive or comfortable for everyone. We realized that giving you the power to customize shortcuts — and fully tailor your workspace to your preferences — was not just a nice-to-have, but absolutely essential.How to Set Up Your Custom ShortcutsSetting up your shortcuts is simple and quick. You’ll find the shortcut customization panel under: Settings → Shortcuts.From there, browse workspaces from a category menu,choose the action for your own shortcut where you see fit,press your preferred key combo — and you're all set!Your custom shortcuts are saved directly in your browser, so they’ll remain set even after you reload the page. They’ll only reset if you clear your browser’s cache. If you use a different browser or device to access CVAT, you’ll need to set your shortcuts again there. And if you ever change your mind about any shortcut — you can easily update them or click "Restore Defaults" to reset everything. Check out this article for detailed step-by-step instructions.Start Customizing Your Shortcuts TodaySign up or log in to your CVAT Online account, or reach out to us to get expert assistance on starting your annotation journey.

In January, we announced AI agents—a new feature for integrating your machine learning models with CVAT. Since then, we’ve been working hard on improving this feature. In this post, we’d like to share our progress.But first, a quick recap: an auto-annotation (AA) function is a Python object that acts as an adapter between the CVAT SDK and your ML model. Once implemented, you can either:Use the cvat-cli task auto-annotate command to annotate a CVAT task, orRegister it with CVAT and then start one agent via the cvat-cli function create-native and cvat-cli function run-agent commands. After that, use the CVAT UI to select a task to annotate and its parameters. If you register in an organization, other members can use it as well.Now, let’s see what’s new.Skeleton supportAn AA function may output any CVAT-known shapes. However, when agents debuted, AA functions couldn’t output skeletons with the agent-based workflow. In CVAT and CVAT CLI version 2.32.0, this has been rectified.Let’s implement an AA function with skeleton outputs using a YOLO11 pose model from the Ultralytics library and see how it works.1) We’ll start with an empty source file yolo11_pose_func.py and add the necessary imports:import PIL.Image from ultralytics import YOLO import cvat_sdk.auto_annotation as cvataa import cvat_sdk.models as models2) Then, we need to create an instance of the YOLO model:_model = YOLO("yolo11n-pose.pt")3) To turn our file into an AA function, we’ll need to add two things. First, a spec – a description of the labels that the function will output.spec = cvataa.DetectionFunctionSpec( labels=[ cvataa.skeleton_label_spec(name, id, [ cvataa.keypoint_spec(kp_name, kp_id) for kp_id, kp_name in enumerate([ "Nose", "Left Eye", "Right Eye", "Left Ear", "Right Ear", "Left Shoulder", "Right Shoulder", "Left Elbow", "Right Elbow", "Left Wrist", "Right Wrist", "Left Hip", "Right Hip", "Left Knee", "Right Knee", "Left Ankle", "Right Ankle", ]) ]) for id, name in _model.names.items() ], )For each class that the model supports, we use skeleton_label_spec to create the corresponding label spec and keypoint_spec ⁣to create a sublabel spec for each keypoint. The Ultralytics library doesn’t provide a way to dynamically determine the supported keypoints, so we have to hardcode them in our function. Our hardcoded list is taken from Ultralytics’s pose estimation documentation.Note that each label and sublabel spec requires a distinct numeric ID. For the skeleton as a whole, we use the class ID that Ultralytics gives us, whereas for the sublabels we just assign sequential IDs.The other thing we need to add is a detect function that performs the actual inference.def detect( context: cvataa.DetectionFunctionContext, image: PIL.Image.Image ) -> list[models.LabeledShapeRequest]: conf_threshold = 0.5 if context.conf_threshold is None else context.conf_threshold return [ cvataa.skeleton( int(label.item()), [ cvataa.keypoint(kp_index, kp.tolist(), outside=kp_conf.item() < 0.5) for kp_index, (kp, kp_conf) in enumerate(zip(kps, kp_confs)) ], ) for result in _model.predict(source=image, conf=conf_threshold) for label, kps, kp_confs in zip(result.boxes.cls, result.keypoints.xy, result.keypoints.conf) ]The first thing detect does is determine the confidence threshold the user has specified (defaulting to 0.5 if they didn’t specify any). Then it calls the model and creates a CVAT skeleton shape for each detection returned by the model. Keypoints with low confidence are marked with the outside property so that CVAT hides them from the view.Having implemented our function, we can integrate it with CVAT in the usual way. First, we’ll need to install CVAT CLI and the Ultralytics library:pip install cvat-cli ultralyticsThen, we can register our function with CVAT and run an agent for it:cvat-cli --server-host <CVAT_BASE_URL> --auth <USERNAME>:<PASSWORD> \ function create-native "YOLO11n-pose" --function-file yolo11_pose_func.py cvat-cli --server-host <CVAT_BASE_URL> --auth <USERNAME>:<PASSWORD> \ function create-native <FUNCTION_ID> --function-file yolo11_pose_func.pywhere:<CVAT_BASE_URL> is the URL of the CVAT instance you want to use (such as https://app.cvat.ai).<USERNAME> and <PASSWORD> are your credentials.<FUNCTION_ID> is the number output by the first command.Now we can try out our function in action. To do so, we’ll need to open CVAT and create a new task with a skeleton label (or add such a label to an existing task). To this label, we’ll add keypoints corresponding to the keypoints in the model:Now we can open the task page, click Actions → Automatic annotation and select our model:Pressing “Annotate” will begin the annotation process. Once it’s complete, we can open a frame from the task and see the results:Attribute supportCVAT allows shapes to include attributes: extra pieces of data that pertain to a given object. The allowed attributes set is configured per label, as well as each attribute’s type and allowed values. Skeleton keypoints may have their individual attributes as well.As of CVAT and CVAT CLI version 2.31.0, AA functions may define labels with attributes, and output shapes with attributes. This works with both the direct annotation and agent-based workflows.To demonstrate this feature, we’ll implement a function that recognizes text via the EasyOCR library. The function will output rectangle shapes, each with an attribute containing the recognized text string.Here’s the function code:import PIL.Image import easyocr import numpy as np import cvat_sdk.auto_annotation as cvataa import cvat_sdk.models as models from cvat_sdk.attributes import attribute_vals_from_dict _reader = easyocr.Reader(['en']) spec = cvataa.DetectionFunctionSpec( labels=[ cvataa.label_spec("text", 0, type="rectangle", attributes=[ cvataa.text_attribute_spec("string", 0), ]) ], ) def detect( context: cvataa.DetectionFunctionContext, image: PIL.Image.Image ) -> list[models.LabeledShapeRequest]: conf_threshold = 0.5 if context.conf_threshold is None else context.conf_threshold input = np.array(image.convert('RGB'))[:, :, ::-1] # EasyOCR expects BGR return [ cvataa.rectangle( 0, list(map(float, [*points[0], *points[2]])), attributes=attribute_vals_from_dict({0: string}), ) for points, string, conf in _reader.readtext(input) if conf >= conf_threshold ]It has the same basic elements as our previous function. To output our attribute, we declare it in the spec (via the attributes argument of label_spec) and specify its value for each output rectangle (via the attributes argument of rectangle).Note that since we have only one label and one attribute, we just hardcode 0 as the ID for both of them.To see our function in action, we’ll need to install the dependencies:pip install cvat-cli easyocrThen, as before, register the function and run an agent:cvat-cli --server-host <CVAT_BASE_URL> --auth <USERNAME>:<PASSWORD> \ function create-native "EasyOCR" --function-file easyocr_func.py cvat-cli --server-host <CVAT_BASE_URL> --auth <USERNAME>:<PASSWORD> \ function create-native <FUNCTION_ID> --function-file easyocr_func.py‍We’ll need to create a task with a label and an attribute that matches our function:Then press "Actions → Automatic annotation", select the model, confirm and see the results:‍Label typesThe last improvement we’d like to discuss is subtle. Let’s revisit the function spec from the previous section:cvataa.label_spec("text", 0, type="rectangle", ...Since the introduction of the AA function interface, the ability to specify a label type in a spec was present but previously ignored. However, since CVAT and CVAT CLI 2.29, specifying this type has small but beneficial effects:When you select which task label to map a function’s label to in the “Automatic annotation” dialog, CVAT will only offer task labels whose type is compatible with the function’s type. For example, if the function label has type “rectangle,” then CVAT will only offer task labels of types “rectangle” and “any.” This prevents you from accidentally adding shapes of an unwanted type via automatic annotation.When you use the “From model” button to copy labels from a function to a task, the specified label type will be set on the copied label.If a function outputs a shape whose type is inconsistent with the declared type of the shape’s label, the CVAT CLI will abort the automatic annotation. This catches function implementation mistakes.Label type declarations are optional. By default, each label spec is assumed to have type “any,” except for labels created with skeleton_label_spec, whose type will be “skeleton.” However, we recommend you declare your label types, as it is easy to do and will help prevent implementation and usage mistakes.ConclusionWith these changes, agent-based functions are catching up to the capabilities of Nuclio-based functions. However, unlike Nuclio-based functions, they can be integrated with CVAT Online and other CVAT instances without server control.We’re continuing to work on this feature to add more capabilities, so stay tuned for updates.

Say goodbye to manual frame-by-frame labeling. Speed up video annotation workflows with SAM 2 Tracker by CVAT. *** Note: This feature is available only to CVAT Enterprise Basic and Premium accounts. We're excited to announce that CVAT now supports automated video annotation with Segment Anything Model 2 (SAM 2) Tracker. SAM 2 is the successor to Meta AI's advanced foundation model, designed for real-time, comprehensive object segmentation in images and video. SAM 2, released in July 2024, allows users to detect and segment any object in an image or video based on specific input prompts, like interactive points, bounding boxes, or masks. Once segmented, the model tracks them across video frames in real-time, ensuring accuracy and consistency. Evolution of SAM-powered annotation in CVAT When SAM's first edition was released in 2023, we integrated it into CVAT's SaaS and on-premises versions within weeks, allowing customers to enhance their image annotation tasks. When SAM 2 was released in 2024, we quickly followed suit to provide faster and more accurate segmentation. An aerial view demonstrating automated segmentation of agricultural fields and crop health conditions using SAM 2 in CVAT. We’ve extended its object segmentation and tracking capabilities to video, building on the success of these integrations. Let's examine how SAM 2 Tracker works and how it can improve your video annotation workflows. Bringing the power of SAM 2 to video annotation Labeling videos is essential for training AI models in industries relying on video data, like autonomous vehicles, sports analysis, and robotics. Those models need a large amount of accurately labeled data—from hundreds to millions of videos—to function reliably. Labeling vast data is a challenging task. Video annotation, unlike image labeling, adds a temporal dimension, increased data volume, and the need for frame consistency. Automated annotation tools like CVAT’s new SAM 2 Tracker are essential to alleviating these challenges by streamlining the process and reducing manual effort. In CVAT, there are a few methods to annotate videos: The old-school manual, frame-by-frame labeling which requires drawing annotations on every frame. And, interpolation-based labeling, where annotations are placed on keyframes and automatically propagated across intermediate frames. While those two options remain viable for simpler scenarios or limited-scope projects, SAM 2 Tracker significantly enhances the convenience and speed of object segmentation and tracking in videos, especially for complex scenes with rapid movements or frequent obstructions. https://elements.envato.com/protozoa-single-cell-organisms-in-microscope-brigh-7BQPJJP https://www.pexels.com/video/aerial-view-of-industrial-site-2073129/ Key features of CVAT’s SAM 2 Tracker Instant segmentation: The SAM 2 Tracker outlines the contours of an object in a single frame when you click on it. Automatic tracking: The tracker preserves the object's shape and position as it moves across frames. Support for complex objects: Works effectively with partial overlaps or changes in the background. Interactive refinement: Adjust annotations at any stage. How to Use SAM 2 Tracker in CVAT Follow these steps to segment and track objects in your videos with SAM 2: Open your CVAT account and select the video you want to annotate from the list of the annotation tasks. In the annotation toolbar, select the "Magic Wand." Then, use the Interactor tab to choose the label and SAM 2 to generate a segmentation mask for your object on the first (zero) frame. Important: For the Tracker to work, don't forget to turn on the "Convert the mask into a polygon" slider, because the mask cannot be further converted into a "Track" mode, and the Tracker will not be able to track an object annotated with a mask as a single element across multiple frames. In the right-hand Objects panel, click the three-dot menu (⋮) next to your polygon, then select "Run annotation action." Select "SAM 2 Tracker" from the pop-up menu and set the number of frames to track. Important: If you annotated the object with a polygon “Shape,” don’t forget to convert them into “Track” mode before running the Tracker. SAM 2 will track your polygon across subsequent frames. Note: Due to deployment requirements, SAM 2 Tracker is currently available exclusively for CVAT’s On-Prem paid accounts (Enterprise Basic and Premium). Getting started For more information about SAM 2 Tracker, visit our documentation. For SAM 2 details, visit its site and GitHub. If you have an Enterprise account and want to install SAM 2 Tracker, contact our support team. If you don’t have a CVAT On-prem account or use CVAT Online and want to try SAM 2 for video annotation, contact our sales team.