The best way to Integrate Computer Vision Pipelines with Generative AI and Reasoning

Ingesting large amounts of video is difficult because the information is unstructured, continuous, and very high-volume, which makes it difficult to look, index, or summarize efficiently. A single video can span hours of footage, include multiple events happening directly, and require heavy compute simply to decode and analyze. Standard computer vision pipelines often can’t sustain at scale, producing isolated detections without the broader context needed to know what’s actually happening.

VSS solves this problem through the use of a GPU-accelerated video ingestion pipeline. As a video file or live stream is available in, it’s broken into smaller chunks and Cosmos Reason VLM generates a wealthy description or caption of every chunk. An LLM then extracts the vital information from the VLM generated captions to construct a knowledge graph that captures the vital details of the video. Once the knowledge graph is built a big language model traverses the graph to reply user’s questions on the videos. 

VSS 2.4 enhances Q&A accuracy and cross camera understanding with: 

• Entity deduplication within the knowledge graph
• Agent-based graph traversal
• CUDA-accelerated graph database 

In previous releases of the VSS Blueprint, constructing the knowledge graph could end in duplicate nodes and edges. In VSS Blueprint 2.4, a knowledge graph post-processing has been added to remove any duplicate entries and merge nodes and edges which are common across videos. Which means that common entities similar to the identical automobile moving across multiple cameras at the moment are merged right into a single entity which improves the power of VSS to know unique objects as they move throughout a video and across cameras. 

Once the knowledge graph has been generated and post processed, an LLM is used to traverse the graph and gather the vital information to reply the user’s query in regards to the videos. 

In VSS 2.4, agentic based-reasoning has been introduced for advanced knowledge graph retrieval. If enabled, an LLM based agent will intelligently decompose the query after which use a set of tools to look the graph, find relevant metadata, reinspect sampled frames from the video and iterate if vital to accurately answer the user’s query. 

It’s now possible to reply questions across multiple camera streams using the knowledge graph postprocessing to merge entities and relationships and the advanced agentic-based retrieval.

To supply developers with the newest tools, the supported Graph database backends have been expanded to incorporate ArangoDB. Users now have the power to configure VSS to make use of either the Neo4J or ArangoDB graph database backend. ArangoDB brings a set of enhancements, including CUDA-accelerated graph functions to speed up knowledge graph generation. For more details on the ArangoDB integration into VSS, see Generate a Video Knowledge Graph: NVIDIA VSS Blueprint with GraphRAG on ArangoDB.

These latest features to enable knowledge graph generation and agentic Q&A are best suited to multi-GPU deployments that may handle large LLMs and multiple concurrent VLM requests.

Augment CV pipelines with VSS Event Reviewer

For smaller-scale and edge deployments, the brand new VSS Event Reviewer feature introduces API endpoints that make it easy to integrate VSS into existing computer vision pipelines for low-latency alerts and direct VLM Q&A on video segments. 

As an alternative of running VSS constantly on all files or streams, Event Reviewer allows VSS to act as an intelligent add-on that delivers VLM insights just for key moments. This approach greatly reduces compute costs, making VSS well-suited for lightweight deployments and edge platforms.

While standard CV pipelines excel at detecting objects and folks or applying analytics to discover events, similar to possible vehicle collisions, they often generate false positives and lack deeper scene understanding.

VSS could be used to reinforce these CV pipelines by analyzing short video clips flagged by the CV system, reviewing the detected events, and uncovering additional insights that traditional methods may miss.  

Figure 4 shows how VSS can augment an existing pipeline. The pc vision pipeline represents any proprietary system that’s able to taking in video files or streams and outputting short clips of interest. The Event Reviewer endpoints can then be called to pass these short video clips to VSS to generate alerts and follow up Q&A questions with a VLM. 

To exhibit this feature, a sample DeepStream detection pipeline is provided within the VSS GitHub repository using GroundingDINO. This instance pipeline ingests a video, runs detection after which outputs clips when the variety of detected objects is larger than a set threshold. The aim of this pipeline is to seek out an important events from the video that have to be inspected by VSS with a VLM. 

VSS will then process each small clip using the VLM by answering a set of yes/no questions defined by the user. These responses are converted to true/false states for every query and could be used to generate low latency alerts to a user. Once the short clip has been processed by VSS, you may ask more detailed follow questions.

This approach selectively uses the VLM only on clips of interest as determined by a light-weight detection pipeline. It will probably drastically reduce compute cost by reducing VLM calls and freeing up the GPU for other workloads.

Deploy flexibly with expanded hardware support

VSS Blueprint 2.4 fully supports several NVIDIA Blackwell platforms, including NVIDIA RTX Pro 6000 server and workstation editions and NVIDIA Jetson Thor for edge deployments. Support for NVIDIA DGX Spark is coming soon. 

	1 NVIDIA Jetson Thor	1-2 NVIDIA RTX PRO 6000 Blackwell WS/SE	4-8 NVIDIA RTX PRO 6000 Blackwell WS/SE
LLM	N/A	Llama 3.1 8B	Llama 3.1 70B
VLM	Cosmos Reason 1	Cosmos Reason 1	Cosmos Reason 1
Really helpful usage	Event Review	Event Review Video Summarization Video Q&A (Vector RAG)	Event Review File Summarization Video Q&A (Graph RAG)

For a full list of supported platforms, see the Supported Platforms section of the VSS documentation. 

Start with visual agentic AI 

The brand new VSS Blueprint 2.4 release brings latest visual agentic AI capabilities to the sting, improvements to spice up Q&A accuracy, cross-camera understanding, and expansion in platform support. The enhancements to knowledge graph creation and traversal improves Q&A accuracy and enables cross camera queries. 

For edge deployments and alerting use cases, the Event Reviewer feature is a solution to use VSS as an intelligent add-on to cv pipelines for low latency alerts. Prolonged platform support to incorporate NVIDIA RTX Pro and NVIDIA Thor.  

To quickly start with the VSS Blueprint, use an NVIDIA Brev Launchable. The launchable provides fast one-click deployment and Jupyter notebooks to walk through learn how to launch VSS, access the Web UI, and use the VSS REST APIs. Visit the NVIDIA-AI-Blueprints/video-search-and-summarization GitHub repo for more technical resources similar to training notebooks and reference code. For more technical questions, visit NVIDIA Developer Forum. 

For details about production deployments and CSPs, see the Cloud section of the VSS documentation.