Real-time computer vision on NVIDIA Jetson

Object detection, segmentation, pose estimation, video analytics and the hardware-accelerated libraries behind them, deployed on Jetson edge devices.

Tasks in the pipeline

Task	What It Does	Real Example
Object Detection	Find and label things in images	Security camera spotting intruders
Segmentation	Outline exact boundaries of objects	Self-driving car knowing road vs sidewalk
Pose Estimation	Track body/hand positions	Fitness app analyzing your workout form
Face Recognition	Identify specific people	Unlocking your phone with your face
Optical Flow	Detect motion patterns	Spotting falls in elderly care

---

What each Jetson module handles

How many cameras a board carries, and at what frame rate, is set first by the module. The three Orin devices sit at different points on that curve.

Device	What You Can Do	Power Usage
Orin Nano	4 cameras, 30 FPS detection	7 watts (phone charger)
Orin NX	8 cameras, 60 FPS detection	15 watts
AGX Orin	16+ cameras, real-time everything	30 watts

Frame rates for three common models, on a regular computer and on an optimized Jetson Orin:

Model	Regular Computer	Jetson Orin (Optimized)
YOLOv8	15 FPS	120 FPS
Pose Detection	10 FPS	60 FPS
Face Recognition	20 FPS	90 FPS

The Jetson Orin series delivers up to 275 TOPS (trillion operations per second) and an 8X performance improvement over previous generations. Pipelines that were previously confined to data center GPUs now run in an embedded form factor.

This post covers detection, segmentation, pose estimation, optical flow, multi-camera synchronization, analytics at scale, action recognition, anomaly detection, face recognition, and the VPI and CV-CUDA libraries.

---

1. Real-time object detection: YOLOv8 and RT-DETR

YOLOv8 performance on Jetson Orin

The YOLO (You Only Look Once) family is the default for real-time object detection. YOLOv8, from Ultralytics, has been benchmarked across the Jetson platforms.

Benchmark results by device

Model	Jetson AGX Orin 32GB	Jetson Orin NX	Jetson Orin Nano
YOLOv8n (INT8)	~120 FPS	65 FPS	43 FPS
YOLOv8s (INT8)	~95 FPS	52 FPS	35 FPS
YOLOv8m (INT8)	~75 FPS	38 FPS	25 FPS
YOLOv8x (INT8)	~75 FPS	28 FPS	18 FPS

With INT8 precision on the YOLOv8x model, you can achieve approximately 75 FPS on the AGX Orin 32GB. For the Orin Nano, YOLOv8n_INT8 achieves an average iteration time of 23.16 ms (43 FPS), while YOLOv8n_FP16 reaches 26.70 ms (37 FPS).

TensorRT optimization code example

from ultralytics import YOLO

# Load YOLOv8 model
model = YOLO('yolov8n.pt')

# Export to TensorRT with INT8 quantization
model.export(
    format='engine',
    device=0,
    half=True,           # FP16 precision
    int8=True,           # INT8 quantization
    data='coco128.yaml', # Calibration dataset
    workspace=4,         # GPU memory workspace (GB)
    batch=1
)

# Load TensorRT engine for inference
trt_model = YOLO('yolov8n.engine')
results = trt_model.predict(source='video.mp4', stream=True)

RT-DETR: transformer-based detection

RT-DETR (Real-Time Detection Transformer) abandons the CNN detector design. Its hybrid encoder architecture decouples intra-scale interaction from cross-scale fusion. Performance stays competitive with CNN detectors and speed remains tunable.

On Jetson AGX Xavier with TensorRT FP16, RT-DETR achieves approximately 50 FPS throughput. The model predicts class, confidence, and bounding box parameters for the top 300 objects, outputting a 300x6 tensor.

# RT-DETR deployment with TensorRT
import tensorrt as trt
import numpy as np

def load_rt_detr_engine(engine_path):
    """Load serialized TensorRT engine."""
    with open(engine_path, 'rb') as f:
        runtime = trt.Runtime(trt.Logger(trt.Logger.WARNING))
        engine = runtime.deserialize_cuda_engine(f.read())
    return engine

# RT-DETR outputs: [batch, 300, 6]
# Format: [x1, y1, x2, y2, confidence, class_id]

---

2. Instance and semantic segmentation on edge

Semantic segmentation with jetson-inference

The jetson-inference library provides optimized FCN-ResNet models for real-time semantic segmentation. For Jetson Nano, fcn_resnet18 is recommended, while Xavier and Orin devices can run deeper backbones like ResNet-34 or ResNet-50.

# Run semantic segmentation on video stream
segnet --network=fcn-resnet18-cityscapes \
       /dev/video0 \
       display://0

NanoSAM: real-time Segment Anything

NanoSAM is a distilled version of the Segment Anything Model (SAM) that runs in real-time on Jetson Orin platforms. It uses a MobileSAM-based image encoder trained on unlabeled images.

from nanosam import NanoSAM
import cv2

# Initialize NanoSAM
sam = NanoSAM(
    encoder_path='nanosam_encoder.trt',
    decoder_path='nanosam_decoder.trt'
)

# Process frame
frame = cv2.imread('image.jpg')
point_coords = np.array([[500, 375]])  # Click point
point_labels = np.array([1])           # Foreground

mask = sam.predict(frame, point_coords, point_labels)

YOLOv11 instance segmentation

YOLO11, released in late 2024, runs instance segmentation on Jetson Orin Nano Super (67 TOPS) at real-time speeds. The Orin NX with 16GB RAM and 100 TOPS handles more demanding segmentation workloads.

---

3. Pose estimation: human, hand, and object

TRT Pose for body keypoint detection

NVIDIA's trt_pose library enables real-time human pose estimation on Jetson devices, detecting 18 body keypoints including eyes, elbows, and ankles.

import torch
import trt_pose.coco
from trt_pose.parse_objects import ParseObjects

# Load model
with open('human_pose.json', 'r') as f:
    human_pose = json.load(f)

topology = trt_pose.coco.coco_category_to_topology(human_pose)
parse_objects = ParseObjects(topology)

# Run inference
counts, objects, peaks = parse_objects(cmap, paf)

Performance benchmarks

Platform	FPS (ResNet18)	FPS (DenseNet121)
Jetson Nano	22 FPS	14 FPS
Jetson Xavier NX	45 FPS	30 FPS
Jetson Orin NX	60+ FPS	45 FPS

Hand pose with trt_pose_hand

The trt_pose_hand extension supports six gesture classes (fist, pan, stop, fine, peace, no hand) and runs in real-time on Jetson Xavier NX. Custom gestures can be added by training an SVM classifier on extracted hand features.

MoveNet on Jetson

MoveNet from Google offers Lightning and Thunder variants. For TensorRT conversion on Jetson, the recommended path is:

MoveNet Lightning achieves under 7ms inference at 192x256 resolution on mobile devices. That budget fits a Jetson Nano.

---

4. Optical flow and motion estimation

NVIDIA Optical Flow Accelerator (OFA)

Starting with Turing architecture, NVIDIA GPUs include a dedicated Optical Flow Accelerator (NVOFA) that computes motion vectors independently of CUDA cores. The Jetson AGX Orin includes this hardware unit.

// NVIDIA Optical Flow SDK usage
#include "nvOpticalFlowCommon.h"
#include "nvOpticalFlowCuda.h"

NvOFCudaAPI *ofAPI;
NV_OF_CUDA_API_FUNCTION_LIST *ofFunctions;

// Initialize optical flow
NvOFInit(NV_OF_MODE_OPTICALFLOW, NV_OF_PERF_LEVEL_SLOW);

// Compute flow between frames
NvOFExecute(inputFrame1, inputFrame2, outputFlow);

VPI optical flow integration

The Vision Programming Interface (VPI) provides unified access to optical flow across CPU, GPU, and OFA backends:

import vpi

# Create optical flow estimator
with vpi.Backend.OFA:  # Use hardware accelerator
    optical_flow = vpi.OpticalFlowPyrLK(
        input_image,
        prev_keypoints,
        cur_keypoints
    )

OpenCV CUDA optical flow

OpenCV provides CUDA-accelerated optical flow algorithms compatible with Jetson:

import cv2

# Create CUDA optical flow object
flow_calculator = cv2.cuda.FarnebackOpticalFlow_create(
    numLevels=5,
    pyrScale=0.5,
    winSize=13,
    numIters=10,
    polyN=5,
    polySigma=1.1,
    flags=0
)

# Upload frames to GPU
gpu_prev = cv2.cuda_GpuMat(prev_gray)
gpu_curr = cv2.cuda_GpuMat(curr_gray)

# Compute optical flow
gpu_flow = flow_calculator.calc(gpu_prev, gpu_curr, None)
flow = gpu_flow.download()

---

5. Multi-camera systems and synchronization

DeepStream multi-camera architecture

NVIDIA DeepStream SDK runs multi-camera video analytics pipelines, adding streams as the module allows. The Jetson TX1 supports up to 6 synchronized camera streams, while Orin devices handle significantly more.

# DeepStream multi-camera pipeline (Python)
import gi
gi.require_version('Gst', '1.0')
from gi.repository import Gst

def create_multi_camera_pipeline(num_cameras):
    pipeline = Gst.Pipeline()

    # Create streammux for batching
    streammux = Gst.ElementFactory.make("nvstreammux", "mux")
    streammux.set_property("batch-size", num_cameras)
    streammux.set_property("width", 1920)
    streammux.set_property("height", 1080)

    # Add camera sources
    for i in range(num_cameras):
        source = Gst.ElementFactory.make("nvarguscamerasrc", f"cam{i}")
        source.set_property("sensor-id", i)
        # Link to streammux sink pad

    # Add inference element
    nvinfer = Gst.ElementFactory.make("nvinfer", "primary-inference")
    nvinfer.set_property("config-file-path", "config_infer.txt")

    return pipeline

Hardware synchronization

For precise frame synchronization across cameras:

1. Master/Slave Configuration: Designate one sensor as master generating timing signals
2. V4L2 Timestamps: Linux kernel assigns timestamps maintained through the GStreamer pipeline
3. Network Time Protocol: For distributed camera systems, use NTP for cross-device synchronization

Deployment: smart parking garage

One production deployment runs 150 360-degree cameras plus 8 license plate cameras on DeepStream. The system monitors entry and exit points and interior spaces. It tracks vehicle movement and reports parking spot occupancy.

---

6. Video analytics at scale

DeepStream pipeline performance

The DeepStream SDK provides a complete GPU-accelerated video processing pipeline:

Key optimizations:

• Hardware decoding: NVDEC offloads video decode from GPU
• Batched inference: Process multiple streams simultaneously
• DLA offload: Run inference on the Deep Learning Accelerator to keep the GPU free

Runtime stream management

For large-scale deployments, DeepStream supports dynamic stream attachment/detachment without pipeline restart:

# Dynamically add camera stream at runtime
def add_stream(pipeline, uri, stream_id):
    source_bin = create_source_bin(stream_id, uri)
    pipeline.add(source_bin)

    streammux = pipeline.get_by_name("streammux")
    sinkpad = streammux.get_request_pad(f"sink_{stream_id}")
    srcpad = source_bin.get_static_pad("src")
    srcpad.link(sinkpad)

    source_bin.set_state(Gst.State.PLAYING)

Performance metrics

Configuration	AGX Orin	Orin NX	Orin Nano
1080p streams (YOLOv8n)	16 streams	8 streams	4 streams
4K streams (YOLOv8n)	4 streams	2 streams	1 stream
CPU utilization	<20%	<25%	<30%

---

7. Action recognition on edge

SlowFast networks

SlowFast architecture uses dual pathways: a Slow pathway (low frame rate, spatial semantics) and Fast pathway (high frame rate, temporal dynamics). After TensorRT optimization, SlowFast achieves 32 FPS on Jetson Xavier with 1080p input.

# SlowFast inference setup
import torch
from slowfast.config.defaults import get_cfg
from slowfast.models import build_model

cfg = get_cfg()
cfg.merge_from_file("configs/SLOWFAST_8x8_R50.yaml")
cfg.NUM_GPUS = 1

model = build_model(cfg)
model.load_state_dict(torch.load("slowfast_r50.pth"))

# Convert to TensorRT
torch.onnx.export(model, dummy_input, "slowfast.onnx")
# Then use trtexec for engine generation

X3D efficient video recognition

X3D reaches the best published accuracy with 4.8x fewer multiply-adds than previous methods. Its progressive expansion approach suits a limited compute budget.

Optimization results

Model	Original FPS	TensorRT FPS	Improvement
SlowFast R50	5.9	32	5.4x
X3D-M	8.2	45	5.5x
I3D ResNet50	4.1	28	6.8x

---

8. Anomaly detection in video streams

End-to-end anomaly detection system

The FICC 2024 benchmark paper demonstrates a complete video-based anomaly detection pipeline on Jetson devices using ResNet50-I3D feature extraction with RTFM (Robust Temporal Feature Magnitude) detection.

Performance comparison

Device	FPS	Power (W)	Efficiency (FPS/W)
Jetson Nano	1.6	5.2	0.31
Jetson AGX Xavier	41.3	18.5	2.23
Jetson Orin Nano	47.6	9.2	5.17

The Jetson Orin Nano achieves 47.56 FPS, 30x faster than Jetson Nano, at half the power consumption of AGX Xavier.

Implementation architecture

# Anomaly detection pipeline
class AnomalyDetector:
    def __init__(self, feature_extractor, anomaly_model):
        self.feature_extractor = feature_extractor  # I3D Non-local
        self.anomaly_model = anomaly_model          # RTFM

    def process_clip(self, frames):
        # Extract spatiotemporal features
        features = self.feature_extractor(frames)

        # Compute anomaly score
        score = self.anomaly_model(features)

        return score > self.threshold

Unsupervised approaches

Modern unsupervised anomaly detection uses Vision Transformers (ViT) combined with convolutional spatiotemporal attention blocks to capture both local and global relationships without labeled data.

---

9. Face recognition and person re-identification

ArcFace/InsightFace pipeline

Production face recognition on Jetson follows a three-stage pipeline:

1. Detection: RetinaFace or SCRFD for face localization
2. Alignment: 5-point landmark alignment to 112x112 canonical view
3. Recognition: ArcFace embedding (512-dimensional feature vector)

# Face recognition with ArcFace
import insightface

# Initialize face analysis
app = insightface.app.FaceAnalysis(
    name='buffalo_l',
    providers=['CUDAExecutionProvider']
)
app.prepare(ctx_id=0, det_size=(640, 640))

# Process image
img = cv2.imread('test.jpg')
faces = app.get(img)

for face in faces:
    embedding = face.embedding  # 512-dim feature
    bbox = face.bbox
    landmarks = face.kps

Multi-camera person re-identification

A production multi-camera ReID system on Jetson Orin Nano (8GB) using DeepStream achieves:

• Real-time tracking across cameras with <200ms latency
• 95%+ accuracy in global ID assignment
• 30% reduction in false positives for perimeter security

DeepStream face recognition pipeline

---

10. VisionWorks and CV-CUDA libraries

Vision Programming Interface (VPI)

VPI is NVIDIA's unified computer vision library providing access to multiple hardware backends:

• CPU: Multi-threaded implementation
• GPU: CUDA-accelerated kernels
• PVA: Programmable Vision Accelerator (1024-bit SIMD)
• VIC: Video Image Compositor for color conversion/scaling
• OFA: Optical Flow Accelerator

Performance comparison (1920x1080)

Backend	Box Filter (ms)	Gaussian (ms)	Harris Corners (ms)
CPU	325.9	412.3	567.8
CUDA	14.2	18.7	23.4
PVA	70.9	85.2	112.6

VPI code example

import vpi

# Create VPI image from NumPy array
with vpi.Backend.CUDA:
    vpi_image = vpi.asimage(numpy_array)

    # Apply Gaussian blur
    blurred = vpi_image.gaussian_filter(
        kernel_size=5,
        sigma=1.4
    )

    # Harris corner detection
    corners = blurred.harris_corners(
        gradient_size=3,
        block_size=3,
        strength_thresh=10
    )

CV-CUDA for cloud-scale vision

CV-CUDA came out of a collaboration between NVIDIA and ByteDance. It supplies GPU-accelerated operators for high-throughput video processing. Starting with v0.14, Jetson builds are available including support for Jetson Thor with Blackwell architecture.

What it adds:

• Zero-copy memory mapping between backends
• Python 3.14 and CUDA 13 support
• OpenCV interoperability

import cvcuda
import torch

# Create CV-CUDA tensor from PyTorch
tensor = torch.randn(1, 3, 1080, 1920, device='cuda')
cv_tensor = cvcuda.as_tensor(tensor)

# Apply operators
resized = cvcuda.resize(
    cv_tensor,
    (720, 1280),
    cvcuda.Interp.LINEAR
)

normalized = cvcuda.normalize(
    resized,
    base=torch.tensor([0.485, 0.456, 0.406]),
    scale=torch.tensor([0.229, 0.224, 0.225])
)

---

Deployment scenarios

Scenario 1: industrial quality inspection

Hardware: Jetson AGX Orin 64GB Pipeline: 4x GigE cameras -> Defect detection (YOLOv8s) -> Instance segmentation -> Classification

# DeepStream config for quality inspection
[primary-gie]
enable=1
model-engine-file=yolov8s_defect.engine
batch-size=4
interval=0
gie-unique-id=1

[secondary-gie]
enable=1
model-engine-file=defect_classifier.engine
operate-on-gie-id=1

Results:

• 4 streams at 60 FPS each
• <50ms end-to-end latency
• Defect classes validated against the line's own reject bin

Scenario 2: retail analytics

Hardware: Jetson Orin NX 16GB Pipeline: 8x IP cameras -> Person detection -> Tracking -> ReID -> Heatmap generation

Results:

• 8 streams at 30 FPS
• Cross-camera person tracking
• Real-time occupancy analytics

Scenario 3: autonomous mobile robot

Hardware: Jetson Orin Nano 8GB Pipeline: Stereo camera -> Depth estimation -> Object detection -> Semantic segmentation -> Path planning

Results:

• 30 FPS perception pipeline
• <100ms obstacle detection latency
• 8W average power consumption

---

Optimization best practices

1. Power mode configuration

# Set maximum performance mode
sudo nvpmodel -m 0

# Enable all clocks at maximum frequency
sudo jetson_clocks

# Verify settings
jetson_clocks --show

2. TensorRT INT8 calibration

import tensorrt as trt

# Configure INT8 calibration
config.set_flag(trt.BuilderFlag.INT8)
config.int8_calibrator = EntropyCalibrator2(
    calibration_data,
    cache_file="calibration.cache"
)

# Build optimized engine
engine = builder.build_serialized_network(network, config)

3. Memory optimization

• Use unified memory for zero-copy CPU-GPU transfers
• Implement ping-pong buffering for continuous streaming
• Use the DLA for inference to free GPU memory

4. Multi-stream processing

# Create multiple CUDA streams for parallel processing
streams = [cuda.Stream() for _ in range(num_cameras)]

for i, frame in enumerate(frames):
    with streams[i]:
        preprocess(frame)
        inference(frame)
        postprocess(frame)

# Synchronize all streams
for stream in streams:
    stream.synchronize()

---

What the numbers add up to

With TensorRT, DeepStream and VPI, a board drawing under 30W reaches inference performance that used to require a data center GPU. The measured points from the sections above:

• YOLOv8 with INT8: 65+ FPS on Orin NX for real-time object detection
• DeepStream: multi-camera analytics at <20% CPU utilization
• VPI: one API across the PVA, VIC and OFA accelerators
• NanoSAM: real-time segmentation on Orin devices
• Multi-camera ReID: 95%+ accuracy at <200ms latency

The module sets the ceiling; the benchmark table above shows how the same pipeline scales across the Orin range.

---

References and resources

• NVIDIA Jetson Benchmarks
• DeepStream SDK Documentation
• VPI Algorithm Performance
• Ultralytics YOLOv8 Documentation
• jetson-inference GitHub
• NanoSAM GitHub
• trt_pose GitHub
• CV-CUDA Releases
• FICC 2024: Benchmarking Jetson Edge Devices
• TensorRT Developer Guide

---

Last updated: January 2026