Edge Video Processing | The Garnet Wiki

Edge video processing is the real-time, low-latency transformation of video streams at the network edge—close to where video is captured or consumed. It matters when latency under 100 ms is required, when bandwidth between camera and cloud is constrained, when live video must be processed locally (e.g., for surveillance, live production, or AR/VR), or when video must be pre-processed before transmission to a central data center.

Real-Time Encoding and Transcoding

Process video streams from cameras, drones, or mobile devices with minimal delay using hardware-accelerated encoders and dynamic bit-rate adaptation.

Use ffmpeg with NVENC (NVIDIA) or VCE (AMD) for GPU-accelerated encoding.
Key configuration for low-latency H.265 (HEVC) encoding:

ffmpeg -i input.mp4 \
  -c:v hevc_nvenc \
  -b:v 4M \
  -maxrate 6M \
  -minrate 2M \
  -g 30 \
  -sc_threshold 40 \
  -profile:v main10 \
  -pix_fmt yuv420p10le \
  -preset slow \
  -rc_lookahead 32 \
  -b_strategy 1 \
  -bf 4 \
  -refs 4 \
  -tune film \
  -rc 2pass \
  -f flv rtmp://edge-server/live/stream1

For 1080p60, use -framerate 60 -s 1920x1080 to ensure correct timing.
Common failure: ffmpeg outputs frames at 59.94 fps but stream expects 60.00. Solution: add -r 60000/1001 to force 59.94 to 60.00.

Latency vs. Quality Trade-offs

Use -tune flag wisely: film improves quality for static scenes; grain preserves texture; zerolatency optimizes for low delay.
Enable tune zerolatency and set b_frames=0 for ultra-low-latency pipelines (< 50 ms).
ffmpeg often defaults to b_frames=3, which adds 150–300 ms latency due to B-frame buffering. For real-time, set -bf 0 and -b_strategy 1 for constant bitrate with minimal buffering.

Dynamic Bitrate Streaming (DASH, HLS)

Generate segmented video on the edge using ffmpeg and mp4box:

ffmpeg -i input.mp4 \
  -c:v libx264 \
  -b:v 1M -maxrate 2M -bufsize 2M \
  -c:a aac -b:a 128k \
  -f hls \
  -hls_time 4 \
  -hls_list_size 10 \
  -hls_segment_filename /var/www/hls/stream1_%05d.ts \
  -hls_flags +append_list \
  -hls_base_url /hls/stream1/ \
  -hls_playlist_type event \
  -hls_segment_type fmp4 \
  -hls_segment_filename /var/www/hls/stream1_%05d.m4s \
  /var/www/hls/stream1.m3u8

Critical: -hls_playlist_type event allows live playlists to be updated continuously without client buffering stalls.
Common mistake: using segment_type mpegts but forgetting -hls_segment_filename with .ts, leading to mismatched segment paths and playback failures.

Video Analytics at the Edge

Run computer vision models directly on video frames captured at the edge to detect, classify, or track objects without sending raw video to the cloud.

Object Detection on Live Streams

Use OpenCV with TensorFlow Lite or ONNX Runtime for inference on edge devices.

import cv2
import numpy as np
import onnxruntime as ort

# Load model
session = ort.InferenceSession("/models/yolov8n-416.onnx")
input_name = session.get_inputs()[0].name

# Open video stream
cap = cv2.VideoCapture(0)  # or RTSP stream
cap.set(cv2.CAP_PROP_FRAME_WIDTH, 416)
cap.set(cv2.CAP_PROP_FRAME_HEIGHT, 416)
cap.set(cv2.CAP_PROP_FPS, 30)

while True:
    ret, frame = cap.read()
    if not ret:
        break

    # Preprocess: resize, normalize, HWC -> CHW
    blob = cv2.dnn.blobFromImage(
        frame,
        1.0 / 255.0,
        (416, 416),
        (0, 0, 0),
        swapRB=True,
        crop=False
    )

    # Run inference
    outputs = session.run(None, {input_name: blob})
    detections = outputs[0]  # (1, 8400, 85)

    # Post-process: NMS, filter by confidence
    conf_threshold = 0.5
    nms_threshold = 0.45

    for detection in detections[0]:
        class_id = int(detection[5])
        confidence = detection[4]
        if confidence < conf_threshold:
            continue

        x, y, w, h = detection[0:4]
        x1 = int((x - w/2) * 416)
        y1 = int((y - h/2) * 416)
        x2 = int((x + w/2) * 416)
        y2 = int((y + h/2) * 416)

        cv2.rectangle(frame, (x1, y1), (x2, y2), (0, 255, 0), 2)
        cv2.putText(frame, f"Class {class_id}", (x1, y1 - 10), cv2.FONT_HERSHEY_SIMPLEX, 0.5, (255, 255, 255), 1)

    # Encode and stream
    _, buffer = cv2.imencode('.jpg', frame, [cv2.IMWRITE_JPEG_QUALITY, 80])
    stream = buffer.tobytes()

    # Send to RTMP
    rtmp_url = "rtmp://edge-server/live/analytics"
    # Use ffmpeg subprocess or direct socket write

Common Edge Analytics Pitfalls

Frame misalignment: cv2.VideoCapture(0) returns frames with timestamps, but cap.set(cv2.CAP_PROP_FPS, 30) does not guarantee 30 fps; use cap.set(cv2.CAP_PROP_BUFFERSIZE, 1) to reduce buffering.
Model input shape mismatch: ONNX model expects (1, 3, 416, 416) but input is (416, 416, 3) — missing batch dimension. Fix: blob = np.expand_dims(blob, axis=0).
Memory exhaustion: ONNX Runtime uses 3× memory during inference (input, output, intermediate nodes). Use ort.SessionOptions() to set enable_mem_pattern=True and enable_cpu_mem_pool=True.
Inference fails silently: model runs but outputs are wrong. Add print(session.get_input_meta().name) and print(session.get_output_meta().name) to verify input/output names.

Video Stream Synchronization and Timestamp Management

Ensure accurate alignment of multiple video streams or video with metadata.

Timecode and Clock Drift

Use ffmpeg with precise timestamps:

ffmpeg -i input1.mp4 \
  -i input2.mp4 \
  -filter_complex "[0:v]setpts=PTS-STARTPTS[cam1]; [1:v]setpts=PTS-STARTPTS+10/TB[cam2]; [cam1][cam2]concat=n=2:v=1:a=0[output]" \
  -map "[output]" \
  -c:v libx264 \
  -r 30 \
  -pix_fmt yuv420p \
  -f flv rtmp://edge-server/live/synced_stream

setpts=PTS-STARTPTS resets timestamps to 0 for each stream.
+10/TB adds 10 seconds offset to second stream; use 10/TB for accuracy (TB = timebase).
TB is defined by -time_base 1/30 or -r 30 in input.

Clock Drift in Multi-Camera Systems

Devices may have different internal clocks. Use NTP with chrony to synchronize clocks across edge devices.

# On each edge node
sudo systemctl enable chronyd
sudo tee /etc/chrony/chrony.conf << EOF
server ntp1.example.com iburst
keyfile /etc/chrony.keys
log measurements statistics tracking
EOF

sudo systemctl restart chronyd

Use chronyc tracking to verify clock offset.
When clock drift exceeds ±50 ms, video streams become out of sync; correct by adjusting timestamps in ffmpeg with setpts.

Edge-Based Video Pre-Processing and Quality Enhancement

Apply real-time filters, stabilization, upscaling, and noise reduction before encoding or transmission.

Video Stabilization with Optical Flow

Use OpenCV to stabilize shaky video:

import cv2

cap = cv2.VideoCapture("shaky.mp4")
ret, prev_frame = cap.read()
prev_gray = cv2.cvtColor(prev_frame, cv2.COLOR_BGR2GRAY)

# Initialize optical flow
lk_params = dict(winSize=(15,15), maxLevel=2, criteria=(cv2.TERM_CRITERIA_EPS | cv2.TERM_CRITERIA_COUNT, 10, 0.03))

# Create video writer
fourcc = cv2.VideoWriter_fourcc(*'H264')
out = cv2.VideoWriter("stabilized.mp4", fourcc, 30.0, (prev_frame.shape[1], prev_frame.shape[0]))

while True:
    ret, frame = cap.read()
    if not ret:
        break

    gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)

    # Calculate optical flow
    prev_pts = cv2.goodFeaturesToTrack(prev_gray, maxCorners=100, qualityLevel=0.01, minDistance=10)
    next_pts, status, error = cv2.calcOpticalFlowPyrLK(prev_gray, gray, prev_pts, None, **lk_params)

    # Filter good points
    good_new = next_pts[status == 1]
    good_old = prev_pts[status == 1]

    # Compute homography
    H, mask = cv2.findHomography(good_old, good_new, cv2.RANSAC, 5.0)

    # Warp frame
    h, w = prev_frame.shape[:2]
    warped = cv2.warpPerspective(frame, H, (w, h))

    out.write(warped)
    prev_gray = gray.copy()
    prev_pts = good_new.reshape(-1, 1, 2)

cap.release()
out.release()

Common failure: cv2.calcOpticalFlowPyrLK returns None for next_pts when tracking fails.
Solution: check status array and recompute features when too many points fail.

Real-Time Upscaling with Super-Resolution

Use ESRGAN model for 4x upscaling:

import cv2
import numpy as np
import onnxruntime as ort

# Load ESRGAN model
session = ort.InferenceSession("esrgan-4x.onnx")
input_name = session.get_inputs()[0].name

# Read low-res frame
frame = cv2.imread("lowres.jpg")
lr_img = frame.astype(np.float32) / 255.0
lr_img = np.transpose(lr_img, (2, 0, 1))  # HWC -> CHW
lr_img = np.expand_dims(lr_img, axis=0)  # Add batch

# Run inference
sr_img = session.run(None, {input_name: lr_img})[0]

# Post-process
sr_img = np.squeeze(sr_img, axis=0)  # Remove batch
sr_img = np.transpose(sr_img, (1, 2, 0))  # CHW -> HWC
sr_img = (sr_img * 255.0).clip(0, 255).astype(np.uint8)

cv2.imwrite("upscaled.png", sr_img)

Critical: ESRGAN expects input in [-1, 1] range, not [0, 1]. Use lr_img = (lr_img - 0.5) * 2 before feeding.
Failure mode: upscaling is blurry but not sharp. Solution: apply cv2.bilateralFilter on output or use cv2.detailEnhance after upscaling.

Edge Video Processing Pipeline: End-to-End Example

# 1. Capture 1080p60 from camera
ffmpeg -f v4l2 -video_size 1920x1080 -framerate 60 -i /dev/video0 \
  -c:v hevc_nvenc \
  -b:v 6M -maxrate 8M -minrate 4M \
  -g 30 -sc_threshold 40 \
  -preset medium -rc 2pass \
  -pix_fmt yuv420p10le \
  -tune film \
  -f flv rtmp://edge-server/live/main

# 2. Run object detection on stream
import cv2
import onnxruntime as ort

session = ort.InferenceSession("/models/yolov8n-640.onnx")
cap = cv2.VideoCapture("rtmp://edge-server/live/main")

while True:
    ret, frame = cap.read()
    if not ret:
        break

    # Preprocess: resize, normalize
    blob = cv2.dnn.blobFromImage(frame, 1.0/255, (640, 640), (0, 0, 0), True, False)
    outputs = session.run(None, {session.get_inputs()[0].name: blob})

    # Filter detections, draw boxes
    # ...

    # Encode and send to RTMP
    _, buffer = cv2.imencode('.jpg', frame, [cv2.IMWRITE_JPEG_QUALITY, 85])
    # Write buffer to RTMP stream using `rtmp://edge-server/live/analyzed`

# 3. Combine with stabilization and upscaling
ffmpeg -i input.mp4 \
  -vf "vidstabdetect=shakiness=20:accuracy=10,vidstabtransform=smoothing=10" \
  -s 3840x2160 \
  -c:v libx264 -crf 20 -preset fast \
  -f flv rtmp://edge-server/live/stabilized_4k

Silent Failures and Pitfalls

ffmpeg ignores -b:v when -crf is present; use -b:v 4M and -crf 23 together.
cv2.VideoCapture fails silently if camera is not ready; use cap.isOpened() before reading.
ONNX model input name mismatch: input vs input.1 vs input_0 — verify with print(session.get_inputs()[0].name).
setpts does not apply to audio streams unless explicitly added: use asetpts for audio.
rtmp:// streams drop frames when encoder output rate exceeds network bandwidth; use ffmpeg with -async 1 and -vsync 1.
chrony clock drift causes frame misalignment over time; use chronyc tracking to monitor and adjust.

Edge video processing is not just about encoding—it's about managing timing, synchronizing streams, running AI models, and handling hardware constraints with precision. Do it right, and video becomes a first-class data stream at the edge.

This page was rewritten on 10 October 2026. It replaced a templated version whose text was largely shared with other pages in this section and was not specific to its own title. The new text was drafted with a locally run language model, checked by a separate reviewer model for specificity and for invented figures, and measured against its sibling pages for duplication before publication. If anything here is wrong, tell us at [email protected] and we will correct it.