How do I set up a scene detection API for incident detection on RTSP security cameras?

To set up a scene/incident detection API for RTSP security cameras, you usually build a pipeline that:

1. connects to RTSP streams
2. samples frames
3. runs detection/inference
4. triggers incident events
5. exposes an API and/or webhook
6. stores alerts, snapshots, and metadata

Here’s a practical architecture and setup path.

---

1) Define what “incident detection” means

“Scene detection” can mean different things:

• Motion detection: detect any movement
• Object detection: person, car, bike, animal, package, etc.
• Event detection: intrusion, loitering, crowding, falling, smoke/fire, tampering
• Scene change detection: camera moved, lighting changed, scene shifted
• Policy-based incident detection: e.g. “person in restricted area after 10 PM”

If you want an API for security cameras, the best approach is usually:

• RTSP ingestion
• AI inference on frames
• rules engine
• event API/webhooks

---

2) Recommended system architecture

Simple architecture

RTSP Camera → Frame Grabber → AI Model → Event Rule Engine → API/Webhook

Production architecture

• Camera ingest service
• Frame sampling service
• Inference service (GPU-enabled if possible)
• Event processor
• Database for events
• Object storage for snapshots/clips
• REST API
• Webhook dispatcher
• Optional: message queue like Redis, RabbitMQ, Kafka

---

3) Choose your detection method

Option A: Motion-based detection

Good for:

• low-cost systems
• quickly detecting activity

Tools:

• OpenCV background subtraction
• frame differencing

Pros:

• lightweight
• easy to implement

Cons:

• lots of false positives
• not semantic (doesn’t know if it’s a person)

Option B: AI object detection

Good for:

• person/vehicle detection
• better incident logic

Common models:

• YOLOv8 / YOLOv9
• Detectron2
• TensorRT-optimized models for edge devices

Pros:

• more accurate
• supports rules like “person in area”

Cons:

• needs more compute

Option C: Scene classification / anomaly detection

Good for:

• detecting unusual events without predefined classes

Tools:

• anomaly detection models
• temporal models
• embeddings + clustering

Cons:

• harder to tune
• usually less interpretable

---

4) RTSP ingest setup

Use FFmpeg or OpenCV to connect to the stream.

Example RTSP URL

rtsp://username:password@192.168.1.100:554/stream1

Python example with OpenCV

import cv2

rtsp_url = "rtsp://user:pass@192.168.1.100:554/stream1"
cap = cv2.VideoCapture(rtsp_url)

while True:
    ret, frame = cap.read()
    if not ret:
        print("Failed to read frame")
        break

    # process frame here
    cv2.imshow("camera", frame)
    if cv2.waitKey(1) & 0xFF == ord('q'):
        break

cap.release()
cv2.destroyAllWindows()

For production, FFmpeg/GStreamer is often more stable than plain OpenCV.

---

5) Sample frames instead of processing every frame

Security cameras often stream at 15–30 FPS, but you usually don’t need every frame.

Typical rates:

• 1–5 FPS for detection
• higher FPS only if you need tracking or fine-grained motion

Example strategy:

• read every frame
• process 1 out of every N frames
• or process every 200–500 ms

This keeps CPU/GPU usage manageable.

---

6) Add an AI detector

Example with YOLOv8

Install:

pip install ultralytics opencv-python fastapi uvicorn

Detection example:

from ultralytics import YOLO

model = YOLO("yolov8n.pt")  # lightweight model

results = model(frame)
for r in results:
    for box in r.boxes:
        cls_id = int(box.cls[0])
        confidence = float(box.conf[0])
        label = model.names[cls_id]

        if label == "person" and confidence > 0.5:
            print("Person detected")

You can then trigger incidents when conditions match.

---

7) Define incident rules

Examples:

• Person detected in restricted zone
• Vehicle detected outside allowed hours
• No person detected where one should be
• Motion after office hours
• Camera covered/tampered
• Crowd detected
• Loitering for more than X seconds

A rule engine can use:

• object labels
• confidence thresholds
• time windows
• regions of interest
• duration
• tracking IDs

---

8) Build the API

A common API design:

Endpoints

• POST /cameras — register a camera
• POST /cameras/{id}/start — start monitoring
• POST /cameras/{id}/stop — stop monitoring
• GET /cameras/{id}/status — stream/detection status
• GET /events — list incidents
• GET /events/{id} — event details
• POST /webhooks — configure incident callbacks

Example event payload

{
  "event_type": "person_detected",
  "camera_id": "cam_01",
  "timestamp": "2026-07-19T10:15:00Z",
  "confidence": 0.93,
  "snapshot_url": "https://storage.example.com/snapshots/123.jpg",
  "metadata": {
    "zone": "front_gate",
    "bbox": [120, 80, 300, 400]
  }
}

---

9) FastAPI example

Here’s a very small skeleton:

from fastapi import FastAPI
from pydantic import BaseModel

app = FastAPI()

class Camera(BaseModel):
    id: str
    rtsp_url: str

cameras = {}

@app.post("/cameras")
def add_camera(camera: Camera):
    cameras[camera.id] = camera
    return {"status": "added", "camera_id": camera.id}

@app.get("/cameras/{camera_id}/status")
def camera_status(camera_id: str):
    return {"camera_id": camera_id, "status": "running"}

@app.get("/events")
def list_events():
    return {"events": []}

You’d extend this with:

• background workers
• frame processing
• event storage
• webhook notifications

Run it:

uvicorn app:app --reload --host 0.0.0.0 --port 8000

---

10) Incident creation flow

A typical incident pipeline:

1. Frame comes in from RTSP
2. Detector finds person/vehicle/etc.
3. Rule engine checks:
   • zone
   • time
   • confidence
   • duration
4. If rule matches:
   • create incident record
   • store snapshot
   • optionally store short clip
   • send webhook/email/SMS
   • expose via API

---

11) Store snapshots and clips

For each incident, save:

• timestamp
• camera ID
• event type
• confidence
• bounding boxes
• snapshot image
• short video clip before/after event

Storage options:

• local disk
• S3-compatible object storage
• MinIO
• Azure Blob / Google Cloud Storage

A clip of 5–10 seconds around the event is often very useful.

---

12) Handle multiple cameras

For multiple RTSP cameras:

• one worker per camera or per stream group
• use async queues
• limit FPS per camera
• isolate failures so one stream doesn’t stop others

For production, decouple components:

• ingest workers
• inference workers
• event service
• API service

---

13) Deployment options

On-prem / edge

Good when:

• privacy matters
• low latency needed
• bandwidth is limited

Use:

• NVIDIA Jetson
• mini PC with GPU
• Intel OpenVINO on CPU

Cloud

Good when:

• cameras can send streams securely
• centralized management needed

Use:

• GPU instances
• autoscaling workers
• secure VPN/tunnels from camera site

---

14) Important engineering concerns

Reliability

• reconnect RTSP automatically
• handle stream drops
• watchdog for worker crashes

Performance

• resize frames before inference
• process fewer frames
• use GPU if available
• use efficient models

Security

• protect RTSP credentials
• use HTTPS for your API
• authenticate API requests
• restrict camera access by network/firewall

False positives

• use zones
• require duration
• use confidence thresholds
• combine motion + object detection

---

15) A practical stack suggestion

If you want a straightforward build:

• Python
• OpenCV or FFmpeg for RTSP
• YOLOv8 for object detection
• FastAPI for the API
• PostgreSQL for events
• Redis for queueing/background jobs
• S3/MinIO for snapshots/clips
• Docker for deployment

---

16) Minimal implementation roadmap

Phase 1

• connect to one RTSP camera
• sample frames
• detect persons using YOLO
• log incidents locally

Phase 2

• expose REST API
• add camera registration
• store events in DB
• save snapshots

Phase 3

• add rules: zones, time, duration
• add webhooks
• add multi-camera support
• add monitoring and retries

Phase 4

• optimize with GPU/TensorRT
• add clip recording
• scale to many cameras

---

17) If you want to go faster

You could also use existing video analytics platforms instead of building from scratch, such as:

• Frigate
• Shinobi
• ZoneMinder
• Milestone integrations
• custom AI pipelines with NVIDIA DeepStream

These can save time if you mainly need incident detection and APIs.

---

18) Example incident API behavior

Example:

• camera sees a person in a restricted zone
• system creates event:
  • person_detected
  • confidence 0.91
  • saves snapshot
  • sends webhook to https://your-app/webhook

Webhook example:

{
  "event_type": "person_detected",
  "camera_id": "cam_01",
  "incident_id": "inc_123",
  "timestamp": "2026-07-19T10:15:00Z"
}

---

If you want, I can also give you one of these next:

1. a full FastAPI + RTSP + YOLO example
2. a Docker Compose setup
3. a production architecture diagram
4. a webhook-based incident detection design
5. a Node.js version