What should I use for low-latency inference at the edge?

For low-latency inference at the edge, the best choice usually depends on your model type and hardware, but the most common options are:

Good general choices

• ONNX Runtime — strong default for edge deployment, good performance across CPUs/GPUs, supports many model formats via ONNX.
• TensorRT — best for NVIDIA GPUs when you need maximum speed and low latency.
• TFLite (TensorFlow Lite) — great for mobile and embedded devices, especially ARM CPUs and Edge TPU setups.
• OpenVINO — excellent for Intel CPUs, iGPUs, and VPUs, often strong for computer vision at the edge.
• ExecuTorch / Lite Interpreter — if you’re in the PyTorch ecosystem and want mobile/edge deployment.

If you want a practical recommendation

• NVIDIA edge device: TensorRT
• Intel CPU / industrial edge box: OpenVINO or ONNX Runtime
• ARM/mobile/embedded: TFLite
• Cross-platform and flexible: ONNX Runtime

For lowest latency, also consider:

• Quantization (INT8 / FP16)
• Pruning / model distillation
• Batch size = 1
• Static input shapes
• Using accelerator-specific backends
• Avoiding Python in the hot path if possible

If you tell me your hardware (e.g. Jetson, Raspberry Pi, x86 CPU, phone, etc.) and model type (CV, LLM, audio), I can recommend the best runtime more precisely.