YOLO Edge Optimization
Took a YOLO detection model from research checkpoint to deployment across three accelerators and four runtimes, measuring what every optimization actually costs in latency, accuracy, and size. Found that export is 3.50× faster on NVIDIA TensorRT and 1.69× slower on Apple Core ML for the identical graph, and traced the regression to ten tensor-indexing operators in the NMS-free detection head that split the Core ML graph into eight partitions instead of five. Naive INT8 quantization collapsed the head to 0.000 mAP across all 5,000 COCO validation images without raising a single error; excluding 95 head nodes recovered it, and TensorRT quantized the same head successfully, establishing the failure as a quantizer limitation rather than an architectural one. Six findings documented alongside four hypotheses that failed under testing.
CASE STUDY
Project breakdown.
Select a section to explore the project one part at a time.
Problem
Vendor-specific advice, read as general.
Deployment guidance for edge computer vision is written one vendor at a time. NVIDIA documents TensorRT; Apple documents Core ML. Each shows its own path working on its own silicon, and each is correct within that scope.
What practitioners inherit is a set of rules stated platform-agnostically but verified only platform-specifically:
- Export to an optimized runtime and inference gets faster.
- Quantize to INT8 and trade a little accuracy for a lot of speed.
- An NMS-free head removes a bottleneck, so it runs quicker.
None of these claims had been tested with the same graph, the same export settings, and the same measurement harness across competing vendors. The published numbers are not wrong — they are single-platform, and they are read as though they were general.
What this project asks
- Does export generalize? With architecture and export settings fixed, does converting a checkpoint to a vendor-optimized runtime reduce latency consistently — or does the sign of the effect depend on whose compiler receives the graph?
- Is INT8 a property of the model or of the quantizer? When post-training quantization destroys accuracy, is the architecture unquantizable, or is one toolchain's calibration strategy failing on it?
- Do precision gains scale with hardware capability? Reduced precision is sold as an efficiency win. Is that win larger on stronger accelerators, or does it shrink as compute headroom grows?
- Does architecture interact with the compiler? An NMS-free head removes a post-processing stage. Does that become an end-to-end gain on every backend, or does the operator set it introduces create new costs elsewhere?
Answering these required a harness trustworthy enough to detect a disagreement — validated by reproducing the published baseline of 0.478 mAP to 0.477 before any cross-vendor comparison was run.