Real-Time Continual Learning on Embedded GPUs

Abstract

Online continual learning (OCL) is an enabling technology for safety-critical autonomous embedded systems whose deployment environments shift faster than fleet-trained models or cloud round-trips can correct. Perception must meet hard sub-second deadlines on a single embedded GPU while training shares the same device; existing OCL runtimes either serialize the two phases or alternate them at coarse cadence, leaving the GPU underutilized and offering no guarantee on inference timing. We present AOCL, a self-adaptive OCL runtime for concurrent training and inference on a single embedded GPU, designed to be timing-scoped (deadline-aware safeguard with bounded recovery), practical (drop-in Avalanche plugin, one default hyperparameter setting across all evaluated cells, no offline workload model), and generalizable (validated across two Jetson platforms, four backbones from ResNet-20 to ResNet-101 and ViT-Tiny, four OCL algorithms, and five benchmarks). AOCL contributes a Unified Adaptation Metric (UAM) that couples streaming-accuracy drift to training-batch granularity and a lightweight feedback controller that co-schedules training and inference at runtime. Analytical properties of the controller bound the per-axis settling envelope at its operating point and the platform-level per-window worst-case latency at the boundary configuration, supporting deployment in real-time execution. Implemented on top of the open-source Avalanche continual-learning framework and deployed on Jetson Orin Nano and AGX Orin, AOCL achieves up to 11.6× higher training throughput (average 5.15×) over the strongest serialized and alternating OCL baselines, while sustaining the lowest or tied-for-lowest deadline-miss rate in 13 out of 15 deadline miss rate measurements. Two realistic case studies: a high-fidelity autonomous-driving simulator and a physical soft-robotic prototype, demonstrate sustained practicality under on-device adaptation.