Residual reinforcement learning for disturbance-resilient control under modeling uncertainties

Abstrak

Modern control systems must operate reliably in the presence of modeling uncertainties and external disturbances, conditions under which conventional fixed-gain controllers often exhibit performance degradation. This paper proposes a residual reinforcement learning framework for disturbance-resilient pitch-rate control of an aircraft longitudinal model. A classical proportional-integral-derivative (PID) controller is employed as a stabilizing baseline, while a deep deterministic policy gradient (DDPG) agent learns a bounded residual control signal to compensate for unmodeled dynamics and external perturbations. To promote favorable transient behavior, the learning process incorporates transient-aware and reference-model-based reward shaping, while actuator constraints are enforced within the environment dynamics. Simulation results demonstrate that the proposed residual controller achieves a superior balance between response speed, overshoot, and tracking accuracy compared with both the standalone PID controller and a pure DDPG-based controller. In particular, the residual architecture significantly reduces overshoot and tracking error while preserving fast transient response and providing robust disturbance rejection under large pitching moment disturbances. These results indicate that residual reinforcement learning offers a practical and effective approach for enhancing robustness and performance in safety-critical flight control applications.