Abstract: Superconducting linear accelerators play a crucial role in high-energy physics, nuclear physics, life sciences, and materials science. As the core component of superconducting linear accelerators, radio-frequency (RF) superconducting cavities are highly susceptible to external disturbances due to their narrow operating bandwidth. The Low Level Radio Frequency (LLRF) system, as an essential part of RF systems, primarily functions to precisely regulate RF signals for stable particle beam acceleration. The LLRF systems in accelerators predominantly employ classical Proportional Integral (PI) controllers. However, due to the nonlinear, time-varying, and delayed characteristics of the controlled object (RF superconducting cavity system), PI parameter tuning heavily relies on expert experience, making it challenging for on-duty operators to quickly optimize these parameters. To address this challenge, this paper proposes a Reinforcement Learning (RL) based approach for adaptive tuning of PI controller gain parameters in LLRF systems. The methodology encompasses experimental data acquisition, feature indicator construction, and reward function design. Experimental results demonstrate that the trained RL agent achieves over 98% accuracy in PI parameter optimization tasks.