This manuscript aims to build a parameterized adaptive controller using Reinforcement Learning (RL) and Deep Neural Networks (DNN). The main objective is to adapt parameters of any given controller structure using RL formulation and achieve better performance. In recent years, reinforcement learning has gained much attention, and its advantages make it ideal for adaptive tuning of parameterized controllers. With the advancement of computational power, it has become easier to approximate a complex policy function using a deep neural network to achieve better accuracy and performance. Conventional Twin Delayed Deep Deterministic Policy Gradient (TD3) algorithm may not be able to provide best possible training for a given number of episodes. To improve performance of the TD3 algorithm, dynamic action space is proposed along with modified reward function, designed to aid faster convergence. The proposed algorithm provides improved performance by dynamically modifying the action space in the existing TD3 algorithm. The effectiveness of the proposed RL-based parameterized controller is demonstrated through a standard first order system by designing an adaptive PI controller. A case study involving a 3 DOF (Degree of Freedom) gyroscope system, which is an unstable plant, is also presented. For the 3 DOF gyroscope system an adaptive Lead controller is designed, where the proposed algorithm provides faster convergence and better performance compared to the original TD3 algorithm. © 2022 IEEE.