In 5G and beyond cellular networks, the typical cell size has been reducing to provide higher data rates and meet the low latency requirements. Such ultra-dense deployments of base stations results in frequent handovers (HOs) of the user equipment. In this poster, we utilize the state-of-the-art double deep reinforcement learning (DDRL) framework to enhance the performance with respect to handover failure, radio link failure, and ping pong effect. Through extensive simulation results, we show that the proposed off-policy DDRL based handover scheme performs better than state-of-the-art handover schemes. © 2023 IEEE.