This work addresses the problem of efficient noreference video quality assessment (NR-VQA). The motivation for this work is that even the best and fastest VQA algorithms do not achieve real-time performance. The speed of quality evaluation is impeded primarily by the spatio-temporal feature extraction stage. This impediment is common to both traditional as well as deep learning models. To address this issue, we explore the efficacy of features used in the action recognition problem for NR- VQA. Specifically, we leverage the efficiency offered by Gate Shift Module (GSM) in extracting spatio-temporal features. A simple yet effective improvement to the GSM model is proposed by adding the self-attention module. We first show that GSM features are indeed effective for NR-VQA. We then demonstrate a speed-up that is orders of magnitude faster than the current state-of-the-art VQA algorithms, albeit at the cost of overall performance. We evaluate the efficacy of our algorithm on both Standard Dynamic Range (SDR) and High Dynamic Range (HDR) datasets like KoNViD-1K, LIVE VQC, HDR. © 2022 IEEE.