The performance of the human visual system is very efficient in many visual tasks such as identifying visual scenes, anticipating future actions based on the past observations, assessing the quality of visual stimuli, etc. A significant amount of effort has been directed towards finding quality aware representations of natural videos to solve the quality prediction task. In this work we present a novel no reference video quality assessment (NR-VQA) algorithm based on the functional Magnetic Resonance Imaging (fMRI) Blood Oxygen Level Dependent (BOLD) signal prediction with voxel-wise encoding models of the human brain. The voxel encoding models are learnt using deep features extracted from the AlexNet model to predict the fMRI response to natural video stimuli. We show that the curvature in the predicted voxel response time series provides good quality discriminability, and forms an important feature for quality prediction. Further, we show that the proposed curvature features in combination with the spatial index, temporal index and NIQE features deliver acceptable performance on the Video Quality Assessment (VQA) task on both synthetic and authentic distortion data-sets. © 2021 IEEE.