A rapid increase in the video traffic together with an increasing demand for higher quality videos has put a significant load on content delivery networks in the recent years. Due to the relatively limited delivery infrastructure, the video users in HTTP streaming often encounter dynamically varying quality over time due to rate adaptation, while the delays in video packet arrivals result in rebuffering events. The user quality-of-experience (QoE) degrades and varies with time because of these factors. Thus, it is imperative to monitor the QoE continuously in order to minimize these degradations and deliver an optimized QoE to the users. Towards this end, we propose a nonlinear state space model for efficiently and effectively predicting the user QoE on a continuous time basis. The QoE prediction using the proposed approach relies on a state space that is defined by a set of carefully chosen time varying QoE determining features. An evaluation of the proposed approach conducted on two publicly available continuous QoE databases shows a superior QoE prediction performance over the state-of-the-art QoE modeling approaches. The evaluation results also demonstrate the efficacy of the selected features and the model order employed for predicting the QoE. Finally, we show that the proposed model is completely state controllable and observable, so that the potential of state space modeling approaches can be exploited for further improving QoE prediction. © 2018 IEEE.