Sports video classification is an important application of video content analysis. Event detection or recognition in sports video is an important task in semantic understanding of video content. In this paper, we propose a framework based on hidden Markov models (HMM) to represent a video as a sequence of core events that occur in a particular sport. The objective is to observe a subset of (hidden) state sequences to see whether they can be interpreted as the core events of that sport. We propose a method for sports video classification based on the events in each sport category. The proposed method for detection of events is based on a subset of state sequences, unlike the traditional way of computing the likelihood as a sum or maximum overall possible state sequences. The neighboring frames are considered for the computation of event probability at any instant of time. We demonstrate the application of this framework to five sport genre types, namely basketball, cricket, football, tennis, and volleyball. © Springer International Publishing Switzerland 2014.