In this paper, we propose a method for classification of sport videos using edge-based features, namely edge direction histogram and edge intensity histogram. We demonstrate that these features provide discriminative information useful for classification of sport videos, by considering five sports categories, namely, cricket, football, tennis, basketball and volleyball. The ability of autoassociative neural network (AANN) models to capture the distribution of feature vectors is exploited, to develop class-specific models using edge-based features. We show that combining evidence from complementary edge features results in improved classification performance. Also, combination of evidence from different classifiers like AANN, hidden Markovmodel (HMM) and support vector machine (SVM) helps improve the classification performance. Finally, the performance of the classification system is examined for test videos which do not belong to any of the above five categories. A low rate of misclassification error for these test videos validates the effectiveness of edge-based features and AANN models for video classification. © Springer-Verlag London Limited 2008.