Tracking athletes performing a particular action in the presence of crowded backgrounds is a challenging task. This task becomes even more difficult in activities like cliff-diving, uneven bars, parallel bars, and others, where athletes perform highly articulated motion. The articulation is rapid and results in highly contorted postures which are quite different from the normal structure of the human body. In low-resolution videos particularly, these rapid deformations in human posture cannot recognized as pose estimation techniques are not readily applicable. Moreover, rapid deformations in successive frames pose a challenge to existing state-of-the-art trackers as they can handle only small-scale deviations in appearance. In this paper, we propose a framework based on sparse representation using dictionary learning which identifies humans in low-resolution videos despite rapid deformations to the body shape. We show that human representation, using deep features obtained using a convolutional neural network, remains sparse even during highly articulated motion. By leveraging this sparsity, we build a dictionary for human postures and show that even highly deformed human body postures can be represented using a linear combination of dictionary atoms. This learned dictionary can also effectively distinguish the actor from the background using sparsity threshold without either annotations or pose estimation. Finally, we incorporate the dictionary into a tracking framework and demonstrate the results on action classes with rapid deformations from the UCF101 datasets. © 2018 ACM.