Feature extraction is one of the most important phases of medical image classification which requires extensive domain knowledge. Convolutional Neural Networks (CNN) have been successfully used for feature extraction in images from different domains involving a lot of classes. In this paper, CNNs are exploited to extract a hierarchical and discriminative representation of X-ray images. This representation is then used for classification of the X-ray images as various parts of the body. Visualization of the feature maps in the hidden layers show that features learnt by the CNN resemble the essential features which help discern the discrimination among different body parts. A comparison on the standard IRMA X-ray image dataset demonstrates that the CNNs easily outperform classifiers with hand-engineered features. © 2016 IEEE.