Accumulation of excess fat in the liver tissue is the leading cause for dysfunction of liver, which can lead to the diseases from fibrosis to end stage cirrhosis. Hence, early detection of fatty liver becomes crucial in avoiding the liver from permanent failure. Depending on the concentration of fat in the tissue, the liver is classified as Normal, Grade 1, Grade 2 and Grade 3 respectively. The texture of liver tissue in ultrasound image is so specific to the concentration of fat, hence classifying the fatty liver is formulated as a texture discrimination problem. In this paper, we present an automated algorithm for grading the tissue of a fatty liver based on the features obtained from the invariant scattering convolution network (ISCN). ISCN, which involves cascade of modulus complex wavelet transforms and averaging operations results in scattering coefficients (SC), these coefficients will give stable invariant representations and also maps the texture of fatty liver image to a discriminative manifold giving good features for classification. SC are of high dimension and hence a compact representation feature is obtained by summing all the SC coefficients. Summed SC features along with cubic SVM classifier gave an accuracy of 96.6% in automatically categorizing the fatty content present in the tissue of a liver. © 2017 IEEE.