Fatty Liver Disease (FLD), if left untreated can progress into fatal chronic diseases (Eg. fibrosis, cirrhosis, liver cancer, etc.) leading to permanent liver failure. Doctors usually use ultrasound scanning as the primary modality for quantifying the amount of fat deposition in the liver tissues, to categorize the FLD into normal and abnormal. However, this quantification or diagnostic accuracy depends on the expertise and skill of the radiologist. With the advent of Health 4.0 and the Computer Aided Diagnosis (CAD) techniques, the accuracy in detection of FLD using the ultrasound by the sonographers and clinicians can be improved. Along with an accurate diagnosis, the CAD techniques will help radiologists to diagnose more patients in less time. Hence, to improve the classification accuracy of FLD using ultrasound images, we propose a novel CAD framework using convolution neural networks and transfer learning (pre-trained VGG-16 model). Performance analysis shows that the proposed framework offers an FLD classification accuracy of 90.6% in classifying normal and fatty liver images. © 2018 IEEE.