Recent years have seen a growing interest in understanding neural networks from an optimization perspective. It is understood now that converging to low-cost local minima is sucient for such models to become eective in practice. However, in this work, we propose a new hypothesis based on recent theoretical ndings and empirical studies that neural network models actually converge to saddle points with high degeneracy. Our ndings from this work are new, and can have a signicant impact on the development of gradient descent based methods for training neural networks. We validated our hypotheses using an extensive experimental evaluation on standard datasets such as MNIST and CIFAR-10, and also showed that recent eorts that attempt to escape saddles nally converge to saddles with high degeneracy, which we dene as ‘good saddles’. We also veried the famous Wigner’s Semicircle Law in our experimental results. © 2018 Copyright held by the owner/author(s).