In this paper, the linear stability of sand waves sheared by a turbulent flow is analyzed. The velocity distribution in the streamwise direction is considered to follow the logarithmic law. The modified pressure distribution owing to the streamline curvature induced by the sand waves is employed in the formulation. The pressure distribution is derived using the Boussinesq approximation for the variation of streamline curvature over the flow depth. The flow model is coupled with the sediment transport model to study the stability of sand waves. Both the modes of sediment transport as bedload and suspended load are considered. The linear stability analysis reveals the favorable region for the formation of dunes and antidunes on the plane formed by the Froude number and the dimensionless wavenumber. The region of instability increases with an increase in particle parameter, while it decreases with an increase in relative roughness. The analysis reveals that the sand waves form beyond a limiting Froude number corresponding to the threshold Shields number. The theoretical results compare well with the available experimental data. © 2021, The Author(s), under exclusive licence to Springer Nature B.V.