The subspace approximation problem with outliers, for given n points in d dimensions, an integer, and an outlier parameter, is to find a k-dimensional linear subspace of that minimizes the sum of squared distances to its nearest points. More generally, the subspace approximation problem with outliers minimizes the sum of p-th powers of distances instead of the sum of squared distances. Even the case of or robust PCA is non-trivial, and previous work requires additional assumptions on the input or generative models for it. Any multiplicative approximation algorithm for the subspace approximation problem with outliers must solve the robust subspace recovery problem, a special case in which the inliers in the optimal solution are promised to lie exactly on a k-dimensional linear subspace. However, robust subspace recovery is Small Set Expansion (SSE)-hard, and known algorithmic results for robust subspace recovery require strong assumptions on the input, e.g., any d outliers must be linearly independent. In this paper, we show how to extend dimension reduction techniques and bi-criteria approximations based on sampling and coresets to the problem of subspace approximation with outliers. To get around the SSE-hardness of robust subspace recovery, we assume that the squared distance error of the optimal k-dimensional subspace summed over the optimal inliers is at least times its squared-error summed over all n points, for some. Under this assumption, we give an efficient algorithm to find a weak coreset or a subset of points whose span contains a k-dimensional subspace that gives a multiplicative-approximation to the optimal solution. The running time of our algorithm is linear in n and d. Interestingly, our results hold even when the fraction of outliers is large, as long as the obvious condition is satisfied. We show similar results for subspace approximation with error or more general M-estimator loss functions, and also give an additive approximation for the affine subspace approximation problem. © 2020, Springer Nature Switzerland AG.