In vivo single molecule tracking has recently developed into a powerful technique for measuring and understanding the transient interactions of transcription factors (TF) with their chromatin response elements. However, this method still lacks a solid foundation for distinguishing between specific and nonspecific interactions. To address this issue, we took advantage of the power of molecular genetics of yeast. Yeast TF Ace1p has only five specific sites in the genome and thus serves as a benchmark to distinguish specific from non-specific binding. Here, we show that the estimated residence time of the short-residencemolecules is essentially the same for Hht1p, Ace1p and Hsf1p, equaling 0.12-0.32 s. These three DNA-binding proteins are very different in their structure, function and intracellular concentration. This suggests that (i) short-residence molecules are bound to DNA non-specifically, and (ii) that nonspecific binding shares common characteristics between vastly different DNA-bound proteins and thus may have a common underlying mechanism. We develop new and robust procedure for evaluation of adverse effects of labeling, and new quantitative analysis procedures that significantly improve residence time measurements by accounting for fluorophore blinking. Our results provide a framework for the reliable performance and analysis of single molecule TF experiments in yeast.