Building security systems is improving at a mammoth rate since the past decade, to cope with the threats of unauthorized access and fraudulent intentions. In high security public places like airports, embassies, corporate offices etc. only facial recognition for verification does not suffice full proof security. To predict the intention of an individual we must capture the objects around the subject/individual and his interactions with them in real time environment with good accuracy besides recognizing them. In this paper we designed an Enterprise Class Deep Neural Network (EcDNN) architecture built on the base architecture of YOLO network. Our proposed multitask learning network architecture recognizes the faces of registered individual as well as objects in the person's vicinity at one shot which achieves significant improvement in performance in terms of speed and model size without loss of precision, if it would have done separately in a cascaded model architecture. Our proposed single network architecture employing multitask learning is achieving state of the art recognition accuracy of 79 mAP at 40 fps with 33 % reduction in model size and an approximately 4x speedup with respect to the benchmark state of the art architectures, validated on standard dataset of PASCAL VOC 2012, FDDB and custom office dataset. © 2019 IEEE.