The real-time detection of traffic rule violators in a city-wide surveillance network is a highly desirable but challenging task because it needs to perform computationally complex analytics on the live video streams from large number of cameras, simultaneously. In this paper, we propose an efficient framework using edge computing to deploy a system for automatic detection of bike-riders without helmet. First, we propose a novel robust and compact method for the detection of the motorcyclists without helmet using convolutional neural networks (CNNs). Then, we scale it for the real-time performance on an edge-device by dropping redundant filters and quantizing the model weights. To reduce the network latency, we place the detector module on edge-devices in the cameras. The edge-nodes send their detected alerts to a central alert database where the end users access these alerts through a web interface. To evaluate the proposed method, we collected two datasets of real traffic videos, namely, IITH-Helmet-1 which contains sparse traffic and IITH-Helmet-2 which contains dense traffic. The experimental results show that our method achieves a high detection accuracy of˜95% while maintaining the real-time processing speed of ˜22fps on NVIDIA-TXI. © 2020 IEEE.