Medical imagery segmentation has been widely using deep learning approaches, which are quickly evolving in semantic segmentation. Nevertheless, due to their poor performance, newly proposed methods like Fully Convolutional Network, U-Net, LinkNet and SegNet still require enhancement to offer better semantic segmentation while identifying breast cancer. Therefore, this article presents a hybrid encoder and decoder framework in histology images of tissue slides. The novel network is designed with DlinkNet and UNet with a fusion of medical imagery data. An Attention Gate Module (AGM) is inserted into the proposed network to enrich the network learning's capacity. This way, semantic features are extracted and semantically segmented in the tumors in the input image set. The effectiveness of the planned network is investigated concerning various parameters like Maximum Symmetric Surface Distance (MSSD 35.98), Accuracy (99.2%), Relative Absolute Volume Difference (RAVD 3.41), Jaccard index (0.83), Average Symmetric Surface Distance (ASSD 0.629), Sensitivity (91%) and Dice Coefficient (DICE) (0.928) with existing networks of FPN and SENet. © 2022 IEEE.