Aiming at the problem that it is difficult to accurately detect irregular defects on the surface of lead frames, a method for detecting defects on the surface of lead frames is proposed by integrating attention and multilevel residuals. First, a multi-scale global attention module is proposed to further acquire the global information of the lead frame and improve the segmentation accuracy by capturing the channel and spatial information of the defective edge region. Then, in order to realize the multiscale fusion of defect information, a multilevel residual fusion attention network module is designed to extract the global semantic information of surface scratch defects. In addition, the encoder employs a Smooth Maximum Unit activation function to improve the detail missing phenomenon during detection. The experimental results show that the MIoU metrics of the proposed lead frame surface defect detection method are improved by 25.05%, 26.79%, 12.11%, and 21.02% compared to the four typical methods on the homemade lead frame surface defect dataset, respectively. The ablation experiments prove that the proposed method has better defect detection performance and can obtain more effective defect information.