Aiming to address the issues of high parameter count and low accuracy in existing pavement crack detection methods, a lightweight detection method named DSCP-UNet is proposed for detecting pavement cracks. Firstly, a diminutive group convolution module (DGConv) and a self-adaptive pooling module (SAPool) are constructed to maintain detection accuracy and reduce model parameters. Additionally, a convolutional block attention module (CBAM) is used to enhance the model's attention towards crack features. Next, the pyramid attention with sampling group (PASG) is designed to further improve the model's ability to discriminate crack features. Then, based on the feature fusion structure, the PixelShuffle up-sampling module is used to solve the problem of blurred edges of the crack feature map. Finally, a hybrid loss function is developed to address the issue of category imbalance in the dataset. Experimental results demonstrate that the proposed method, with only 3.07 MB of parameters, achieves a recall of 84.53% and an F1 score of 72.76% on the CFD task test, and 71.36% recall with an F1 score of 75.14% on the Crack500 task test. This performance surpasses that of UNet, CPFNet, PIDNet, and DDRNet in terms of overall effectiveness. DSCP-UNet stands out for its lightweight design, high precision, and real-time detection capability, offering a robust solution for semantic segmentation of pavement cracks.