This paper proposes an effective fabric defect detection algorithm to address the challenges of complex fabric backgrounds and diverse defects in the inspection process. First, the downsampling operation in the backbone feature extraction network is improved by introducing space-to-depth conversion convolution (SPD-Conv), which reduces information loss during feature extraction and enhances its accuracy. Second, a multi-scale fused feature pyramid (MFFPN) is constructed to mitigate interference caused by information conflicts during feature fusion, further strengthening cross-layer feature interaction. Additionally, the loss function is optimized to Shape-IoU, which computes the loss by focusing on the shape and scale of bounding boxes, thereby achieving more accurate bounding box regression. Experimental results demonstrate that the proposed method achieves significant progress in fabric defect detection tasks, effectively handling defect inspection scenarios for fabrics with varying colors and textures.