Addressing the demand for rapid and cost-effective detection of diverse defects on wafer surfaces, this paper presents a dual-interference quantitative phase imaging method that integrates the principles of quadriwave lateral shearing interferometry and off-axis interferometry. A common-path design for the illumination and microscopic imaging modules of both interferometric modalities eliminates the need for optical reconfiguration, enabling low-cost imaging of various defect types. Moreover, the topography of the defects under test can be quantitatively reconstructed from only two interferograms, ensuring high system stability and detection efficiency. Experimental results demonstrated that the total time for detecting linear defects on a gallium nitride wafer surface—including both image acquisition and processing—was under 2 seconds. The extracted maximum defect depth was 195.4 ± 1.21 nm, which is consistent with the value of 197 ± 1.09 nm obtained using a commercial white-light interferometer, while the processing time was significantly reduced. Furthermore, the measured sizes for surface slip lines, scratches, particles, and residues were 173, 109, 796 and 380 nm, respectively. All relative errors, referenced to nominal values, were within 3%. These comprehensive results robustly validate the effectiveness of the proposed detection system.