This paper uses the Gaussian noise (GN) model to study the performance of the modulation recognition of different modulation formats for dense wavelength divi sion multiplexing (DWDM) systems.There are twenty-three kinds of modulation formats studied in this work in total.Besides the common modulation formats,such as M-ary amplitud e shift keying (MASK),M-ary phase shift keying (MPSK) and M-ary quadrature amp litude modulation (MQAM),2-dimensional optimal constellation and higher-order amplitude phase shift keying (APSK) constellation with optimized performance are also included in this study.In view of the situation that some feature parameters of MASK,MQAM and MAPSK are not constant if the signal-to-noise-ra tio (SNR) changes, this paper proposes a method which uses the training sequence to find the thresh old.The simulation results show that this threshold optimization method improves the pro bability of correct classification (Pcc) up to 93% for the region with the low SNR.Finally,this paper obtains the Pcc of the 23modulation formats for t he DWDM systems with various system parameters, such as the dispersion,nonlinearity and system total transmission distance.The modulation recognition performance of different modulation formats are analyzed through sim ulation. The simulation results show that MAPSK is the optimal format with Pcc above 99%,and 2-dimensional optimal constellation is the suboptimal one,with P cc above 93%.