In order to further improve the spectr al efficiency (SE) and to extend the transmission distance of Nyquist wavelength -division-multiplexed (Nyquist-WDM) system,this paper uses an improved nonbinary low-density parity-check (LDPC) code for channel coding in the Nyquist-WDM system and polarization multiplexing (PM) 64-ary quadrature amplitude modulation (QAM) tec hnique to study the performance of the 28.8Tbit/s Nyquist-WDM coherent detection system.With the s imulation conditions employed, the results show that with 4900km transmission distance in s ingle mode fiber (SMF),the inter-symbol interference (ISI) of the Nyquist-WDM PM 64QAM transmission system with baudrat e of 30Gbaud and channel grid of 25GHz can be suppressed effectively via adopting the improved NB-LDPC coding,resulting in 2.58dB improvement in bit error rate (BER) performance when com pared with the uncoded Nyquist-WDM PM 64QAM transmission system,and leading to a high spectral efficiency of 11.63bit·s－1·Hz－1.At the same time,due to the improved nonb inary LDPC coding,the transmission distance has been extended largely and the tolerance of the system to fiber nonlinear impairments has been enhanced eff ectively.