Photonic assisted millimeter-wave communication has a wide applicatio n prospect in the next generation broadband wireless access network due to its super-bandwidth advantage.In this paper,polarization multiplexed millimeter-wave signals with stable frequency are generated based on the optica l path structure of a single laser source and a dual-polarization Mach-Zehnder modulator (DP-MZM).Combined with VPI and MA TLAB simulation environment,the effective transmission of single sideband (SSB) polarization-division-multiplexing (PDM) q uadrature phase shift keying (QPSK), 16-ary quadrature amplitude modulation (16QAM) and 32-ary quadrature ampl itude modulation (32QAM) signals is achieved at 70 km,65 km and 50 km,respectively.The bit error rate (BER) per formance of uniform 16QAM and probability shaping (PS) 16QAM,uniform 32QAM and PS-32QAM vector millimeter wave (MMW) signals in optical fiber transmission is compared and analyzed with the same net bit transmission r ate.The simulation results show that at the same net bit rate,the optical power of PS-16QAM/32QAM signal in the common single-mode fiber (SMF) transmission system is about 0.3dBm with the hard-decision threshold 3.8×10－3.In nonlinear fiber (NLF) transmission system,the optical power of PS- 16QAM signal is increased by 0.8 dBm,and the optical power of PS-32QAM signal i s increased by 0.5 dBm.The results show that the transmission performance of MMW signal is improved obviously after PS.