Abstract: In order to solve the optical power limitation problem of the transmitter of the Radio over Fiber (RoF) communication system and to improve the reliability of information transmis-sion, An LDPC-PS-64QAM scheme is proposed by combining low-density parity-check (LDPC) codes and probabilistic shaping (PS) techniques to optimise the transmission characteristics of RoF systems. The scheme uses the sign inversion method to probabilistically shape the 64-quadrature amplitude modulation (QAM) signal, while the LDPC code specified by 3GPP is used as the channel coding scheme. The simulation results show that at a symbol rate of 10 Gbaud, the BER of uniformly distributed (UD) and probabilistic shaping signals are compared, and the transmission power of PS signal is reduced by 1.4 dBm compared with UD signal, and the transmission power of the transmitter is further reduced by using LDPC code; at the fiber length of 20 km, the probabilistic shaping signal is reduced by 2.1 dB compared with uniformly distributed signal OSNR at the same BER. The result shows that the LDPC-PS-64QAM system has higher reliability by changing the transmission length of the fiber.