In underwater wireless optical communication systems, the effects of water absorption, scattering, and turbulence make channel estimation and signal detection challenging, leading to increased communication bit error rates and even communication failure. To address the difficulties of channel estimation and signal detection in complex underwater channels for optical communication, a machine learning-based channel estimation and demodulation algorithm is proposed, and its performance in underwater channel estimation and signal detection in direct current biased optical-orthogonal frequency division multiplexing optical communication systems is studied. Firstly, based on the proposed channel estimation and demodulation algorithm (deep neural networks and unsupervised learning k-means constellation demodulator), simulation modeling of complex channel frequency response, second-order equalization, and bit error analysis is completed. Secondly, studies on the signal-to-noise ratio gains in complex optical communication channels are conducted, comparing traditional least squares, linear minimum mean square error channel estimation algorithms, and minimum distance demodulation algorithms. In the simulation results, in an underwater channel with a turbulence scintillation index of 0.18 and a distance of 10 meters, the proposed channel estimation algorithm provides a signal-to-noise ratio gain of >6 dB and 1 dB compared to the least squares and linear minimum mean square error estimation for 8-QAM subcarriers at a bit error rate of 10-5. Additionally, using the proposed signal detection algorithm, a signal-to-noise ratio gain of >1 dB is achieved compared to traditional algorithms. The simulation results demonstrate that the proposed machine learning-based channel estimation and demodulation algorithm can improve the performance of complex underwater optical communication channels. The research results provide a reference for the design of long-distance, high-speed complex underwater optical communication systems.