A fusion algorithm for infrared and visible light images combining visible light image enhancement and multiscale decomposition is proposed to address the problems of detail loss and low contrast in infrared and visible light image fusion in recent years. Firstly, an adaptive visible light image enhancement method is proposed to improve the overall contrast of visible light images and enhance the detail information in visible light images. Then, a multiscale decomposition algorithm based on Gaussian filtering and rolling guided filtering is proposed to decompose the source image into small-scale layers, large-scale layers, and base layers. The small-scale layer fusion adopts a fusion rule based on maximum absolute value, the large-scale layer fusion injects the infrared spectral features into the visible light image using nonlinear weight coefficients, and the base layer adopts a fusion rule based on visual saliency mapping to avoid contrast loss. Finally, each scale layer is reconstructed to generate the fusion image. Experimental results show that compared with other algorithms, the proposed method has improved the objective evaluation indicators such as edge preservation, human-inspired perceptual metrics, spatial frequency, standard deviation, and edge intensity by an average of 23.50%, 30.38%, 46.67%, 50.41%, 20.17%, and 54.19%, respectively, and the generated fusion image also performs well in subjective evaluation.