The rate control mechanism of high efficiency video coding (HEVC) is using the average of the bandwidth in frame layer,rather than adjusing the target bits acc o rding to the complexity of the current video,which makes the encoding process of high comple xity frame layers′ bandwidth insufficient relatively,while the complexity of the l ower frame layers′ bandwidth surplus relatively.To solve those problems,in order to reduce the comp utational complexity and improve the quality of coding,firstly,this paper has researched the effect of complexity in the process of coding frame layer on the rate allocation.According to th e complexity of the video content per frame,it allocates bits by the frame layer complexity model reasonably.Secondly,in order to establish R-Q curve,it combines the larges t coding unit (LCU) tree characteristics and the Gaussian mixture distribu tion characteristics of the LCUs′s residual signal to calculate parameters,so as to make the output bit rate equal to the target bits.Experimental results demonstrate that the proposed method is better than HEVC′s rate control algorithm and it is of better performance and strong robustness,the ra te of error is in 0.01% and the average peak signal-to-noise ratio (PSNR) is up to 0.21dB.