The popularity of drones is gradually posing a threat to public safety, with issues such as small targets and high background noise in thermal infrared images. In response to the above issues, this article proposes a thermal infrared image unmanned aerial vehicle target tracking model based on a lightweight improved Siamese drone tracker (Siamese DT) network. Firstly, Ghost convolution is used to improve the utilization of model computation and memory resources. Secondly, the introduction of native sparse attention (NSA) mechanism improves the detection accuracy of the model with minimal computational cost. The experimental results on the Anti-UAV410 dataset show that under the OPE (one pass evaluation) standard, the state accuracy (SA) of the model is 67.93%, the parameter size is 38.503M, and the floating-point number is 62.647GFLOPs. Compared with the baseline network, the model proposed in this paper reduces computational complexity and memory usage while improving accuracy. It is suitable for deployment on mobile terminals and has good detection performance for TIR images.