The governing equation of a gear system is established.In the equation

the time-varying mesh stiffness

mesh damping

input and output torque fluctuations are considered.Due to the mesh stiffness of a gear pair

the governing equation represents a linear dynamic system with periodic time-varying coefficient.A-operator method (AOM) is used to obtain the approximate analytic solution.According to the results

A-operator method (AOM) overcomes the defaults of harmonic balance method and can keep all frequency components.In the response of the system

fundamental

double and composed frequency etc of time-varying mesh stiffness and torque fluctuation frequency coexist.A-operator method (AOM) takes less time than other numerical method.