Time-varying mesh stiffness (TVMS) and loaded static transmission error (LSTE) are considered the main sources of vibration in a gear system. TVMS and LSTE of a helical gear pair have been calculated by using an instantaneous pressure angle. However, this method could not consider the root profile accurately. In this study, an analytical model is proposed to calculate TVMS and LSTE of a helical gear pair. To consider the trochoidal root profile, the profile model is also developed by a virtual rack reflecting the cutter information. The finite element (FE) model of a helical gear is used to validate the proposed analytical model. The LSTE results are highly similar to FE analysis results and relative errors in peak-to-peak values of LSTE are also small. The proposed model is found to be more effective than traditional models in calculating the TVMS and LSTE of both helical gear pairs and spur gears. The changes in the radii of the base circle and root circle according to the helix angle are also investigated.