Time drift within Virtual machines is very common. Time is calculated within Operating Systems based on the assumption that it has 100% access to the CPU0 for timing. In a shared Virtual Environment, that is not the case and can cause time drifts in relation to CPU workload.
If your Time Sources are Virtual Machines, the Time Sources (i.e. Domain Controllers) should not be set to sync time from the hosts via VMware Tools.
The NTP client on each ESX host should be set to a physical Time Source. If none are available within the organization, round robin DNS records have been configured to gather time from the internet via UDP port 123.