I was having trouble getting the time right on my ESXi hosts. Every hosts had NTP enabled and 4 external NTP servers was added.
No matter what I did the time was running 4 minutes too fast. After some digging around I finally figured what the problem was.
1. My ESXi servers were added to my active directory so I could login with my windows users directly to the host.
2. My domain controller was syncing the time through VMware tools with the host. (So my DC was running 4 minutes too fast as well) (I know best practise is NOT to sync time through VMware tools)
I suspected that my ESXi hosts were syncing with my DC instead of NTP. To verifiy I removed my ESXi hosts from the domain and voila the time was right again!
Conclusion: If your ESXi hosts has been added to a domain it will sync it time to the domain and ignore the NTP settings. Set your primary DC to sync from an external source and you will be alright.