It is that simple. You already have to account for latency because everyone but one player (who you also can't trust no matter how many rootkits you install) is not the server.

Client side validation cannot possibly provide any actual security, but even if that wasn't the case and it was actually flawless, it would still be unconditionally unacceptable for a game to ever have kernel level access.