But surely couldn't the argument be made that since all political power comes from the voters (in theory), they deserve to be the ultimate judge of whether someone is fit for office or not?

From a purely theoretical standpoint