You should look into IPMI console access, that's usually the real 'only way out of this'
SSH has a lot of complexity but it's still the happy path with a lot of dependencies that can get in your way- is it waiting to do a reverse dns lookup on your IP? Trying to read files like your auth key from a saturated or failing disk? syncing logs?
With that said i am surprised people are having responsiveness issues under full load, are you sure you weren't running out of memory and relying heavily on swapping?
it's typically up to the distribution to configure things like that, and many Linux distributions do come in both server and desktop or workstation variants like Ubuntu desktop vs Ubuntu server, or RHEL server vs RHEL Workstation
I can't say how well they tune these things as I haven't ran them personally, but they do exist.