As a man I never considered pants to be uncomfortable. They are the perfect mix of range of motion, hiding the private parts. and support. Dresses, skirts, and robes seem like a nightmare to wear. You either restrict your range of motion or have to constantly worry about showing your underwear. And I would wear boxer briefs underneath by choice anyhow, so I still end up wearing pants.

I think the bigger issue is that most of us men are too lazy to look for different kinds of pants and end up wearing the same stuff year-round. Which can indeed become uncomfortable due to changes in temperature. But that issue wouldn't be fixed by having access to dresses or robes. As again, some would either be too cold or too hot.