Interesting analysis, but the conclusion is a bit contrived. People often prefer type safety over performance. When it doesn't impact type safety, they still prefer better performance over worse performance. That's not a contradiction.
You could argue that introducing the EnumSet-type isn't worth it for that, but I don't really see a disadvantage from having it.
Those who know about EnumSet, can use it. Those who don't know it, can continue to use HashSet. And if you don't know about EnumSet, but find one in a code base, it's not hard to guess what it does...