A new marriage combined with kids makes transition difficult, and it sounds like you both have had a lot of transition in the last 3 years.
I know this isn't a relationship advice thread, it is AITA. This situation is so much more complicated than yes or no. Leaving your wife to take on all the responsibility of being a parent and vacationing without her is an asshole move.
At the same time, he seems to be hurting from the little you shared. If he is saying that the move ruined his life, he is likely struggling from loneliness. You didn't describe any connection between the two of you at all. You guys may just need to have some sincere conversations about all of the change in the last 3 years and what you each need.