I got a quest 2 a few years back, and it blew my mind. We ended up getting my wife her own so we could play together. Now, my daughter plays a lot of gorilla tag, but other than that, they collect dust.
For me, the biggest thing that prevents me from using it more, is the isolation. You need to find an empty space and remove yourself completely from the world.
On my phone or Xbox, I still know what's going on around me, and I can hop in, play for a bit, and still know what's going on in my house. I can walk away for a moment and get back to what I was doing. In VR, it feels like more of an investment. If I'm not sure that I have plenty of time to disengage from reality, I'm not going to bother putting on the headset.
Also, I'm a sweater, and a soggy, foggy headset is just eww.