What is your upload speed? Many ISPs give you 50 download but <5 upload, that would be a huge bottleneck
The biggest issue is security though. Unless you're setting up a VPN that only works when you set up a secured client on each device, I wouldn't trust that server to have access anywhere on the network. I would strongly recommend against opening any ports on your firewall as well. Tailscale and Cloudflare Tunnels are popular for homelabs that might be useful here and free for your use case