This is possible! At least, I’ve seen it described in other people’s setups. Having the integrated GPU should help in this case.
If I remember correctly, you need to make sure the vm shuts down properly so that it shuts down the graphics card properly. Then you can unload the vfio-pci module and load the correct module for your card.
Two great resources are the subreddit (unfortunately) r/vfio and the arch wiki page: https://wiki.archlinux.org/title/PCI_passthrough_via_OVMF