In the end, it was a firmware configuration. In my case - I have a 24G GPU, so passing it through I need to increase the MIMO Size. See https://edk2.groups.io/g/discuss/topic/ovmf_resource_assignment/59340711 for details.
In the end, the fix was adding the QEMU config: raw.qemu: -fw_cfg name=opt/ovmf/X-PciMmio64Mb,string=65536