On Thu, Nov 05, 2020 at 04:30:50PM +0000, Alex Bennée wrote:
Well this is where we may need to consider supporting another message format from Xen. Remember the individual daemons are attempting to be hypervisor agnostic with a minimal shim dealing with the specifics of setting up on each hypervisor. That said maybe terminating IOREQ and vhost-user messages isn't that tricky or maybe another message stream makes more sense (vfio user messages?).
I just remembered about the IOTLB feature in the vhost-user protocol. I'm mentioning it since I haven't seen it discussed so far, but I don't think it can help with the memory mapping problem.
The vhost protocol has bidirectional IOTLB messages, to deal with virtio devices behind vIOMMUs. It works like this: * The vhost device wants to read a buffer described by the vring, but it is behind an IOVA. The device sends an "IOTLB miss" message to the hypervisor. * The hypervisor sends an "IOTLB update" message with the IOVA->PA translation. Or "IOTLB access fail" on error. * When the mapping disappears the hypervisor sends "IOTLB invalidate".
It's inefficient, though can be accelerated with page tables (not upstream, I have a prototype somewhere). But I don't think we need it. In the scenarios were the shared memory is static, this feature isn't necessary (vhost has SET_MEM_TABLE to communicate static memory translations). And even when using a vIOMMU, for zero-copy, there doesn't need to be memory mapping messages between device and hypervisor, because a range of GPA could be reserved at boot for the device side and the hypervisor could populate stage-2 when the driver side issues mappings.
Thanks, Jean