Crate Addition Request: memory-model

[sameo]

Crate Name

memory-model

Short Description

Trying to summarize discussions at #16:

This crate will fundamentally allow for accessing guest memory and translating guest addresses into host memory mapping.

The crate APIs will allow for writing to and reading from the guest memory.
This API should be able to handle raw slices but also DataInit types, i.e. types that can be safely initialized from a byte array.

Why is this crate relevant to the rust-vmm project?

Handling guest memory is a fundamental piece of a vmm...

[sameo]

@jiangliu I'm creating that issue so that we can create a memory-model repo and you can send your initial code drop there. Discussing through a separate issue makes it a little hard to follow.

[jiangliu]

Thanks, sameo! Will try implement a PoC by following @bonzini 's suggestion.

[sameo]

Thanks @jiangliu

[sboeuf]

LGTM

@andreeaflorescu @zachreizner @aghecenco PTAL

[andreeaflorescu]

Can we do a bit of brainstorming for the name of this crate? In the context of crates.io memory-model doesn't say much in my opinion.

[sboeuf]

True, maybe vmm-memory-model would be more appropriate.

[jiangliu]

This crate deals with virtual machine memory, so how about vm-mem or vmmem?

[sboeuf]

vm-memory?

[bonzini]

vmm-memaccess?

[sameo]

vm-memory is quite descriptive imho.

[jiangliu]

Design documentation for the memory-model crate (https://github.com/jiangliu/memory-model/tree/v2)

memory-model

A library to manage and access virtual machine's physical memory.

The memory-model crate aims to provide a set of stable traits for consumers to access virtual machine's physical memory. Based on these common traits, typical consumers like hypervisors, virtio backend drivers, vhost-user drivers could access guest's physical memory without knowing the implementation details. And thus virtual device backend drivers based on this crate may be reused by different hypervisors.

On the other hand, this crate dosen't define the way how the underline mechanism is implemented to access guest's physical memory. For light-wieght hypervisors like crosvm and firecracker, they may make some assumptions about the structure of virtual machine's physical memory and implement a light-weight backend to access guest's memory. For hypervisors like qemu, a high performance and full functionality backend may be implemented with less assumptions.

This crate is derived from two upstream projects:

• crosvm project commit 186eb8b0db644892e8ffba8344efe3492bb2b823
• firecracker project commit 80128ea61b305a27df1f751d70415b04b503eae7

To be hypervisor neutral, the high level abstraction has been heavily refactored. It could be divided into four parts as:

Abstraction of Generic Address Space

Build generic abstractions to describe and access an address space as below:

• AddressValue: stores the raw value of an address. Typically u32, u64 or usize is used to store the raw value. But pointers, such as *u8, can't be used because it doesn't implement the Add and Sub traits.
• Address: encapsulates an AddressValue object and defines methods to access it.
• AddressRegion: defines methods to access content within an address region. An address region may be continuous or it may have holes within the range [min_addr, max_addr) managed by it.
• AddressSpace: extends AddressRegion to build hierarchy architecture. An AnddressSpace object contains a group of non-intersected AddressRegion objects, and the contained AddressRegion object may be another AddressSpace object. By this way, a hierarchy tree may be built to describe an complex address space structure.

To make the abstraction as generic as possible, all the core traits only define methods to access the address space are defined here, and they never define methods to manage (create, delete, insert, remove etc) address spaces. By this way, the address space consumers (virtio device drivers, vhost-user drivers and boot loaders etc) may be decoupled from the address space provider (typically a hypervisor).

Specialization for Virtual Machine Physical Address Space

The generic address space crates are specialized to access guest's physical memory with following traits:

• GuestAddress: represents a guest physical address (GPA). On ARM64, a 32-bit hypervisor may be used to support a 64-bit guest. For simplicity, u64 is used to store the the raw value no matter the guest a 32-bit or 64-bit virtual machine.
• GuestMemoryRegion: used to represent a continuous region of guest's physical memory.
• GuestMemory: used to represent a collection of GuestMemoryRegion objects. The main responsibilities of the GuestMemory trait are:
  - hide the detail of accessing guest's physical address.
  - map a request address to a GuestMemoryRegion object and relay the request to it.
  - handle cases where an access request spanning two or more GuestMemoryRegion objects.

The virtual machine memory consumers, such as virtio device drivers, vhost drivers and boot loaders etc, should only rely on traits defined here to access guest's memory.

A Sample and Default Backend Implementation Based on mmap()

Provide a default and sample implementation of the GuestMemory trait by mmapping guest's memory into current process. Three data structures are introduced here:

• MmapRegion: mmap a continous range of guest's physical memory into current and provide methods to access the mmapped memory.
• GuestRegionMmap: a wrapper structure to map guest physical address into (mmap_region, offset) tuple.
• GuestMemoryMmap: manage a collection of GuestRegionMmap objects for a virtual machine.

One of the main responsibilities of the GuestMemoryMmap object is to handle the use cases where an access request crosses the memory region boundary. This scenario may be triggered when memory hotplug is supported. So there's a tradeoff between functionality code and complexity:

• use following pattern for simplicity which fails when the request crosses region boundary. It's current default behavior in the crosvm and firecracker project.

        let guest_memory_mmap: GuestMemoryMmap = ...
        let addr: GuestAddress = ...
        let buf = &mut [0u8; 5];
        let result = guest_memory_mmap.find_region(addr).unwrap().write_slice(buf, addr);

• use following pattern for functionality to support request crossing region boundary:

        let guest_memory_mmap: GuestMemoryMmap = ...
        let addr: GuestAddress = ...
        let buf = &mut [0u8; 5];
        let result = guest_memory_mmap.write_slice(buf, addr);

Utilities and Helpers

Following utility and helper traits/macros are imported from the crosvm project with minor changes:

• DataInit: Types for which it is safe to initialize from raw data. A type T is DataInit if and only if it can be initialized by reading its contents from a byte array. This is generally true for all plain-old-data structs. It is notably not true for any type that includes a reference.
• {Le,Be}_{16,32,64}: Explicit endian types useful for embedding in structs or reinterpreting data.
• VolatileMemory: Types for volatile access to memory.

Relationship among Traits and Structs

• AddressValue
• Address
• AddressRegion
• AddressSpace: AddressRegion
• GuestAddress: Address<u64>
• GuestMemoryRegion: AddressRegion<A = GuestAddress, E = Error>
• GuestMemory: AddressSpace<GuestAddress, Error> + AddressRegion<A = GuestAddress, E = Error>
• MmapAddress: Address<usize>
• MmapRegion: AddressRegion<A = MmapAddress, E = Error>
• GuestRegionMmap: AddressRegion<A = GuestAddress, E = Error> + GuestMemoryRegion
• GuestMemoryMmap: AddressSpace<GuestAddress, Error> + AddressRegion<A = GuestAddress, E = Error> + GuestMemoryRegion + GuestMemory

[bonzini]

Nice! I tried to make some more simplifications at https://github.com/bonzini/memory-model/tree/v2

In particular, MmapRegion gets its Bytes<usize> implementation directly from VolatileMemory, and I removed completely AddressSpace and AddressRegion for now. Because of this I also removed MmapAddress in favor of just using usize. This made sense to me because VolatileMemory and VolatileSlice are also accessed with usize offsets.

I tried also moving GuestMemoryMmap's Bytes<GuestAddress> implementation to generic code, but I couldn't do it because of the generic try_access method. However, try_access itself can be moved to GuestMemory.

Also, it may be useful to place more methods (formerly in AddressRegion, which I removed) in GuestMemoryRegion, but I didn't do it until we have experience with more than 1 implementation of GuestMemoryRegion and GuestMemory.

[jiangliu]

Thanks for quick review:)
I feel there's still one fundamental decision we need to make.
Currently qemu and firecracker maps all guest's memory into current process and kvm handles EPT faults by walking GPA->HVA->GPA. With vhost-user drivers, there's a requirement to limit access to partial of guest's memory due to security concerns.
One step further, for a hypervisor dedicated to server serverless workloads with customized guest kernel, do we still need to map all guest's memory into the hypervisor process address space? I think the answer may be 'no'.

The key idea is to manage guest's physical memory as a dedicated object, and assign part of guest's memory to related components (vhost drivers, hypervisor workers etc) according to access rules.
For example, vhost-user shouldn't access guest memory hosting guest's kernel text/data/bss sections.
Another point, currently hypervisors and guest kernels model all guest physical memory as normal RAM. We hope to enhance the hypervisor to provide guest's physical memory with additional properties, such R, W, X, SysExecute etc. This is analogous to memory type defined E820 and ACPI tables. Thus we could make use of the advanced control flags in EPT table.

So we hope we could build a layered abstraction as:
GuestPhysicalMemory: manage guest's physical memory resources. This is a new layer and have proposed to firecracker community but rejected.
GuestMemory: interfaces to access partial/all of the guest's memory. This is the current implementation in crosvm/firecracker.

And the address space abstraction is the common abstraction for both GuestPhysicalMemory and GuestMemory. And to support above mentioned usages, the AddressRegion traits need to be enhanced to provide memory property/type information.

Based on this design, it may help to address the security concern mentioned by @alexandruag . And it may also help to implement vIOMMU/vIOTLB in vhost-user drivers(more work needed to prove this claim).

So the fundament question is:

1. Should this crate provide traits to manage guest physical memory or only provide traits to access guest's memory?
2. Should the guest physical management functionality be treated as a dedicated abstraction or be merged into the memory access traits(GuestMemory)?
   I prefer this crate could cover both guest memory management and guest memory access with dedicated objects and abstractions.
   Thanks,
   Gerry

[jiangliu]

Please refer to #16 for previous discussions.

[bonzini]

Should this crate provide traits to manage guest physical memory or only provide traits to access guest's memory?

I have no objections to providing both, however I would start with a restricted implementation until it is clear what is common between the various users.

In fact I don't have any problem with adding more information to AddressRegion or GuestMemoryRegion. All I am doing in my refactoring is making sure that the interface is as simple as possible, and the implementation is as generic as possible.

I have pushed a new version that is nicely split into multiple commits and also adds a generic implementation of Bytes<GuestAddress>.

By the way, it might even be possible to make a separate crate for DataInit, Bytes, endian types and VolatileMemory. What do you think?

[jiangliu]

On Feb 26, 2019, at 6:58 PM, Paolo Bonzini ***@***.*** ***@***.***>> wrote: Should this crate provide traits to manage guest physical memory or only provide traits to access guest's memory? I have no objections to providing both, however I would start with a restricted implementation until it is clear what is common between the various users. In fact I don't have any problem with adding more information to AddressRegion or GuestMemoryRegion. All I am doing in my refactoring is making sure that the interface is as simple as possible, and the implementation is as generic as possible.

Great, so let's start small and add needed interfaces on demand.

I have pushed a new version that is nicely split into multiple commits and also adds a generic implementation of Bytes<GuestAddress>. By the way, it might even be possible to make a separate crate for DataInit, Bytes, endian types and VolatileMemory. What do you think?

Seems reasonable because these traits have more generic usage scenarios than GuestMemory.

…

You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#22 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AB14_Nj4VeWbuzrxR0HQm9RLva72Ps76ks5vRRMNgaJpZM4a98W8>.

[jiangliu]

Hi buddies,
Paolo and I have cooperated to shape out version 2 of the memory-model crate, you may access it at my personal repository at https://github.com/jiangliu/memory-model/tree/v2. Please help to review it and give your feedbacks. If the overall design is OK, please help to create repository vm-memory under project rust-vmm then we could send out PRs.

Note: as we have discussed, the 'memory-model' crate will be renamed as 'vm-memory'.

[sboeuf]

@jiangliu This looks very good! I am looking forward to the future pull requests on the brand new vm-memory repo :)

[bonzini]

@sameo This can be closed now

[sameo]

Fixed with rust-vmm/vm-memory#1