Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

is-swiotlb_force_bounce(dev)==false for CVM IO device #34

Open
enterJazz opened this issue Dec 8, 2023 · 4 comments
Open

is-swiotlb_force_bounce(dev)==false for CVM IO device #34

enterJazz opened this issue Dec 8, 2023 · 4 comments

Comments

@enterJazz
Copy link
Contributor

enterJazz commented Dec 8, 2023

  • dma_direct_map has the is_swiotlb_force_bounce check for the struct device ( dma device )
  • the drivers (nvme, virtio) hold this device in enclosing structures
  • we need to find where the struct device originally is located, and set the force_bounce flag in device->dma_io_tlb_mem to false

another difficulty:

  • we need to be able to differentiate between the cvm io device and the qcow2 accessing device
  • cvm io device goes through disk mapper -> only what goes through disk mapper has the dma pages already in shared mem
@mmisono
Copy link
Member

mmisono commented Dec 8, 2023

struct virtio_blk has struct virito_device *dev, and struct virtio_device has struct device dev

https://github.com/torvalds/linux/blob/5e3f5b81de80c98338bcb47c233aebefee5a4801/drivers/block/virtio_blk.c#L66C14
https://github.com/torvalds/linux/blob/5e3f5b81de80c98338bcb47c233aebefee5a4801/include/linux/virtio.h#L129

struct virtio_blk is initialized when the os probe PCI devices
https://github.com/torvalds/linux/blob/5e3f5b81de80c98338bcb47c233aebefee5a4801/drivers/block/virtio_blk.c#L1302

So I guess we can set the flag here. But the important question is

we need to be able to differentiate between the cvm io device and the qcow2 accessing device

Maybe we can use virtio_device_id or other information in struct dev

@enterJazz
Copy link
Contributor Author

Both approaches should work, and are fine for a prototype. However, they are both driver specific- would be nice if we can later on have these modifications in driver generic code, to avoid driver specific mods

@enterJazz
Copy link
Contributor Author

enterJazz commented Dec 11, 2023

Regarding IDing devices:

we should be able to ID devices as sysfs does; in particular, there should be a way to pass a device id via qemu, and then check for said device ID in the kernel.

we should be able to e.g. use some attribute in /sys/block/<device-id> ( or we can somehow set device id beforehand )

see https://github.com/torvalds/linux/blob/master/block/blk-sysfs.c

@enterJazz
Copy link
Contributor Author

enterJazz commented Dec 11, 2023

So my current idea is:

  • set device name / mount with QEMU (e.g. to /dev/cvm-io)
  • have the device ( driver ? ) check its name -> if its name is cvm-io disable swiotlb

CURRENT IDEA:

struct device dev;
struct device_driver dev_drver = dev->device_driver;

we then should be able to identify the device driver by one of its fields:

struct device_driver {
	const char		*name;
	const struct bus_type	*bus;

	struct module		*owner;
	const char		*mod_name;	/* used for built-in modules */

	bool suppress_bind_attrs;	/* disables bind/unbind via sysfs */
	enum probe_type probe_type;

	const struct of_device_id	*of_match_table;
	const struct acpi_device_id	*acpi_match_table;

	int (*probe) (struct device *dev);
	void (*sync_state)(struct device *dev);
	int (*remove) (struct device *dev);
	void (*shutdown) (struct device *dev);
	int (*suspend) (struct device *dev, pm_message_t state);
	int (*resume) (struct device *dev);
	const struct attribute_group **groups;
	const struct attribute_group **dev_groups;

	const struct dev_pm_ops *pm;
	void (*coredump) (struct device *dev);

	struct driver_private *p;
};

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants