Linux userspace DMA with iommu=on and without VFIO

Question

Is it possible for a Linux user-space application to use its own memory for DMA without being blocked by the IOMMU and without using VFIO?

Our application works fine when the iommu is disabled (intel_iommu=off) or in passthrough mode (intel_iommu=passthrough). However, it does not work when the IOMMU is enabled (intel_iommu=on) because the memory that we allocate in userspace is not allowed for DMA.

The official solution would be to use the Linux VFIO interface to manage the IOMMU, however we feel that the VFIO feature is not very mature and would prefer to find a simpler solution.

Can we somehow instruct the IOMMU to allow DMA for the physical memory that we have allocated? That would be wonderful because then we would not have to instruct our users to change their kernel boot parameters.

If there is a simple and reliable solution based on VFIO then that would also be interesting.

See related previous question for details of how we allocate memory: mremap(2) with HugeTLB to change virtual address?

Answer 1

You can write a simple char device driver to do this.

Inside driver(pseudo code):

static ssize_t device_write(struct file *filp, const char __user *buf, size_t count, loff_t * ppos)
{
    struct page *page;
    dma_addr_t DmaBusAddress;

    copy_from_user(addrstr, buf, sizeof(addrstr));

    uaddr = simple_strtoul(addrstr, NULL, 0);

    /* Get page structure which describes your user space 
       memory area 
    */
    res = get_user_pages(...,uaddr,...&page);

    /* (optional)Get the kernel virtual address for your user space page  
    */
    kernel_virtual_address = kmap(page);

    DmaBusAddress = dma_map_page(...,page)

    /*
    or the function below if you use address instead of page

    DmaBusAddress  = dma_map_single(...kernel_virtual_address,count...);

    */

    dev->dma_dir = DMA_TO_DEVICE;/* Write operation */
    dev->dma_size = count;
    dev->dma_addr = DmaBusAddress;

    /* Assume you have already memory map your DMA controller's I/O space 
      with remap_pfn_range() */

    writeRegister(DMA_ADDRESS,dev->dma_addr);
    ........................

    writeRegister(START_DMA_TRANSFER,1); /* enable DMA controller */

}

The code above shows how to use generic DMA layer.

Quote from ldd3

"The IOMMU can arrange for any physical memory to appear within the address range accessible by the device, and it can cause physically scattered buffers to look contiguous to the device. Making use of the IOMMU requires using the generic DMA layer; virt_to_bus is not up to the task"

and

"The generic DMA layer goes to great lengths to ensure that things work correctly on all architectures,but, as we will see, proper behavior requires adherence to a small set of rules."

The chapter "Memory Mapping and DMA" might answer all you questions.

Here it is the link: http://free-electrons.com/doc/books/ldd3.pdf