1

I am writting a Kernel Module that is going to trigger and external PCIe device to read a block of data from my internel memory. To do this I need to send the PCIe device a pointer to the physical memory address of the data that I would like to send. Ultimately this data is going to be written from Userspace to the kernel with the write() function (userspace) and copy_from_user() (kernel space). As I understand it, the address that my kernel module will see is still a virtual memory address. I need a way to get the physical address of it so that the PCIe device can find it.

1) Can I just use mmap() from userspace and place my data in a known location in DDR memory, instead of using copy_from_user()? I do not want to accidently overwrite another processes data in memory though.

2) My kernel module reserves PCIe data space at initialization using ioremap_nocache(), can I do the same from my kernel module or is it a bad idea to treat this memory as io memory? If I can, what would happen if the memory that I try to reserve is already in use? I do not want to hard code a static memory location and then find out that it is in use.

Thanks in advance for you help.

whh4000
  • 905
  • 1
  • 12
  • 30

1 Answers1

3

You don't choose a memory location and put your data there. Instead, you ask the kernel to tell you the location of your data in physical memory, and tell the board to read that location. Each page of memory (4KB) will be at a different physical location, so if you are sending more data than that, your device likely supports "scatter gather" DMA, so it can read a sequence of pages at different locations in memory.

The API is this: dma_map_page() to return a value of type dma_addr_t, which you can give to the board. Then dma_unmap_page() when the transfer is finished. If you're doing scatter-gather, you'll put that value instead in the list of descriptors that you feed to the board. Again if scatter-gather is supported, dma_map_sg() and friends will help with this mapping of a large buffer into a set of pages. It's still your responsibility to set up the page descriptors in the format expected by your device.

This is all very well written up in Linux Device Drivers (Chapter 15), which is required reading. http://lwn.net/images/pdf/LDD3/ch15.pdf. Some of the APIs have changed from when the book was written, but the concepts remain the same.

Finally, mmap(): Sure, you can allocate a kernel buffer, mmap() it out to user space and fill it there, then dma_map that buffer for transmission to the device. This is in fact probably the cleanest way to avoid copy_from_user().

Peter
  • 14,559
  • 35
  • 55
  • Ok, I am not actually initiating a DMA transfer from my end. The end point on the PCIe bus is doing a DMA read from my memory space. Do I still need to use dma_map_page()? And ultimately I would like to not have to change my BAR for the PCIe device so I would like my buffer to be located in a certain space. Any options? (I have LDD3 in front of me) – whh4000 Jul 03 '14 at 20:11
  • 2
    Yes, that's how bus-mastering DMA works. You are telling the device what addresses to read in the host memory space. The BAR is not related to this at all. It's only for the registers in your device that are mapped into kernel space. You use those registers to tell the device to fetch data from elsewhere (anywhere!) in the memory space, and that's what `map_page` is for. My answer is exactly what you need to do, and is the standard way to do bus-mastered DMA. – Peter Jul 03 '14 at 20:28