3

I am writing a device driver for a Fibre Channel card that will move large amounts of data. The card can act as PCI master and will DMA the data into system memory. This is on an x86_64 linux system running kernel 3.0.35.

I first tried allocating the buffers using kmalloc(), but found that I could not allocate buffers large enough. However, as a learning exercise, I continued with the development using small buffers allocated with kmalloc(). I got the driver to work in this case. I allocated small buffers with kmalloc() (and the GFP_DMA flag), passed the address returned by kmalloc to dma_map_single(), and returned the address returned to the registers on the card.

I am now trying to modify the driver to use large buffers - on the order of 200MB. I have reserved a block of memory at boot time using the mem= kernel parameter. I map the reserved memory using ioremap(). I have mmap-ed the memory to user space and I have verified that the user application can write to the mmap-ed region and that the driver can read the data using the virtual address returned by ioremap(). But I have not been able to get the board to DMA data to my buffers.

I have tried to map the region that I allocated with dma_map_single(). I first passed the virtual address returned from ioremap(). I thought this was the correct address because the DMA-API-HOWTO file says:

"the driver can give a virtual address X to an interface like dma_map_single(), which sets up any required IOMMU mapping and returns the DMA address Z. The driver then tells the device to do DMA to Z, and the IOMMU maps it to the buffer at address Y in system RAM."

But I got no data from the card. I have also tried passing dma_map_single() the physical address of the region, but that didn't work. I have tried writing the address returned by dma_map_single() to the cards registers, and I have tried writing the physical address to the registers. Neither of those efforts worked, either.

Is there a step that I'm missing? Is my procedure in this case completely wrong? Can a card even perform DMAs to a memory region like the one I have reserved on an x86_64?

There is a possibility that I could upgrade the system to a 3.12.28 kernel. I understand that that kernel supports CMA. Would CMA work in this case?

Any help is appreciated.

user3822149
  • 31
  • 5
  • read about http://stackoverflow.com/help/mcve . Good luck. – shellter Jul 28 '16 at 00:31
  • `Mb` is Mega *bits*. `MB` is Mega bytes. Sorry I'm not a kernel hacker, but why do you want to use a 4 year old kernel? 4.x is current. If you're going for stability, 3.0.35 is nowhere near the last 3.0.x stable kernel. https://www.kernel.org/pub/linux/kernel/v3.0/ lists 3.0.101 as having more than a year of backported bugfixes beyond what 3.0.35 has. – Peter Cordes Jul 28 '16 at 04:06
  • Edited. Thanks. I don't have much control over the kernel version. My development machine is running SLES11 SP2. The machine where it will actually be used is running SLES11 SP4. I might be able to get them to upgrade to SLES12. Beyond that, I have very little influence in the organization. – user3822149 Jul 28 '16 at 14:34
  • kmalloc() is not the right way to allocate large buffers, especially for DMA. __get_dma_pages() may be a better option. – igorinov Jul 28 '16 at 16:07
  • 1
    Pretty late response here, but I did finally answer my own question w.r.t. how I tackled this problem. https://stackoverflow.com/questions/19460544/how-do-i-allocate-a-dma-buffer-backed-by-1gb-hugepages-in-a-linux-kernel-module – muusbolla Jun 26 '17 at 10:20

0 Answers0