How would one prevent MMAP from caching values?

Question

I've written a linux driver that ioremaps exports PCI BAR0 for a particular device to a sysfs binary attribute allowing userspace to directly control it.

The problem rears when I attempt to MMAP on top of the attribute to directly access that bit of memory (from a userland program). Reads succeed just fine and return expected values, though when I write to that memory it appears to be cached somewhere between the kernel and memory and not delivered to the GMCH root complex (and therefore the device). What I'd like to do is have an implicit write memory barrier after each access.

Is there any way to prevent the kernel from caching writes to a mmap-ed bit of memory?

Follow ups:

Is calling msync() after every access the "accepted" way to do this?

To answer the follow-up ... you don't need `msync()` at all here because there is no file backing the mmap. What you might need though are some architecture specific instructions for various things ... for example to ensure correct order of hardware accesses on powerpc you may need the [eieio](https://www-01.ibm.com/support/knowledgecenter/ssw_aix_61/com.ibm.aix.alangref/idalangref_eieio_instrs.htm) instruction (someone has a sense of humour :-) ... same comment made [here](https://unix.stackexchange.com/questions/237783/accessing-memory-mapped-i-o-is-slow). — Murray Jensen, Nov 04 '15 at 12:42

score 29 · Accepted Answer · answered Mar 27 '12 at 18:44

29

Going to go ahead and answer this one myself with my solution.

In the Kernel driver from my sysfs mmap function, there is a macro in /include/asm/pgtable.h that sets the proper flags for a nocache'd pfn remap. It looks like this:

vma->vm_page_prot = pgprot_noncached(vma->vm_page_prot);
if (io_remap_pfn_range(vma, vma->vm_start, vma->vm_pgoff,
               vma->vm_end - vma->vm_start,
               vma->vm_page_prot))
    return -EAGAIN;

Additionally, in the userland mmap, I used the MAP_SHARED flag in the mmap flags argument.

The combination of the two ultimately did the trick.

answered Mar 27 '12 at 18:44

Sean Madden

1,069
2
10
22

Thx so much man ! I had exactly the same problem as u and couldn't find a solution... – Julien Jun 06 '12 at 14:29
2

+1 Worked for me too. Thank you for taking the time to post the solution here once you found it. :-) – TheCodeArtist Mar 06 '14 at 11:10

score 0 · Answer 2 · answered Mar 27 '12 at 13:58

0

Might ioremap_nocache() help?

answered Mar 27 '12 at 13:58

blueshift

6,742
2
39
63

I'm using io_remap_pfn_range to remap and translate the kernel pages into the userspace addresses. I've looked (and am using) ioremap_nocache() elsewhere and that manually sets the KERNEL_NOCACHE attribute on each page manually. I can't find any mention of that happening (or a flag to set) using io_remap_pfn_range. – Sean Madden Mar 27 '12 at 14:17
2

Additional: ioremap_nocache() is used pretty much only for mapping IO memory into kernel space and not down into userspace - that's why io_remap_pfn_range is handy. – Sean Madden Mar 27 '12 at 14:53

How would one prevent MMAP from caching values?

2 Answers2

Linked