Disable Cache/Buffer on Specific File (Linux)

Question

I am currently working in a Yocto Linux build and am trying to interface with a hardware block on an FPGA. This block is imitating an SD card with a FAT16 file system on it; containing a single file (cam.raw). This file represents the shared memory space between the FPGA and the linux system. As such, I want to be able to write data from the linux system to this memory and get back any changes the FPGA might make (Currently, the FPGA simply takes part of the data from the memory space and adds 6 to the LSB of a 32-bit word, like I write 0x40302010 and should get back 0x40302016 if I read back the data). However, due to some caching somewhere, while I can write the data to the FPGA, I cannot immediately get back the result.

I am currently doing something like this (using python because its easy):

% mount /dev/mmcblk1 /memstick
% python
>> import mmap
>> import os
>> f = os.open("/memstick/cam.raw", os.O_RDWR | os.O_DIRECT)
>> m = mmap.mmap(f, 0)
>> for i in xrange(1024):
...  m[i] = chr(i % 256)
...
>> m.flush() # Make sure data goes from linux to FPGA
>> hex(ord(m[0])) # Should be 0x6
'0x0'

I can confirm with dd that the data is changed (though I frequently run into buffering issues with that too) and using the tools for the FPGA (SignalTap/ChipScope) that I am indeed getting correct answer (ie, the first 32-bit word in this case is 0x03020106). However, someone, whether its python or linux or both are buffering the file and not reading from the "SD card" (FPGA) again and storing the file data in memory. I need to shut this completely off so all reads result in reads from the FPGA; but Im not sure where the buffering is taking place or how to do that.

Any insight would be appreciated! (Note, I can use mmap.flush() to take any data I write from python to dump it to the FPGA, but I need like a reverse flush or something to have it reread the file data into the mmap!)

Update:

As suggested in the comments, the mmap approach might not be the best one to implement what I need. However, I have now tried both in python and C, but using basic I/O functions (os.read/write in python, read/write in C) using the O_DIRECT flag. For most of these operations, I end up getting errno 22. Still looking into this....

Hmmm, good to know. Is there a better way of handling the reads and writes then? (I really like the mmap interface though....) — Unn, Apr 28 '15 at 21:42
You're doing something a little weird here: you have a file on a filesystem mounted normally, but the contents of the underlying block device is changing, and nobody is telling the kernel that the block device contents has changed. Normally, when you plug a drive into your system, all changes to the drive go through the kernel first, so it can invalidate all of the caches (who knows how many layers of caching?) all the way down. You can either try `mmap()` on `/dev/mmcblk1`, parsing the filesystem yourself, or you can ask a kernel expert. — Dietrich Epp, Apr 28 '15 at 21:57
The reason `O_DIRECT` doesn't work is because `O_DIRECT` is basically an instruction to bypass the page cache, and `mmap()` is a function which is designed to expose the page cache to your program, so they are logically incompatible. — Dietrich Epp, Apr 28 '15 at 22:01
The `mmap.flush` method forces it to sync the memory buffer with what's actually on disk. But I'm not sure that's sufficient for your problem--if any of the in-memory pages are dirty, they'll overwrite whatever's on disk, rather than the other way around. — abarnert, Apr 28 '15 at 22:39
@DietrichEpp Thanks for all the great information. I think I am seeing some issue with caching; but Im just not sure how to tell the kernel: "this is a volatile file, please don't cache." I can try direct reads and writes minus the mmap though I haven't had any luck with that yet (got errors with both os.read and os.write) — Unn, Apr 28 '15 at 23:38

score 2 · Accepted Answer · edited May 23 '17 at 11:46

After doing digging, I found out what I was doing wrong with the O_DIRECT flag. In my C and Python versions, I wasnt using memalign to create the buffer and wasn't doing block reads/writes. This post has a good explanation:

How can I read a file with read() and O_DIRECT in C++ on Linux?

So, in order to achieve what I am doing, this C program works as a basic example:

#include <stdio.h>
#include <fcntl.h>
#include <errno.h>

#define BLKSIZE 512

int main() {
  int fd;
  int x;
  char* buf;

  fd = open("/home/root/sd/fpga/cam.raw", O_RDWR | O_SYNC | O_DIRECT);
  if (!fd) {
    printf("Oh noes, no file!\n");
    return -1;
  }

  printf("%d %d\n", fd, errno);

  buf = (char*) memalign(BLKSIZE, BLKSIZE*2);

  if (!buf) {
    printf("Oh noes, no buf!\n");
    return -1;
  }

  x = read(fd, buf, BLKSIZE);
  printf("%d %d %x %x %x %x\n", x, errno, buf[0], buf[1], buf[2], buf[3]);

  lseek(fd, 0, 0);

  buf[0] = '1';
  buf[1] = '2';
  buf[2] = '3';
  buf[3] = '4';
  x = write(fd, buf, BLKSIZE);

  printf("%d %d\n", fd, errno);

  lseek(fd, 0, 0);

  x = read(fd, buf, BLKSIZE);
  printf("%d %d %x %x %x %x\n", x,errno, buf[0], buf[1], buf[2], buf[3]);

  return 0;
}

This will work for my purposes, I didnt look how to do proper memory alignment to use Python's os.read/os.write functions in a similar way.

Disable Cache/Buffer on Specific File (Linux)

1 Answers1