4

I'm working on a research project that requires me to perform a memory capture from custom hardware. I am working with a Zedboard SoC (dual-core ARM Cortex-A9 with FPGA fabric attached). I have designed a device driver that allows me to perform virtual memory captures and physical memory captures (using an AXI4-Lite peripheral that controls the Xilinx AXI DMA IP).

My goal is to capture all mapped pages, so I check /proc/pid/maps for mapped regions, then obtain PFNs from /proc/pid/pagemaps, pass the physical addresses into my device driver, and then pass them to my custom hardware (which invokes the Xilinx AXI DMA to obtain the contents from physical memory).

NOTE: I am using Xilinx's PetaLinux distribution, which is built on Linux version 4.14.

My device driver implements the following procedure through a series of IOCTL calls:

  1. Stop the target process.
  2. Perform virtual memory capture (using the access_process_vm() function).
  3. Flush the cache (using the flush_user_range() function).
  4. Perform physical memory capture.
  5. Resume the target process.

What I'm noticing, however, is that the virtual memory capture and the physical memory capture differ in the [heap] section (which is the first section that extends past one page). The first page matches, but none of the other pages are even close. The [stack] section does not match at all. I should note that for the first two memory sections, .text and .rodata, the captures match exactly. The conclusion for now is that data that does not change during runtime matches between virtual and physical captures while data that does change during runtime does not match.

So this leaves me wondering: am I using the correct function to ensure coherency between the cache and the RAM? If not, what is the proper function to use to force a cache flush to RAM? It is necessary that the data in RAM is up-to-date to the point when the target process is stopped because I cannot access the caches from the custom hardware.

Edit 1: In regards to this question being marked as a possible duplicate of this question, I am using a function from the accepted answer to initiate a cache flush. However, from my perspective, it doesn't seem to be working as the physical memory does not match the virtual memory as I would expect if a cache flush were occurring.

ngc6027
  • 101
  • 11
  • Sound to me like you want to look into how swapping is implemented in linux. – Flavius Mar 21 '19 at 20:30
  • Possible duplicate of [How to flush the CPU cache for a region of address space in Linux?](https://stackoverflow.com/questions/22701352/how-to-flush-the-cpu-cache-for-a-region-of-address-space-in-linux) – ReAl Mar 22 '19 at 10:54
  • @Flavius When I grab the PFN from /proc/pid/pagemap, I check the swap bit for each page, as well as the present bit. Each page that I attempt to pull from physical memory claims to be present and not swapped. Is it possible for the memory to be swapped between when I check that bit and when I initate the physical memory transfer? – ngc6027 Mar 22 '19 at 16:42

1 Answers1

4

For anyone coming across this question in the future, the problem was not what I thought it was. The flush_user_range() function I mentioned is the correct function to use to push pages back to main memory from cache.

However, what I didn't think about at the time was that pages that are virtually contiguous are not necessarily (and are very often not) physically contiguous. In my kernel code, I passed the length of the mapped region to my hardware, which requested that length of data from the AXI DMA. What I should have done was a virtual-to-physical translation to get the physical address of each page in each region, then requested one page length of data from main memory, repeating that process for each page in each mapped region.

I realize this is a pretty specific problem that likely won't help anyone else doing the same thing I was doing, but hopefully the learned lesson can help someone in the future: Linux allocates pages in physical memory (that are usually 4kB in size, though you shouldn't assume that's the case), and a collection of physical pages is contained in one mapped region. If you're working on any code that requires you to check physical memory, be sure to be wary of where your data may cross physical page boundaries and act accordingly.

ngc6027
  • 101
  • 11
  • One should never assume that virtual memory is contiguous in physical address space. There are a couple of ways around this in kernel programming. The old fashioned way is to reserve kernel memory and manage it yourself (kernel parameter memmap=). This is the mode I generally use with user-mode embedded programming. A fairly recent new way is to use the Contiguous Memory Allocator (CMA), which guarantees that kernel allocations are contiguous in physical space. There is no public, open-source user-mode interface for CMA, but that doesn't mean there couldn't be one. – Nufosmatic Jul 30 '21 at 13:09
  • Another virtual-vs-physical area is NVIDIA GPUDirect. GPUDirect makes up to 256MB of CUDA memory resources visible on the PCIe bus on GPUs capable of GPUDirect (Quadro for one - GeForce is not supported). This is a means of driving data directly into a GPU from a DMA device such as a Mellanox network card. I have never found GPUDirect pages to not be contiguous, but I check every time anyway... – Nufosmatic Jul 30 '21 at 13:15