0

Can I allocate one large and guaranteed continued range physical memory (100 MB consecutive without breaks) on Linux, and if I can, then how can I do this?

It is necessary to mapping this a continuous block of memory through the PCI-Express BAR from one CPU1 to the other CPU2 located behind the PCIe Non-Transparent Bridge.

Alex
  • 12,578
  • 15
  • 99
  • 195
  • 2
    Can you compile your driver into the kernel? That'd let you allocate the memory when it boots. That's the only way I can think of (and I admit - I only read about the possibility and never did it myself) . – Benjamin Gruenbaum Nov 19 '13 at 11:22
  • I don't understand if you are coding a user-level application or a kernel module. – Basile Starynkevitch Nov 19 '13 at 11:36
  • @Benjamin Gruenbaum If this cann't be done at the user-space, but can only be done at the kernel-space (yes I can compile my driver into the kernel), then how? – Alex Nov 19 '13 at 11:37
  • Did you already wrote your driver? Where is it? – Basile Starynkevitch Nov 19 '13 at 11:41
  • 1
    Here's a related question from the Unix stackexchange: http://unix.stackexchange.com/q/37729/17046 – Nathan Fellman Nov 19 '13 at 11:43
  • @Nathan Fellman Did you use memory allocation at a boot time with say `mem=3GB` and use `void* virt_ptr = ioremap(3GB, 1GB)` to get this 1GB of top of 4GB RAM in user-space? – Alex Nov 19 '13 at 13:46
  • 1
    no, I wrote a device driver that used `kmalloc` to allocate the block of memory that I wanted – Nathan Fellman Nov 19 '13 at 13:58
  • @Nathan Fellman Thanks! I.e. you done: `void* virt_krenel_space_ptr = kmalloc(size, GFP_ATOMIC);` to allocate the contiguous memory in virtual addressing of kernel-space? And then didn't map it to user-space? But here said that we can allocate only 128KB - 4MB by `kmalloc()` http://stackoverflow.com/questions/116343/what-is-the-difference-between-vmalloc-and-kmalloc – Alex Nov 19 '13 at 14:35
  • I successfully allocated about 50MB. I didn't try more. – Nathan Fellman Nov 19 '13 at 14:37
  • @Nathan Fellman And then did you use `remap_pfn_range(vma, vma->vm_start, RAW_DATA_OFFSET >> PAGE_SHIFT, RAW_DATA_SIZE, PAGE_SHARED);` and `mmap()` to mapping it to user-space? – Alex Nov 19 '13 at 14:47
  • @Alex: I didn't need the user-space address, I needed a physical address for DMA – Nathan Fellman Nov 19 '13 at 20:06
  • @Nathan Fellman And `kmalloc()` works only with physical addressing, not with virtual in kernel-space, isn't it? – Alex Nov 19 '13 at 20:34
  • 1
    well, the kernel does provide methods to map the ranges to user space. I simply didn't use them. – Nathan Fellman Nov 19 '13 at 20:49

2 Answers2

1

You don't allocate physical memory in user applications (physical memory only makes sense inside the kernel).

I don't understand if you are coding a kernel module or some Linux application (e.g. a numerical finite-element code=.

Inside applications, you can allocate virtual memory with e.g. mmap(2) (and then you can allocate a big contiguous segment of address space)

I guess that some GPU cards give access to a large amount of GPU memory thru mmap so I believe it is possible to do what you want.

You might be interested by numa(7) man page. Probably the numa(3) library should give you what you want. Did you consider also open MPI? See also msync(2) and mlock(2)

Basile Starynkevitch
  • 223,805
  • 18
  • 296
  • 547
  • Thanks! I need to allocate range of virtual memory in user-space with (commited) allocated big contiguous segment of physical memory. But how can I do this with `mmap()`, can you write an example please? – Alex Nov 19 '13 at 11:27
  • I don't understand how exactly you access to that particular segment of physical memory from user applications... Please explain more whay you want to do.... And I don't understand if you are coding inside a kernel driver or inside userland applications. – Basile Starynkevitch Nov 19 '13 at 11:29
  • I only want what I have allocated to 100MB of virtual memory as usual, but that on the physical level, it was not segmented. That is, was pinned(no pageable) and contiguous(on physical level). – Alex Nov 19 '13 at 11:33
  • If this can be done at the user level, it is best. If this can only be done at the kernel level, then how? – Alex Nov 19 '13 at 11:35
  • @Alex: This can't be done in uesr space. You'll need to write a device driver. – Nathan Fellman Nov 19 '13 at 11:42
  • @Basile Starynkevitch NUMA requires QPI interface, or HT, but does not work on PCIe. OpenMPI does not yet support the connection of several CPU via PCIe. Yes, `mlock()` allows you to lock pages of the memory (pinned-memory), but is this function ensures that on a physical level, this block of memory will be continuous? – Alex Nov 19 '13 at 11:43
0

From user space -- there is no guarantee depends on you luck.

if you compile your driver into the kernel -- you can use the mmap and allocate the required amount of memory.

if it is required to use it as storage or some other work not specifically for a driver then you should set the memmap parameter in the boot command line.

e.g. memmap=200M$1700M it will block 200 MB memory starting from the end of 1700M (address).

Later it can be used to as FS as well ;)

Kanishka
  • 19
  • 6
  • Additional hint if you are willing to reserve it. you will need to do "modprobe ioatdma" – Kanishka Nov 19 '13 at 13:10
  • Thanks! But how can I allocate pages in virtual addressing for this physical range and get a pointer to it in user-space? – Alex Nov 19 '13 at 13:16
  • As said here, it is not good approach to use `memmap`: http://unix.stackexchange.com/questions/37729/how-can-i-reserve-a-block-of-memory-from-the-linux-kernel And they are advised to read on page 442 of the PDF: http://lwn.net/images/pdf/LDD3/ch15.pdf or http://www.xml.com/ldd/chapter/book/ch13.html – Alex Nov 19 '13 at 13:52
  • One of the reason why it is not a good practice.If once it has been blocked by you you will have to specifically apply each kernel functionality on this reserved area. still if you are interested in doing so arch/x86/kernel/setup.c arch/x86/mm/pageattr.c are the code files you might wanna look into. in short with this technique you will need a dedicated device driver.(similar to that of RAM) – Kanishka Nov 20 '13 at 08:05