1

Novice c programmer here. This may be nonsense.

I've learned about some basics of dynamic memory allocation using malloc and free. When I allocate memory, for concreteness say for an array of 10 ints via malloc, malloc returns to me a pointer p to the beginning of the memory block allocated. Typically I'd like to pass that pointer, as well as some relevant metadata about the memory allocation (namely, that it was for a length 10 array of ints) to the rest of my code to support subsequent informed and safe read/write access. So that is what I do, ferrying this metadata around my code.

It's interesting to me that when the time comes to actually free the memory, free is perfectly capable of doing the job given only the pointer p; it does not need to be told e.g. the length of the array. I don't have knowledge of any of the details here, but from what I've read, the implementations of malloc/free involve storage of and access to metadata that is written adjacent to each allocation, and that metadata is accessed by free at de-allocation time (and is evidently sufficient to for it know precisely which memory locations should be freed). I imagine this must mean in particular that in my example, the length of the allocated array of ints can be gleaned from the metadata that free is looking up.

This leads me to a few questions:

  1. Are the high level details of my understanding of the existence of this metadata correct?

  2. Can this allocation metadata be accessed by a c programmer using the standard library?

  3. If not, can it be readily accessed from c by means other than the standard library?

  4. If the answer to 2 or 3 is negative for reasons related to standards and rules, are there specific contexts or examples (e.g. that only work on a specific OS, or with a specific non-standard implementation of c, etc.) where this is possible and/or useful?

  5. If this is a bad idea, regardless of whether it is possible, I'd be curious to hear about why. My only guess so far is that the details of how the metadata are stored might vary by system, so that code using it naively might end up being intolerant to system changes. Still, given that free can successfully make sense of it on different systems, I'd think there'd be a way to encapsulate the system dependencies to still get useful introspection into the metadata.

For 2 or 3, I'm imagining something like:

// Allocate some memory
int size = 10;
int *arr;
arr = (int *)malloc(sizeof(int) * size)

// Do other stuff.
// Choose not to keep track of storage details necessary for informed read/write access to arr, e.g. size;

// Acquire access to whatever `free` would access if asked to free `arr`
_type_ metadata = get_pointer_metadata(arr);

// Use metadata to perform read/writes

But don't know whether there is anything like get_pointer_metadata in the c standard library, or third party packages?

I've tried searching google for such a function, but my understanding of the details of the situation and language is too poor to make meaningful searches.

It'sRecreational
  • 13
  • 4
  • 1
    Different `malloc`s will do it differently and there may even be different metadata formats different allocations from the same allocator (think 16-byte pair of pointers vs. many-KB buffer), so there's no portable way to do this. – twotwotwo Feb 22 '23 at 16:49
  • 1
    There is no reliable way to access the metadata maintained by `malloc()` et al. Consequently, it is not advisable to try to access it. However, on many systems there is a header `` which provides some rudimentary access to metadata. This header is widely abused — it should not be used to gain access to `malloc()` (use `` for that) but rather to the `mallinfo()` function. Even that is limited because it doesn't report on all possible memory allocations – see `man mallinfo` on Linux. – Jonathan Leffler Feb 22 '23 at 16:54
  • Makes sense. Thank you all. My only follow up, @JonathanLeffler. How does one square `There is no reliable way to access the metadata maintained by malloc()` with the fact that the implementation(s) of free seem to do it reliably? Are you just saying that there is no reliable way to do this across different malloc implementations? Or is free also not reliable in getting the relevant metadata? – It'sRecreational Feb 22 '23 at 17:07
  • There's no documented external interface (other than the `mallinfo()` one I mentioned) to do it. How the library does things behind the scenes is its problem; it doesn't have to expose its dirty internal workings to the general public, and it doesn't do that. Replacing `malloc()` ain't easy — you have to replace `calloc()`, `realloc()`, `free()` too, and that's a bare minimum. Those functions have to understand each other — they don't have to let you know how they understand each other. – Jonathan Leffler Feb 22 '23 at 17:16
  • I do not believe your question is nonsense, I have wondered about similar things, in particular about the way the OS decides where, and how much space to allocate per call, and why does free appear to not give the memory back immediately. (It actually does, but appears not to when viewing memory usage metrics.) FWIW, there is some discussion on the [inner working of how memory allocation works here](https://stackoverflow.com/questions/64029219/why-does-malloc-call-mmap-and-brk-interchangeably/64029494#64029494) – ryyker Feb 22 '23 at 17:19
  • @JonathanLeffler Got it, makes good sense, thanks so much for your patience and help. I certainly understand there is no obligation for such things to be exposed to the public, and low motivation to do so, particularly if they are dirty, but I was hoping perhaps this aspect of the internals was not so dirty, and might be exposed. I'm purely curious and trying to learn, not actually intending to attempt working this way to access metadata given your feedback, if that's not clear. – It'sRecreational Feb 22 '23 at 17:23
  • @It'sRecreational Tip; Avoid allocation mistakes and simplify: `arr = (int *)malloc(sizeof(int) * size)` --> `arr = malloc(sizeof arr[0] * size)`. – chux - Reinstate Monica Feb 22 '23 at 17:39
  • Makes sense, thanks for the tip (and answer) @chux-ReinstateMonica – It'sRecreational Feb 23 '23 at 00:39

1 Answers1

3

Is it possible/advisable to access the metadata stored by malloc via the standard library?

No. The standard C library does not offer such access.

  1. Are the high level details of my understanding of the existence of this metadata correct?

Meta data does exist - but where and what exist is more varied. Meta data could be encoded in the pointer itself. Original allocation size might not exist.

  1. Can this allocation metadata be accessed by a c programmer using the standard library?

No.

  1. If not, can it be readily accessed from c by means other than the standard library?

Perhaps. Some compilers/libraries offer such access. It is implementation dependent.

  1. If the answer to 2 or 3 is negative for reasons related to standards and rules, are there specific contexts or examples (e.g. that only work on a specific OS, or with a specific non-standard implementation of c, etc.) where this is possible and/or useful?

Yes, there are specific examples. Since I code for portability, I do not use them and instead have code keep track of relevant meta data itself.

  1. If this is a bad idea, regardless of whether it is possible, I'd be curious to hear about why. ...

Not so much a bad idea, but a weak one.

Meta data is often implementation specific and so if the library obliged access to such, that would constrain an implementation to follow a restrictive memory model.

Instead of a implementing specific solutions consider forming a structure with your meta-data and pass that around.

typedef struct {
  void *ptr;
  size_t size;
} ptr_sz;
chux - Reinstate Monica
  • 143,097
  • 13
  • 135
  • 256