6

The question is in the title... I searched but couldn't find anything.


Edit:

I don't really see any need to explain this, but because people think that what I'm saying makes no sense (and that I'm asking the wrong questions), here's the problem:

Since people seem to be very interested in the "root" cause of all the problem rather than the actual question asked (since that apparently helps things get solved better, let's see if it does), here's the problem:

I'm trying to make a D runtime library based on NTDLL.dll, so that I can use that library for subsystems other than the Win32 subsystem. So that forces me to only link with NTDLL.dll.

Yes, I'm aware that the functions are "undocumented" and could change at any time (even though I'd bet a hundred dollars that wcstombs will still do the same exact thing 20 years from now, if it still exists). Yes, I know people (especially Microsoft) don't like developers linking to that library, and that I'll probably get criticized for the right here. And yes, those two points above mean that programs like chkdsk and defragmenters that run before the Win32 subsystem aren't even supposed to be created in the first place, because it's literally impossible to link with anything like kernel32.dll or msvcrt.dll and still have NT-native executables, so we developers should just pretend that those stages are meant to be forever out of our reaches.

But no, I doubt that anyone here would like me to paste a few thousand lines of code and help me look through them and try to figure out why memory allocations that aren't failing are being rejected by the source code I'm modifying. So that's why I asked about a different problem than the "root" cause, even though that's supposedly known to be the best practice by the community.

If things still don't make sense, feel free to post comments below! :)


Edit 2:

After around ~8 hours of debugging, I finally found the problem:

It turns out that RtlReAllocateHeap() does not automatically work like RtlAllocateHeap() if the pointer given to it is NULL.

Baum mit Augen
  • 49,044
  • 25
  • 144
  • 182
user541686
  • 205,094
  • 128
  • 528
  • 886
  • 1
    Just to complete other answers.. malloc() keeps a record of the blocks it has allocated and when free()ing something it compares the pointer against that record to check if it has been allocated. – BlackBear Jan 03 '11 at 23:13
  • @BlackBear: Yes, which is why it made me wonder why it can't just check to see if a pointer is inside the block (how hard can that be??)... – user541686 Jan 03 '11 at 23:19
  • 1
    @BlackBear: Actually, in most implementations the heap only holds information on blocks that are free. Data required by free() (such as the size of the block) is prepended by malloc() to the allocated block, the size of the record is subtracted from the pointer passed to free() to access the record. If the implementation includes a validation signature, error detection is possible, if not an invalid record will cause heap corruption. – Clifford Jan 03 '11 at 23:30
  • Ouch: RtlReAllocateHeap() doesn't handle being given a NULL pointer? It's too bad MS didn't make this behave like the standard `realloc()` function, especially since it only takes a couple lines of code. That's a hard lesson to learn. – Michael Burr Jan 04 '11 at 04:47
  • Yeah, I completely assumed it accepted that and worked as intended, then spent 8 (or more? no idea) hours debugging it, and I only finally found out when I compared the printf()'s from the hooked realloc()s side-by-side, and noticed that one of my prints was being inhibited. :\ – user541686 Jan 04 '11 at 05:11

4 Answers4

14

It has to point to the beginning of the block. It's safe to pass a null pointer to free(), but passing any pointer not allocated by malloc() or one of its relatives will cause undefined behaviour. Some systems will give you a runtime error - something along the lines of "Deallocation of a pointer not malloced".

Edit:

I wrote a test program to get the error message. On my machine, this program:

#include <stdlib.h>

int main(int argc, char **argv)
{
  int *x = malloc(12);
  x++;

  free(x);

  return 0;
}

Crashes with this message:

app(31550) malloc: *** error for object 0x100100084: pointer being freed was not allocated
Carl Norum
  • 219,201
  • 40
  • 422
  • 469
  • Huh... so in that case, is there a way to (portably) request memory that's, say, 64-byte aligned? I was going to allocate more and extract a piece, but since I'm passing around the pointer, I can't keep track of the size of the unused bytes in the beginning... – user541686 Jan 03 '11 at 22:37
  • @Lambert, sure you can. Writing 'aligned malloc' is actually a pretty popular job interview question. Just allocate enough space to account for your alignment, and also an additional space for a pointer. Then stash the pointer right before the aligned memory, and write a `free()` function that pulls this information out and frees the 'real' pointer. – Carl Norum Jan 03 '11 at 22:38
  • Well it's not that I *can't*, it's that I'd have to make a whole new table and keep track of stuff inside that... and while I know how to do it, it's just pain I'd like to avoid if I can. :) – user541686 Jan 03 '11 at 22:39
  • 1
    It's not that hard. But you can also use `memalign()` and friends. Check out this answer: http://stackoverflow.com/questions/3839922/aligned-malloc-in-gcc – Carl Norum Jan 03 '11 at 22:40
  • @Santiago: But what happens is, the library is giving me a pointer to *anywhere* inside the region so I can free it... so how does that help? – user541686 Jan 03 '11 at 22:40
  • You can allocate a bit more in any case (even if it's not required) and write some data before the passing pointer, so you can access it like `*(x - 1)`. (Not very robust, yeah). – crazylammer Jan 03 '11 at 22:41
  • @Carl: Thanks for the link, but I'm under Windows and using NTDLL.dll, and that function doesn't exist. But +1 for your answer! :) – user541686 Jan 03 '11 at 22:42
  • @Lambert, then you might have to write your own. It's quite possible that windows has similar routines, though. – Carl Norum Jan 03 '11 at 22:43
  • @Carl: All right, not the answer I was hoping for, but nevertheless correct. Thanks! :) – user541686 Jan 03 '11 at 22:44
  • @Carl: The "should give you a runtime error" part is misleading. Passing an "interior" pointer to `free` results in undefined behavior, most likely severe memory corruption. – R.. GitHub STOP HELPING ICE Jan 03 '11 at 23:06
  • @R. I cleaned up the wording a bit. – Carl Norum Jan 03 '11 at 23:12
  • 1
    *"should give you a runtime error"* is not *required* behaviour; the results are undefined and implementation dependent. In some cases it will just corrupt the heap by adding an invalid block, you may not notice until a subsequent memory allocation occurs, when sh*t happens! – Clifford Jan 03 '11 at 23:13
  • @lambert Windows has _aligned_malloc() , http://msdn.microsoft.com/en-us/library/8z34s9c6%28VS.80%29.aspx – nos Jan 03 '11 at 23:15
  • @nos: "Windows" is 4 GB of data, including MSVCRT.dll. NTDLL.dll is a library. And NTDLL.dll (which is the only library I'm linking to) doesn't have `malloc` *or* `_aligned_malloc`, so this doesn't help. Thank you for the response, though. – user541686 Jan 03 '11 at 23:18
4

The only thing you can pass to free is a pointer returned to you from malloc (or calloc, realloc, etc), or NULL.

I'm guessing you're asking because you have some function where you do a malloc, and you want to mess with the data block and then get rid of it when you're done, and you don't want to be bothered keeping a copy of the original pointer around. It doesn't work that way.

Andy Lester
  • 91,102
  • 13
  • 100
  • 152
  • Haha it's not that I don't want to bother, it's that it's not really easy to do that in my situation. But thank you for the answer. :) – user541686 Jan 03 '11 at 22:38
  • How is it not easy to do? Are you calling some API function that doesn't hand you back that `malloc`ed pointer? If it's specifically hiding it, then you should leave it up to the API function to do the memory management. Be careful, because double-free bugs are terribly annoying because you have to figure out who did the freeing before you, and why. – Andy Lester Jan 03 '11 at 22:42
  • Hahaha no, I'm trying to implement an aligned malloc on top of NTDLL's `RtlAllocateHeap`, so I can pass it to another library. I *could* make a new table to keep track of everything, but then I'd have to worry about threading, etc. and things would be a pain. – user541686 Jan 03 '11 at 22:43
  • 1
    @Lambert - IOW, you don't want to bother. Don't be ashamed of it. A desire to avoiding unnessecary work is a good quality for a software developer. Particularly if you are willing to do work up-front to achieve that goal. Just don't take it to extremes. – T.E.D. Jan 03 '11 at 22:50
  • 2
    @Lambert, don't make a table. Just add a header to the memory block in your aligned malloc implementation. – Carl Norum Jan 03 '11 at 22:50
  • @Carl: I *just* thought of the same thing and came here to write it, and voila, you'd written it too! :) it's just a little tricky, since after thinking about it it seems that I would need to *always* subtract `2 * alignment - 1` from the pointer the user gives me, align it to the next boundary, and store the size of the nonuser buffer there, since if I just subtract `alignment - 1`, there might not be any space before the buffer. Thanks for the suggestion, +1. – user541686 Jan 03 '11 at 22:59
1

From the comments you have added to existing answers, it seems you are asking the wrong question. If memory alignment is what you need, why don't you ask that question? Ask about the root problem rather than asking about your perceive solution!

So if you don't mind I'll answer the question you should have asked:

Aligned memory allocation is supported in Win32 by _aligned_malloc(). It is more of less equivalent to POSIX memalign()

If you need an implementation of aligned allocation, it is fairly simple to implement:

void* aligned_malloc( size_t size, size_t alignment )
{
    void* unaligned = malloc( size + alignment ) ;
    void* aligned = (void*)(((intptr_t)unaligned + alignment) & ~(alignment - 1));
    void** free_rec_ptr = ((void**)aligned - 1) ;
    *free_rec_ptr = unaligned ;

    return aligned ;
}

void aligned_free( void* block )
{
    void** free_rec_ptr = ((void**)block - 1) ;
    free( *free_rec_ptr ) ;
}

The alignment argument must be a power of two.

Clifford
  • 88,407
  • 13
  • 85
  • 165
  • @Clifford: Thank you for your answer, but unfortunately, I already knew about `_aligned_malloc` -- it's just not present in NTDLL.dll (neither is `malloc`), the sole library I'm linking to, and I was just wondering if I even needed to bother through this entire trouble of keeping track of the alignment of pointers I pass to other libraries, if I just didn't have to. So no, I wasn't avoiding asking about the root cause, I really *did* mean to ask this question. – user541686 Jan 03 '11 at 23:13
  • @Lambert: It is part of the MSVCRT library which is normally linked implicitly. Since malloc()/free() are also in that library, you *must* have access to it. What compiler are you using? – Clifford Jan 03 '11 at 23:21
  • @Clifford: I'm specifically trying *not* to link to MSVCRT for various reasons -- so by definition the function isn't available. It's not a compiler issue, it's something I've done on purpose. – user541686 Jan 03 '11 at 23:23
  • @Lambert: You are not making sense: if you can call free(), you *have* linked MSVCRT! The reason I ask about the compiler is because MSDN only documents _aligned_malloc() for VC++ 2003 and later. I am not sure that it is available in earlier versions. You'd do well to answer investigative questions, rather than assume that the question is irrelevant. – Clifford Jan 03 '11 at 23:35
  • @Clifford: I'm making complete sense -- it's *your* incorrect assumption that I'm actually calling `free()`. I'm *supplying* `free()` for the caller, and I'm wrapping `RtlFreeHeap()`. Does that make sense now? Also, I **know** I'm not linking to MSVCRT because I've disassembled my executable. So when I say I'm not linking to it, I'm not joking, I really mean it. :) – user541686 Jan 03 '11 at 23:37
  • @Lambert: Given the question, that was hardly an unfair assumption! If you are providing your own free() implementation, then the entire premise of your question is flawed! If it is your implementation of free() then only you know the answer! The documentation for RtlFreeHeap is even clearer than for free() - the pointer is that *returned* by RtlAllocateHeap, not some other pointer. – Clifford Jan 03 '11 at 23:45
  • @Clifford: I'm creating free() as a **wrapper** for RtlFreeHeap(), so I'm not really coding free() from scratch. I don't see where there's a flaw with this. See the edit in my question, and if you still think this isn't making sense, please let me know; thanks. – user541686 Jan 03 '11 at 23:50
  • @Lambert: Yes, but the behaviour is defined by RtlHeapFree() which may not be the same as that for free(). As it happens it is, but say the standard *did* define free() to work as you suggested; that is not to say that RtlHeapFree() would work the same way. – Clifford Jan 03 '11 at 23:53
  • @Clifford: This post was related to [my previous post](http://stackoverflow.com/questions/4588660/is-there-a-fundamental-difference-between-malloc-and-heapalloc-aside-from-the-po)... so the question is, (edited) *do the Heap API follow all the requirements for the C memory allocation API*? (Answer: apparently not, but I can't find the difference... I was trying to program a wrapper and find the difference, hence this post.) – user541686 Jan 03 '11 at 23:55
  • @Clifford: If you're interested, here's a different way of explaining the problem: I'm modifying a garbage collector, which uses malloc(), to use RtlAllocateHeap(). RtlAllocateHeap() works fine, but after a few allocations, the GC tells me "unable to allocate memory", even though nothing is out of memory and even though I've completely wrapped all calls correctly. (My code doesn't link to *any* C memory function.) I thought maybe the GC required stricter alignment, so I tried to make malloc() return memory that was more aligned... hence this post. Does that make sense? – user541686 Jan 04 '11 at 00:02
  • @Lambert: Somehow I doubt that alignment is the issue, but if you need it, perhaps the example implementation in my edit will help. Use your existing malloc/free wrappers or replace my malloc/free calls with Rtl functions as necessary. I have tried to keep the answer generic, matching your original question, rather than the subsequent detail. – Clifford Jan 04 '11 at 02:20
  • @Clifford: Thank you very much for your edit. I was actually experimenting with something similar myself, but it seems like, as you say, alignment isn't the issue (although it's very baffling right now). Just something I'd like to point out, though: your implementation doesn't work in some cases: if malloc() gives you the address 7, you're writing a pointer to the address 4, which doesn't belong to you. I think I've taken care of that in my code, though... something else seems to be the problem, I have no idea what. :\ – user541686 Jan 04 '11 at 02:25
  • @Clifford: Nope, alignment isn't the issue... I made my code return a 256-byte aligned buffer, but it still doesn't work. This is getting interesting... I'll have to hook the malloc() from the CRT to see how it behaves differently! – user541686 Jan 04 '11 at 02:29
  • @Clifford: Turns out that the `realloc()` of my code had to first do a null-check and call `malloc()` instead of calling `RtlReAllocateHeap()`, since they don't behave the same way if the pointer is null. Thanks for all your help anyhow! :) – user541686 Jan 04 '11 at 03:19
  • @Lambert: That's a shame, I was particularly pleased with my alligned_malloc() implementation! ;) BTW I believe it does work; if malloc() gave you address 7 and the alignment were 64, the pointer is written to 60 not 4: `(7 + 64) & ~(64 - 1) == 0x47 & ~0x3f == 0x47 & 0xFFFFFFC0 == 0x40 == 60' – Clifford Jan 04 '11 at 11:29
  • @Clifford: Haha yeah, it's a shame. :\ Hm... So let's say the implementation gave you the address 63 -- then you'd round to 64, subtract 4, then write the value at address 60, right? – user541686 Jan 04 '11 at 19:14
  • @Lambert: You are right it is not quite correct, in more than one way. The idea is that an address of 63 would be aligned to 128 and teh address written to 124. The expression does not work in this case, and even if it did the over allocation of 64 is too little, it must be twice the alignment for worst case. Probably worth fixing just in case someone comes along and tries to use it! However I will have to come back to it. – Clifford Jan 04 '11 at 22:12
  • Yeah, how I did it was: I allocate `4 * align` more than I need to, when calling RtlAllocateHeap (to be safe); then I take the pointer `ptr` I get, add `align` to it, round *up* to the next boundary -- this is the user buffer, `p`. Then go *down* by one `align`, and over there, write (using just 1 byte, assuming `align < 256`) the value `p - ptr`. Then, when freeing the pointer, do the reverse to find the real beginning, and pass it to RtlFreeHeap. (I didn't handle `align >= 256`, but I could use a `ushort` in that case, just making sure that align is never `< 2`.) – user541686 Jan 04 '11 at 22:26
0

FWIW, the last time I looked into how the C run time library worked with malloc and free, the block that malloc gives you actually has some extra words at negative offsets relative to the pointer it gives you. These say things like how big the block is, and other stuff. free relied on being able to find those things. Don't know if that sheds any light.

Mike Dunlavey
  • 40,059
  • 14
  • 91
  • 135
  • It's exactly what I ended up doing, except that it gets more complicated with differing alignments. Thanks for the answer, though. – user541686 Jan 05 '11 at 19:15