93

Just to be clear: I do know that malloc and free are implemented in the C library, which usually allocates chunks of memory from the OS and does its own management to parcel out smaller lots of memory to the application and keeps track of the number of bytes allocated. This question is not How does free know how much to free.

Rather, I want to know why free was made this way in the first place. Being a low-level language, I think it would be perfectly reasonable to ask a C programmer to keep track not only of what memory was allocated but how much (in fact, I commonly find that I end up keeping track of the number of bytes malloced anyway). It also occurs to me that explicitly giving the number of bytes to free might allow for some performance optimisations, e.g. an allocator that has separate pools for different allocation sizes would be able to determine which pool to free from just by looking at the input arguments, and there would be less space overhead overall.

So, in short, why were malloc and free created such that they're required to internally keep track of the number of bytes allocated? Is it just a historical accident?

A small edit: A few people have provided points like "what if you free a different amount than what you allocated". My imagined API could simply require one to free exactly the number of bytes allocated; freeing more or less could simply be UB or implementation defined. I don't want to discourage discussion about other possibilities, though.

Community
  • 1
  • 1
  • You know, I was thinking about this a few days ago. I came to the conclusion that there really isn't a good reason for it. I hope I'm proven wrong. – user541686 Jun 13 '14 at 11:12
  • 1
    Such a good question!, this is been bugging me for quiet some time as well – Montaldo Jun 13 '14 at 11:12
  • 19
    Because it already is a pain to keep track of the allocations themselves, and it would even complicate the code more if you'd additionally have to keep track of the size. – Jens Gustedt Jun 13 '14 at 11:14
  • 11
    I can think of several reasons: Why make the user do it if they don't have to? What if the user messes it up? It's kind of a redundant question anyway. If they'd made the other choice, you'd still be asking why. – BoBTFish Jun 13 '14 at 11:14
  • 16
    @BoBTFish: This is *C* we're talking about, not Python or even C++. The user already has to do a $h!1 ton he doesn't have to. That's not a reason. – user541686 Jun 13 '14 at 11:15
  • @JensGustedt: Memory management is already complicated enough that this would be just icing on the cake. Consider that if this were allowed, the user could also be potentially allowed to merge contiguous blocks or free sub-blocks as needed, but can't do that in the current system. – user541686 Jun 13 '14 at 11:17
  • Because C already keep track of the allocated chunks sizes to know where to allocated following chunks. The space overhead is constant and the lookup time is probably constant too... – jean-loup Jun 13 '14 at 11:18
  • 1
    @Mehrdad Yes, but there's always some freedom there (well, that's the idea). Here there is only ever one correct value, and passing anything else would be wildly wrong. There doesn't seem to be any mention of this issue in the [C99 rationale](http://www.open-std.org/jtc1/sc22/wg14/www/C99RationaleV5.10.pdf#168). – BoBTFish Jun 13 '14 at 11:19
  • 1
    @jean-loup: It's not the same kind of information, and it's not stored in the same way. To allocate more memory, you only need to know where the *empty* space is. You do *not* need to know how the *already allocated* space is split up between multiple allocations by the caller -- either way, they're "in use" and it doesn't make any difference why or how. – user541686 Jun 13 '14 at 11:19
  • @Mehrdad The empty space is after the allocated space or at least not at the same place ... – jean-loup Jun 13 '14 at 11:20
  • 1
    @jean-loup: I know, but that only requires storing intervals representing where the empty space is. It does not require keeping track of individual allocations. Think of an interval tree that keeps track of free space only. – user541686 Jun 13 '14 at 11:20
  • @Mehdrad That's true, so C would have to 'merge' the chunks and this is linear to the number of already allocated chunks (test each chunks for overlap) – jean-loup Jun 13 '14 at 11:23
  • 4
    K&R has nothing to say about this either. We can speculate all we like, but I think the original reason may be [lost in history](http://www.bbc.co.uk/news/technology-15287391). – BoBTFish Jun 13 '14 at 11:25
  • 1
    @jean-loup: Merging is linear-time if done naively. I believe sure you can keep it to logarithmic if you're careful. (It may not be cache-friendly but it will provide a guaranteed time bound.) – user541686 Jun 13 '14 at 11:26
  • @Mehrdad It would also have to test for freeing non-allocated chunks, which looks dangerous – jean-loup Jun 13 '14 at 11:31
  • 1
    @jean-loup: I'd stop using the word "dangerous" as a reason for not doing something in K&R C. I don't think you realize how silly it is. – user541686 Jun 13 '14 at 11:35
  • It's difficult to find a way to temper any reply to this insane plan. FFS why would anyone want to keep track of this shit when the memory-manager is already doing it? – Martin James Jun 13 '14 at 19:20
  • 38
    You can't require the programmer to correctly pass in the size of the block, because **the caller of `malloc` does not know the size of the block returned**. `malloc` often returns a block that is larger than requested. At best, the programmer could pass in the size requested in the `malloc()` call, which would not help the implementer of `free()` at all. – Ben Voigt Jun 13 '14 at 19:20
  • @MartinJames: Like I said numerous times, it's because the memory manager has no reason to actually store the size of each allocated block, it only needs to store the size of free space, which does not imply the former. – user541686 Jun 13 '14 at 19:21
  • @BenVoigt: I don't follow. If `malloc` always adds some amount of memory then `free` will be aware of that and can adjust the user-provided size appropriately. That overhead is a fixed amount and doesn't need to be stored per-block. – user541686 Jun 13 '14 at 19:22
  • 2
    @Mehrdad: The added amount is not necessarily constant nor predictable. Basically, if there's an available region only slightly larger than the request, `malloc` will return the whole thing instead of splitting it, because the remainder after split would be too small to use. See http://stackoverflow.com/a/21185118/103167 and http://stackoverflow.com/q/5813078/103167 – Ben Voigt Jun 13 '14 at 19:25
  • @BenVoigt: There's no fundamental reason for `malloc` to behave that way though. All you're doing is stating the current behavior, which is obviously not going to motivate an alternative. It could very well just return the requested amount and then fail later when the remaining size is too small. – user541686 Jun 13 '14 at 19:29
  • 2
    @Mehrdad: The next allocation will just move along to the next available area, not fail. But each available area requires some metadata storage, so tracking really small areas (especially if you already have some of the same size) is counter-productive. More areas affects not only the size of metadata but the time needed to iterate through it. – Ben Voigt Jun 13 '14 at 19:32
  • @BenVoigt: How is tracking each free area any more counterproductive than tracking each used area? It seems like a worse idea to me, the latter requires more information and more time to iterate through it. – user541686 Jun 13 '14 at 19:53
  • 1
    @Mehrdad: What's counterproductive is allowing the areas to be arbitrarily small. – Ben Voigt Jun 13 '14 at 19:55
  • 1
    @BenVoigt: I don't see why you have to; you can just round each allocation up to the next allocation boundary (i.e. which is probably the alignment, like 16 bytes typically). – user541686 Jun 13 '14 at 19:57
  • 1
    @Mehrdad The malloc/free interface had to allow flexibility for a range of different implementations. Requiring the actual allocated memory to be a fixed function of the request size would have constrained the implementation. – Patricia Shanahan Jun 13 '14 at 20:28
  • 1
    @PatriciaShanahan: I don't see why requiring the caller to pass *more* information to `free` would *constrain* the implementation. We're talking about the interface here, the implementation can decide for itself whether or not it wants to take advantage of the extra information somehow. All I'm saying is that the extra information *could* allow implementations that allow sub-blocks to be freed; I'm not saying every implementation must necessarily do so. – user541686 Jun 13 '14 at 21:09
  • 2
    @Mehrdad: If you require the size to be sent to `free` you require the caller the track it. If the memory manager in many, most, or even a significant proportion of cases ignores the information then you have a setup that is _strictly_ inferior to not having that information included. – Jack Aidley Jun 13 '14 at 21:23
  • 1
    @JackAidley: What kind of a caller **doesn't** already know what size to pass to `free`?! Either the caller is allocating a block of a variable size (like an array), in which case it **already** keeps track of the size during the entire time the block is in use (buffer overflows anyone?!), or it's a constant size (like a node), in which case the caller can just use a `sizeof` and calculate it and pass it to `free`. I really hope you have a good example to justify why such a thing would happen in a "significant proportion" of situations as you claim because I don't think you thought it through. – user541686 Jun 14 '14 at 01:04
  • "...might allow for some performance optimisations..." It's highly unlikely that compiled code will ever get optimized as well as existing library code. Best that could be hoped for is _approaching_ the current performance while needing to add additional (also less efficient) code to track/manage the allocated sizes. – user2338816 Jun 14 '14 at 03:13
  • "determine which pool to free from " - huh?? You have to locate the start of the block anyway, based on the address being freed – M.M Jun 14 '14 at 05:18
  • @BenVoigt "which would not help the implementer of free() at all." - well, the allocator could repeat the calculation it used to determine the size to allocate in the first place. – M.M Jun 14 '14 at 05:24
  • @Mehrdad: Have you _ever_ actually either written or even _looked_ at a real memory manager? It is _very_ common for memory managers to actually return a block of memory that is a different size from the requested block for a whole host of efficiency and alignment reasons. This means that the _asked for_ size is completely useless for the free call, to be any use _at all_ the `malloc` must return the actually allocated size and the `free` call pass back this actually allocated size. Without this it is just a completely useless piece of information that will inevitably slow things down. – Jack Aidley Jun 14 '14 at 07:10
  • @JackAidley: Yes, but I don't think you're understanding what I'm telling you. **My last comment was *only* trying to tell you why the requested size is already something callers must keep track of in most cases**. It **never** stated that *today's* memory managers would find that useful, **nor** did it ever say that a memory manager that *did* find the size useful would benefit *speedwise*. I *only* stated that such a memory manager could have some benefits over others, that speed is *not* necessarily one such factor, and that the size is in most cases *already available* and trivial to pass. – user541686 Jun 14 '14 at 07:57
  • 2
    What's actually hilarious is that no one here seems to realize that in C++, [`std::allocator::deallocate`](http://en.cppreference.com/w/cpp/memory/allocator/deallocate) ***does* in fact require the size to be passed as the second argument**. If this isn't enough to shut down your mindless "it's a useless idea" arguments, I don't think anything else will be. @jaymmer: I suggest you take this into account before deciding the correct answer is the "it wouldn't make sense" argument that you previously selected. – user541686 Jun 14 '14 at 09:10
  • @Mehrdad: No, it **doesn't**. Do you not even _read_ what the things you linked to? The second argument is the _number of elements allocated_ **not** the size of memory allocated. – Jack Aidley Jun 14 '14 at 10:50
  • 1
    @JackAidley: WTF?! 1st of all, since you love being extra nitpicky: it's the number of elements **requested** to be allocated, **not** the number of elements allocated. 2nd, it's only a `sizeof(T)` multiple away from the number of bytes, and "size" can mean **either bytes or number of elements** depending on the context (`vector::size`??). And 3rd, you seem to be arguing just for the sake of arguing *without understanding the **points*** I'm trying to make (namely, that **there is *actually* a size parameter present**), so I'm done arguing with you. Hope you enjoyed your downvote on my answer. – user541686 Jun 14 '14 at 11:08
  • @Mehrdad: Your grasp of this is clearly too poor for it to be worth wasting more time on this. Everyone else, have a little think about how a general purpose memory allocator might differ from one that only allocates a specific type of known size and thus why Mehrdad's argument regarding `std::allocator` is absurd. – Jack Aidley Jun 14 '14 at 11:17
  • This is not "primarily opinion based" there are a whole load of important, practical design resigns why not passing the size is a much better way to do. Just read the answers given and this becomes clear, especially that from Nathaniel J. Smith. – Jack Aidley Jun 14 '14 at 19:42
  • @BenVoigt: "*object* management"?? since when does `deallocate` call *destructors*?! That's the job of `destroy`, not `deallocate`! The standard literally says the objects must be destroyed **prior** to the call, not *via* the call! And what exactly prevents it from storing the number somewhere else that you think made it such a different case from `malloc`? They could have done exactly the same thing here, but they didn't. – user541686 Jun 14 '14 at 19:50
  • @BenVoigt: *"Ok, so it doesn't call destructors. The number of elements is more useful in a pooled scenario, which is what `std::allocator` is designed to facilitate."*... if you weren't making stuff random reasons earlier, you're certainly doing that now. That's nonsense for so many reasons I don't even have the motivation to list them, you'll just make up more reasons. Instead of doing everything you can to support the silly position that passing the size is a good idea for `deallocate` but somehow a bad idea for `free`, just acknowledge that the same rationale applies to both and move on. – user541686 Jun 14 '14 at 23:43
  • 1
    @Mehrdad: `free` isn't an extensible polymorphic system. `deallocate` is (`std::allocator` is consumed using template-based duck-typed polymorphism). For some implementers of STL-compatible allocators, that data might be useful. For the system allocator, it never is, because the system allocator has to handle requested-actual block size mismatch. But in order for the duck-typed call to provide that data to replacement allocators that need it, the argument has to be accepted by the system allocator. – Ben Voigt Jun 14 '14 at 23:53
  • 1
    @Mehrdad: If there were APIs which accepted pointers to malloc/free, then it would make sense for malloc/free to have a signature that was friendly to usage-specific malloc/free replacements. But they aren't used that way... malloc/free replacements aren't locally used the way `std::allocator` replacements are, a malloc/free replacement has to take over all allocation duties. – Ben Voigt Jun 14 '14 at 23:56
  • @BenVoigt: *"If there were APIs which accepted pointers to malloc/free, then it would make sense for malloc/free to have a signature that was friendly to usage-specific malloc/free replacements."*... this is so much nonsense; do you not realize this is a chicken/egg problem? They're not there **precisely because** the C API (from how many decades ago?) wasn't designed to need or use them. You can't use the lack of something in the present as though it's some sort of justification against the presence of some alternative scenario... – user541686 Jun 15 '14 at 00:06
  • @Mehrdad: A prerequisite for them being useful is that allocated blocks have consistent size. That's true for `std::allocator` replacements, which are used locally for one or a few containers managing instances of a single type, using template arguments on the containers. It's not true for malloc/free or malloc/free replacements, because malloc/free calls are global symbols. malloc/free signatures don't need to support the use case "this allocator is optimized for managing a pool of a single type". – Ben Voigt Jun 15 '14 at 00:19
  • @BenVoigt: Uh...huh? Now try applying that reasoning to justify why `some_allocator::deallocate(p, n)` requires `n`, whereas `free(n)` doesn't. – user541686 Jun 15 '14 at 00:28
  • @Mehrdad: Because `some_allocator` doesn't have to deal with padding, ever. So having the number of elements removes the need to store the actual block size. Whereas with `free`, the number of elements doesn't provide the block size, that has to be internally stored. – Ben Voigt Jun 15 '14 at 00:30
  • @BenVoigt: Again, you're speaking nonsense. I could replace all calls to `free` by calls to `deallocate` on some allocator if `free` happened to take in the number of bytes as well, removing the need to store the size internally in the same way it removes the need for allocators. There's absolutely nothing regarding padding or anything else that prevents me from doing so. That's been my point the entire time, and since we've gone full circle I'm not going to continue arguing in a circle again. – user541686 Jun 15 '14 at 00:35
  • @Mehrdad: Actually you'd have to replace both `malloc` and `free`, and then suddenly you can't, because `std::allocator::allocate` doesn't satisfy `malloc` postconditions. – Ben Voigt Jun 15 '14 at 00:37
  • @BenVoigt: Way to bring up an irrelevant point. If I gave you a memory manager that **actually does** what I'm claiming, would that satisfy you? I honestly don't know how to convince you, you seem to make reasons up as you go. – user541686 Jun 15 '14 at 00:52
  • @Mehrdad: Shall we just delete all comments related to `std::allocator`, because it operates under a completely different set of rules? (the relevant rule: the storage is properly sized and aligned for an array of type `T`, and so is every other allocation made with the same allocator) – Ben Voigt Jun 15 '14 at 02:26
  • @BenVoigt: No, and I'm not sure why your response to "what would it take to convince you?" was "shall we delete all comments regarding allocator?" instead of actually telling me what it would take to convince you. Which I take to mean you're stuck on your position no matter what. I'm certainly *not* going to delete my comments regarding `std::allocator`, because in my view it's the perfect example of why your line of reasoning is wrong. You're free to disagree. – user541686 Jun 15 '14 at 02:29
  • AFAIK std::allocator is required to handle varying allocation sizes anyway (think about rebind and scoped allocators, for example) – jaymmer - Reinstate Monica Jun 15 '14 at 02:33
  • @jaymmer: Yeah, that's another reason why I [said earlier](http://stackoverflow.com/questions/24203940?noredirect=1#comment37410150_24203940) that his response made no sense. I just didn't have any motivation to list the reasons because no matter how many arguments I propose against it, people seem very determined to make up another obscure rationale on the spot for why it must be wrong. – user541686 Jun 15 '14 at 02:38
  • @jaymmer: You may be interested in the update on my answer. – user541686 Jul 16 '14 at 02:50

12 Answers12

106

One-argument free(void *) (introduced in Unix V7) has another major advantage over the earlier two-argument mfree(void *, size_t) which I haven't seen mentioned here: one argument free dramatically simplifies every other API that works with heap memory. For example, if free needed the size of the memory block, then strdup would somehow have to return two values (pointer + size) instead of one (pointer), and C makes multiple-value returns much more cumbersome than single-value returns. Instead of char *strdup(char *) we'd have to write char *strdup(char *, size_t *) or else struct CharPWithSize { char *val; size_t size}; CharPWithSize strdup(char *). (Nowadays that second option looks pretty tempting, because we know that NUL-terminated strings are the "most catastrophic design bug in the history of computing", but that's hindsight speaking. Back in the 70's, C's ability to handle strings as a simple char * was actually considered a defining advantage over competitors like Pascal and Algol.) Plus, it isn't just strdup that suffers from this problem -- it affects every system- or user-defined function which allocates heap memory.

The early Unix designers were very clever people, and there are many reasons why free is better than mfree so basically I think the answer to the question is that they noticed this and designed their system accordingly. I doubt you'll find any direct record of what was going on inside their heads at the moment they made that decision. But we can imagine.

Pretend that you're writing applications in C to run on V6 Unix, with its two-argument mfree. You've managed okay so far, but keeping track of these pointer sizes is becoming more and more of a hassle as your programs become more ambitious and require more and more use of heap allocated variables. But then you have a brilliant idea: instead of copying around these size_ts all the time, you can just write some utility functions, which stash the size directly inside the allocated memory:

void *my_alloc(size_t size) {
    void *block = malloc(sizeof(size) + size);
    *(size_t *)block = size;
    return (void *) ((size_t *)block + 1);
}
void my_free(void *block) {
    block = (size_t *)block - 1;
    mfree(block, *(size_t *)block);
}

And the more code you write using these new functions, the more awesome they seem. Not only do they make your code easier to write, they also make your code faster -- two things which don't often go together! Before you were passing these size_ts around all over the place, which added CPU overhead for the copying, and meant you had to spill registers more often (esp. for the extra function arguments), and wasted memory (since nested function calls will often result in multiple copies of the size_t being stored in different stack frames). In your new system, you still have to spend the memory to store the size_t, but only once, and it never gets copied anywhere. These may seem like small efficiencies, but keep in mind that we're talking about high-end machines with 256 KiB of RAM.

This makes you happy! So you share your cool trick with the bearded men who are working on the next Unix release, but it doesn't make them happy, it makes them sad. You see, they were just in the process of adding a bunch of new utility functions like strdup, and they realize that people using your cool trick won't be able to use their new functions, because their new functions all use the cumbersome pointer+size API. And then that makes you sad too, because you realize you'll have to rewrite the good strdup(char *) function yourself in every program you write, instead of being able to use the system version.

But wait! This is 1977, and backwards compatibility won't be invented for another 5 years! And besides, no-one serious actually uses this obscure "Unix" thing with its off-color name. The first edition of K&R is on its way to the publisher now, but that's no problem -- it says right on the first page that "C provides no operations to deal directly with composite objects such as character strings... there is no heap...". At this point in history, string.h and malloc are vendor extensions (!). So, suggests Bearded Man #1, we can change them however we like; why don't we just declare your tricky allocator to be the official allocator?

A few days later, Bearded Man #2 sees the new API and says hey, wait, this is better than before, but it's still spending an entire word per allocation storing the size. He views this as the next thing to blasphemy. Everyone else looks at him like he's crazy, because what else can you do? That night he stays late and invents a new allocator that doesn't store the size at all, but instead infers it on the fly by performing black magic bitshifts on the pointer value, and swaps it in while keeping the new API in place. The new API means that no-one notices the switch, but they do notice that the next morning the compiler uses 10% less RAM.

And now everyone's happy: You get your easier-to-write and faster code, Bearded Man #1 gets to write a nice simple strdup that people will actually use, and Bearded Man #2 -- confident that he's earned his keep for a bit -- goes back to messing around with quines. Ship it!

Or at least, that's how it could have happened.

igk
  • 41
  • 11
Nathaniel J. Smith
  • 11,613
  • 4
  • 41
  • 49
  • 16
    Err, just in case it's unclear, this is a flight of fancy, with corroborating detail thrown in to provide artistic versimilitude. Any resemblance to persons living or dead is purely because everyone involved [kinda looked the same](http://electronicdesign.com/site-files/electronicdesign.com/files/uploads/2013/11/thompson-ritchie-and-kernighan.gif). Please do not confuse with actual history. – Nathaniel J. Smith Jun 14 '14 at 20:30
  • 6
    You win. This looks to me the most plausible explanation (and the best piece of writing). Even if everything here is proven incorrect or invalid, this stands the best answer for the excellent depictions of bearded men. – jaymmer - Reinstate Monica Jun 15 '14 at 00:36
  • Wow, an answer on this page that actually sounds plausible. +1 from me. – user541686 Jun 15 '14 at 01:36
  • Even better -- An answer on this page that is actually enjoyable! +1 as well. – David C. Rankin Jul 02 '14 at 21:10
  • I wonder if any notable Pascal systems used a garbage-collected string pool in a fashion similar to microcomputer BASIC interpreters? The semantics of C wouldn't work with such a thing, but in Pascal such a thing could be handled quite nicely if code maintained traceable stack frames (which many compilers did anyway). – supercat Feb 25 '15 at 20:36
  • RIP Bearded Man, +1 from me. – chqrlie Feb 27 '15 at 01:31
  • Nice history lesson. However freeing the `strdup` result is easy if a bit cumbersome. `mfree(string, strlen(string) + 1);` works. – Joshua Jun 05 '19 at 17:18
  • @Joshua: easy but slow and risky: what if the allocated string was torn apart with `strtok()`? – chqrlie Aug 26 '20 at 18:07
31

"Why does free in C not take the number of bytes to be freed?"

Because there's no need for it, and it wouldn't quite make sense anyway.

When you allocate something, you want to tell the system how many bytes to allocate (for obvious reasons).

However, when you have already allocated your object, the size of the memory region you get back is now determined. It's implicit. It's one contiguous block of memory. You can't deallocate part of it (let's forget realloc(), that's not what it's doing anyway), you can only deallocate the entire thing. You can't "deallocate X bytes" either -- you either free the memory block you got from malloc() or you don't.

And now, if you want to free it, you can just tell the memory manager system: "here's this pointer, free() the block it is pointing to." - and the memory manager will know how to do that, either because it implicitly knows the size, or because it might not even need the size.

For example, most typical implementations of malloc() maintain a linked list of pointers to free and allocated memory blocks. If you pass a pointer to free(), it will just search for that pointer in the "allocated" list, un-link the corresponding node and attach it to the "free" list. It didn't even need the region size. It will only need that information when it potentially attempts to re-use the block in question.

  • If I borrow $100 from you and then borrow $100 from you again and then do that another five times, do you really care that I borrowed money seven times from you (unless you're actually charging interest!)?? Or do you just care that I borrowed $700 from you? Same thing here: the system only cares about memory that is unallocated, it doesn't (and needn't and shouldn't) care about how the allocated memory is divided up. – user541686 Jun 13 '14 at 11:57
  • 1
    @Mehrdad: No, and it doesn't. C, though, does. _Its entire purpose is to make things (a little) safer._ I really don't know what you're looking for here. – Lightness Races in Orbit Jun 13 '14 at 12:14
  • There are variety of different memory management schemes so it's difficult to speak in generalities but many (I think most) memory managers will combine blocks on free thus they need to know the size to check whether the block can be merged into another free block and thus form a larger contiguous memory chunk. – Jack Aidley Jun 13 '14 at 13:10
  • @JackAidley that's why I wrote "most", not "all". (This is an implementation detail anyway.) – The Paramagnetic Croissant Jun 13 '14 at 17:08
  • 12
    @user3477950: *It does not need to be passed the number of bytes*: Yeah, because it was designed this way. The OP asked **Why was it designed this way?** – Deduplicator Jun 14 '14 at 01:57
  • 5
    "Because it doesn't need to" - it could have just as well been designed so that it does need to. – user253751 Jun 14 '14 at 02:14
  • You hit on something with _contiguous_ also - blocks aren't _really_ next to one another, the OS just gives a process this sort of walled garden view of it being that way. If a process could free a selective amount of bytes, fragmentation would get so horribly bad that things would theoretically slow to a crawl due to the VMM looking like a spaghetti-cabled server room. – Tim Post Jun 14 '14 at 07:45
  • @Deduplicator because it doesn't really make sense otherwise. – The Paramagnetic Croissant Jun 14 '14 at 09:16
  • @user3477950: "because it doesn't really make sense otherwise" *That* could become a good answer, if you explained **why it does not make sense otherwise**. – Deduplicator Jun 14 '14 at 10:17
  • @Deduplicator I think I've already explained it pretty well: "You can't deallocate part of it [...], you can only deallocate the entire thing. You can't "deallocate X bytes" either -- you either free the memory block you got from malloc() or you don't" – The Paramagnetic Croissant Jun 14 '14 at 10:34
  • @user3477950 I was, actually, agreeing with you - just from the viewpoint of the kernel. – Tim Post Jun 14 '14 at 10:42
  • 5
    @Mehrdad that a completely unclear and faulty analogy. If you allocate 4 byte a hundred time, it most definitely does matter exactly which one you free. freeing the 1st one is not the same as freeing the second one. with the money on the other hand it does not matter if you repay the first or the second borrow first, it's just one big pile – user2520938 Jun 14 '14 at 10:46
  • @TimPost Um, OK, sorry for the confusion. – The Paramagnetic Croissant Jun 16 '14 at 16:10
14

Actually, in the ancient Unix kernel memory allocator, mfree() took a size argument. malloc() and mfree() kept two arrays (one for core memory, another one for swap) that contained information on free block addresses and sizes.

There was no userspace allocator until Unix V6 (programs would just use sbrk()). In Unix V6, iolib included an allocator with alloc(size) and a free() call which did not take a size argument. Each memory block was preceded by its size and a pointer to the next block. The pointer was only used on free blocks, when walking the free list, and was reused as block memory on in-use blocks.

In Unix 32V and in Unix V7, this was substituted by a new malloc() and free() implementation, where free() did not take a size argument. The implementation was a circular list, each chunk was preceded by a word that contained a pointer to the next chunk, and a "busy" (allocated) bit. So, malloc()/free() didn't even keep track of an explicit size.

ninjalj
  • 42,493
  • 9
  • 106
  • 148
13

C may not be as "abstract" as C++, but it's still intended to be an abstraction over assembly. To that end, the lowest-level details are taken out of the equation. This prevents you from having to furtle about with alignment and padding, for the most part, which would make all your C programs non-portable.

In short, this is the entire point of writing an abstraction.

Lightness Races in Orbit
  • 378,754
  • 76
  • 643
  • 1,055
  • 10
    Not sure what alignment or padding have to do with this. The answer doesn't really answer anything. – user541686 Jun 13 '14 at 11:15
  • 1
    @Mehrdad C is not an x86 language, it tries (more or less) to be portable and thus frees the programmer of that significant burden. You can reach that level anyway in various other ways (e.g. inline assembly) but abstraction is the key. I agree with this answer. – Marco A. Jun 13 '14 at 11:17
  • 5
    @Mehrdad: If you asked `malloc` for N bytes and it returned a pointer to the beginning of a whole page instead (because of _alignment, padding, or other constraints_, there would be no way for the user to keep track of it - forcing them to do it would be counterproductive. – Michael Foukarakis Jun 13 '14 at 11:18
  • 4
    @MichaelFoukarakis: `malloc` can simply always return an aligned pointer without ever storing the size of the allocation. `free` could then round up to the appropriate alignment to ensure everything is freed properly. I don't see where the problem is. – user541686 Jun 13 '14 at 11:22
  • @MarcoA.: You would have a point if it was possible to achieve this at a "lower" level. However, the fact is that (at least on Windows) lower-level functions (even down to the kernel) like `VirtualFree` necessarily require that the entire set of pages allocated by `VirtualAlloc` be operated on as a whole, and they store the size of the allocation somewhere (to be queried by `VirtualQuery`). Hence there has to be a more fundamental reason than convenience for the user. – user541686 Jun 13 '14 at 11:24
  • 7
    @Mehrdad: There is no benefit for all that extra work you just mentioned. Additionally, passing a `size` parameter to `free` opens up another source of bugs. – Michael Foukarakis Jun 13 '14 at 11:27
  • @MichaelFoukarakis: There is: doing so allows the user can freely merge and split contiguous pieces of memory, allocating in one large chunk but freeing a little bit at a time. Currently that is not possible, and instead requires another layer of memory management on top of the existing one, which basically duplicates the work. – user541686 Jun 13 '14 at 11:30
  • 1
    Did you just describe heap fragmentation as a benefit? (at this point, this belongs in the chat) – Michael Foukarakis Jun 13 '14 at 11:31
  • @MichaelFoukarakis: The alternative is even worse: *allocating* in small chunks and still freeing them individually, with more overhead and no benefit. I'm not sure what alternative you have in mind that's better. – user541686 Jun 13 '14 at 11:32
  • I did upvote for the abstraction point, although we could still argue that C would be abstract enough even without this or that should be in C but isn't. I don't, however, understand the point about alignment and padding. – jaymmer - Reinstate Monica Jun 13 '14 at 11:36
  • @jaymmer: What I've been trying to say is the question is at a deeper level than the C standard library; not all current systems allow this at the kernel level (see my comment about virtual memory in Windows), even if you wanted to write your own library. So the reason likely isn't a design issue. It might be a theoretical or an implementation issue unrelated to C itself. – user541686 Jun 13 '14 at 11:43
  • @Mehrdad: The benefit you're suggesting is not what was asked about. According to the question, passing a size smaller than the true size would be UB. The question still requires freeing the entire allocation at once. – Ben Voigt Jun 13 '14 at 19:24
  • @BenVoigt: The whole point of mentioning the benefits of such an interface is to motivate the alternative. What I mentioned is a possible benefit that motivates this scheme, I'm not saying it's the only benefit or that it's the same one the OP had in mind. The point is, such a scheme *would* have benefits and therefore people shouldn't assume it wouldn't, that's all. Which of those benefits would actually be implemented is entirely another question. – user541686 Jun 13 '14 at 19:27
  • `malloc` could just as easily use a different, slightly lower-level abstraction where it does not remember the size of the allocated block for you. – user253751 Jun 14 '14 at 02:15
  • I'll just leave this here for you to enjoy http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2013/n3778.html – user541686 Jul 16 '14 at 02:52
10

Five reasons spring to mind:

  1. It's convenient. It removes a whole load of overhead from the programmer and avoids a class of extremely difficult to track errors.

  2. It opens up the possibility of releasing part of a block. But since memory managers usually want to have tracking information it isn't clear what this would mean?

  3. Lightness Races In Orbit is spot on about padding and alignment. The nature of memory management means that the actual size allocated is quite possibly different from the size you asked for. This means that were free to require a size as well as a location malloc would have to be changed to return the actual size allocated as well.

  4. It's not clear that there is any actual benefit to passing in the size, anyway. A typical memory manager has 4-16 bytes of header for each chunk of memory, which includes the size. This chunk header can be common for allocated and unallocated memory and when adjacent chunks come free they can be collapsed together. If you're making the caller store the free memory you can free up probably 4 bytes per chunk by not having a separate size field in allocated memory but that size field is probably not gained anyway since the caller needs to store it somewhere. But now that information is scattered in memory rather than being predictably located in the header chunk which is likely to be less operationally efficient anyway.

  5. Even if it was more efficient it's radically unlikely your program is spending a large amount of time freeing memory anyway so the benefit would be tiny.

Incidentally, your idea about separate allocators for different size items is easily implemented without this information (you can use the address to determine where the allocation occurred). This is routinely done in C++.

Added later

Another answer, rather ridiculously, has brought up std::allocator as proof that free could work this way but, in fact, it serves as a good example of why free doesn't work this way. There are two key differences between what malloc/free do and what std::allocator does. Firstly, malloc and free are user facing - they're designed for the general programmers to work with - whereas std::allocator is designed to provide specialist memory allocation to the standard library. This provides a nice example of when the first of my points doesn't, or wouldn't, matter. Since it's a library, the difficulties of handling the complexities of tracking size are hidden from the user anyway.

Secondly, std::allocator always works with the same size item this means that it is possible for it to use the originally passed number of elements to determine how much of free. Why this differs from free itself is illustrative. In std::allocator the items to be allocated are always of the same, known, size and always the same kind of item so they always have the same kind of alignment requirements. This means that the allocator could be specialised to simply allocate an array of these items at the start and dole them out as needed. You couldn't do this with free because there is no way to guarantee that the best size to return is the size asked for, instead it is much more efficient to sometimes return larger blocks than the caller asks for* and thus either the user or the manager needs to track the exact size actually granted. Passing these kinds of implementation details onto the user is a needless headache that gives no benefit to the caller.

-* If anyone is still having difficultly understanding this point, consider this: a typical memory allocator adds a small amount of tracking information to the start of a memory block and then returns a pointer offset from this. Information stored here typically includes pointers to the next free block, for example. Let's suppose that header is a mere 4 bytes long (which is actually smaller than most real libraries), and doesn't include the size, then imagine we have a 20 byte free block when the user asks for a 16 byte block, a naive system would return the 16byte block but then leave a 4byte fragment that could never, ever be used wasting time every time malloc gets called. If instead the manager simply returns the 20 byte block then it saves these messy fragments from building up and is able to more cleanly allocate the available memory. But if the system is to correctly do this without tracking the size itself we then require the user to track - for every, single allocation - the amount of memory actually allocated if it is to pass it back for free. The same argument applies to padding for types/allocations that don't match the desired boundaries. Thus, at most, requiring free to take a size is either (a) completely useless since the memory allocator can't rely on the passed size to match the actually allocated size or (b) pointlessly requires the user to do work tracking the real size that would be easily handled by any sensible memory manager.

Jack Aidley
  • 19,439
  • 7
  • 43
  • 70
  • #1 is true. #2: not sure what you mean. Not so sure about #3, alignment doesn't require storing extra information. #4 is circular reasoning; memory managers only require that overhead per chunk *because* they store the size, so you can't use that as an argument for why they store the size. And #5 is very debatable. – user541686 Jun 13 '14 at 11:45
  • I accepted and then unaccepted - this looks like a good answer but the question seems to be getting a lot of activity, I think I might have been a bit premature. This definitely gives me something to think over though. – jaymmer - Reinstate Monica Jun 13 '14 at 11:50
  • 2
    @jaymmer: Yeah, it's premature. I suggest waiting more than a day or two before accepting, and I suggest thinking about it on your own. It's a really interesting question and most/all of the answers you'll get at first for any such "why" question on StackOverflow will just be semi-automatic attempts to justify the current system rather than to actually address the underlying question. – user541686 Jun 13 '14 at 11:52
  • @Mehrdad: You've misunderstood what I'm saying in #4. I'm not saying it will take extra size, I'm saying (a) it'll move who has to store the size so it won't actually save any space and (b) the resulting change is actually likely to make it less efficient not more. As to #5, I'm not convinced it's debatable at all: we're - at most - talking about saving a couple of instructions from the free call. Compared to the costs of the free call that will be miniscule. – Jack Aidley Jun 13 '14 at 13:07
  • 1
    @Mehrdad: Oh, and on #3, no it doesn't require extra information, it does require extra memory. A typical memory manager that's designed to work with 16 byte alignment will return a pointer to a 128 byte block when asked for a 115 byte block. If the `free` call is to correctly pass the size to be freed it must know this. – Jack Aidley Jun 13 '14 at 13:12
  • @JackAidley: I don't understand, doesn't memory alignment *always* require extra memory because of the alignment overhead? It has nothing to do with whether or not you also store the size, so if that's what you meant it seems to be completely irrelevant, or perhaps I'm still not understanding #3. – user541686 Jun 13 '14 at 19:13
  • @JackAidley: The point of moving the size information to the caller *isn't* to save space. Rather, it's to allow for the possibility that two allocated blocks can be potentially contiguous, and that a single allocated block could be broken down into multiple blocks by the caller. It's not an efficiency issue, so I'm not arguing whether or not it's minuscule. It's an issue of whether such a thing is even *possible*, and the extra per-memory overhead before each block makes it impossible for the caller to recombine or split blocks. – user541686 Jun 13 '14 at 19:14
  • @Mehrdad: How much extra memory is required and when is an implementation issue that depends on the memory manager. You cannot, therefore, write code that _assumes_ that the memory you get back is the size you ask for. This means that you're automatically putting the burden to _track_ size onto the caller. You cannot just assume that the sizes are the same. This is transparent in the system as designed but requires effort from the caller in a pass-the-size-with-free system. – Jack Aidley Jun 13 '14 at 21:20
  • @JackAidley: The caller *must* assume the size is the same in either case. It's only the implementation that gains additional flexibility. – user541686 Jun 13 '14 at 21:49
  • Regarding your edit: consider a hypothetical malloc&free implementation that internally uses allocators that return fixed-size blocks (much like std::allocator), and simply delegates malloc/free calls to the appropriate internal allocator based on the size parameter. Would this not defeat the "malloc/free has to handle variable-size allocations" reasoning? – jaymmer - Reinstate Monica Jun 15 '14 at 01:29
  • @jaymmer: Not really, no, because - apart from anything else - that's a really poor way to design a general purpose memory manager. I've written managers that work that way in C++, but it's only worth deploying them for small sized allocations for the general case it's just woefully inefficient at managing the available memory without fragmentation in the general case and since `malloc` and `free` are designed to be general case managers it does not make sense for them to work this way. – Jack Aidley Jun 15 '14 at 06:49
  • I'll leave this link here for you to enjoy too, since I just found out about it. http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2013/n3778.html – user541686 Jul 16 '14 at 02:53
10

Why does free in C not take the number of bytes to be freed?

Because it doesn't need to. The information is already available in the internal management performed by malloc/free.

Here are two considerations (that may or may not have contributed to this decision):

  • Why would you expect a function to receive a parameter it doesn't need?

    (this would complicate virtually all client code relying on dynamic memory, and add completely unnecessary redundancy to your application). Keeping track of pointer allocation is already a dificult problem. Keeping track of memory allocations along with associated sizes would increase the complexity of client code unnecessarily.

  • What would the altered free function do, in these cases?

    void * p = malloc(20);
    free(p, 25); // (1) wrong size provided by client code
    free(NULL, 10); // (2) generic argument mismatch
    

    Would it not free (cause a memory leak?)? Ignore the second parameter? Stop the application by calling exit? Implementing this would add extra failure points in your application, for a feature you probably don't need (and if you need it, see my last point, below - "implementing solution at application level").

Rather, I want to know why free was made this way in the first place.

Because this is the "proper" way to do it. An API should require the arguments it needs to perform it's operation, and no more than that.

It also occurs to me that explicitly giving the number of bytes to free might allow for some performance optimisations, e.g. an allocator that has separate pools for different allocation sizes would be able to determine which pool to free from just by looking at the input arguments, and there would be less space overhead overall.

The proper ways to implement that, are:

  • (at the system level) within the implementation of malloc - there is nothing stopping the library implementer from writing malloc to use various strategies internally, based on received size.

  • (at application level) by wrapping malloc and free within your own APIs, and using those instead (everywhere in your application that you may need).

utnapistim
  • 26,809
  • 3
  • 46
  • 82
  • 6
    @user3477950: *It does not need to be passed the number of bytes*: Yeah, because it was designed this way. The OP asked **Why was it designed this way?** – Deduplicator Jun 14 '14 at 02:01
  • 4
    "Because it doesn't need to" - it could have just as well been designed so that it does need to, by not saving that information. – user253751 Jun 14 '14 at 02:13
  • 1
    As to your point 2, you have me wondering if free(NULL) is defined behaviour. Aha, "All standards compliant versions of the C library treat free(NULL) as a no-op" - source http://stackoverflow.com/questions/1938735/does-freeptr-where-ptr-is-null-corrupt-memory/1938791#1938791 – Mawg says reinstate Monica May 04 '15 at 09:17
5

I'm only posting this as an answer not because it's the one you're hoping for, but because I believe it's the only plausibly correct one:

It was probably deemed convenient originally, and it could not be improved thereafter.
There is likely no convincing reason for it. (But I'll happily delete this if shown it's incorrect.)

There would be benefits if it was possible: you could allocate a single large piece of memory whose size you knew beforehand, then free a little bit at a time -- as opposed to repeatedly allocating and freeing small chunks of memory. Currently tasks like this are not possible.


To the many (many1!) of you who think passing the size is so ridiculous:

May I refer you to C++'s design decision for the std::allocator<T>::deallocate method?

void deallocate(pointer p, size_type n);

All n T objects in the area pointed to by p shall be destroyed prior to this call.
n shall match the value passed to allocate to obtain this memory.

I think you'll have a rather "interesting" time analyzing this design decision.


As for operator delete, it turns out that the 2013 N3778 proposal ("C++ Sized Deallocation") is intended to fix that, too.


1Just look at the comments under the original question to see how many people made hasty assertions such as "the asked for size is completely useless for the free call" to justify the lack of the size parameter.

Community
  • 1
  • 1
user541686
  • 205,094
  • 128
  • 528
  • 886
  • 5
    For the `malloc()` implementation it would also remove the necessity to remember *anything* about an allocated region, reducing the allocation overhead to the alignment overhead. `malloc()` would be able to do all bookkeeping within the freed chunks themselves. There are use-cases in which this would be a large improvement. Deallocating a large chunk of memory in several small chunks would have to be discouraged, though, this would drastically increase fragmentation. – cmaster - reinstate monica Jun 13 '14 at 12:11
  • 1
    @cmaster: Those use cases were exactly the kind I was referring to, you hit the nail on the head, thanks. Regarding fragmentation: not sure how that increases fragmentation relative to the alternative, which is to both allocate and free memory in small chunks. – user541686 Jun 13 '14 at 19:06
  • `std::allocator` allocates _only_ elements of a specific, known size. It is not a general purpose allocator, the comparison is apples to oranges. – Jack Aidley Jun 14 '14 at 11:19
  • It appears to me that a partly philosophical, partly design decision was made to make the standard library in C a minimal set of primitives from which virtually anything can be built - originally intended as a systems level language and portable to may systems this *primitive* approach makes sense. With C++, a different decision was made to make the standard library very extensive (and getting larger with C++11). Faster development hardware, larger memories, more complex architectures and a need to address higher-level application development contribute to this change of emphasis perhaps. – Clifford Jun 14 '14 at 11:53
  • 1
    @Clifford: Exactly -- that's why I said there's no convincing reason for it. It's just a decision that was made, and there's no reason to believe that it's strictly better than the alternatives. – user541686 Jun 14 '14 at 11:58
2

malloc and free go hand in hand, with each "malloc" being matched by one "free". Thus it makes total sense that the "free" matching a previous "malloc" should simply free up the amount of memory allocated by that malloc - this is the majority use case that would make sense in 99% of cases. Imagine all the memory errors if all uses of malloc/free by all programmers around the world ever, would need the programmer to keep track of the amount allocated in malloc, and then remember to free the same. The scenario you talk about should really be using multiple mallocs/frees in some kind of memory management implementation.

Marius George
  • 526
  • 2
  • 6
  • 5
    I think the "Imagine all the [...] errors" is moot when you think of other bug factories like `gets`, `printf`, manual loops (off-by-one), undefined behaviours, format-strings, implicit conversions, bit tricks, et cetera. – Sebastian Mach Jun 13 '14 at 11:26
1

I would suggest that it is because it is very convenient not to have to manually track size information in this way (in some cases) and also less prone to programmer error.

Additionally, realloc would need this bookkeeping information, which I expect contains more than just the allocation size. i.e. it allows the mechanism by which it works to be implementation defined.

You could write your own allocator that worked somewhat in the way you suggest though and it is often done in c++ for pool allocators in a kind of similar way for specific cases (with potentially massive performance gains) though this is generally implemented in terms of operator new for allocating pool blocks.

Pete
  • 4,784
  • 26
  • 33
1

I don't see how an allocator would work that does not track the size of its allocations. If it didn't do this, how would it know which memory is available to satisfy a future malloc request? It has to at least store some sort of data structure containing addresses and lengths, to indicate where the available memory blocks are. (And of course, storing a list of free spaces is equivalent to storing a list of allocated spaces).

M.M
  • 138,810
  • 21
  • 208
  • 365
  • It doesn't even need an explicit size field. It can just have a pointer to the next block, and an allocated bit. – ninjalj Jun 14 '14 at 11:51
0

Well, the only thing you need is a pointer that you'll use to free up the memory you previously allocated. The amount of bytes is something managed by the operating system so you don't have to worry about it. It wouldn't be necessary to get the number of bytes allocated returned by free(). I suggest you a manual way to count the number of bytes/positions allocated by a running program:

If you work in Linux and you want to know the amount of bytes/positions malloc has allocated, you can make a simple program that uses malloc once or n times and prints out the pointers you get. In addition, you must make the program sleep for a few seconds (enough for you to do the following). After that, run that program, look for its PID, write cd /proc/process_PID and just type "cat maps". The output will show you, in one specific line, both the beginning and the final memory addresses of the heap memory region (the one in which you are allocating memory dinamically).If you print out the pointers to these memory regions being allocated, you can guess how much memory you have allocated.

Hope it helps!

0

Why should it? malloc() and free() are intentionally very simple memory management primitives, and higher-level memory management in C is largely up to the developer. T

Moreover realloc() does that already - if you reduce the allocation in realloc() is it will not move the data, and the pointer returned will be the the same as the original.

It is generally true of the entire standard library that it is composed of simple primitives from which you can build more complex functions to suit your application needs. So the answer to any question of the form "why does the standard library not do X" is because it cannot do everything a programmer might think of (that's what programmers are for), so it chooses to do very little - build your own or use third-party libraries. If you want a more extensive standard library - including more flexible memory management, then C++ may be the answer.

You tagged the question C++ as well as C, and if C++ is what you are using, then you should hardly be using malloc/free in any case - apart from new/delete, STL container classes manage memory automatically, and in a manner likely to be specifically appropriate to the nature of the various containers.

Clifford
  • 88,407
  • 13
  • 85
  • 165
  • Moreover realloc() does that already - if you reduce the allocation in realloc() is it will not move the data, and the pointer returned will be the the same as the original. Is this guaranteed behaviour? It seems to be common in several embedded allocators, but I wasn't sure if this behavior was specified as part of standard-c. – rsaxvc Jun 14 '14 at 16:10
  • @rsaxvc : Good question - [cplusplus.com](http://www.cplusplus.com/reference/cstdlib/realloc/) documents *"If the new size is larger, the value of the newly allocated portion is indeterminate."*, implying that if smaller it is *determinate*. [opengroup.org() says *"If the new size of the memory object would require movement of the object, the space for the previous instantiation of the object is freed."* - if it were smaller, the data would not need to be moved. Again the implication is that smaller will not reallocate. I am not sure what the ISO standard says. – Clifford Jun 14 '14 at 17:55
  • 1
    `realloc` is *absolutely* allowed to move the data. According to the C standard it's totally legal to implement `realloc` as a `malloc`+`memcpy`+`free`. And there are good reasons why an implementation might want to move an allocation that has been reduced in size, e.g. to avoid fragmenting memory. – Nathaniel J. Smith Jan 17 '15 at 23:06