9

The language shootout benchmarks at http://benchmarksgame.alioth.debian.org/ indicate that FPC programs use about 1/50th of the memory that comparable programs using g++ use. Do these benchmarks unintentionally favor fpc or is it really true that FPC is this much better than g++? I've always considered these benchmarks as a collection of decent micro-benchmarks, so I'm surprised at these results, since a factor of 50 times is pretty significant IMHO.

References:

http://benchmarksgame.alioth.debian.org/u32/pascal.php http://benchmarksgame.alioth.debian.org/u64q/pascal.html

Edit: This is becoming even more interesting since this page claims that pascal used only 8KB for some of the programs, which seems amazingly low

igouy
  • 2,547
  • 17
  • 16
Sudarshan S
  • 1,235
  • 2
  • 16
  • 25
  • One of the common meanings of bias is prejudice - are you making an accusation or asking a question? Rather than "biased" it's always possible that somethings wrong with the way that the FPC program memory use is being measured. Have you looked at [how those measurements are being made](http://shootout.alioth.debian.org/help.php#memory)? – igouy Jun 27 '12 at 15:30
  • 3
    @igoguy I think interpreting this as anything other than trying to understand the wildly-disparate memory numbers is a stretch. It's a perfectly reasonable way of asking this question. – Dave Newton Jun 27 '12 at 16:12
  • @Dave Newton -- It's easy to ask why such a large difference in memory use is shown, without any accusation of bias. For example, just omit any mention of bias and ask -- "Is it really true that FPC is this much better than g++?" – igouy Jun 27 '12 at 16:19
  • 2
    @igoguy I don't understand. "Biased" can simply mean "favors", without implying any nefarious intent. Geez. Unintentional bias is, in fact, just one reason why micro-benchmarks are so unreliable at predicting the real world. – Dave Newton Jun 27 '12 at 16:21
  • @Dave Newton -- You don't understand because you are ignoring the fact that bias (unqualified) is also commonly used to mean prejudice and intentional distortion. (And NO - that's not usually the problem with micro-benchmarks. The problem is that software performance measurements are very brittle.) – igouy Jun 27 '12 at 16:30
  • 2
    @igoguy Whatever. Bias is a completely common, reasonable term, especially in this context--you're choosing to ignore a different usage of the word and would rather ascribe malice to a pretty straight-forward question with no negative connotations. – Dave Newton Jun 27 '12 at 16:57
  • @Dave Newton -- Actually I would prefer that the question was straight-forward, with no negative connotations, and there's a real easy fix. – igouy Jun 27 '12 at 17:00
  • 2
    @igouy No, really, this is a completely normal way to ask this question in English. – Marcin Jun 27 '12 at 18:05
  • 4
    @igouy I've made it explicit I am *not* accusing the site of intentional bias – Sudarshan S Jun 27 '12 at 18:28
  • @Marcin -- I'm a native English speaker -- Unqualified accusations of bias have negative connotations, and some very negative connotations. – igouy Jun 27 '12 at 18:50
  • @igouy I'm a native English speaker, and I expect that Dave Newton is too (I wouldn't be surprised that Sudarshan S is also). Your unwarranted focus on the word "bias", and inability to accept that it has a completely neutral meaning rather suggests you are not a native speaker; you are certainly out of step with everyone else commenting on this question. – Marcin Jun 27 '12 at 21:44
  • bias: "often supporting or opposing a particular person or thing in an unfair way by allowing personal opinions to influence your judgment" http://dictionary.cambridge.org/dictionary/british/bias_1?q=bias – igouy Jun 27 '12 at 23:48
  • @Sudarshan S -- Thank you for making that clear. – igouy Jun 28 '12 at 14:39
  • Unfortunately, the links to benchmarksgame.alioth.debian.org are no longer available, but archived versions can be found at https://web.archive.org/web/20180406195444/https://benchmarksgame-team.pages.debian.net/benchmarksgame/compare/pascal.html and https://web.archive.org/web/20150923114120/http://benchmarksgame.alioth.debian.org/u32/pascal.php, respectively. – jwdietrich Nov 29 '20 at 11:15

2 Answers2

11

Note that startup time is IIRC another benchmark where FPC peaks

I think the answer must be primarily sought in the fact that Free Pascal statically links programs by default, avoiding libc and other auxiliary libraries

This has several consequences:

  • For the simple programs that are being benchmarked, FPC programs are static using only the own RTL (no static copy of libc) and have no dynamic linking overhead (both in time and memory). Including mapping shared glibc segments (is this so?) that might be mistaken for application memory use.
  • libc might do potentially unneeded but involved initializations that FPC doesn't do for these simple programs. (like initializing zoneinfo)
  • since FPC uses a totally independent memory manager, the initial block of the heap suballocator might have a different size. Possibly FPC's is systematically smaller.
  • For threads, the size of the new thread's stack might cause various differences (size and maybe the fact if it is (partially) only a reservation or committed memory, or whatever the *nix equivalent is of that)

All in all, I think this observed behaviour is less about FPC, and more about lack of variation amongst the other benchmarked development systems . FPC merely stands out, because nearly everything else is built on top of the gcc/glibc technology (either because they are a direct gcc derivate or because their VM/interpreters are built on top of gcc), and thus all share libc's general treats. FPC being different merely highlights (g?)libc's bad scaling towards simple programs. (*)

The shootout probably might be biassed in the sense that either shared adress space is counted rather than actually used private bytes, or because it doesn't differentiate enough between private bytes allocated by the suballocator and private bytes actually used by the process. It would probably require a libc/libmalloc core devel however to sort this out, and since the shootout is open source, the question if you can provide a better measurement is open.

Either that or there is something fundamentally wrong with (g)libc. (I'm no expert at that). A possible solution to get more relevant information would be to run the benchmark on FreeBSD, or a Linux with uclibc. In short anything else but glibc.

As Igouy's post states, when linking to libc, FPC gets the (bad) characteristics of the other development systems. This is another indicator that the question should be "why do glibc using binaries perform bad in the shootout memory benchmark" rather than "why does FPC perform well in the shootout benchmark"

Note that FPC originally avoided libc because of cross-distribution compatibility concerns, and not performance or filesize.

So to all that assume this is a fluke wrt to measuring the memory usage of FPC, ever considered it is a problem with glibc memory use or the measurement of it? Or rather, that the high glibc number is wrong, and not the low FPC number....

.... A FPC developer ....

(*) and before you can say it is merely developed to be efficient for "sizable" applications, remember that the Unix philosophy is about chaining small tools together, and many Unix processes are shorted lived.

Marco van de Voort
  • 25,628
  • 5
  • 56
  • 89
  • >>when linking to libc<< (For convenience I'll just refer to the Memory usage displayed by the Gnome system monitor for the process.) For the single threaded mandelbrot Free Pascal #3 program [28KiB], for the same program compiled with "uses cthreads;" [300KiB] – igouy Jun 29 '12 at 01:19
5

Yes, it's really true that the unix utility top reports that those Pascal programs use that much memory, and those C++ programs use that much memory.

For example, on x64, while the Free Pascal n-body program is run and while the C++ n-body program is run, top reports these measurements --

VIRT        RES       SHR
 608          4         0 FPC
7208        420       332 C++

The memory use top reports for the Free Pascal programs is the memory use that the benchmarks game reports for the Free Pascal programs.

Now look at that x64 quad-core comparison

We can see 2 different cases:

  1. Both the Pascal and C++ programs use multiple MBs and the memory use is very similar, differing by less than ~2x. When additional memory is allocated to solve the task, there's not much difference between the programs.

  2. The C++ programs uses a few hundred KB and the Pascal program uses a few KB. When additional memory is not allocated to solve the task, the Pascal programs use a few hundred KB less.

The question presents 2 alternatives, but there's usually a third alternative - Have I misunderstood what's going on?

The fact that a C++ mandelbrot program could use 4000x more memory than a Pascal mandelbrot program was too much for the OP to believe, it seemed impossible to the OP - but there's a simple enough explanation, time/space tradeoff.

The C++ program is multi threaded, written to use multi core; but the Pascal program is single threaded, written to use single core.

The memory use of a Pascal multi threaded mandelbrot program is very similar to that of the C++ multi threaded program.

igouy
  • 2,547
  • 17
  • 16
  • I'm not sure I understand your point, is this an attempt to show that for this particular benchmark it's not 50, but only an order of magnitude or two? – Dave Newton Jun 27 '12 at 16:15
  • The point is that the memory use **top** reports ***is*** the memory use that the benchmarks game reports. – igouy Jun 27 '12 at 16:50
  • @Marcin -- What exactly do you think is *the issue at hand* ? – igouy Jun 27 '12 at 19:17
  • @igouy Whether or not FPC is really more memory efficient in any meaningful sense, or if this is an artifact of some unknown biasing factor. – Marcin Jun 27 '12 at 21:41
  • @Marcin -- I don't think we need to find out exactly what you mean by "in any meaningful sense" because you can look at the data and decide for yourself. But NO, this is not an artifact of some unknown biasing factor. – igouy Jun 28 '12 at 16:00
  • 5
    igouy: that is a flawed assumption. The reason why the threaded FPC version eats more is because it links to pthreads and thus to libc, and thus suddenly gets the same behaviour as the other libc linking program. Threading is an userland API, and we thus have no choice except to link to libc, incurring all the penalties the other processes have too. – Marco van de Voort Jun 28 '12 at 19:21
  • @Marco van de Voort -- Yes and No. (For convenience I'll just refer to the Memory usage displayed by the Gnome system monitor for the process.) For the single threaded mandelbrot Free Pascal #3 program [28KiB], for the same program compiled with "uses cthreads;" [300KiB], but for the multi threaded mandelbrot Free Pascal #4 program [**30MiB**]. So Yes linking to pthreads does increase the memory use, but No that doesn't seem to cause the big increase. Please correct any mistakes made in my reasoning. – igouy Jun 28 '12 at 23:52
  • So the challenge is simple then. Show a single threaded C++ program that eats 28KiB ? – Marco van de Voort Sep 27 '12 at 10:50
  • (probably it is the default stack size of the created threads that create the enormous increase) – Marco van de Voort Oct 27 '14 at 21:07