13

While answering to this question a debate began in comments about complexity of QuickSort. What I remember from my university time is that QuickSort is O(n^2) in worst case, O(n log(n)) in average case and O(n log(n)) (but with tighter bound) in best case.

What I need is a correct mathematical explanation of the meaning of average complexity to explain clearly what it is about to someone who believe the big-O notation can only be used for worst-case.

What I remember if that to define average complexity you should consider complexity of algorithm for all possible inputs, count how many degenerating and normal cases. If the number of degenerating cases divided by n tend towards 0 when n get big, then you can speak of average complexity of the overall function for normal cases.

Is this definition right or is definition of average complexity different ? And if it's correct can someone state it more rigorously than I ?

Community
  • 1
  • 1
kriss
  • 23,497
  • 17
  • 97
  • 116
  • 1
    In regards to the argument, I think that if you give big-O notation for running time and don't qualify it, then you should be talking about worst case, simply because you are saying that time is bounded by a function with the specified big-O. If time is bounded, that means worst-case time is bounded, by definition of "bound". If you say, "this is O(n log n) average case", though, then that's well-defined and means what you say in this question. – Steve Jessop Oct 11 '10 at 10:37
  • Might be worth trying http://cstheory.stackexchange.com/ for the question – Chris S Oct 11 '10 at 11:01
  • @Chris: although the FAQ for that site says that "typical homework problems in textbooks" are too basic, and I think this is about as basic as that. – Steve Jessop Oct 11 '10 at 11:29
  • You'll get a page of mathematical notation on there, not beginner friendly – Chris S Oct 11 '10 at 13:11
  • @Chris S: I work as a programmer but did some math in my old time, so it's not really a problem for me. But for others it could indeed be nice to have something more accessible. – kriss Oct 11 '10 at 13:48

5 Answers5

12

You're right.

Big O (big Theta etc.) is used to measure functions. When you write f=O(g) it doesn't matter what f and g mean. They could be average time complexity, worst time complexity, space complexities, denote distribution of primes etc.

Worst-case complexity is a function that takes size n, and tells you what is maximum number of steps of an algorithm given input of size n.

Average-case complexity is a function that takes size n, and tells you what is expected number of steps of an algorithm given input of size n.

As you see worst-case and average-case complexity are functions, so you can use big O to express their growth.

sdcvvc
  • 25,343
  • 4
  • 66
  • 102
  • It's not quite right to write f=O(g) if we're being pedantic. Big O is a set, so we should write f \in O(g) – jhclark Sep 24 '12 at 19:15
  • 2
    @jhclark: It's a _very_ strong custom to write = when using big O, see http://en.wikipedia.org/wiki/Asymptotic_notation#Equals_sign or Concrete Mathematics on asymptotic notation. In fact I have never seen any textbook that uses \in except to point out this pecularity. – sdcvvc Sep 24 '12 at 21:55
  • 1
    Agreed that it's customary -- I'm being pedantic. However, as wikipedia says, many consider "f = O(g) to be an abuse of notation since pure mathematics typically defines = to indicate a bidirectional equality. I'm definitely in the camp that considers this use of = to be rather odious. – jhclark Sep 26 '12 at 01:18
3

If you're looking for a formal definition, then:

Average complexity is the expected running time for a random input.

Yakov Galka
  • 70,775
  • 16
  • 139
  • 220
  • quote please. Wiki article does not really directly relate. – Unreason Oct 11 '10 at 11:36
  • 2
    actually found it http://en.wikipedia.org/wiki/Average-case_complexity, +1 for your answer, it seems (if we believe wikipedia), that formal definition is really on random input. – Unreason Oct 11 '10 at 11:53
  • also regarding the concept of case vs complexity check http://en.wikipedia.org/wiki/Best,_worst_and_average_case; and also it seems you use term 'running time' for bounding function. – Unreason Oct 11 '10 at 11:55
  • @Unreason: I linked for people who don't know what expected value means in mathematics. You're right that this article is not related directly to the question. – Yakov Galka Oct 11 '10 at 14:19
2

Let's refer Big O Notation in Wikipedia:

Let f and g be two functions defined on some subset of the real numbers. One writes f(x)=O(g(x)) as x --> infinity if ...

So what the premise of the definition states is that the function f should take a number as an input and yield a number as an output. What input number are we talking about? It's supposedly a number of elements in the sequence to be sorted. What output number could we be talking about? It could be a number of operations done to order the sequence. But stop. What is a function? Function in Wikipedia:

a function is a relation between a set of inputs and a set of permissible outputs with the property that each input is related to exactly one output.

Are we producing exacly one output with our prior defition? No, we don't. For a given size of a sequence we can get a wide variation of number of operations. So to ensure the definition is applicable to our case we need to reduce a set possible outcomes (number of operations) to a single value. It can be a maximum ("the worse case"), a minimum ("the best case") or an average.

The conclusion is that talking about best/worst/average case is mathematically correct and using big O notation without those in context of sorting complexity is somewhat sloppy.

On the other hand, we could be more precise and use big Theta notation instead of big O notation.

Alexey
  • 9,197
  • 5
  • 64
  • 76
  • The paragraph you pointed out as wrong merely states we can compute the limit of average toward infinity using Taylor expansion and ignoring insignificant terms. Of course the average value is affected and I'm pointing a specific case when degenerating case is *not significant*. Obviously that is not always true. In other words, of course average value is always definable, but that does not help much when what we want is not *defining* it but *computing* it. – kriss Dec 17 '17 at 23:38
  • @kriss, I just don't understand "then you can speak of average complexity of the overall function for normal cases". – Alexey Dec 18 '17 at 09:17
  • OK, I should change wording. I just mean ignore the term tending toward zero (exceptional cases) and only keep the "normal case" term. – kriss Dec 18 '17 at 18:15
1

I think your definition is correct, but your conclusions are wrong.

It's not necessarily true that if the proportion of "bad" cases tends to 0, then the average complexity is equal to the complexity of the "normal" cases.

For example, suppose that 1/(n^2) cases are "bad" and the rest "normal", and that "bad" cases take exactly (n^4) operations, whereas "normal" cases take exactly n operations.

Then the average number of operations required is equal to:

(n^4/n^2) + n(n^2-1)/(n^2)

This function is O(n^2), but not O(n).

In practice, though, you might find that time is polynomial in all cases, and the proportion of "bad" cases shrinks exponentially. That's when you'd ignore the bad cases in calculating an average.

Steve Jessop
  • 273,490
  • 39
  • 460
  • 699
  • OK, I agree with you. I'm really doing exactly what you suggest to compute the complexity of the average. That's even the calculus I had put in comments of the linked question. I was too fast saying we keep the normal case, that's obviously not always true and depends on the complexity of the degenerating cases. Stated the way I did, the bad case could even continue forever and program never stop and that's definitely not good for average. – kriss Oct 11 '10 at 10:57
  • @kriss: yes, the non-halting case is a simpler example than mine, although then it's not technically an "algorithm" you're analysing. – Steve Jessop Oct 11 '10 at 11:08
0

Average case analysis does the following:

Take all inputs of a fixed length (say n), sum up all the running times of all instances of this length, and build the average.

The problem is you will probably have to enumerate all inputs of length n in order to come up with an average complexity.

phimuemue
  • 34,669
  • 9
  • 84
  • 115