O(n^3)
Each i
would be run n*(n+1)/2 times which gives Triangular numbers. Well, to be more exact, it would be run (n-2)*(n-1)/2, but that's the same O notation.
If you start with n=3, you get 1 run.
Every time n goes up, you get what you had before, plus (n-2).
But, of course what you need is the sum of all these, that's called triangular pyramidal and it's n*(n+1)*(n+2)/6 which is O(n^3).
Sorry my last answer incorrectly didn't sum over i
and so under counted.
Update with additional explanation
A good place to start is with small n
cases.
For example, at n = 3
, the loop runs once, since i
is 1~3, j
is 2~3 and k
is only 3.
When we move to n = 4
, then j
is 2~4 and k
is 3~4. On the j = 2
run, k
runs twice, and on the j = 3
run, k
runs only once. That's 3 runs total.
Of course, we can't keep running through all the possibility like this, so I made an excel sheet. For any given n
, what i
, j
, and k
values occur.

First, start with an n
. They are in green. For any n
, what i
values are possible? They are listed under i
so that any i
that's right of an n
will happen. For example, at n = 7
, the possible values for i
are [7, 6, 5, 4, 3]
. Ok, so now we know which i
values will happen, next what j
values will happen for any given i
? I decided to be verbose here. For every i
, I listed all the possible j
values next to it. Let's say we're looking at n = 7
still. One possible i
value in that case is 5. When i = 5
, then we can see the possible j
values are [4, 3, 2]
. Ok, getting close now. For each of those j
values, how many times will k
run? Well, k
will always run one fewer time than j
, so in the k
column, this is the value I wrote.
With all this is written out, now it's time to work backwards and summing up as we go. Let's zero in on n = 7
, i = 5
, j = 3
, k = 1 or 2
, I marked that cell in blue. There are 2 possible values for k
at this j
value. For this i = 5
value, there are 3 possible j
values, namely 4, 3, and 2. Summing up the times k
runs for each of these, we get 3 + 2 + 1 = 6
which we see in the i total
column. The last step is to sum up all the i
values that happen for any given n
. If n = 7
, then we have to sum up all the i
values in [7, 6, 5, 4, 3]
which is 15 + 10 + 6 + 3 + 1
or 35
.
The final step is finding a general solution. We can see that to find i total
we are summing up all integers from 1 to (i-2). Then to find the n total
we are summing up the i total
s. Well, summing up all integers from 1 to x gives Triangular numbers, as I mentioned before, and summing up Triangular numbers gives Triangular pyramidal numbers. The formula for that is n*(n+1)*(n+2)/6
. In this case, we start from n = 3
so this particular case would be (n-2)*(n-1)*(n)/6
. Plugging in some n
values to this matches our hand-derived values. With O notation, we can basically strip out constants, so it would be n * n * n
or O(n^3).
This is basically representing a 4D array in excel, so I understand it's not clear at first glance, but if you step through, I think you'll find each step stands. I'm open to corrections or better explanations.