4

I am aware of the existence of this, and this on this topic. However, I would like to finalize on an actual implementation in Python this time.

My only problem is that the elbow point seems to be changing from different instantiations of my code. Observe the two plots shown in this post. While they appear to be visually similar, the value of the elbow point changed significantly. Both the curves were generated from an average of 20 different runs. Even then, there is a significant shift in the value of the elbow point. What precautions can I take to make sure that the value falls within a certain bound?

My attempt is shown below:

def elbowPoint(points):
  secondDerivative = collections.defaultdict(lambda:0)
  for i in range(1, len(points) - 1):
    secondDerivative[i] = points[i+1] + points[i-1] - 2*points[i]

  max_index = secondDerivative.values().index(max(secondDerivative.values()))
  elbow_point = max_index + 1
  return elbow_point

points = [0.80881476685027154, 0.79457906121371058, 0.78071124401504677, 0.77110686192601441, 0.76062373158581287, 0.75174963969985187, 0.74356408965979193, 0.73577573557299236, 0.72782434749305047, 0.71952590556748364, 0.71417942487824781, 0.7076502559300516, 0.70089375208028415, 0.69393584640497064, 0.68550490458450741, 0.68494440529025913, 0.67920157634796108, 0.67280267176628761]
max_point = elbowPoint(points)

enter image description here enter image description here

Community
  • 1
  • 1
Legend
  • 113,822
  • 119
  • 272
  • 400
  • 1
    Just a small sidenote: You shouldn't use a defaultdict for a function with a default value, because then a dictionary entry will be created at `yourDefaultDict[i]` for all values `i` you query. Here I can't see an issue with it. – ninjagecko Jul 12 '11 at 05:27
  • @ninjagecko: Oh! You're right! I will fix it my version. Thank you for pointing it out. – Legend Jul 12 '11 at 05:31

1 Answers1

3

Its sounds like your actual concern is how to smooth your data as it contains noise? in which case perhaps you should fit a curve to the data first, then find the elbow of the fitted curve?

Whether this will work would depend on the source of the noise, and if the noise is important for your application? by the way you may want to see how sensitive your fit is to your data by seeing how it changes (or hopefully doesn't) when a point is omitted from the fit (obviously with a high enough polynomial you will always get a good fit to a specific set of data, but you are presumably interested in the general case)

I have no idea if this approach is acceptable, intuitively though i'd think that sensitivity to small errors is bad. ultimately by fitting a curve you are saying that the underlying process is, in the ideal case, modelled by the curve, and any deviation from the curve is an error/noise

jk.
  • 13,817
  • 5
  • 37
  • 50
  • +1 Thank you. I tried to fit a curve using a polynomial of degree 4 and it looks good to me. The only question now is to find out if smoothing is considered an accepted approach. Any pointers to that would be great. – Legend Jul 12 '11 at 22:06