1

I'm a novice with python, but I have timeseries data and I'm looking to plot the number of times my threshold was exceeded as a function of my threshold value. For example, as my threshold value increases, the number of times it is exceeded should decrease.

I have tried initializing an array and appending each exceeding value into it within a while loop, iterating through each of my threshold values as well as my dataset simultaneously, then counting up the number of values for each threshold and plotting those.

After this, I'm interested in including a constraint that it has to last for 2 or more consecutive datapoints.

What I've tried:

count = []    # row i includes all values over threshold x[i]
counter = 0 
x = np.linspace(10, 40, 1440)
y = np.empty(1440,dtype=object)    # element i is the count over threshold with threshold x[i]
d = minutedata[1]
i=0
j=0

# outer is x threshold data, inner is checking each observation datapoint with a certain x threshold value
for j, x_thresh in enumerate(x):
    for i, el in enumerate(data):
        if el > x_thresh:    # in jth x element, running through data
            counter += 1   # adding in values over threshold
            y[i] = counter            # makes a vector for each threshold containing values over threshold
            counter = 0

but currently all of my y values come out as None. I know there's an issue with my loops somewhere but I'm not sure what.

Any help or suggestions are very appreciated!

liveFreeOrπHard
  • 189
  • 9
  • There's a line that goes `for j in ...` then one that goes `while i in ...`. I believe this is a mistake, both should start with `for`. – LoneCodeRanger Sep 12 '22 at 18:29
  • Also it is generally advised in python to use `for x_element in x:` rather than `for j in range(len(x)):`. Once you get used to the new paradigm (if you're coming from another language), this should be easier to read an less error prone. – LoneCodeRanger Sep 12 '22 at 18:32
  • 1
    Usually you don't if you use the tools that python provides, but if you ever need to have the index of the element, you can use `for j, x_element in enumerate(x):`. – LoneCodeRanger Sep 12 '22 at 18:34
  • Also you don't need to initialize i and j, this is done by the for loops, more specifically by the range function (you can give it a param to start counting at 1 for instance, or with a different increment). – LoneCodeRanger Sep 12 '22 at 18:38
  • 1
    I appreciate your help with my for loops. I have changed them to enumerates so I can have access to the indices just in case I want/need them, but I am still having the same issue with my code. – liveFreeOrπHard Sep 12 '22 at 19:06
  • It works, the problem is that the original `y` is filled with `None`, and the value of `y[i]` is updated **only** if el > x_thresh. So, `None`s are never replaced by `0`. Solutions could be: put the `y[i] = counter` outside the inner loop, or use `np.zeros`to create `y`. – Ignatius Reilly Sep 12 '22 at 19:44
  • 1
    As a side comment, I would put `counter = 0` right after `for j, x_thresh in enumerate(x):` instead of defining it at the beginning of the code and then resetting at the end of the inner loop. Not just to save one line, but for readability: it puts the iteration algorithm all together. – Ignatius Reilly Sep 12 '22 at 19:50
  • I was able to get it working properly for a simple example!! If I only wanted to count the value as passing the threshold if it stayed over for at least 2 iterations, any idea how I'd implement that? Perhaps with an "and" to check the i+1th element as well? – liveFreeOrπHard Sep 12 '22 at 19:50
  • No sure, because is going to ignore the last element in the streak. – Ignatius Reilly Sep 12 '22 at 19:59
  • 1
    Not sure the most efficient way... but the first solution that comes to my mind is to generate a vector of 0 - 1 (or True - False) representing whether each value passes the threshold, and then counting the consecutives. Some ideas [here](https://stackoverflow.com/questions/6352425/whats-the-most-pythonic-way-to-identify-consecutive-duplicates-in-a-list) and [here](https://stackoverflow.com/questions/39340345/how-to-count-consecutive-duplicates-in-a-python-list) – Ignatius Reilly Sep 12 '22 at 20:09

0 Answers0