Iterating and averaging over some messy data

Question

I have some messy data that I'm passing through a function. The function below tries to take an average. Sometimes items in the list aren't numbers, and will throw an error.

I tried to use regex to replace non numeric characters, but some stuff is still getting through. Any time a bad value shows up (due to messy data) I just want a 0 recorded for that item in the list.

def mean(vals):
    if len(vals) == 0:
        return 0.0

    for val in vals:
        val = re.sub("[^0-9.]", "", str(val))
    print vals
    vals = [float(val) for val in vals]
    return sum(vals) / len(vals)

I'm printing the list of vals just to see where I'm throwing an error. The last vals list is:

['</a>']

How is this possible, given I've regexed everything that isn't a number or a period?

Are you getting this data from an HTML source? – Jon Clements Jun 05 '15 at 20:05 — Jon Clements, Jun 05 '15 at 20:05

score 1 · Answer 1 · edited May 23 '17 at 12:05

1

You are not changing vals's value by (see Modification of the list items in the loop (python))

val = re.sub("[^0-9.]", "", str(val))

Instead, you could loop through the index of the list and change its content directly.

edited May 23 '17 at 12:05

Community

1
1

answered Jun 05 '15 at 20:01

zw324

26,764
16
85
118

Jon Clements · Accepted Answer · 2015-06-05T20:08:50.453

1

Instead of an re.sub, use try/except...

def mean(vals):
    total = 0.0
    length = 0
    for val in vals:
        try:
            total += float(val)
        except (ValueError, TypeError):
             pass
        length += 1
    return total / length if length else 0.0

edited Jun 05 '15 at 20:08

answered Jun 05 '15 at 20:02

Jon Clements

138,671
33
247
280

1

@ZiyaoWei that's not the way I interpreted *I just want a 0 recorded for that item in the list* – Jon Clements Jun 05 '15 at 20:04
Mmm, you are right - OP's words and code doesn't match. – zw324 Jun 05 '15 at 20:05
Sorry, you're right about the inconsistency in the code. I didn't want to return 0 for the mean, just 0 for that item, then calculate the mean for all items in list. – Chris J. Vargo Jun 05 '15 at 20:10

score 1 · Answer 3 · answered Jun 05 '15 at 20:13

1

You are not changing your list in the for loop, you are just setting a variable inside the loop scope that doesn't reflects on the list.

To change your list you should do something like this:

>>> vals = [re.sub("[^0-9.]", "", str(val)) for val in vals]

answered Jun 05 '15 at 20:13

Rodrigo López

4,039
1
19
26

Iterating and averaging over some messy data

3 Answers3