Fast Pythonic way to turn many lists of strings into lists of floats while catching ValueErrors

Question

I have about 50 million lists of strings in Python like this one:

["1", "1.0", "", "foobar", "3.0", ...]

And I need to turn these into a list of floats and Nones like this one:

[1.0, 1.0, None, None, 3.0, ...]

Currently I use some code like:

def to_float_or_None(x):
    try:
        return float(x)
    except ValueError:
        return None

result = []
for record in database:
    result.append(map(to_float_or_None, record))

The to_float_or_None function is taking in total about 750 seconds (according to cProfile)... Is there a faster way to perform this conversion from a list of strings to a list of floats/Nones?

Update
I had identified the to_float_or_None function as the main bottleneck. I can not find a significant difference in speed between using map and using list comprehensions. I applied Paulo Scardine's tip to check the input, and it already saves 1/4 of the time.

def to_float_or_None(x):
    if not(x and x[0] in "0123456789."):
        return None
    try:
        return float(x)
    except:
        return None

The use of generators was new to me, so thank you for the tip Cpfohl and Lattyware! This indeed speeds up the reading of the file even more, but I was hoping to save some memory by converting the strings to floats/Nones.

Answer 1

Edit: I have just realised I misread the question, and we are talking about a list of lists, not just a list. Updated to suit that.

You can use a list comprehension here to produce something a bit faster and nicer to read:

def to_float_or_None(x):
    try:
        return float(x)
    except ValueError:
        return None

database = [["1", "1.0", "", "foobar", "3.0"], ["1", "1.0", "", "foobar", "3.0"]]

result = [[to_float_or_None(item) for item in record] for record in database]

Giving us:

[[1.0, 1.0, None, None, 3.0], [1.0, 1.0, None, None, 3.0]]

Edit: As noted by Paolo Moretti in the comments, if you want the absolute fastest result, then using map may be faster as we are not using a lambda function:

def to_float_or_None(x):
    try:
        return float(x)
    except ValueError:
        return None

database = [["1", "1.0", "", "foobar", "3.0"], ["1", "1.0", "", "foobar", "3.0"]]

result = [list(map(to_float_or_None, record)) for record in database]

Giving us the same result. I would note, however, that premature optimization is a bad thing. If you have identified this as a bottleneck in your application, then fair enough, but otherwise stick with the more readable over the fast.

We still use a list comprehension for the outer loop as we would need a lambda function to use map again given it's dependence on record:

result = map(lambda record: map(to_float_or_None, record), database)

Naturally, if you want to evaluate these lazily, you can use generator expressions:

((to_float_or_None(item) for item in record) for record in database)

Or:

(map(to_float_or_None, record) for record in database)

This would be the preferred method unless you need the entire list at once.

Answer 2

I don't know about the performance aspect, but this should work for your case.

list_floats = [to_float_or_None(item) for item in original_list]

Answer 3

Or, if you really have that much data in lists, maybe use something like pandas Series and apply() a lambda function to convert:

import pandas,re

inlist = ["1", "1.0", "", "foobar", "3.0"] # or however long...
series = pandas.Series(inlist)
series.apply(lambda x: float(x) if re.match("^\d+?(\.\d+?)*$",x) else None)

Out[41]: 
0     1
1     1
2   NaN
3   NaN
4     3

Lots of other advantages - not least for specifying afterwards how you want to handle those missing values...

Answer 4

The answers given thus far don't really fully answer the question. try...catch vs a validating if then can result in different performance (see: https://stackoverflow.com/a/5591737/456188). To summarize that answer: depends on the ratio of failures to successes and the MEASURED time of a failure and success in both cases. Basically we can't answer this, but we can tell you how to:

1. Look at a few representative cases to get a ratio.
2. Write an if/then that tests the same as the try/catch optimize it and then measure how long it takes both version of the to_float_or_None to fail 100 times and measure how long it takes both versions of the to_float_or_None to succeed 100 times.
3. Do a little math to figure out which will be faster.

Side note about the list comprehension issue:

Depending on whether the you want to be able to index the results of this, or whether you just want to iterate over it a generator expression would actually be even better than a list comprehension (just replace the [ ] characters with ( ) characters).

It takes essentialy no time to create, and the actual execution of to_float_or_None (which is the expensive part) can be delayed until the result it needed.

This is useful for many reasons, but won't work if you're going to need to index it. It will however, allow you to zip the original collection with the generator so you can still have access to the original string along with its float_or_none result.