31

I have this line in my code which converts my data to numeric...

data["S1Q2I"] = data["S1Q2I"].convert_objects(convert_numeric=True)

The thing is that now the new pandas release (0.17.0) said that this function is deprecated.. This is the error:

FutureWarning: convert_objects is deprecated.  
Use the data-type specific converters pd.to_datetime, 
pd.to_timedelta and pd.to_numeric. 
data["S3BD5Q2A"] = data["S3BD5Q2A"].convert_objects(convert_numeric=True)

So, I went to the new documentation and I couldn't find any examples of how to use the new function to convert my data...

It only says this:

"DataFrame.convert_objects has been deprecated in favor of type-specific functions pd.to_datetime, pd.to_timestamp and pd.to_numeric (new in 0.17.0) (GH11133)."

Any help would be nice!

Hooked
  • 84,485
  • 43
  • 192
  • 261
guidebortoli
  • 659
  • 3
  • 8
  • 16
  • 1
    You mean [this](http://pandas.pydata.org/pandas-docs/stable/generated/pandas.to_timedelta.html) and [this](http://pandas.pydata.org/pandas-docs/stable/generated/pandas.to_numeric.html#pandas.to_numeric)? What are you expecting this to do/not do? – EdChum Oct 14 '15 at 13:21
  • 7
    It's just this: `data['S1Q2I'] = pd.to_numeric(data['S1Q2I'])` – Evan Wright Oct 14 '15 at 14:03
  • Thanks! Now I understood!! – guidebortoli Oct 15 '15 at 12:12
  • 6
    What if I want the entire DataFrame converted if it can be? .convert_objects will act on a DataFrame, but .to_numeric only acts on a Series. – Michael Currie Dec 27 '15 at 05:22
  • 4
    Same request. How do you convert an entire DataFrame ? – PBrockmann Jan 07 '16 at 16:48
  • 3
    The newly preferred methods simply do not capture the functionality of `.convert_objects`, which can _infer_ datatypes. This is CRUCIAL if you don't know in advance the types of your columns. – abalter Sep 19 '16 at 10:58
  • "The newly preferred methods simply do not capture the functionality of .convert_objects" is there an alternative non-deprecated method? – Chip McCormick Mar 16 '20 at 04:48
  • data["S1Q2I"] = data["S1Q2I"].astype(int) did not work? – P.R. Jun 18 '20 at 06:15

5 Answers5

22

As explained by @EvanWright in the comments,

data['S1Q2I'] = pd.to_numeric(data['S1Q2I'])

is now the prefered way of converting types. A detailed explanation in of the change can be found in the github PR GH11133.

Hooked
  • 84,485
  • 43
  • 192
  • 261
10

You can effect a replacement using apply as done here. An example would be:

>>> import pandas as pd
>>> a = pd.DataFrame([{"letter":"a", "number":"1"},{"letter":"b", "number":"2"}])
>>> a.dtypes
letter    object
number    object
dtype: object
>>> b = a.apply(pd.to_numeric, errors="ignore")
>>> b.dtypes
letter    object
number     int64
dtype: object
>>>

But it sucks in two ways:

  1. You have to use apply rather than a non-native dataframe method
  2. You have to copy to another dataframe--can't be done in place. So much for use with "big data."

I'm not really loving the direction pandas is going. I haven't used R data.table much, but so far it seems superior.

I think a data table with native, in-place type conversion is pretty basic for a competitive data analysis framework.

Community
  • 1
  • 1
abalter
  • 9,663
  • 17
  • 90
  • 145
5

It depends on which version of Pandas...... if you have Pandas's version 0.18.0 this type will work ........

df['col name'] = df['col name'].apply(pd.to_numeric, errors='coerce')

another versions ........ 

df['col name']=df.col name .astype(float)
Stephen Rauch
  • 47,830
  • 31
  • 106
  • 135
Samer
  • 51
  • 1
  • 1
4

If you convert all columns to numeric at once, this code may work.

data = data.apply(pd.to_numeric, axis=0)
user8866085
  • 71
  • 1
  • 5
1

You can get it to apply correctly to a particular variable name in a dataframe without having to copy into a different dataframe like this: 

>>> import pandas as pd
>>> a = pd.DataFrame([{"letter":"a", "number":"1"},{"letter":"b", "number":"2"}])
>>> a.dtypes
letter    object
number    object
dtype: object
>>> a['number'] = a['number'].apply(pd.to_numeric, errors='coerce')
>>> a.dtypes
letter    object
number     int64
dtype: object

An example based on the original question above would be something like:

data['S1Q2I'] = data['S1Q2I'].apply(pd.to_numeric, errors='coerce')

This works the same way as your original:

data['S1Q2I'] = data['S1Q2I'].convert_objects(convert_numeric=True)

in my hands, anyway....

This doesn't address the point abalter made about inferring datatypes which is a little above my head I'm afraid!

magsmanston
  • 11
  • 1
  • Ahh I've just read the discussion [here](https://github.com/pandas-dev/pandas/issues/11221) and appreciate the limitations, now. Most annoying. But, if you have a particular dataframe column you want to convert, this is how you could do it. – magsmanston Dec 07 '16 at 08:23