How to convert column with dtype as object to string in Pandas Dataframe

Question

When I read a csv file to pandas dataframe, each column is cast to its own datatypes. I have a column that was converted to an object. I want to perform string operations for this column such as splitting the values and creating a list. But no such operation is possible because its dtype is object. Can anyone please let me know the way to convert all the items of a column to strings instead of objects?

I tried several ways but nothing worked. I used astype, str(), to_string etc.

a=lambda x: str(x).split(',')
df['column'].apply(a)

or

df['column'].astype(str)

Sometimes, string operations fail when there are unrecognized characters present. Paste your data into notepad and see if there is weird symbols where a blank space (or something else) is expected. — Dimanjan, Apr 13 '22 at 07:20
everything here is outdated, answer in duplicate question: `df['id'] = df['id'].astype("string")` — AJ AJ, Nov 08 '22 at 10:18

score 75 · Answer 1 · answered Jun 22 '17 at 23:10

75

since strings data types have variable length, it is by default stored as object dtype. If you want to store them as string type, you can do something like this.

df['column'] = df['column'].astype('|S80') #where the max length is set at 80 bytes,

or alternatively

df['column'] = df['column'].astype('|S') # which will by default set the length to the max len it encounters

answered Jun 22 '17 at 23:10

Siraj S.

3,481
3
34
48

19

which python version are you using? it does not work for me – VMEscoli Mar 09 '18 at 19:52
4

got `TypeError: data type "bytes256" not understood`, any suggestion why? – Jia Gao May 30 '19 at 22:59
Since `pandas` inherits almost the entire `numpy`'s type system (apart from `category`) please refer to https://docs.scipy.org/doc/numpy/reference/arrays.scalars.html#built-in-scalar-types for more information about type shortcuts. – ayorgo Jan 10 '20 at 19:29
1

Works in Python 3.8.2 – gies0r May 29 '20 at 00:52
1

this ends with an error with non-latin characters for me (like `á`) – Michal Skop Feb 15 '23 at 17:59

score 54 · Answer 2 · edited May 23 '17 at 12:18

54

Did you try assigning it back to the column?

df['column'] = df['column'].astype('str')

Referring to this question, the pandas dataframe stores the pointers to the strings and hence it is of type 'object'. As per the docs ,You could try:

df['column_new'] = df['column'].str.split(',')

edited May 23 '17 at 12:18

Community

1
1

answered Nov 27 '15 at 12:51

Hypothetical Ninja

3,920
13
49
75

88

Yeah I tried that. The datatype of that column remained as object even after trying that. – Pranav Nov 27 '15 at 14:54
could you paste a sample of your dataframe? – Hypothetical Ninja Nov 27 '15 at 15:25
I have edited the answer, please check if it works – Hypothetical Ninja Nov 27 '15 at 15:39
16

Both of them dont work :( – ihmpall Oct 07 '16 at 10:39
4

http://stackoverflow.com/questions/21018654/strings-in-a-dataframe-but-dtype-is-object – Keith May 16 '17 at 21:32

score 34 · Answer 3 · answered Mar 30 '18 at 10:58

Not answering the question directly, but it might help someone else.

I have a column called Volume, having both - (invalid/NaN) and numbers formatted with ,

df['Volume'] = df['Volume'].astype('str')
df['Volume'] = df['Volume'].str.replace(',', '')
df['Volume'] = pd.to_numeric(df['Volume'], errors='coerce')

Casting to string is required for it to apply to str.replace

pandas.Series.str.replace
pandas.to_numeric

score 6 · Answer 4 · answered Aug 10 '17 at 15:09

6

You could try using df['column'].str. and then use any string function. Pandas documentation includes those like split

answered Aug 10 '17 at 15:09

koshmaster

329
3
4

2

Nope, pandas will store the pointer to the string and the final column type will be 'object' – asa Nov 21 '19 at 15:05
7

I believe pandas will ALWAYS store string columns as objects – koshmaster Nov 21 '19 at 15:50

How to convert column with dtype as object to string in Pandas Dataframe

or

4 Answers4

Linked

Related