removing newlines from messy strings in pandas dataframe cells?

Question

I've used multiple ways of splitting and stripping the strings in my pandas dataframe to remove all the '\n'characters, but for some reason it simply doesn't want to delete the characters that are attached to other words, even though I split them. I have a pandas dataframe with a column that captures text from web pages using Beautifulsoup. The text has been cleaned a bit already by beautifulsoup, but it failed in removing the newlines attached to other characters. My strings look a bit like this:

"hands-on\ndevelopment of games. We will study a variety of software technologies\nrelevant to games including programming languages, scripting\nlanguages, operating systems, file systems, networks, simulation\nengines, and multi-media design systems. We will also study some of\nthe underlying scientific concepts from computer science and related\nfields including"

Is there an easy python way to remove these "\n" characters?

score 112 · Accepted Answer · edited Aug 24 '21 at 18:02

112

EDIT: the correct answer to this is:

df = df.replace(r'\n',' ', regex=True)

I think you need replace:

df = df.replace('\n','', regex=True)

Or:

df = df.replace('\n',' ', regex=True)

Or:

df = df.replace(r'\\n',' ', regex=True)

Sample:

text = '''hands-on\ndev nologies\nrelevant scripting\nlang
'''
df = pd.DataFrame({'A':[text]})
print (df)
                                                   A
0  hands-on\ndev nologies\nrelevant scripting\nla...

df = df.replace('\n',' ', regex=True)
print (df)
                                                A
0  hands-on dev nologies relevant scripting lang

edited Aug 24 '21 at 18:02

garlic_rat

99
6

answered May 28 '17 at 13:22

jezrael

822,522
95
1,334
1,252

Thanks! For some reason it still doesn't seem to work, i've copied the code exactly and tried to run it. Then I've tried to only apply it to the column it needs to be applied to: no luck. Then i've tried a for loop looping over every cell of that column and calling cell.replace('n', ''). Also nothing. I can't use regex in the for loop replace call since I'm iterating over individual strings. – Calvin May 28 '17 at 13:32
I think this task are data dependent and are really hard (if simple replace doesnt work). So is possible create [pickle](http://pandas.pydata.org/pandas-docs/stable/io.html#io-pickle) and send it to my email in my profile? If data are confidental, is possible anonymize it? – jezrael May 28 '17 at 13:42
2

For anyone who finds this later: I think the expression should be: `df.replace(r'\n', ' ', regex=True)`, i.e., don't use two backslashes. – Will May 26 '21 at 19:33

score 19 · Answer 2 · answered Jul 31 '20 at 21:29

19

df.replace(to_replace=[r"\\t|\\n|\\r", "\t|\n|\r"], value=["",""], regex=True, inplace=True)

worked for me.

Source:

https://gist.github.com/smram/d6ded3c9028272360eb65bcab564a18a

answered Jul 31 '20 at 21:29

LinuxUser

635
1
7
19

not sure what value=["",""] does? Gist also doesn't explain. But it worked like charm. – Michel K May 05 '21 at 20:19

score 15 · Answer 3 · answered May 14 '21 at 23:23

15

To remove carriage return (\r), new line (\n) and tab (\t)

df = df.replace(r'\r+|\n+|\t+','', regex=True)

answered May 14 '21 at 23:23

Emeeus

5,072
2
25
37

score 9 · Answer 4 · answered Oct 29 '17 at 12:31

9

in messy data it might to be a good idea to remove all whitespaces df.replace(r'\s', '', regex = True, inplace = True).

answered Oct 29 '17 at 12:31

Pawel Piela

301
3
5

score 6 · Answer 5 · answered Jun 12 '19 at 21:56

6

   df = 'Sarah Marie Wimberly So so beautiful!!!\nAbram Staten You guys look good man.\nTJ Sloan I miss you guys\n'

   df = df.replace(r'\\n',' ', regex=True)

This worked for the messy data I had.

answered Jun 12 '19 at 21:56

Harshini Kanukuntla

91
1
3

removing newlines from messy strings in pandas dataframe cells?

5 Answers5

Linked