23

I want to remove all types of escape sequences from a list of strings. How can I do this? input:

['william', 'short', '\x80', 'twitter', '\xaa', '\xe2', 'video', 'guy', 'ray']

output:

['william', 'short', 'twitter', 'video', 'guy', 'ray']

http://docs.python.org/reference/lexical_analysis.html#string-literals

Bruce
  • 33,927
  • 76
  • 174
  • 262
  • 2
    The final string object does not carry any information on wether the string literal that contrcuted it contained an escape sequence. If you can't even tell if there are any, how would you "remove" them? – Sven Marnach Nov 13 '11 at 22:30

6 Answers6

42

If you want to strip out some characters you don't like, you can use the translate function to strip them out:

>>> s="\x01\x02\x10\x13\x20\x21hello world"
>>> print(s)
 !hello world
>>> s
'\x01\x02\x10\x13 !hello world'
>>> escapes = ''.join([chr(char) for char in range(1, 32)])
>>> t = s.translate(None, escapes)
>>> t
' !hello world'

This will strip out all these control characters:

   001   1     01    SOH (start of heading)
   002   2     02    STX (start of text)
   003   3     03    ETX (end of text)
   004   4     04    EOT (end of transmission)
   005   5     05    ENQ (enquiry)
   006   6     06    ACK (acknowledge)
   007   7     07    BEL '\a' (bell)
   010   8     08    BS  '\b' (backspace)
   011   9     09    HT  '\t' (horizontal tab)
   012   10    0A    LF  '\n' (new line)
   013   11    0B    VT  '\v' (vertical tab)
   014   12    0C    FF  '\f' (form feed)
   015   13    0D    CR  '\r' (carriage ret)
   016   14    0E    SO  (shift out)
   017   15    0F    SI  (shift in)
   020   16    10    DLE (data link escape)
   021   17    11    DC1 (device control 1)
   022   18    12    DC2 (device control 2)
   023   19    13    DC3 (device control 3)
   024   20    14    DC4 (device control 4)
   025   21    15    NAK (negative ack.)
   026   22    16    SYN (synchronous idle)
   027   23    17    ETB (end of trans. blk)
   030   24    18    CAN (cancel)
   031   25    19    EM  (end of medium)
   032   26    1A    SUB (substitute)
   033   27    1B    ESC (escape)
   034   28    1C    FS  (file separator)
   035   29    1D    GS  (group separator)
   036   30    1E    RS  (record separator)
   037   31    1F    US  (unit separator)

For Python newer than 3.1, the sequence is different:

>>> s="\x01\x02\x10\x13\x20\x21hello world"
>>> print(s)
 !hello world
>>> s
'\x01\x02\x10\x13 !hello world'
>>> escapes = ''.join([chr(char) for char in range(1, 32)])
>>> translator = str.maketrans('', '', escapes)
>>> t = s.translate(translator)
>>> t
' !hello world'
sarnold
  • 102,305
  • 22
  • 181
  • 238
  • 1
    Sorry, that loop just makes me cringe. `escapes = ''.join([chr(char) for char in range(1, 32)])` `s.translate(None, escapes)` – Rebs May 21 '15 at 23:26
  • @AdamGriffiths, that's a nice change. Thanks. – sarnold May 26 '15 at 23:55
  • I like this answer rather than the first one. it works well and very flexible. – weefwefwqg3 Jan 11 '17 at 01:28
  • 2
    This answer is a bit outdated. In Python 3.1 and above you would need to add these two lines after the declaration of `escapes`. `translator = str.maketrans('', '', escapes)` and then `t = s.translate(translator)` – Cold Fish Sep 17 '19 at 06:18
  • 2
    @yvihs, thanks for the fix! I've edited my answer for the newer pythons. – sarnold Sep 19 '19 at 12:43
17

Something like this?

>>> from ast import literal_eval
>>> s = r'Hello,\nworld!'
>>> print(literal_eval("'%s'" % s))
Hello,
world!

Edit: ok, that's not what you want. What you want can't be done in general, because, as @Sven Marnach explained, strings don't actually contain escape sequences. Those are just notation in string literals.

You can filter all strings with non-ASCII characters from your list with

def is_ascii(s):
    try:
        s.decode('ascii')
        return True
    except UnicodeDecodeError:
        return False

[s for s in ['william', 'short', '\x80', 'twitter', '\xaa',
             '\xe2', 'video', 'guy', 'ray']
 if is_ascii(s)]
Fred Foo
  • 355,277
  • 75
  • 744
  • 836
4

You could filter out "words" that are not alphanumeric using a list comprehension and str.isalnum():

>>> l = ['william', 'short', '\x80', 'twitter', '\xaa', '\xe2', 'video', 'guy', 'ray']
>>> [word for word in l if word.isalnum()]
['william', 'short', 'twitter', 'video', 'guy', 'ray']

If you wish to filter out numbers, too, use str.isalpha() instead:

>>> l = ['william', 'short', '\x80', 'twitter', '\xaa', '\xe2', 'video', 'guy', 'ray', '456']
>>> [word for word in l if word.isalpha()]
['william', 'short', 'twitter', 'video', 'guy', 'ray']
johnsyweb
  • 136,902
  • 23
  • 188
  • 247
  • 1
    This is a good answer for many applications, although it should be noted that other non-alphanumerical characters like whitespaces will also suffer the axe. – Dr K Aug 04 '17 at 15:37
3

The following worked for me in Python 3.6:

word = '\t\t\t\t\t\ntest\t msg'
filter = ''.join([chr(i) for i in range(1, 32)])
word.translate(str.maketrans('', '', filter))

Output:

'test msg'

Source here

Jeril
  • 7,858
  • 3
  • 52
  • 69
2

This cannot be done, at least at the broad scope you are asking. As others have mentioned, runtime python doesn't know the difference between the something with escape sequences, and something without.

Example:

print ('\x61' == 'a')

prints True. So there's no way to find the difference between these two strings, unless you try some static analysis of your python script.

Adam Wagner
  • 15,469
  • 7
  • 52
  • 66
0

I had similar issues while converting from hexadimal to String.This is what finally worked in python Example

list_l = ['william', 'short', '\x80', 'twitter', '\xaa', '\xe2', 'video', 'guy', 'ray']
decode_data=[]
for l in list_l:
    data =l.decode('ascii', 'ignore')
    if data != "":
        decode_data.append(data)

# output :[u'william', u'short', u'twitter', u'video', u'guy', u'ray']
royhowie
  • 11,075
  • 14
  • 50
  • 67
AKV
  • 171
  • 1
  • 4
  • 12