How to remove non-ascii chars using sed

Question

I want to remove non-ascii chars from some file. I have already tried these many regexs.

sed -e 's/[\d00-\d128]//g'  # not working

cat /bin/mkdir | sed -e 's/[\x00-\x7F]//g' >/tmp/aa

but this file contains some non-ascii chars.

[root@asssdsada ~]$ hexdump /tmp/aa |more
          00 01 02 03 04 05 06 07 - 08 09 0A 0B 0C 0D 0E 0F  0123456789ABCDEF

00000000  45 4C 46 B0 F0 73 38 C0 - C0 BC BC FF FF 61 61 61  ELF..s8......aaa
00000010  A0 A0 50 E5 74 64 50 57 - 50 57 50 57 D4 D4 51 E5  ..P.tdPWPWPW..Q.
00000020  74 64 6C 69 62 36 34 6C - 64 6C 69 6E 75 78 78 38  tdlib64ldlinuxx8
00000030  36 36 34 73 6F 32 47 4E - 55 42 C8 C0 80 70 69 42  664so2GNUB...piB
00000040  44 47 BA E3 92 43 45 D5 - EC 46 E4 DE D8 71 58 B9  DG...CE..F...qX.
00000050  8D F1 EA D3 EF 4B 86 FC - A9 DA 79 ED 63 B5 51 92  .....K....y.c.Q.
00000060  BA 6C FC D1 69 78 30 ED - 74 F1 73 95 CC 85 D2 46  .l..ix0.t.s....F
00000070  A5 B4 6C 67 DA 4A E9 9A - 4B 58 77 A4 37 80 C0 4F  ..lg.J..KXw.7..O
00000080  F3 E9 B2 77 65 97 74 F9 - A2 C0 F2 CC 4A 9C 58 A1  ...we.t.....J.X.

Thor · Accepted Answer · 2013-02-28T11:57:35.597

21

This doesn't seem to work with sed. Perhaps tr will do?

tr -d '\200-\377'

Or with the complement:

tr -cd '\000-\177'

edited Feb 28 '13 at 11:57

answered Feb 28 '13 at 11:36

Thor

45,082
11
119
130

@EladTabak: It should work. Can you produce an example where is does not work? – Thor Feb 17 '17 at 16:33
On macOS High Sierra, I get this error: `tr: Illegal byte sequence`. – codeforester Oct 15 '18 at 18:11
1

@codeforester: I tested this with tr from GNU coreutils – Thor Oct 15 '18 at 18:22
1

tr -cd '\001-\177' removes the NULL character do, worth removing, many tools such as grep recognize input as binary if it contains nulls. Binary file (standard input) matches. – ericcurtin Oct 11 '19 at 10:50

score 8 · Answer 2 · answered Jul 18 '13 at 15:33

8

Did you try

cat /bin/mkdir | tr -cd "[:print:]"

I think it solves the problem ?

If only text content interest you, you can also use

cat /bin/mkdir | strings

answered Jul 18 '13 at 15:33

sebtic

198
1
6

score 3 · Answer 3 · edited May 23 '17 at 12:02

3

Do you know what encoding the file is currently using? If so, you can use iconv to convert it. It's a utility to convert from one character encoding to another. So if the original file is in UTF-8 and you want to convert to ASCII you can use the following:

iconv -f utf8 -t ascii <inputfile>

The file command on the input file might tell you the current encoding.

Interestingly, there's a command called enca which will do its best to determine the character encoding being used if you know the language of the contents of the file.

This other question might be the answer.

edited May 23 '17 at 12:02

Community

1
1

answered Feb 28 '13 at 10:56

chooban

9,018
2
20
36

I am working on linux system, which is having very limited pool of commands and 'iconv' is not available. – user87005 Feb 28 '13 at 11:00
Looking at the output from hexdump, is this a binary file? (Guessing from ELF at the start) If so, what's the purpose of removing non-ascii characters? The binary will be corrupted. – chooban Feb 28 '13 at 11:01
it is just an example, friend. – user87005 Feb 28 '13 at 11:02
Ah, cool. I've added a link to a related question which might solve your problems. – chooban Feb 28 '13 at 11:04
1

same is working fine with this perl command, But I need sed. cat /bin/mkdir | perl -ne 's/[^[:ascii:]]//g;print $_;' – user87005 Feb 28 '13 at 11:06
enca also not available on my machine. % enca enca: Command not found. – user87005 Feb 28 '13 at 11:08
Rather than cat'ing the file, have you tried just passing it to sed as an argument? – chooban Feb 28 '13 at 11:15

score 1 · Answer 4 · answered May 22 '20 at 00:20

The solutions offered here did not work for me. Maybe my problem was different, but I needed to strip the ASCII colors and other characters from the otherwise pure ASCII text.

The following worked for me, however:

Stripping Escape Codes from ASCII Text

sed -E 's/\x1b\[[0-9]*;?[0-9]+m//g'

In context (BASH):

$ printf "\e[32;1mhello\e[0m\n"
hello

$ printf "\e[32;1mhello\e[0m\n" | cat -vet
^[[32;1mhello^[[0m$

$ printf "\e[32;1mhello\e[0m\n" | sed -E 's/\x1b\[[0-9]*;?[0-9]+m//g' | cat -vet
hello$

score 0 · Answer 5 · edited Aug 24 '17 at 11:15

0

Try with sed -i option, eg.

sed -i 's/[\d128-\d255]//g' MYFILE.txt

it will replace all non-ascii characters in the file.

edited Aug 24 '17 at 11:15

oz123

27,559
27
125
187

answered Oct 12 '15 at 14:07

Dharmender Rawat

645
8
5

For me at least -- running this on my file removed all lowercase ascii characters – Uncle Iroh May 18 '21 at 20:32

How to remove non-ascii chars using sed

5 Answers5