Convert non-ASCII chars to Space

Question

I want to replace all occurrences of non-ASCII chars in Unix to space but group of all the characters should to converted to a single space like :

CHAVEZ MONTAï¿½O   should be converted to CHAVEZ MONTAO<followed by one space>

How can I do so. I used below Perl command :

grep --color='auto' -P -n "[\x80-\xFF]" file.xml

But this is converting one single char in one space this is not what I want.

EDIT1:

I know CHAVEZ MONTAï¿½O converted to CHAVEZ MONTA O makes more sense. But would be better if it is

CHAVEZ MONTAO<followed by one space>

But please suggest for CHAVEZ MONTA O too.

i think the output must be `CHAVEZ MONTA O` – Avinash Raj Dec 22 '14 at 08:42 — Avinash Raj, Dec 22 '14 at 08:42
shouldn't it be `CHAVEZ MONTA O`? – gniourf_gniourf Dec 22 '14 at 08:43 — gniourf_gniourf, Dec 22 '14 at 08:43

Avinash Raj · Answer 1 · 2014-12-22T09:18:45.213

1

Seems like you want something like this,

$ echo 'CHAVEZ MONTAï¿½O' | perl -pe 's/[^[:ascii:]]+/ /g'
CHAVEZ MONTA O

$ echo 'CHAVEZ MONTAï¿½O' | perl -pe 's/([^[:ascii:]]+)(.)/\2 /g'
CHAVEZ MONTAO 

$ echo 'CHAVEZ MONTAï¿½O' | perl -pe 's/ï¿½/ /g'
CHAVEZ MONTA O

$ echo 'CHAVEZ MONTAï¿½O' | perl -pe 's/ï¿½([[:ascii:]])/\1 /g'
CHAVEZ MONTAO

edited Dec 22 '14 at 09:18

answered Dec 22 '14 at 08:43

Avinash Raj

172,303
28
230
274

Also what if I just want to change ï¿½ to one space ? – Mayank Jain Dec 22 '14 at 08:46
Yep, the above changes to one space . see my update for the second one... – Avinash Raj Dec 22 '14 at 08:47
My question is how can I search for ï¿½ , just this group of non-ascii chars in unix? Is it possible? – Mayank Jain Dec 22 '14 at 08:48
just directly give that to the search pattern. – Avinash Raj Dec 22 '14 at 09:19

score 0 · Answer 2 · answered Dec 22 '14 at 08:44

0

Pure Bash:

shopt -s extglob
var="CHAVEZ MONTAï¿½O"
echo "${var//+([^[:ascii:]])/ }"

answered Dec 22 '14 at 08:44

gniourf_gniourf

44,650
9
93
104

score 0 · Answer 3 · answered Dec 22 '14 at 09:18

0

If you use this sed, you can do it like this:

sed -s 's/[\d128-\d255]\+/ /' < INPUTFILE

It replaces all chars > 128 with one whitespace.

answered Dec 22 '14 at 09:18

Biber

709
6
19

Convert non-ASCII chars to Space

3 Answers3