Remove all special characters and case from string in bash

Question

I am writing a bash script that needs to parse filenames.

It will need to remove all special characters (including space): "!?.-_ and change all uppercase letters to lowercase. Something like this:

Some_randoM data1-A
More Data0

to:

somerandomdata1a
moredata0

I have seen lots of questions to do this in many different programming languages, but not in bash. Is there a good way to do this?

Lowercase example here: http://stackoverflow.com/questions/2264428/converting-string-to-lower-case-in-bash-shell-scripting — Jesan Fafon, May 22 '14 at 20:17
Remove spaces here: http://stackoverflow.com/questions/369758/how-to-trim-whitespace-from-bash-variable — Jesan Fafon, May 22 '14 at 20:18
More generally, Bash regular expressions: http://www.tldp.org/LDP/abs/html/bashver3.html#REGEXMATCHREF — Jesan Fafon, May 22 '14 at 20:19

score 79 · Accepted Answer · edited May 22 '14 at 22:14

79

cat yourfile.txt | tr -dc '[:alnum:]\n\r' | tr '[:upper:]' '[:lower:]'

The first tr deletes special characters. d means delete, c means complement (invert the character set). So, -dc means delete all characters except those specified. The \n and \r are included to preserve linux or windows style newlines, which I assume you want.

The second one translates uppercase characters to lowercase.

edited May 22 '14 at 22:14

mklement0

382,024
64
607
775

answered May 22 '14 at 20:37

Dan Bliss

1,694
13
10

14

Just if you (just like me) came here looking for a solution the get rid of characters like \r \n or ^C in a variable var2=`echo $var | tr -d '[:cntrl:]'` is the solution. Thanks to this answer I figured this out. – Arsen Mar 19 '16 at 18:04

score 36 · Answer 2 · edited Apr 04 '19 at 18:22

36

Pure BASH 4+ solution:

$ filename='Some_randoM data1-A'
$ f=${filename//[^[:alnum:]]/}
$ echo "$f"
SomerandoMdata1A
$ echo "${f,,}"
somerandomdata1a

A function for this:

clean() {
    local a=${1//[^[:alnum:]]/}
    echo "${a,,}"
}

Try it:

$ clean "More Data0"
moredata0

edited Apr 04 '19 at 18:22

Mike Q

6,716
5
55
62

answered May 22 '14 at 20:45

gniourf_gniourf

44,650
9
93
104

4

+1; the `,,` operator (convert entire string to lowercase) requires `bash 4+`. – mklement0 May 22 '14 at 21:55

score 20 · Answer 3 · answered Mar 21 '17 at 00:07

20

if you are using mkelement0 and Dan Bliss approach. You can also look into sed + POSIX regular expression.

cat yourfile.txt | sed 's/[^a-zA-Z0-9]//g'

Sed matches all other characters that are not contained within the brackets except letters and numbers and remove them.

answered Mar 21 '17 at 00:07

Unwastable

629
4
5

1

Nice answer! You saved my life! – Johnny Aug 28 '19 at 04:43
Very nice. This is useful if you want to use regex. – Partha Prateem Patra Dec 02 '22 at 13:02

score 17 · Answer 4 · answered Aug 03 '17 at 14:17

17

I've used tr to remove any characters that are not part of [:print:] class

cat file.txt | tr -dc '[:print:]'

or

echo "..." | tr -dc '[:print:]'

Additionally you might want to | (pipe) the output to od -c to confirm the result

cat file.txt | tr -dc '[:print:]' | od -c

answered Aug 03 '17 at 14:17

luka5z

7,525
6
29
52

`od` was useful for me – Sohail Si Dec 11 '20 at 14:22

Remove all special characters and case from string in bash

4 Answers4

Linked