Removing diacritical marks from a Greek text in an automatic way

Question

I have a decompiled stardict dictionary in the form of a tab file

κακός <tab> bad

where <tab> signifies a tabulation.

Unfortunately, the way the words are defined requires the query to include all diacritical marks. So if I want to search for ζῷον, I need to have all the iotas and circumflexes correct.

Thus I'd like to convert the whole file so that the keyword has the diacritic removed. So the line would become

κακος <tab> <h3>κακός</h3> <br/> bad

I know I could read the file line by line in bash, as described here [1]

while read line           
do           
    command           
done <file 

But what is there any way to automatize the operation of converting the line? I heard about iconv [2] but didn't manage to achieve the desired conversion using it. I'd best like to use a bash script.

---

Besides, is there an automatic way of transliterating Greek, e.g. using the method Perseus has?

---

/edit: Maybe we could use the Unicode codes? We can notice that U+1F0x, U+1F8x for x < 8, etc. are all variants of the letter α. This would reduce the amount of manual work. I'd accept a C++ solution as well.

[1] http://en.kioskea.net/faq/1757-how-to-read-a-file-line-by-line
[2] How to remove all of the diacritics from a file?

Answer 1

I'm not so familiar with Ancient Greek as I am with Modern Greek (which only really uses two diacritics)

However I went through the vowels and found out which combined with diacritics. This gave me the following list:

ἆἂᾶὰάἀἄ 
ἒὲέἐἔ 
ἦἢῆὴήἠἤ 
ἶἲῖὶίἰἴ 
ὂὸόὀὄ 
ὖὒῦὺύὐὔ 
ὦὢῶὼώὠὤ  

I saved this list as a file and passed it to this sed

cat test.txt | sed -e 's/[ἆἂᾶὰάἀἄ]/α/g;s/[ἒὲέἐἔ]/ε/g;s/[ἦἢῆὴήἠἤ]/η/g;s/[ἶἲῖὶίἰἴ]/ι/g;s/[ὂὸόὀὄ]/ο/g;s/[ὖὒῦὺύὐὔ]/υ/g;s/[ὦὢῶὼώὠὤ]/ω/g'

^{Credit to hungnv}

It's a simple sed. It takes each of the options and replaces it with the unmarked character. The result of the above command is:

ααααααα
εεεεε
ηηηηηηη
ιιιιιιι
οοοοο
υυυυυυυ
ωωωωωωω

---

Regarding transliterating the Greek: the image from your post is intended to help the user type in Greek on the site you took it from using similar glyphs, not always similar sounds. Those are poor transliterations. e.g. β is most often transliterated as v. ψ is ps. φ is ph, etc.

Answer 2

You can remove diacritics from a string relatively easily using Perl:

$_=NFKD($_);s/\p{InDiacriticals}//g;

for example:

$ echo 'ὦὢῶὼώὠὤ ᾪ' | perl -CS -MUnicode::Normalize -pne '$_=NFKD($_);s/\p{InDiacriticals}//g'
ωωωωωωω Ω

This works as follows:

• The -CS enables UTF8 for Perl's stdin/stdout
• The -MUnicode::Normalize loads a library for Unicode normalisation
• -e executes the script from the command line; -n automatically loops over lines in the input; -p prints the output automatically
• NFKD() translates the line into one of the Unicode normalisation forms; this means that accents and diacritics are decomposed into separate characters, which makes it easier to remove them in the next step
• s/\p{InDiacriticals}//g removes all characters that Unicoded denotes as diacritical marks

This should in fact work for removing diacritics etc for all scripts/languages that have good Unicode support, not just Greek.