Convert Hi-Ansi chars to Ascii equivalent (é -> e)

Question

Is there a routine available in Delphi 2007 to convert the characters in the high range of the ANSI table (>127) to their equivalent ones in pure ASCII (<=127) according to a locale (codepage)?

I know some chars cannot translate well but most can, esp. in the 192-255 range:

À → A
à → a
Ë → E
ë → e
Ç → C
ç → c
– (en dash) → - (hyphen - that can be trickier)
— (em dash) → - (hyphen)

score 30 · Accepted Answer · edited Oct 08 '14 at 17:08

WideCharToMultiByte does best-fit mapping for any characters that aren't supported by the specified character set, including stripping diacritics. You can do exactly what you want by using that and passing 20127 (US-ASCII) as the codepage.

function BestFit(const AInput: AnsiString): AnsiString;
const
  CodePage = 20127; //20127 = us-ascii
var
  WS: WideString;
begin
  WS := WideString(AInput);
  SetLength(Result, WideCharToMultiByte(CodePage, 0, PWideChar(WS),
    Length(WS), nil, 0, nil, nil));
  WideCharToMultiByte(CodePage, 0, PWideChar(WS), Length(WS),
    PAnsiChar(Result), Length(Result), nil, nil);
end;

procedure TForm1.Button1Click(Sender: TObject);
begin
   ShowMessage(BestFit('aÀàËëÇç–—€¢Š'));
end;

Calling that with your examples produces results you're looking for, including the emdash-to-minus case, which I don't think is handled by Jeroen's suggestion to convert to Normalization form D. If you did want to take that approach, Michael Kaplan has a blog post the explicitly discusses stripping diacritics (rather than normalization in general), but it uses C# and an API that was introduces in Vista. You can get something similar using the FoldString api (any WinNT release).

Of course if you're only doing this for one character set, and you want to avoid the overhead from converting to and from a WideString, Padu is correct that a simple for loop and a lookup table would be just as effective.

Thanks Craig. That's a more generic solution than the lookup. It had a typo in the magic number, so I corrected it and used a constant instead. But anyway, it works on D2007 as well as D2009. — Francesca, Dec 14 '09 at 18:20
One thing we noticed with this, is that 'β' (unicode 1E9E latin capital letter sharp s) isn't converted, so we do this beforehand : StringReplace(aStr, 'β', 'SS', [rfReplaceAll]) — PatrickvL, Oct 16 '15 at 16:09
Same thing with Char(539) -> t and Char(537) -> s (also with oxo's answer) — ciuly, Aug 17 '22 at 10:38

score 8 · Answer 2 · answered Mar 19 '13 at 12:52

Just to extend Craig's answer for Delphi 2009:

If you use Delphi 2009 and newer, you can use a more readable code with the same result:

function OStripAccents(const aStr: String): String;
type
  USASCIIString = type AnsiString(20127);//20127 = us ascii
begin
  Result := String(USASCIIString(aStr));
end;

Unfortunately, this code does work only on MS Windows. On Mac, the accents are not replaced by best-fitted characters but by question marks.

Obviously, Delphi internally uses WideCharToMultiByte on Windows whereas on Mac iconv is used (see LocaleCharsFromUnicode in System.pas). The question is if this different behaviour on different OS should be considered as bug and reported to CodeCentral.

iconv does have a `//TRANSLIT` option, but `LocaleCharsFromUnicode()` does not use it. — Remy Lebeau, Jun 25 '15 at 07:10

score 3 · Answer 3 · answered Dec 11 '09 at 22:22

3

I believe your best bet is creating a lookup table.

answered Dec 11 '09 at 22:22

Padu Merloti

3,219
3
33
44

Also, if you're using a decent regex library with delphi, that could be used as well, but it still is kind of a lookup table. – Padu Merloti Dec 11 '09 at 22:28
Thanks Padu. That's what I thought. I'll nevertheless accept Craig's answer because it's more generic. – Francesca Dec 14 '09 at 18:23

score 1 · Answer 4 · edited Oct 08 '14 at 17:08

1

What you are looking for is normalization.

Michael Kaplan wrote a nice blog article about normalization.

It does not immediately solve your problem, but points you in the right direction.

--jeroen

edited Oct 08 '14 at 17:08

Community

1
1

answered Dec 11 '09 at 23:19

Jeroen Wiert Pluimers

23,965
9
74
154

1

NFKD + removal of combining marks works a lot of the time. However, there are characters like `ÆÐØÞßæðøþ` that do not decompose and have to be dealt with manually. – dan04 Jul 02 '10 at 02:30

Convert Hi-Ansi chars to Ascii equivalent (é -> e)

4 Answers4

Linked

Related