Is there an easy way to work around a Delphi utf8-file flaw?

Question

I have discovered (the hard way) that if a file has a valid UTF-8 BOM but contains any invalid UTF8 encodings, and is read by any of the Delphi (2009+) encoding-enabled methods such as LoadFromFile, then the result is a completely empty file with no error indication. In several of my applications, I would prefer to simply lose a few bad encodings, even if I get no error report in this case either.

Debugging reveals that MultiByteToWideChar is called twice, first to get the output buffer size, then to do the conversion. But TEncoding.UTF8 contains a private FMBToWCharFlags value for these calls, and this is initialized with a MB_ERR_INVALID_CHARS value. So the call to get the charcount returns 0 and the loaded file is completely empty. Calling this API without the flag would 'silently drop illegal code points'.

My question is how best to weave through the nest of classes in the Encoding area to work around the fact that this is a private value (and needs to be, because it is a class var for all threads). I think I could add a custom UTF8 encoding, using the guidance in Marco Cantu's Delphi 2009 book. And it could optionally raise an exception if MultiByteToWideChar has returned an encoding error, after calling it again without the flag. But that does not solve the problem of how to get my custom encoding used instead of Tencoding.UTF8.

If I could just set this up as a default for the application at initialization, perhaps by actually modifying the class var for Tencoding.UFT8, this would probably be sufficient.

Of course, I need a solution without waiting to lodge a QC report asking for a more robust design, getting it accepted, and seeing it changed.

Any ideas would be very welcome. And can someone confirm this is still an issue for XE4, which I have not yet installed?

Answer 1

I ran into the MB_ERR_INVALID_CHARS issue when I first updated Indy to support TEncoding, and ended up implementing a custom TEncoding-derived class for UTF-8 handling to avoid specifying MB_ERR_INVALID_CHARS. I didn't think to use a class helper.

However, this issue is not just limited to UTF-8. Any decoding failure of any of the TEncoding classes will result in a blank result, not an exception being raised. Why Embarcadero chose that route, when most of the RTL/VCL uses exceptions instead, is beyond me. Not raising an exception on error caused a fair amount of issues in Indy that had to be worked around.

Answer 2

This can be done pretty simply, at least in Delphi XE5 (have not checked earlier versions). Just instantiate your own TUTF8Encoding:

procedure LoadInvalidUTF8File(const Filename: string);
var
  FEncoding: TUTF8Encoding;
begin
  FEncoding := TUTF8Encoding.Create(CP_UTF8, 0, 0); 
                      // Instead of CP_UTF8, MB_ERR_INVALID_CHARS, 0
  try
    with TStringList.Create do
    try
      LoadFromFile(Filename, FEncoding);
      // ...
    finally
      Free;
    end;
  finally
    FEncoding.Free;
  end;
end;

The only issue here is that the IsSingleByte property for the newly instantiated TUTF8Encoding is then incorrectly set to False, but this property is not currently used anywhere in the Delphi sources.

Answer 3

A partial workaround is to force the UTF8 encoding to suppress MB_ERR_INVALID_CHARS globally. For me, this avoids the need for raising an exception, because I find it makes MultiByteToWideChar not quite 'silent': it actually inserts $fffd characters (Unicode 'replacement character') which I can then find in the cases where this is important. The following code does this:

unit fixutf8;
interface
uses System.Sysutils;
type
  TUTF8fixer = class helper for Tmbcsencoding
  public
    procedure setflag0;
  end;

implementation
procedure TUTF8fixer.setflag0;
{$if CompilerVersion = 31}
asm
  XOR ECX,ECX
  MOV Self.FMBToWCharFlags,ECX
end;
{$else}
begin
  Self.FMBToWCharFlags := 0;
end;
{$endif}

procedure initencoding;
begin
  (Tencoding.UTF8 as TmbcsEncoding).setflag0;
end;

initialization
  initencoding;
end.

A more useful and principled fix would require changing the calls to MultiByteToWideChar not to use MB_ERR_INVALID_CHARS, and to make an initial call with this flag so that an exception could be raised after the load is complete, to indicate that characters will have been replaced.

There are relevant QC reports on this issue, including 76571, 79042 and 111980. The first one has been resolved 'as designed'.

(Edited to work with Delphi Berlin)

Answer 4

Your "global" approach is not really global - it relies upon the assumption that all the code would only use one and the same instance of TUTF8Encoding. The same instance where you hacked the flags field.

But it would not work if one obtain TUTF8Encoding object(s) by other means than TEncoding.GetUTF8, for example in XE2 another method - TEncoding.GetEncoding(CP_UTF8) - would create a new instance of TUTF8Encoding instead of re-using FUTF8 shared one. Or some function might run TUTF8Encode.Create directly.

So i'd suggest two more approaches.

Approach with patching the class implementation, somewhat hacky. You introduce your own class for the sake of obtaining new "fixes" constructor body.

type TMyUTF8Encoding = class(TUTF8Encoding)
  public constructor Create; override;
end;

This constructor would be the copycat of TUTF8Encoding.Create() implementation, except for setting the flag as you want it ( in XE2 it is done by calling another, inherited Create(x,y,z) so u would not need an access to the private field ) instead.

Then you can patch the stock TUTF8Encoding VMT overriding its virtual constructor to that new constructor of yours.

You may read Delphi documentation about "internal formats" and so forth, to get the VMT layout. You would also need calling VirtualProtect (or other platform-specific function) to remove protection from VMT memory area before patching and then to restore it.

Examples to learn from

• How to change the implementation (detour) of an externally declared function
• https://stackoverflow.com/a/1482802/976391

Or you may try using Delphi Detours library, hopefully it can patch virtual constructors. Then... it might be an overkill here to use that rather complex lib for that single goal.

After you hacked the TUTF8Encoding class do call the TEncoding.FreeEncodings to remove the already created shared instances (if any) if any and thus trigger recreating the UTF8 instances with your modifications.

---

Then, if you compile your program as a single monolithic EXE , without using runtime BPL modules, you just can copy the SysUtils.pas sources to your application folder and then to include that local copy into your project explicitly.

How to patch a method in Classes.pas

There you would change the very TUTF8Encoding implementation as you see fit in the sources and Delphi would use it.

This brain-deadly simplistic (hence - equally reliable) approach would not work though if your projects would be built to reuse rtlNNN.bpl runtime package instead of being monolithic.