Save Hex value or UTF-16 [LE] inside a file in Delphi 7 [Error in using these codes]

Question

Continuing from my previous question: How to save UTF-16 (Little Endian) and String value inside a file in Delphi 7?

When I use the following code from this answer:

uses
  Classes;
    
const
  BOM: WideChar = $FEFF;
var
  W: WideString;
  FS: TFileStream;
begin
  W := 'ᗿABC';
  FS := TFileStream.Create('text.txt', fmCreate);
  try
    FS.WriteBuffer(BOM, Sizeof(BOM));
    FS.WriteBuffer(PWideChar(W)^, Length(W) * Sizeof(WideChar));
  finally
    FS.Free;
  end;
end;

I have these problems:

1. when I used this: W := 'ᗿABC'; my IDE showed W := '?ABC'; Can I fix this problem? Because [?] is not my goal

2. When running the code, I get the error in the image:

Answer 1

1. The code editor in Delphi 7 does not support Unicode characters very well, if at all. You really should stop using a 25-year-old compiler and upgrade.

   In any case, try this instead:

   W := #$15FF'ABC';
   

   Or, you may have to resort to something more like this:

   W := WideChar($15FF) + WideString('ABC');
   

2. Try type-casting the integer value to WideChar:

   const
     BOM: WideChar = WideChar($FEFF);
   

   Otherwise, use Word instead of WideChar:

   const
     BOM: Word = $FEFF;
   

Answer 2

Using Delphi 7 kind of works with Unicode in WideStrings, but it is not consistent:

var
  RegExprWLineSeparators: WideString;
begin
  // In the following line the literal ends up producing ASCII question marks for the 5th and 6th character.
  RegExprWLineSeparators:= #$d#$a#$b#$c+ WideChar($2028)+ WideChar($2029)+ #$85;

  // But assigning characters individually will make both correct - so first do the one above
  // (or provide anything, because you want to reassign it anyway) and later make it per character.
  RegExprWLineSeparators[5]:= WideChar($2028);
  RegExprWLineSeparators[6]:= WideChar($2029);

A few characters aren't even assignable this way (neither via text literal, nor via ordinal literal), so you use a different approach to test against these:

var
  sText: Widestring;
begin
  sText:= <something>;

  // Checking if the first character is a UTF-16 BE or LE BOM
  case Word(sText[1]) of
    $FEFF,
    $FFFE: Delete( sText, 1, 1 );  // Remove such a character
  end;

Thumb of rules are:

• use Words and cast them to WideChar when using text literals
• use Word over WideChar when comparing/checking
• noncharacters (like U+FFFE and U+FFFF) are usually unassignable

Using Delphi 7 for Unicode as a beginner should be avoided - do this when you're confident with Unicode, UTF-16 and Pascal in general. I started this since Windows 2000 on Delphi 5 and later continued with Delphi 7, having experiences like these in different occasions (regular expressions, amongst others).

As an alternative to a dated Delphi version you could try the free Lazarus IDE for FPC - it uses UTF-8 as an approach to Unicode and should treat/support text literals in code much better. The IDE even looks like the robust Delphi 7 one.