How to save classic Delphi string to disk (and read them back)?

Question

I want to achieve a very very basic task in Delphi: to save a string to disk and load it back. It seems trivial but I had problems doing this TWICE since I upgraded to IOUtils (and one more time before that... this is why I took the 'brilliant' decision to upgrade to IOUtils).

I use something like this:

procedure WriteToFile(CONST FileName: string; CONST uString: string; CONST WriteOp: WriteOperation);    
begin
   if WriteOp= (woOverwrite)
   then IOUtils.TFile.WriteAllText (FileName, uString)  //overwrite
   else IOUtils.TFile.AppendAllText(FileName, uString); //append
end;    

Simple right? What could go wrong? Well, I recently stepped into a (another) bug in IOUtils. So, TFile is buggy. The bug is detailed here.

Anyone has can share an alternative (or simply your thoughts/ideas) that is not based on IOUtils and it is known to work? Well... the code above also worked for a while for me... So, I know if difficult to guaranty that a piece of code (no matter how small) will really work!

Also I would REALLY like to have my WriteToFile procedure to save the string to an ANSI file when it is possible (the uString contains only ANSI chars) and as Unicode otherwise.
Then the ReadAFile function should automagically detect the encoding and correctly read the string back.
The idea is that there are still text editors out there that will wrongly open/interpret an Unicode/UTF file. So, whenever possible, give a good old ANSI text file to the user.

So:
- Overwrite/Append
- Save as ANSI when possible
- Memory efficient (don't eat 4GB of ram when the file to load is 2GB)
- Should work with any text file (up to 2GB, obviously)
- No IOUtils (too buggy to be of use)

Answer 1

Then the ReadAFile function should automagically detect the encoding and correctly read the string back.

This is not possible. There exists files that are well-formed if interpreted as any text encoding. For instance see The Notepad file encoding problem, redux.

This means that your goals are unattainable and that you need to change them.

My advice is to do the following:

• Pick a single encoding, UTF-8, and stick to it.
• If the file does not exists, create it and write UTF-8 bytes to it.
• If the file exists, open it, seek to the end, and append UTF-8 bytes.

A text editor that does not understand UTF-8 is not worth supporting. If you feel inclined, include a UTF-8 BOM when you create the file. Use TEncoding.UTF8.GetBytes and TEncoding.UTF8.GetString to encode and decode.

Answer 2

Just use TStringList, until size of file < ~50-100Mb (it depends on CPU speed):

procedure ReadTextFromFile(const AFileName: string; SL: TStringList);
begin
  SL.Clear;
  SL.DefaultEncoding:=TEncoding.ANSI; // we know, that old files has this encoding
  SL.LoadFromFile(AFileName, nil); // let TStringList detect real encoding.
  // if not - it just use DefaultEncoding.
end;

procedure WriteTextToFile(const AFileName: string; const TextToWrite: string);
var
  SL: TStringList;
begin
  SL:=TStringList.Create;
  try
    ReadTextFromFile(AFileName, SL); // read all file with encoding detection
    SL.Add(TextToWrite);
    SL.SaveToFile(AFileName, TEncoding.UTF8); // write file with new encoding.
    // DO NOT SET SL.WriteBOM to False!!!
  finally
    SL.Free;
  end;
end;

Answer 3

The Inifiles unit should support unicode. At least according to this answer: How do I read a UTF8 encoded INI file?

Inifiles are quite commonly used to store strings, integers, booleans and even stringlists.

    procedure TConfig.ReadValues();
    var
        appINI: TIniFile;
    begin
        appINI := TIniFile.Create(ChangeFileExt(Application.ExeName,'.ini'));

        try
            FMainScreen_Top := appINI.ReadInteger('Options', 'MainScreen_Top', -1);
            FMainScreen_Left := appINI.ReadInteger('Options', 'MainScreen_Left', -1);
            FUserName := appINI.ReadString('Login', 'UserName', '');
            FDevMode := appINI.ReadBool('Globals', 'DevMode', False);
        finally
            appINI.Free;
        end;
    end;

    procedure TConfig.WriteValues(OnlyWriteAnalyzer: Boolean);
    var
        appINI: TIniFile;
    begin
        appINI := TIniFile.Create(ChangeFileExt(Application.ExeName,'.ini'));

        try
            appINI.WriteInteger('Options', 'MainScreen_Top', FMainScreen_Top);
            appINI.WriteInteger('Options', 'MainScreen_Left', FMainScreen_Left);
            appINI.WriteString('Login', 'UserName', FUserName);
            appINI.WriteBool('Globals', 'DevMode', FDevMode);
        finally
            appINI.Free;
        end;
    end;

Also see the embarcadero documentation on inifiles: http://docwiki.embarcadero.com/Libraries/Seattle/en/System.IniFiles.TIniFile

Answer 4

Code based on David's suggestions:

{--------------------------------------------------------------------------------------------------
 READ/WRITE UNICODE
--------------------------------------------------------------------------------------------------}

procedure WriteToFile(CONST FileName: string; CONST aString: String; CONST WriteOp: WriteOperation= woOverwrite; WritePreamble: Boolean= FALSE); { Write Unicode strings to a UTF8 file. It can also write a preamble }
VAR
   Stream: TFileStream;
   Preamble: TBytes;
   sUTF8: RawByteString;
   aMode: Integer;
begin
 ForceDirectories(ExtractFilePath(FileName));

 if (WriteOp= woAppend) AND FileExists(FileName)
 then aMode := fmOpenReadWrite
 else aMode := fmCreate;

 Stream := TFileStream.Create(filename, aMode, fmShareDenyWrite);   { Allow read during our writes }
 TRY
  sUTF8 := Utf8Encode(aString);                                     { UTF16 to UTF8 encoding conversion. It will convert UnicodeString to WideString }

  if (aMode = fmCreate) AND WritePreamble then
   begin
    preamble := TEncoding.UTF8.GetPreamble;
    Stream.WriteBuffer( PAnsiChar(preamble)^, Length(preamble));
   end;

  if aMode = fmOpenReadWrite
  then Stream.Position:= Stream.Size;                               { Go to the end }

  Stream.WriteBuffer( PAnsiChar(sUTF8)^, Length(sUTF8) );
 FINALLY
   FreeAndNil(Stream);
 END;
end;


procedure WriteToFile (CONST FileName: string; CONST aString: AnsiString; CONST WriteOp: WriteOperation);
begin
 WriteToFile(FileName, String(aString), WriteOp, FALSE);
end;


function ReadFile(CONST FileName: string): String;  {Tries to autodetermine the file type (ANSI, UTF8, UTF16, etc). Works with UNC paths }
begin
 Result:= System.IOUtils.TFile.ReadAllText(FileName);
end;