Encounter encoding problem while copying an email from Thunderbird via Delphi using native messaging

Question

I'm writing a plugin in thunderbird using native messaging (following the ping pong example in python) to call a Delphi program to copy an e-mail locally as an ".eml" file. The problem I am facing seems to be the encoding. In addition, the resulting file contains double quotes ("") at the start and the end of the file as well as escaped double quotes (\"). I just want to have a 1 to 1 copy and not to change its content.

Example of a mail content:

"test"
€‚ƒ„…†‡ˆ‰Š‹ŒŽ‘’“”•–—˜™š›œžŸ
éöàäèüâêû 

However, in the file, it looks more like this:

\"test\"
€‚ƒ„…†‡ˆ‰Š‹ŒŽ‘’“â€â€¢â€“—˜™š›œžŸ
éöà äèüâêû

I might have found the problem, which is explained here:

https://www.i18nqa.com/debug/utf8-debug.html

However, I do not really know how to adapt my code to solve this problem.

Thank you for your help!

Here is my background.js:

async function main() {
        messenger.menus.create({
            contexts : ["message_list"],
            id: "copy@mail.lu",
            onclick : passMsg,
            title: messenger.i18n.getMessage("lang.menuTitle")
        });
    }
    
    async function passMsg(OnClickData) {
        if (OnClickData.selectedMessages && OnClickData.selectedMessages.messages.length > 0) {
            let MessageHeader = OnClickData.selectedMessages.messages[0];
            let raw = await messenger.messages.getRaw(MessageHeader.id);
            let port=browser.runtime.connectNative("copymail");
    
            port.onMessage.addListener((message) => {
              port.disconnect();
            });
    
            port.postMessage(raw);
        } else {
            console.log("No message selected");
        }
    }
    main();

Here is my Delphi code:

procedure WriteSTDInputToFile(const Filename: String);
    var
       Buffer:    array [0 .. 3] of Byte;
       msgLen:    LongInt;
       msg:       UTF8String;
       myFile:    TextFile;
       StdIn:     THandleStream;
       jsonValue: TJSONValue;
    begin
       StdIn  := THandleStream.Create(GetStdHandle(STD_INPUT_HANDLE));
    
       try
          msgLen    := 0;
          if StdIn.Read(Buffer, SizeOf(msgLen)) > 0 then
             msgLen := PLongInt(@Buffer)^;
    
          if msgLen > 0 then
          begin
             SetLength(msg, msgLen);
             StdIn.Read(PUTF8Char(msg)^, msgLen);
    
             if msg <> '' then
             begin
                AssignFile(myFile, Filename, CP_UTF8);
                ReWrite(myFile);
    
                jsonValue := TJSONObject.ParseJSONValue(msg);
    
                try
                   write(myFile, UTF8Encode(jsonValue.ToString));
                finally
                   jsonValue.Free;
                end;
    
                CloseFile(myFile);
    
             end;
          end;
    
       finally
          if Assigned(StdIn) then
             StdIn.Free;
       end;
    
    end;

Resulting file content:

"X-MDAV-Result: clean
X-MDAV-Processed: mail.test.lu, Wed, 28 Oct 2020 08:13:22 +0100
X-Spam-Processed: mail.test.lu, Wed, 28 Oct 2020 08:13:22 +0100
Return-path: <copy@mail.lu>
X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on MAIL01E
X-Spam-Level: 
X-Spam-Status: No, score=0.7 required=10.0 tests=HTML_MESSAGE,MPART_ALT_DIFF
    shortcircuit=no autolearn=disabled version=3.4.2
Authentication-Results: test.lu;
    auth=pass (plain) smtp.auth=ascholtes@test.lu
Received: from [172.16.17.35] [(172.16.17.35)] by test.lu (172.31.3.6) with ESMTPSA id md50033234892.msg; 
    Wed, 28 Oct 2020 08:13:21 +0100
X-MDRemoteIP: 172.16.17.35
X-MDArrival-Date: Wed, 28 Oct 2020 08:13:21 +0100
X-Authenticated-Sender: ascholtes@test.lu
X-Rcpt-To: copy@mail.lu
X-MDRcpt-To: copy@mail.lu
X-Return-Path: copy@mail.lu
X-Envelope-From: copy@mail.lu
X-MDaemon-Deliver-To: ascholtes@test.lu
To: Ayuth Scholtes <copy@mail.lu>
From: Ayuth Scholtes <copy@mail.lu>
Subject: Test
Organization: CISS
Message-ID: <7eb36f7c-a7af-c272-c189-eded642c3e1c@test.lu>
Date: Wed, 28 Oct 2020 08:13:21 +0100
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:68.0) Gecko/20100101
 Thunderbird/68.10.0
MIME-Version: 1.0
Content-Type: multipart/alternative;
 boundary=\"------------6068A746223BB2C9F1771938\"
Content-Language: lb-LU

This is a multi-part message in MIME format.
--------------6068A746223BB2C9F1771938
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: 8bit

|\"test\" â¬âÆââ¦â â¡Ëâ°Å â¹ÅŽâââââ¢ââËâ¢Å¡âºÅžŸ éöàäèüâêû|


--------------6068A746223BB2C9F1771938
Content-Type: text/html; charset=utf-8
Content-Transfer-Encoding: 8bit

<html>
  <head>

    <meta http-equiv=\"content-type\" content=\"text/html; charset=UTF-8\">
  </head>
  <body>
    <pre class=\"lang-pascal s-code-block hljs delphi\"><code>\"test\"
â¬âÆââ¦â â¡Ëâ°Å â¹ÅŽâââââ¢ââËâ¢Å¡âºÅžŸ
éöàäèüâêû</code></pre>
  </body>
</html>

--------------6068A746223BB2C9F1771938--
"

Answer 1

At first let me say that you did a good job in transferring data between web extension (Thunderbird add-on) and native application using native messaging. It isn't easy to understand it and set it up, but you managed to transfer required data with some tiny glitches you describe in your question.

... the resulting file contains double quotes (") at the start and the end of the file as well as escaped double quotes (\")

In the add-on you obtain raw email data as a string - console.log(typeof raw) gives string which you then pass to port.postMessage. Although the documentation says it takes JSON object representing the message to send, but it seems to accept single string value which is valid JSON according to some standards. In Delphi code you receive the message via STDIN and parse it using TJSONObject.ParseJSONValue into TJSONValue. It will in fact create instance of TJSONString. You can verify that by examining the value of jsonValue.ClassName. The problem with quotes arises when you use jsonValue.ToString which returns quoted version of the string that is basically the same what you had before parsing. Use the Value property to return raw string value.

Using jsonValue.Value alone will not help you with the encoding issue. The raw message data that you obtain from the e-mail client is in EML format. It conforms to RFC-822 and that means it is ASCII encoded, but it can contain arbitrarily encoded message parts (see your own sample EML). Since you only want to save EML file as is not taking any encoding into account, the best would be to transfer raw bytes of EML, but this isn't out-of-the-box supported by Javascript and native messaging API. Therefore I'd suggest you to send Base64-encoded data string to native application where you decode it into raw bytes that you can write straight to disk.

To encode raw message data as Base64 string in add-on use function btoa:

port.postMessage(btoa(raw));

To receive the message in native application you can do the following:

uses
  System.SysUtils, System.Classes, System.IOUtils, System.JSON, System.NetEncoding, Winapi.Windows;

procedure WriteSTDInputToFile(const FileName: string);
var
  StdIn: THandleStream;
  MsgLen: Cardinal;
  Data: TBytes;
  JSONValue: TJSONValue;
begin
  StdIn := THandleStream.Create(GetStdHandle(STD_INPUT_HANDLE));
  try
    StdIn.ReadBuffer(MsgLen, SizeOf(MsgLen));
    SetLength(Data, MsgLen);
    StdIn.ReadBuffer(Data, MsgLen);
    JSONValue := TJSONObject.ParseJSONValue(Data, 0);
    Data := TNetEncoding.Base64.DecodeStringToBytes(JSONValue.Value);
    TFile.WriteAllBytes(FileName, Data);
  finally
    StdIn.Free;
  end;
end;

Notice several improvements to the original code:

• Usage of Cardinal type for MsgLen. The protocol defines that the first 4 bytes on input indicate message length in bytes represented as 32-bit unsigned integer. Cardinal is Delphi's native type for such a value, or you can use UInt32 alias as well.
• I used ReadBuffer method instead of Read to read from STDIN, which makes the program crash in case of some unexpected circumstances. Ideally you should handle such circumstances, send error message in response via STDOUT and handle the response in add-on.
• I don't mix traditional I/O routines with streams. I didn't even use stream for writing to output file in my code. Creating the file is one-liner thanks to File.WriteAllBytes from System.IOUtils.
• I don't check if Assigned(StdIn) then StdIn.Free;. This is what Free already does for you.

Knowing that the incoming message is a quoted Base64-encoded string it would be possible to leave out JSON processing, so that the code becomes:

procedure WriteSTDInputToFile(const FileName: string);
var
  StdIn: THandleStream;
  MsgLen: Cardinal;
  Msg: RawByteString;
  Data: TBytes;
begin
  StdIn := THandleStream.Create(GetStdHandle(STD_INPUT_HANDLE));
  try
    StdIn.ReadBuffer(MsgLen, SizeOf(MsgLen));
    StdIn.Seek(1, soFromCurrent); { skip double quote }
    SetLength(Msg, MsgLen - 2); { minus leading and trailing double quotes }
    StdIn.ReadBuffer(Msg[Low(Msg)], MsgLen);
    Data := TNetEncoding.Base64.DecodeStringToBytes(UTF8ToString(Msg));
    TFile.WriteAllBytes(FileName, Data);
  finally
    StdIn.Free;
  end;
end;