Why are carriage return line feed (CR LF) not properly handled by TPerlRegEx when specified as Replacement

Question

I try to replace spaces with a new line using the TPerlRegEx class.

with RegExp do
begin
  Subject:=Memo1.Lines.Text;
  RegEx:=' ';
  Replacement:='\r\n';
  ReplaceAll;
  Memo1.Lines.Text:=Subject;
end;

The problem is that it treats the \r\n replacement as literal text.

Answer 1

Use #13#10

program Project29;

{$APPTYPE CONSOLE}

uses
  SysUtils, PerlRegEx;

var RegEx: TPerlRegEx;

function CStyleEscapes(const InputText:string):string;
var i,j: Integer;

begin
  SetLength(Result, Length(InputText));
  i := 1; // input cursor
  j := 1; // output cursor
  while i <= Length(InputText) do
    if InputText[i] = '\' then
      if i = Length(InputText) then
        begin
          // Eroneous quotation...
          Result[j] := '\';
          Inc(i);
          Inc(j);
        end
      else
        begin
          case InputText[i+1] of
            'r', 'R': Result[j] := #13;
            'n', 'N': Result[j] := #10;
            't', 'T': Result[j] := #9;
            '\':
              begin
                Result[j] := '\';
                Inc(j);
                Result[j] := '\';
              end;
            else
              begin
                Result[j] := '\';
                Inc(j);
                Result[j] := InputText[i+1];
              end;
          end;
          Inc(i,2);
          Inc(j);
        end
    else
      begin
        Result[j] := InputText[i];
        Inc(i);
        Inc(j);
      end;
  SetLength(Result, j-1);
end;

begin
  RegEx := TPerlRegEx.Create;
  try

    RegEx.RegEx := ' ';
    RegEx.Replacement := CStyleEscapes('\t\t\t');;
    RegEx.Subject := 'FirstLine SecondLine';
    RegEx.ReplaceAll;
    WriteLn(RegEx.Subject);

    ReadLn;

  finally RegEx.Free;
  end;
end.

Answer 2

I really wanted to know why it doesn't do the matching as expected.

Processing of \ escape sequences in the Replacement text is performed in TPerlRegEx.ComputeReplacement. If you take a look at the code you will find that there are no sequences that yield the carriage return and line feed characters. In fact ComputeReplacement is all about back references.

The processing of the matching phase of the regex is performed by the PCRE code. However, the replacement phase is pure Pascal code. And it's easy enough to inspect the code to see what it does. And it doesn't do what you think and expect it to do.

The conclusion is that you cannot specify the characters you want using escape sequences. I think you will need to devise your own rules for escaping non-printable characters and apply those rules in an OnReplace event handler.

Answer 3

Edit, as I learned something new today.

I ran in the same problem as the question a while ago, and took the wrong conclusion that
TRegEx does not do any C-style backslash escape expansion at all.

The correct conclusion should have been that
TRegEx does not do C-style backslash escape expansion in the replacement string parameters, and I should research if it does in the pattern string parameters.

I knew support of character escaping mechanisms varies by development tool.

For instance, C, C#, Java, Perl, PHP, Ruby, bash, and many more do backslash escape expansion.
But since the Delphi compiler (since it is not a C-style compiler) doesn't.
It will expand Pascal-style escapes (like #13#10, or ^M^J) into CRLF though.

So I did that research today (thanks David for pointing me to my initial mistake), and came up with two examples (one in Delphi and one in C#) that has a function that basically does this:

• show the pattern match result of a known CRLF string, and a pattern that contains a string
• show the replacement of space by a string

Then the example function is called by:

• a string that in source code is backslash escaped \r\n string, so might be parsed by the compiler
• a string that is put together character so it becomes a backslash escaped \r\n string runtime to it might get parsed by the RegEx engine

From the output in both examples, you see that:

• The Delphi compiler does not parse the \r\n string
• The C# compiler does parse the \r\n string
• The RegEx engine in both Delphi and C# parses the pattern \r\n string at run-time (RegEx documentation)
• The RegEx engine in both Delphi and C# do not parse the replace \r\n string at run-time (RegEx documentation)

The recommendation stil stands:

So either use the Pascal-style escapes, or use a C-Style backslash expansion function like Cosmin wrote.

As a side note: When using any expansion function, you should keep in mind that it will alter the meaning of text. Delphi users might not expect C-style expansion of strings.