14

I am having a problem with a large Delphi codebase where I work where as a side effect of porting from Delphi 2007 to XE2, we now encounter the following strange, related issues:

  • You can not set breakpoints or single step through code even in the debug build because the line numbering is all messed up, only in some of the units.

  • Introducing a syntax error deliberately at line 2010 will cause the cursor to focus at line 2020, give or take 3 or 4 lines, something like this:

.

 procedure Correct;
 begin
    DoSomething; // syntax error reported HERE but the real error is below.
    // more stuff here.
 end;

 procedure OllKorrect;  
 begin
        ThisLineIsFine();
        __VARIABLE_NOT_DEFINED__ := 1; // intentional error
 end

I am hoping someone has seen this before. Elements of the issue may include:

The code contains many odd compiler directives like {$REALCOMPATIBILITY ON} and {$H-}/{$H+} directives, thousands of {$H+}/{$H-} directives in the code.

Secondly, the code uses a lot of {$I INCLUDE} directives, and I suspect that include files might directly mess up the line-numbering of the compiler.

I am unable to say for sure, but I suspect all these old "make it work like turbo pascal for DOS" compiler switches are the reason behind it. I'd like to know if anyone knows something about this for sure. It only happens in some places in the code, and with a project that has over 500 units, some of which reach 10K/20KLOC in size, it is definitely frustrating. What I can say is that it not only units that have {$I include.inc} directives that mess up, and that many units that contain a lot of {$H-}/{$H+} or {$REALCOMPATIBILITY} directives do not have this problem. If I could see what the units that misbehave have in common I could figure this out.

Update: The line termination issue makes sense. I ran this code which detected problems. The fix code is commented out because if you uncomment it and it erases all your source code, that's your problem. It is loading a non-unicode file into a unicode TStringList and saving it back out. That's okay in my world because its all version controlled and backed up. Your mileage may vary.

program linefeedsProject1;

{$APPTYPE CONSOLE}

uses
  IOUtils,
  Classes,
  Types,
  SysUtils;


  var
    broken,notBroken:Integer;

  function fix(filename:String):Boolean;
  var
    sl:TStringList;
  begin
    sl := TStringList.Create;
    try
    sl.LoadFromFile(filename);
    //TODO:Change file extensions.
    sl.SaveToFile(filename);
    finally
      sl.Free;
    end;
  end;

  function scan(filename:String):Boolean;
  var
  crFlag:Boolean;
  lfFlag:Boolean;
  missingCr:Integer;
  missingLf:Integer;
   f:TFileStream;
   buf:Array[0..1024] of AnsiChar;
   n:Integer;
   procedure scanChars;
   var
    i:Integer;
   begin
     for i := 0 to n-1 do
     begin
       if buf[i]=#13 then
       begin
          crFlag := true;
          lfFlag := false;
       end
       else if buf[i]=#10 then
       begin
           if not crFlag then
            inc(missingCr);
          lfFlag := true;
          crFlag := false;
       end
       else begin
         if (crFlag) then
          inc(missingLf);
         crFlag := false;
         lfFlag := false;
       end;
     end;
   end;
  begin
   result := false;
   crFlag := false;
   lfFlag := false;
   missingCr := 0;
   missingLf := 0;
    f := TFileStream.Create(filename, fmOpenRead);
    try
      while f.Position< f.Size do
      begin
        n := f.Read(buf[0],1024);
        scanChars;
      end;

     if (missingCr>0) or (missingLf>0) then
     begin
          WriteLn('  ', filename);
          Inc(broken);
          result := true;
     end
     else
     begin
        Inc(notBroken);
     end
    finally
      f.Free;
    end;

  end;
var
 files:TStringDynArray;
 afile:String;
 begin
  try
  broken := 0;
  notBroken := 0;
    files := TDirectory.GetFiles('C:\dev\abackupcopyofyoursourcecode',  '*.pas',
    TSearchOption.soTopDirectoryOnly );
    // tried TSearchOption.soAllDirectories and it exploded. not recommended.

    for afile in files do
    begin
       if scan(afile) then
       begin
           // fix(afile); // uncomment at your own risk and only on a backup copy of your code.
       end;

    end;


    WriteLn('Broken ', broken);
    WriteLn('not broken ',notBroken);

   // readln;

     except
    on E: Exception do
      Writeln(E.ClassName, ': ', E.Message);
  end;
end.

Update 2: If you want a scanner/fixer for this issue you can download mine (with source) here. Link is Google Drive. You can view the source code from the link, but click the "File" pull down menu (part of the google drive web user interface) and then click "Download" to download it.

Warren P
  • 65,725
  • 40
  • 181
  • 316
  • +1 , I have experienced the same behavior using the {$I INCLUDE} directive. – RRUZ Apr 24 '13 at 23:42
  • I wonder if it's line-termination characters inside the .inc include files instead of the main file, that is messing me up. – Warren P Apr 24 '13 at 23:46
  • 1
    If you've added (or removed) lines of code and hit F9, it's possible the DCUs being used in the binary (and therefore the tokens for the debugger) haven't been rebuilt (I've had that issue many times). In such cases, I just do a "Clean" then "Build" and the problem goes away. Tried that? – LaKraven Apr 25 '13 at 00:04
  • +1 I have seen this before, but have always disregarded it and haven't followed any patterns. – Jerry Dodge Apr 25 '13 at 02:00
  • @LaKraven I do the same, and my problems go away too. It could have to do with certain elements not being aware of changes in the code, and therefore it relies on line positions as compiled in the DCU. – Jerry Dodge Apr 25 '13 at 02:02
  • 1
    PS - I encounter exactly the same issues when debugging errors in SQL Server script (very large, like 20k lines in one file) – Jerry Dodge Apr 25 '13 at 02:07
  • 1
    (That is, SQL Server Management Studio 2008 R2 specifically) – Jerry Dodge Apr 25 '13 at 02:13
  • 2
    Lakraven; Clean and rebuild has no effect. The line termination answers seem rational and probable. – Warren P Apr 25 '13 at 02:20
  • I must add a clean/rebuild hasn't *always* worked, I recall times when I could not figure out why it brought me to the wrong line. But like I said, I never really paid much attention and followed any patterns. – Jerry Dodge Apr 25 '13 at 02:28
  • @Warren P, line termination characters is the most likely explanation - drove me crazy when it hit me some time ago as one cannot see why this mismatch is happening. – iamjoosy Apr 25 '13 at 07:43
  • i had suchexperience when my \Windows\system32 fodler had rtl.bpland vcl.bpl of xe2u2 and i put update #4 over Delphi - thus the program (and debug info) was compiled against newer BPL/DCP files, that it loaded when executed actually. – Arioch 'The Apr 25 '13 at 10:39
  • I always have rigorous cleaning procedures to prevent such things. – Warren P Apr 25 '13 at 11:08
  • Possible duplicate of [Delphi XE - All blue dots are shifted with one line up](http://stackoverflow.com/questions/6609898/delphi-xe-all-blue-dots-are-shifted-with-one-line-up) – Gabriel Mar 25 '17 at 20:48

3 Answers3

23

I've seen things like this before, and IME it's generally due to an compiler bug in counting line numbers. If you have nonstandard line breaks (not CRLF) at some points--which can happen--the IDE will do proper line breaks but the compiler doesn't count them as new lines, so everything afterwards gets thrown off by one.

What I do when I encounter a file like this is open it in EditPad, convert all linebreaks to some other style (Unix or Mac style) and then convert all linebreaks to Windows style and save it. This ensures that every line in the file ends with CRLF, and after a rebuild the issue with the blue dots not lining up right goes away.

Mason Wheeler
  • 82,511
  • 50
  • 270
  • 477
  • This happens in Visual Studio too. There I use `CodeRush` which warns me for any file that has mixed line endings and has an option to convert them to one style. – Jeroen Wiert Pluimers Apr 25 '13 at 08:17
  • 1
    I wrote a little utility to help out and appended it to the answer above. If it's useful to people I will write something less rushed and make it open source. – Warren P Apr 25 '13 at 11:07
  • Confirmed. After running the code I posted above to my answer, I am no longer able to reproduce these problems. Good call. Thanks both of you guys who took time to answer. – Warren P Apr 25 '13 at 11:27
  • I wrote a utility, see the link in the updated question if anyone needs to scan their code. (It will fix it too, if you pass in the correct parameters, but by default to be careful it only reads things and doesn't write things.) – Warren P Apr 25 '13 at 13:59
10

This is a common problem caused by mismatched line termination characters. (Code with a missing CR or LF on the end of the line.) It can also be caused by having a .dcu that doesn't match the source file that's open in the editor.

For the first, the easiest fix is to open the source file in a regular text editor (such as Notepad). Make a small change (insert a blank line and then delete it), and save the file. Notepad will fix the line endings.

If this isn't the issue, look for extra copies of the .dcu (or .pas) file that might be on your drive in the IDE's search path. Sometimes the compiler sees a different version than what's open in the editor.

Ken White
  • 123,280
  • 14
  • 225
  • 444
  • Is there a tool to fix the line termination automatically on a large codebase? If this is the issue, I'd rather deal with it thoroughly. I might write such a tool if I can't find one out there. – Warren P Apr 24 '13 at 23:43
  • Not that I've seen. I once wrote one (a Delphi app that iterated each .pas file in a folder, loaded the file into a `TStringList`, and then did a `SaveToFile`). (This has been a known issue in the IDE since D2005, when the current editor codebase was introduced. I don't know why it hasn't been fixed yet; probably a low-priority issue compared to xPlatform support.) – Ken White Apr 24 '13 at 23:46
  • That would do it I guess. Unless it kept the invalid line-termination characters as a mid-point in one of the lines. – Warren P Apr 24 '13 at 23:46
  • It fixed it in the cases I used it without fail. :-) – Ken White Apr 24 '13 at 23:49
  • 5
    It's sad that we have to resort to third-party tools to resolve dumb little IDE and compiler bugs. These are all things that theoretically should be dealt with directly. – Jerry Dodge Apr 25 '13 at 02:10
  • 1
    @Jerry: This is not only a Delphi IDE issue. Visual Studio has problems with this also (mostly because of people opening *nix files in it), and offers a "Normalize line endings" option in the editor context menu to fix it. (See [the question where I posted this answer](http://stackoverflow.com/a/553560/62576) for instance.) It also frequently happens when code is copy/pasted from web pages from some browsers into the IDE code editor. – Ken White Apr 25 '13 at 02:30
  • @Ken Indeed, I commented in the question that I've seen the precise behavior in SQL Server Management Studio 2008 R2 (among others) – Jerry Dodge Apr 25 '13 at 02:31
  • 1
    Speaking of which: http://meta.stackexchange.com/questions/169766/code-not-formatted-as-code – Jerry Dodge Apr 25 '13 at 02:34
  • For what it is worth: Notepad regularly barfs on non-standard line endings on my systems (ranging from NT 4 to Windows 8). Usually I take `SourcePad` for the correction, or a plain simple `unix2dos` http://en.wikipedia.org/wiki/Unix2dos – Jeroen Wiert Pluimers Apr 25 '13 at 08:10
  • 1
    Now you can use my handy dandy recursive descending Fixer Upper. – Warren P Apr 25 '13 at 18:08
0

It definitely is connected to line endings as explained in previous posts. It is extremely hard to try to edit it away, since the Embarcadero editor tries to do magic of its own and preserves the line endings of the current section (or so it seems),

Solution: Right click in the IDE editor window, select "Format Source" (Ctrl-D) and accept.

This will fix all line endings and removes the problem (for me any way). As a by-product you will get correctly formatted source code :-)