How to handle thousands of files that only have differences of eol characters CRLF / LF?

Question

I've received files from a collaborator by manual transfer i.e. not from a git way - not from a merge request - that I want to put into my worktree. My issue is that:

the eol characters don't respect my policy, which is:

CRLF for most file formats in the working directory;
LF in the local and remote repositories.

there are thousands of such files, with a huge majority having only these eol differences, and less than a hundred having meaningful differences.

Here is what I tried:

I have tried different git settings:
- git config --get core.autocrlf=true to have CRLF in the working directory and LF in local & remote repositories as explained in Git replacing LF with CRLF and I have done renormalization and git rm --cached -r . and git reset --hard afterwards;
- setting the .gitattributes to this template one with additional C# settings and eol=crlf to be extra sure these settings aren't lost later as explained in What's the strategy for handling CRLF (carriage return, line feed) with Git? - but fundamentally this adds nothing to the previous point;
- whitespace = cr-at-eol so that git diff ignores differences on eol characters as explained in this answer to Make 'git diff' ignore ^M.
but then my git GUI client SourceTree ignores these settings and still list thousands of file changes. It is able to not show differences, but will still list the file as shown in Force SourceTree ignore line endings in git files. So I've tried several others and I've found no setting in which files with only eol character changes are skipped: SourceTree, SmartGit, Gittyup, GitHub Desktop, GitAhead, TortoiseGit, GitGUI, Git Extensions. GitAhead is the best in that particular instance because it collapses the files with only eol character differences in the list of file differences. But it still lists them, and it is in general very slow.

Here are the strategies that I see:

Manually reviewing all of the files - I'd rather not;
Finding a git GUI client that skips files with only eol character differences;
Replacing all the LF by CRLF using e.g. Notepad++ on the appropriate file formats;
Creating a branch in which I add the thousands of files, then merging it into my development branch, forcing in the process all the eol characters to be LF in the local & remote repositories.

Any better idea?

Of course, next time I have to do that I'll encourage my collaborator to use git to send me files!

"Replacing all the LF by CRLF using e.g. Notepad++ on the appropriate file formats" Yep. What's wrong with that? On a Mac with BBEdit you'd have already done this by now; it's trivially quick and easy to set up and run this sort of batch conversion. — matt, Feb 18 '23 at 15:08
I've tried doing just that with Sublime on Windows and for some reason it failed. Maybe it's because it's not managing eol characters properly, maybe it's the number of files, maybe I used a bad regexp. There's also the list of file types to correctly manage. As you can see from the gitattributes template I use, it's a Unity project and for some reason for some file types LF is preferred. I was searching for a more elegant, robust, and systematic way, there isn't yet. — BobbyVolley, Feb 18 '23 at 21:43
Look at `unix2dos` or `dos2unix` depending the eol you want. — Philippe, Feb 18 '23 at 21:48

score 0 · Answer 1 · answered Feb 18 '23 at 15:10

You can move the files in a folder and use this piece of code to convert crlf to lf. Also change the directoryPath variable to the folder path

This will recursively read all files and change the crlf to lf and saves it.

Make sure the file will fit into memory

const fs = require('fs');
const path = require('path');

const directoryPath = 'path/to/directory';

// Function to recursively read all files in a directory
function readDirectory(directoryPath) {
  fs.readdir(directoryPath, (err, files) => {
    if (err) {
      console.error(err);
      return;
    }

    files.forEach((file) => {
      const filePath = path.join(directoryPath, file);
      fs.stat(filePath, (err, stat) => {
        if (err) {
          console.error(err);
          return;
        }

        if (stat.isDirectory()) {
          readDirectory(filePath); // Recursively read subdirectories
        } else {
          replaceCRLF(filePath); // Replace CRLF in files
        }
      });
    });
  });
}

// Function to replace CRLF with LF in a file
function replaceCRLF(filePath) {
  fs.readFile(filePath, 'utf8', (err, data) => {
    if (err) {
      console.error(err);
      return;
    }

    const newData = data.replace(/\r\n/g, '\n'); // Replace CRLF with LF
    if (newData !== data) {
      fs.writeFile(filePath, newData, 'utf8', (err) => {
        if (err) {
          console.error(err);
          return;
        }

        console.log(`Replaced CRLF with LF in file: ${filePath}`);
      });
    }
  });
}

readDirectory(directoryPath);```

How to handle thousands of files that only have differences of eol characters CRLF / LF?

1 Answers1