3

I have ISCSI node filenames with colons stored in GIT repository on Debian 10 Linux.

Example:

'iscsi/nodes/iqn.2000-01.com.synology:NAS01-DS916.nas/ff11::111:11ff:ff1f:1ff1,3260,1/default'
'iscsi/send_targets/1.2.3.4,3260/iqn.2000-01.com.synology:NAS01-DS916.nas,ff11::111:11ff:ff1f:1ff1,3260,1,default'

But checkout fails on Windows, because the colon is invalid character in Windows filename.

I get following GIT errors at Windows checkout:

error: invalid path 'iscsi/nodes/iqn.2000-01.com.synology:NAS01-DS916.nas/ff11::111:11ff:ff1f:1ff1,3260,1/default'
...
error: invalid path 'iscsi/send_targets/1.2.3.4,3260/iqn.2000-01.com.synology:NAS01-DS916.nas,ff11::111:11ff:ff1f:1ff1,3260,1,default'

Questions:

1) How to list all path having colon : in full GIT repo history?

2) How to remove all files from GIT repo history with path having at least one colon : in filename?

SOLUTION for 1) :

Works1:

git log --all --name-only -m --pretty= -- '*:*' | sort -u

Works2 (only for the named repo master):

git ls-tree -r master --name-only | grep ":"

Works3: Finally I used this to list files with colons in filename:

git log --format="reference" --name-status --diff-filter=A "*:*" >/opt/git_repo_files_w_colons.txt

UPDATE1 for 2):

I got

Aborting: Refusing to destructively overwrite repo history since
this does not look like a fresh clone.
  (expected freshly packed repo)
Note: when cloning local repositories, you need to pass
      --no-local to git clone to avoid this issue.
Please operate on a fresh clone instead.  If you want to proceed
anyway, use --force.

when executing

git filter-repo --invert-paths --path-match "*:*"

UPDATE2 for 2) :

Clone a copy of the repo:

git clone --no-local /source/repo/path/ /target/path/to/repo/clone/
# Cloning into '/target/path/to/repo/clone'...
# remote: Enumerating objects: 9534, done.
# remote: Counting objects: 100% (9534/9534), done.
# remote: Compressing objects: 100% (4776/4776), done.
# remote: Total 9534 (delta 4216), reused 8042 (delta 3136), pack-reused 0
# Receiving objects: 100% (9534/9534), 7.40 MiB | 17.08 MiB/s, done.
# Resolving deltas: 100% (4216/4216), done.

Remove the files with colon from repo history:

git filter-repo --invert-paths --path-match "*:*"
# Parsed 591 commits
# New history written in 0.47 seconds; now repacking/cleaning...
# Repacking your repo and cleaning out old unneeded objects
# HEAD is now at 501102d daily autocommit
# Enumerating objects: 9534, done.
# Counting objects: 100% (9534/9534), done.
# Delta compression using up to 8 threads
# Compressing objects: 100% (3696/3696), done.
# Writing objects: 100% (9534/9534), done.
# Total 9534 (delta 4216), reused 9534 (delta 4216), pack-reused 0
# Completely finished after 1.33 seconds.

Checking still shows filenames with colon:

git log --format="reference" --name-status --diff-filter=A "*:*"
# A    iscsi/nodes/iqn.2000-01.com.synology:NAS01-DS916.nas/ff11::111:11ff:ff1f:1ff1,3260,1/default
# ...

Unfortunately it seems filter-repo was executed, but log still lists filenames with colon :-(

Christopher Moore
  • 15,626
  • 10
  • 42
  • 52
klor
  • 1,237
  • 4
  • 12
  • 37
  • What's the git checkout command you are running in Windows? Perhaps there's a way to change that so it will work with colons. – nofinator Jan 13 '23 at 17:30
  • I use Git Extensions client, so there is no way to change the git commands. – klor Jan 13 '23 at 17:37
  • 1
    Just a reminder to all folks what symbols and filenames are forbidden in Windows, https://stackoverflow.com/questions/1976007/what-characters-are-forbidden-in-windows-and-linux-directory-names I have ended up in this situation because I have added a files in GitHUB using a Web interface, it allowed me to add a file with colon :) but it was a pain in the arse later, could not fix it in Linux, (terminal didn't start on it) thanks god I have also MacOS. – Nusrat Nuriyev Jun 08 '23 at 15:31

3 Answers3

3

The better approach might be to not check those files out on Windows,see the sparse checkout facility, or to rename them with characters Windows can handle, but to answer the questions as asked:

  1. How to list all path having colon : in full GIT repo history?
git log --all --name-only -m --pretty= -- '*:*' | sort -u
  1. How to remove all files from GIT repo history with path having at least one colon : in filename?
git filter-branch --prune-empty --index-filter '
        git ls-files "*:*" | git update-index --remove --stdin
' -- --all

which will rewrite your entire history starting from the first commit you have to change. Do this in a scratch clone.

VonC
  • 1,262,500
  • 529
  • 4,410
  • 5,250
jthill
  • 55,082
  • 5
  • 77
  • 137
  • 2
    Another good approach indeed. You could also list those files and [pass them to a `git filter-repo --invert-path`](https://stackoverflow.com/a/70387140/6309) command. – VonC Jan 14 '23 at 21:00
  • The `git filter-branch --prune-empty --index-filter ' git ls-files "*:*" | git update-index --remove --stdin ' -- --all` resulted a warning: `use an alternative filtering tool such as 'git filter-repo' instead` – klor Jan 16 '23 at 08:40
  • Otherwise the `git filter-branch ...` worked. Listing the filenames with colons results empty, so the filter-branch was successful. – klor Jan 16 '23 at 09:13
  • @vonc: I would prefer the `git filter-repo --invert-path command` solution, but it results a `Aborting: Refusing to destructively overwrite repo history since this does not look like a fresh clone. (expected freshly packed repo)` error. – klor Jan 16 '23 at 09:15
  • 1
    @klor Yes, `git filter-repo` only works on a fresh clone: that way, you do not touch your current working local repository, which means if the `filter-repo` "breaks" anything, you can delete the local clone, re-clone and retry the command. – VonC Jan 16 '23 at 09:17
1

link #1 did not provide solution to list paths having colo

Check:

git ls-tree -r master --name-only | grep ":"

But the approach suggested was to reset all files without ":", and delete the rest:

git ls-tree -r master --name-only | grep -v ":" | xargs git reset HEAD
git commit -m "deleting all files with a colon in the name"
git restore -- .

The OP klor reports listing those files with a git log pretty format "reference" (which is <abbrev-hash> (<title-line>, <short-author-date>)):

git log --format="reference" --name-status --diff-filter=A "*:*" >/opt/git_etc_repo_files_w_colons.txt

The OP suggested:

# Clone repository, to be executed on a safe repo:
git clone --no-local /source/repo/path/ /target/path/to/repo/clone/
# Cloning into '/target/path/to/repo/clone'...
# remote: Enumerating objects: 9534, done.
# remote: Counting objects: 100% (9534/9534), done.
# remote: Compressing objects: 100% (4776/4776), done.
# remote: Total 9534 (delta 4215), reused 8043 (delta 3136), pack-reused 0
# Receiving objects: 100% (9534/9534), 7.41 MiB | 16.78 MiB/s, done.
# Resolving deltas: 100% (4215/4215), done.

cd /target/path/to/repo/clone/

# List the files with colon from repo history into a list file:
git log --all --name-only -m --pretty= -- '*:*' | sort -u >/opt/git_repo_files_w_colons.txt

# Remove the files with colon from repo history:
git filter-repo --invert-paths --paths-from-file /opt/git_repo_files_w_colons.txt
# Parsed 591 commits
# New history written in 0.74 seconds; now repacking/cleaning...
# Repacking your repo and cleaning out old unneeded objects
# HEAD is now at e5fdf93 daily autocommit
# Enumerating objects: 9347, done.
# Counting objects: 100% (9347/9347), done.
# Delta compression using up to 8 threads
# Compressing objects: 100% (3696/3696), done.
# Writing objects: 100% (9347/9347), done.
# Total 9347 (delta 4078), reused 9345 (delta 4076), pack-reused 0
# Completely finished after 1.59 seconds.

# List files with colon to check result:
git log --format="reference" --name-status --diff-filter=A "*:*"
# Empty result, so git filter-repo was successful, filenames with colon were removed!
VonC
  • 1,262,500
  • 529
  • 4,410
  • 5,250
  • The `git ls-tree -r master --name-only | grep ":"` lists filenames with colons only from a branch, not the whole history. – klor Jan 16 '23 at 08:22
  • @klor Yes, I focus on the miniaml fix you need for a clone (using the default branch) to succeed. – VonC Jan 16 '23 at 08:25
  • I executed `git ls-tree -r live_master --name-only | grep -v ":" | xargs git reset HEAD `, but I got `error: unknown switch '/'` – klor Jan 16 '23 at 08:33
  • @klor Are you executing that on Windows or your Debian server? With which version of Git? And in which shell? – VonC Jan 16 '23 at 08:37
  • Executing on Debian server. Using Putty to SSH to Debian server, Bash shell, Git v2.28. – klor Jan 16 '23 at 08:51
  • OK I just tested in a simple CMD on Windows, and it worked. In a shell, just in case, replace double-quote by simple quote. – VonC Jan 16 '23 at 08:53
  • I executed `git ls-tree -r live_master --name-only | grep -v ':' | xargs git reset HEAD`, but got same `error: unknown switch '/'` – klor Jan 16 '23 at 09:09
  • The `error: unknown switch '/'` relates to `git reset HEAD`. `error: unknown switch '/' usage: git reset ......` – klor Jan 16 '23 at 09:26
  • Let us [continue this discussion in chat](https://chat.stackoverflow.com/rooms/251168/discussion-between-vonc-and-klor). – VonC Jan 16 '23 at 09:27
0
  1. How to list all path having colon : in full GIT repo history?

You may be out of luck here. There's more talk about this issue and workarounds in Windows here.

  1. How to remove all files from GIT repo history with path having at least one colon : in filename?

Please be careful with this approach if you are developing with others. When existing branches are modified in the origin, it will cause a lot of conflicts when others try to pull them down.

But if you are ok to proceed, you can try an approach here.

nofinator
  • 2,906
  • 21
  • 25
  • Thank you for your answer, but: link #1 did not provide solution to list paths having colon link #2 did not provide solution to remove files having filename with colon. – klor Jan 13 '23 at 18:05