git p4 submit: patch does not apply

Question

Lately I've been running into issues using git-p4.

My current workflow is:

git checkout -b some_feature
# do some work and commit to some_feature branch
git checkout master
git merge --no-ff some_feature
git p4 rebase
git p4 submit

Not always, but occasionally when executing git p4 submit, the changes do not actually apply and instead I see:

error: some_file.extension: patch does not apply

Doing some research, I've tried doing a hard reset on master and merging again without the --no-ff flag, but that doesn't seem to help.

Any ideas are greatly appreciated.

Answer 1

I ran into this issue today. In my case, the problem was because my text editor (Visual Studio) was adding a Unicode BOM to every file it touched. Meanwhile, the Perforce server was set up to strip the Unicode BOM from every file, preventing a Unicode BOM from appearing in Perforce. This ultimately caused my git p4 submit to fail with the following message:

error: patch failed: path/to/file.csproj:1
error: path/to/file.csproj: patch does not apply

Final solution

I added the following filter definition in my ~/.gitconfig file:

[filter "utf8-autobom"]
        clean = sed -b -e '1!b' -e 's/^\\xEF\\xBB\\xBF//'
        smudge = sed -b -e '1!b' -e 's/\\(^\\|^\\xEF\\xBB\\xBF\\)/\\xEF\\xBB\\xBF/'

Then I applied the utf8-autobom filter to the offending files by adding the following line in .gitattributes:

*.csproj filter=utf8-autobom

Then I forced Git to apply the filter to its index with:

rm .git/index
git reset

Then I committed the edited files to Git and submitted my commit to Perforce as usual:

git add .
git commit --amend
git p4 submit

How it works

The filter definition is based on the following sed commands, where "abc" is a placeholder for the appropriate Unicode BOM byte sequence:

# Remove 'abc' from beginning of file, if present
sed -b -e '1!b' -e 's/^abc//'

# Add 'abc' to beginning of file, if not present
sed -b -e '1!b' -e 's/\(^\|^abc\)/abc/'

For a UTF-8 BOM, we use the byte sequence EF BB BF instead of "abc".

The filter removes the BOM on commit by running the clean command, and adds the BOM on checkout by running the smudge command. This keeps the BOM in the working tree files, but prevents the BOM from being committed to Git, or submitted to Perforce.

(Refer to the gitattributes documentation for details on clean and smudge.)

Diagnosing the problem

I thought the error message was interesting:

error: patch failed: path/to/file.csproj:1
error: path/to/file.csproj: patch does not apply

It said the patch failed on line 1 even though my commit didn't edit line 1 of the file.

To see what was going on, I ran git diff p4/master HEAD. The diff showed that my commit was adding a strange <U+FEFF> character at the beginning of the file. I suspected it was related to file encoding, so I used Notepad++ to open the file in my Git working tree. Then I opened the corresponding file in my Perforce workspace. Notepad++ showed the encoding as "UTF-8-BOM" in my Git working tree, and "UTF-8" in my Perforce workspace. (Linux and Cygwin users: use the file <path-to-file> command to show information about file encoding.)

Googling "UTF-8-BOM" got me on the right path.

Notice

Some Git operations will be slower with this filter installed.

Tip

You can remove the smudge portion of the filter if you don't need the BOM in your working tree. This will speed up some Git operations (like git checkout). With the smudge line removed, the filter definition is:

[filter "utf8-autobom"]
        clean = sed -b -e '1!b' -e 's/^\\xEF\\xBB\\xBF//'

Answer 2

I had this same problem and I think the root cause was the line endings. You can try to run stuff like dos2unix to fix it, or set this config:

git config --global core.autocrlf true

Sometimes, file modes can cause problems to. This config can fix it (if you don't care about file modes):

git config core.filemode false

Answer 3

From jlapolla's 2016 answer

Then I applied the utf8-autobom filter

Such a filter might not be needed anymore with Git 2.37 (Q3 2022)

See commit fbe5f6b (04 Apr 2022) by Tao Klerks (TaoK).
^{(Merged by Junio C Hamano -- gitster -- in commit 804ec03, 20 May 2022)}

git-p4: preserve utf8 BOM when importing from p4 to git

^{Signed-off-by: Tao Klerks}

Perforce has a file type "utf8" which represents a text file with explicit BOM.
utf8-encoded files without BOM are stored as regular file type "text".
The "utf8" file type behaves like text in all but one important way: it is stored, internally, without the leading 3 BOM bytes.

git-p4 has historically imported utf8-with-BOM files (files stored, in Perforce, as type "utf8") the same way as regular text files - losing the BOM in the process.

Under most circumstances this issue has little functional impact, as most systems consider the BOM to be optional and redundant, but this is a correctness failure, and can have lead to practical issues for example when BOMs are explicitly included in test files, for example in a file encoding test suite.

Fix the handling of utf8-with-BOM files when importing changes from p4 to Git, and introduce a test that checks it is working correctly.

That means:

The type utf8 explicitly means utf8 with BOM.
These are streamed just like regular text files, however, without the BOM in the stream.

Therefore, to accurately import these files into git, we need to explicitly re-add the BOM before writing.
'contents' is a set of bytes in this case, so create the BOM prefix as a b'' literal.

So a p4 submit -d "add utf8 test files" followed by git p4 clone --dest="$git" //depot@all should give utf-8 files in the Git repository, with BOM preserved.

---

Furthermore, still with Git 2.37 (Q3 2022):

See commit f7b5ff6 (30 Apr 2022) by Tao Klerks (TaoK).
^{(Merged by Junio C Hamano -- gitster -- in commit 3af1df0, 20 May 2022)}

git-p4: improve encoding handling to support inconsistent encodings

^{Signed-off-by: Tao Klerks}

git-p4 is designed to run correctly under python2.7 and python3, but its functional behavior wrt importing user-entered text differs across these environments:

Under python2, git-p4 "naively" writes the Perforce bytestream into git metadata (and does not set an "encoding" header on the commits); this means that any non-utf-8 byte sequences end up creating invalidly-encoded commit metadata in git.

Under python3, git-p4 attempts to decode the Perforce bytestream as utf-8 data, and fails badly (with an unhelpful error) when non-utf-8 data is encountered.

Perforce clients (especially p4v) encourage user entry of changelist descriptions (and user full names) in OS-local encoding, and store the resulting bytestream to the server unmodified - such that different clients can end up creating mutually-unintelligible messages.
The most common inconsistency, in many Perforce environments, is likely to be utf-8 (typical in linux) vs cp-1252 (typical in windows).

Make the changelist-description- and user-fullname-handling code python-runtime-agnostic, introducing three "strategies" selectable via config:

• 'passthrough', behaving as previously under python2,
• 'strict', behaving as previously under python3, and
• 'fallback', favoring utf-8 but supporting a secondary encoding when> utf-8 decoding fails, and finally escaping high-range bytes if the decoding with the secondary encoding also fails.

Keep the python2 default behavior as-is ('legacy' strategy), but switch the python3 default strategy to 'fallback' with default fallback encoding 'cp1252'.

git p4 now includes in its man page:

git-p4.metadataDecodingStrategy

Perforce keeps the encoding of a changelist descriptions and user full names as stored by the client on a given OS. The p4v client uses the OS-local encoding, and so different users can end up storing different changelist descriptions or user full names in different encodings, in the same depot.

Git tolerates inconsistent/incorrect encodings in commit messages and author names, but expects them to be specified in utf-8. git-p4 can use three different decoding strategies in handling the encoding uncertainty in Perforce:

• 'passthrough' simply passes the original bytes through from Perforce to git, creating usable but incorrectly-encoded data when the Perforce data is encoded as anything other than utf-8.
• 'strict' expects the Perforce data to be encoded as utf-8, and fails to import when this is not true.
• 'fallback' attempts to interpret the data as utf-8, and otherwise falls back to using a secondary encoding - by default the common windows encoding 'cp-1252' - with upper-range bytes escaped if decoding with the fallback encoding also fails.

Under python2 the default strategy is 'passthrough' for historical reasons, and under python3 the default is 'fallback'. When 'strict' is selected and decoding fails, the error message will propose changing this config parameter as a workaround. If this option is passed into a p4 clone request, it is persisted into the resulting new git repo.

git-p4.metadataFallbackEncoding

Specify the fallback encoding to use when decoding Perforce author names and changelists descriptions using the 'fallback' strategy (see git-p4.metadataDecodingStrategy).
The fallback encoding will only be used when decoding as utf-8 fails.

This option defaults to cp1252, a common windows encoding.
If this option is passed into a p4 clone request, it is persisted into the resulting new git repo.