running git-upload-pack which send objects packed back to git-fetch-pack which receives missing objects from another repository.
You can see more about git fetch-pack with Git 2.19 (Q3 2018), since "git fetch-pack --all" used to unnecessarily fail upon seeing an annotated tag that points at an object other than a commit.
See commit c12c9df (13 Jun 2018) by Kirill Smelkov (navytux).
Helped-by: Junio C Hamano (gitster).
See commit e9502c0 (11 Jun 2018) by Jeff King (peff).
Helped-by: Junio C Hamano (gitster).
(Merged by Junio C Hamano -- gitster -- in commit 0079732, 28 Jun 2018)
Fetch-pack --all became broken with respect to unusual tags in
5f0fc64 (fetch-pack: eliminate spurious error messages, 2012-09-09, Git v1.8.0),
and was fixed only recently in e9502c0 (fetch-pack: don't try to fetch
peel values with --all, 2018-06-11).
fetch-pack: don't try to fetch peel values with --all
When "fetch-pack --all" sees a tag-to-blob on the remote, it tries to fetch both the tag itself ("refs/tags/foo") and the peeled value that the remote advertises ("refs/tags/foo^{}").
Asking for the object pointed to by the latter can cause upload-pack to complain with "not our ref", since it does not mark the peeled objects with the OUR_REF (unless they
were at the tip of some other ref).
Arguably upload-pack should be marking those peeled objects. But it never has in the past, since clients would generally just ask for the tag and expect to get the peeled
value along with it.
And that's how "git fetch" works, as well as older versions of "fetch-pack --all".
Let's explicitly test all relevant cases with 4 tag objects pointing to
-
- a blob,
-
- a tree,
-
- a commit, and
-
- another tag objects.
The referenced tag objects themselves are referenced from under regular refs/tags/*
namespace.
Before e9502c0 (Git 2.19) fetch-pack --all was failing e.g. this way:
.../git/t/trash directory.t5500-fetch-pack/fetchall$ git ls-remote ..
44085874... HEAD
...
bc4e9e1f... refs/tags/tag-to-blob
038f48ad... refs/tags/tag-to-blob^{} # peeled
520db1f5... refs/tags/tag-to-tree
7395c100... refs/tags/tag-to-tree^{} # peeled
.../git/t/trash directory.t5500-fetch-pack/fetchall$ git fetch-pack --all ..
fatal: A git upload-pack: not our ref 038f48ad...
fatal: The remote end hung up unexpectedly
With Git 2.25 (Q1 2020), fetch-pack includes trace2 annotation.
See commit 9e5afdf (19 Nov 2019) by Erik Chen (erikchen).
(Merged by Junio C Hamano -- gitster -- in commit 76c6824, 05 Dec 2019)
fetch: add trace2 instrumentation
Signed-off-by: Erik Chen
Add trace2 regions to fetch-pack.c to better track time spent in the various phases of a fetch:
- parsing remote refs and finding a cutoff
- marking local refs as complete
- marking complete remote refs as common
All stages could potentially be slow for repositories with many refs.
Example: (you can then follow what a git fetch does)
D:\git\git>set GIT_TRACE2_EVENT=1
D:\git\git>git fetch
23:58:43.225088 exec-cmd.c:237 trace: resolved executable dir: D:/newenv/prgs/gits/PortableGit-2.24.0.2-64-bit.7z/mingw64/bin
{"event":"version","sid":"20200111T225843.226090Z-H91a50d96-P00007560","thread":"main","time":"2020-01-11T22:58:43.234089Z","file":"common-main.c","line":48,"evt":"2","exe":"2.24.0.windows.2"}
{"event":"start","sid":"20200111T225843.226090Z-H91a50d96-P00007560","thread":"main","time":"2020-01-11T22:58:43.234089Z","file":"common-main.c","line":49,"t_abs":0.011593,"argv":["git.exe","fetch"]}
{"event":"data_json","sid":"20200111T225843.226090Z-H91a50d96-P00007560","thread":"main","time":"2020-01-11T22:58:43.260090Z","file":"compat/win32/trace2_win32_process_info.c","line":118,"repo":0,"t_abs":0.037686,"t_rel":0.037686,"nesting":1,"category":"process","key":"windows/ancestry","value":["git.exe","cmd.exe","explorer.exe"]}
{"event":"def_repo","sid":"20200111T225843.226090Z-H91a50d96-P00007560","thread":"main","time":"2020-01-11T22:58:43.261089Z","file":"repository.c","line":130,"repo":1,"worktree":"D:/git/git"}
23:58:43.262092 git.c:439 trace: built-in: git fetch
{"event":"cmd_name","sid":"20200111T225843.226090Z-H91a50d96-P00007560","thread":"main","time":"2020-01-11T22:58:43.262092Z","file":"git.c","line":440,"name":"fetch","hierarchy":"fetch"}
{"event":"region_enter","sid":"20200111T225843.226090Z-H91a50d96-P00007560","thread":"main","time":"2020-01-11T22:58:43.281090Z","file":"builtin/fetch.c","line":1361,"repo":1,"nesting":1,"category":"fetch","label":"remote_refs"}
{"event":"child_start","sid":"20200111T225843.226090Z-H91a50d96-P00007560","thread":"main","time":"2020-01-11T22:58:43.284088Z","file":"run-command.c","line":735,"child_id":0,"child_class":"remote-https","use_shell":false,"argv":["git","remote-https","origin","https://github.com/git/git"]}
23:58:43.284088 run-command.c:663 trace: run_command: GIT_DIR=.git git remote-https origin https://github.com/git/git
With Git 2.31 (Q1 2021), the ls-refs protocol operation (used also by git fetch) has been optimized to narrow the sub-hierarchy of refs/ it walks to produce response.
It furhter illustrates what a git fetch does.
See commit b3970c7, commit 16b1985 (20 Jan 2021) by Taylor Blau (ttaylorr).
See commit 83befd3 (20 Jan 2021) by Jacob Vosmaer (jacobvosmaer).
(Merged by Junio C Hamano -- gitster -- in commit 6254fa1, 05 Feb 2021)
ls-refs.c: traverse prefixes of disjoint "ref-prefix" sets
Original-patch-by: Jacob Vosmaer
Signed-off-by: Taylor Blau
ls-refs performs a single revision walk over the whole ref namespace, and sends ones that match with one of the given ref prefixes down to the user.
This can be expensive if there are many refs overall, but the portion of them covered by the given prefixes is small by comparison.
To attempt to reduce the difference between the number of refs traversed, and the number of refs sent, only traverse references which are in the longest common prefix of the given prefixes.
This is very reminiscent of the approach taken in b31e268 ("ref-filter.c: find disjoint pattern prefixes", 2019-06-26, Git v2.23.0-rc0 -- merge listed in batch #6) which does an analogous thing for multi-patterned 'git for-each-ref'(man) invocations.
The callback 'send_ref' is resilient to ignore extra patterns by discarding any arguments which do not begin with at least one of the specified prefixes.
Similarly, the code introduced in b31e268 is resilient to stop early at metacharacters, but we only pass strict prefixes here.
At worst we would return too many results, but the double checking done by send_ref will throw away anything that doesn't start with something in the prefix list.
Finally, if no prefixes were provided, then implicitly add the empty string (which will match all references) since this matches the existing behavior (see the "no restrictions" comment in "ls-refs.c:ref_match()").
With Git 2.38 (Q3 2022), the common ancestor negotiation exchange during a "git fetch"(man) session now leaves trace log.
Again, that helps seeing what a git fetch does.
See commit a29263c (02 Aug 2022) by Josh Steadmon (steadmon).
(Merged by Junio C Hamano -- gitster -- in commit 098b7bf, 25 Aug 2022)
fetch-pack: add tracing for negotiation rounds
Signed-off-by: Josh Steadmon
Acked-by: Jeff Hostetler
Currently, negotiation for V0/V1/V2 fetch have trace2 regions covering the entire negotiation process.
However, we'd like additional data, such as timing for each round of negotiation or the number of "haves" in each round.
Additionally, "independent negotiation" (AKA push negotiation) has no tracing at all.
Having this data would allow us to compare the performance of the various negotation implementations, and to debug unexpectedly slow fetch & push sessions.
- Add per-round
trace2 regions for all negotiation implementations (V0+V1, V2, and independent negotiation), as well as an overall region for independent negotiation.
- Add trace2 data logging for the number of haves and "in vain" objects for each round, and for the total number of rounds once negotiation completes.
Example:
GIT_TRACE2_EVENT="$(pwd)/trace2"
git -C myclient fetch --progress origin main
...
"key":"total_rounds","value":"6"
...
