31

I'm in the process of setting up a deployment script. The basic process is:

  1. Push changes to a bare repository on the server
  2. Then based on new tags will create a new folder for the release.
  3. Use git archive to move the files into the release directory
  4. Runs some migrations scripts and puts it live (if all is successful).

The issue is my repository contains a submodule, which doesn't get put in the archive, and therefore doesn't get put in the release directory.

I've seen git-archive-all, but that doesn't work on a bare repository.

If its not possible, I'm considering,

  1. making the repository not bare, and updating the working copy, which would allow me to use git-archive-all. Or
  2. having a second bare repository of the submodule on the server, which I could get an archive from (would have to look into this to make sure I'm getting the right revision).
Jacob
  • 8,278
  • 1
  • 23
  • 29

6 Answers6

9

Based on answers and comments here. You can create non-bare repo and run:

git ls-files --recurse-submodules | tar caf ../prog.tar.gz -T-
  • To handle files starting with hypen, use --verbatim-files-from
  • To put files in folder inside the archive, use --xform s:^:prog/:

Full version:

git ls-files --recurse-submodules | tar caf ../prog.tar.gz --xform s:^:prog/: --verbatim-files-from -T-
Alek
  • 634
  • 7
  • 7
  • Thanks! Version to go directly to a directory of files (bypassing tar.gz): `git ls-files --recurse-submodules | tar c -T- | tar x -C TARGET_DIR` (Effectively converts a git non-bare repo to a git-free directory) – Akom Dec 13 '21 at 16:00
  • 1
    It's worth noting that this will give the files as they exist in the (possibly dirty) worktree, rather than what's committed to a particular ref. – Andrea Reina Jan 24 '22 at 08:54
  • 1
    Note: This might behave unexpectedly with symlinks. Ex: `foo/bar -> ../baz` becomes `prog/foo/bar -> prog/../baz` – Grammargeek Apr 12 '22 at 22:21
  • Of you are on macOS, then consider to look for solution in this SO answer: https://unix.stackexchange.com/a/725152/205472 – Vlad Nov 16 '22 at 16:32
6

I use this python package https://github.com/Kentzo/git-archive-all. You can install it by using

pip install git-archive-all

On OSX, you can install it also using brew install git-archive-all

ismailsunni
  • 1,458
  • 1
  • 24
  • 32
3

If your submodule is in a repo accessible from the server, I would rather have a post-receive hook which would

  • update a full non-bare repo (so a second repo beside your original bare one), including the submodule (git submodule update --init)
  • git archive from that second repo (you would be sure to get the right version since the non-bare repo would reference the right version of the submodule)
    Since the non-bare repo would contain the parent repo and its submodules, git archive-all would be able to detect the .git subdirectories and would archive everything.

If the submodule isn't accessible from the server, that means:

  • it needs to be pushed in its own repo on the server
  • the submodule in the parent repo needs to be reference with a relative path, in order for the parent repo to still be able to retrieve said submodule once pushed on the server.
VonC
  • 1,262,500
  • 529
  • 4,410
  • 5,250
  • 6
    Even with submodules initialized and updated. They don't appear in an archive produced with git-archive. Hence [git-archive-all](https://github.com/meitar/git-archive-all.sh). Would there be any reason to keep both a bare and non-bare repository? – Jacob Mar 30 '11 at 06:38
  • @Jacob: the submodules won't appear if their repo referenced in the `.gitsubmodules` file isn't accessible from the server. On the other point, `git archive-all` will work better on a non-bare repo (which will contain the extra `.git` subdirectories) with all the right submodules in place than in a bare repo (without any `.git` to grep and parse). – VonC Mar 30 '11 at 07:21
  • @Jacob: that is why I recommended a non-bare repo with submodules checked out in it: `git archive-all` will detect and archive everything. I have edited my answer to add that point. – VonC Mar 30 '11 at 08:32
  • 1
    If you update a full non-bare repo (with modules updated using `git submodule update --init`), what does `git-archive-full.sh` provide that tar does not already do for you? Like `tar --exclude-vcs --exclude=.gitmodules -c .` – Jarl Nov 08 '12 at 10:13
1

This should do the trick: https://github.com/meitar/git-archive-all.sh/wiki

Brandon
  • 953
  • 10
  • 12
  • 1
    Actually, looks like it has a few small issues so it might not be ready for prime time yet. – Brandon Aug 23 '12 at 15:35
  • this worked for me after changing it to use the `-o output_file` instead of `> output_file` format for `git archive` – hilcharge Jul 26 '15 at 14:37
  • 1
    It's worth noting that this is a different git-archive-all from the one mentioned in @ismailsunni's answer. – Robie Basak May 21 '17 at 13:53
1

Here it is as a few-liner:

prefix=$(basename "$(pwd -P)")
{
  git ls-files
  git submodule foreach --recursive --quiet \
                'git ls-files --with-tree="$sha1" | sed "s#^#$path/#"'
} | sed "s#^#$prefix/#" | xargs tar -c -C.. -f "$prefix.tar.xz" --
sam hocevar
  • 11,853
  • 5
  • 49
  • 68
Tobu
  • 24,771
  • 4
  • 91
  • 98
  • Downside to this is that you can't specify a ref for the root repository :P Always runs for whatever is currently checked out. – Qix - MONICA WAS MISTREATED Jul 25 '17 at 05:52
  • 1
    Another problem is that if the repo (including submodules) have more files than fits on tar's command line, the archive contains only a subset of files. Solution is to use `--files-from` rather than xargs: `tar -c -C.. -f "$prefix.tar.xz" --verbatim-files-from --files-from=-` – Michal Sojka May 02 '18 at 20:04
  • 1
    ```git ls-files --recurse-submodules | cpio -o --format=ustar > HEAD.tar``` – noelbk Jul 15 '19 at 08:34
0

Run the following after the regular archive command:
git submodule foreach 'cd REPO_ROOT/$path && git archive HEAD | tar -x -C TARGET_ROOT/$path'

Here the REPO_ROOT is where your repo stands and the TARGET_ROOT is where you put your archived version. (I assume that you have a TARGET_ROOT folder where the expanded version of the first git archive call. If you want a final zip/tar, you can tar/zip the final folder)

git submodule foreach provides the $path variable. See git help submodule foreach section for more details.

Cagdas Tulek
  • 67
  • 2