Deleting files after adding to tar archive

Question

Can GNU tar add many files to an archive, deleting each one as it is added?

This is useful when there is not enough disk space to hold both the entire tar archive and the original files - and therefore it is not possible to simply manually delete the files after creating an archive in the usual way.

Are ALL of the files which exist in the directory must be compressed? — Eugene S, May 28 '12 at 08:48
yes, I have a directory called images. I was doing tar cvjf images.tar.bz2 images/ but I ran out of disk space. — Ivy, May 28 '12 at 08:49
One solution can be to do ftp for entire files to another machine --> archive it --> then move back "archived files" to the same machine — Pradeep, May 28 '12 at 09:57
I'd rather not do that, as there are a lot of files, and I may need to do it again with even more — Ivy, May 28 '12 at 13:05
@lanzz No, as the bsd tar *doesn't* have the `--remove-files` option as shown in the gnu tar man page. — bfcoder, Aug 27 '18 at 14:32
@bfcoder No, it would still answer the question. If you're on bsd, `man tar` would show you the man page for bsd tar, not for gnu tar. — lanzz, Aug 27 '18 at 14:41

score 103 · Accepted Answer · edited Dec 21 '18 at 03:33

103

With GNU tar, use the option --remove-files.

edited Dec 21 '18 at 03:33

xinthose

3,213
3
40
59

answered May 28 '12 at 08:50

Fred Foo

355,277
75
744
836

4

Are you sure that by using this flag will remove each file after adding it to the archive rather than deleting all the files in the end of the process? – Eugene S May 28 '12 at 09:54
16

@EugeneS: I checked the GNU tar source code and there is some (configurable) delay between archiving and removal, but tar will not wait for the entire tarball to be created. – Fred Foo May 28 '12 at 10:15
How about bsdtar? Anyone? – dpb Aug 08 '13 at 15:25
Works great. Helps to make backups by pg_dump -Fd – Viacheslav Dobromyslov Mar 03 '15 at 12:34

score 6 · Answer 2 · edited Apr 17 '21 at 14:27

6

I had a task - archive files and then remove into OS installed "tar" without GNU-options.

Method:

Use "xargs"

Suppose, we are have a directory with files.
Need move all files, over the week into tar and remove it.
I do one archive (arc.tar) and added files to it. (You can create new archive every try)

Solution:

find ./ -mtime +7 | xargs -I % sh -c 'tar -rf arc.tar % && rm -f %'

edited Apr 17 '21 at 14:27

Ivy

3,393
11
33
46

answered Nov 28 '13 at 19:45

Mikro Koder

1,056
10
13

5

1. I would use `&&` instead of `;` between `tar` and `rm` commands, so only remove files if their addition to the archive was successful. 2. Consider using `-exec` option if your `find` has it. https://stackoverflow.com/a/6043896/711006 – Melebius Nov 27 '17 at 07:51

score 4 · Answer 3 · answered Feb 21 '19 at 14:36

4

For non GNU tar, you can use "-u" to proccess file per file in a loop

tar -cf archive.tar afile
for myfile in dir/*.ext
do
    tar -uf archive.tar $myfile && rm $myfile || echo "error tar -uf archive.tar $myfile"
done

answered Feb 21 '19 at 14:36

E D

41
2

Welcome to Stack Overflow! While this code may solve the question, [including an explanation](https://meta.stackexchange.com/questions/114762/explaining-entirely-code-based-answers) of how and why this solves the problem would really help to improve the quality of your post, and probably result in more up-votes. Remember that you are answering the question for readers in the future, not just the person asking now. Please [edit] your answer to add explanation, and give an indication of what limitations and assumptions apply. – double-beep Feb 21 '19 at 14:57
note the flag that's truly necessary here is `-r`, "append", although `-u`, "update", also includes it. – SpinUp __ A Davis Jun 23 '23 at 17:28

score 0 · Answer 4 · edited May 12 '16 at 12:14

I'm not sure if you can add files to bzip2 archives without first extracting. However here is one solution that just came to my mind (giving you the pseudoish algorithm):

1. For each [file] in [all small files]
    1.1 compress [file] into [file].bz2
    1.2 (optionally verify the process in some way)
    1.3 delete [file]
2. For each [bzfile] in [all bzip files from step 1]
    2.1 append to tar (tar rvf compressedfiles.tar [bzfile]
    2.2 (optionally verify the process in some way)
    2.3 delete [bzfile]

Now you should have a tar file containing all files individually bzip2:ed files. The question is how much overhead bzip2 adds to the individual files. This needs to be tested.

Deleting files after adding to tar archive

4 Answers4

Method:

Solution:

Linked