How to check if a Unix .tar.gz file is a valid file without uncompressing?

Question

I have found the question How to determine if data is valid tar file without a file?, but I was wondering: is there a ready made command line solution?

Rob Wells · Accepted Answer · 2015-07-23T15:58:46.007

153

What about just getting a listing of the tarball and throw away the output, rather than decompressing the file?

tar -tzf my_tar.tar.gz >/dev/null

Edited as per comment. Thanks zrajm!

Edit as per comment. Thanks Frozen Flame! This test in no way implies integrity of the data. Because it was designed as a tape archival utility most implementations of tar will allow multiple copies of the same file!

edited Jul 23 '15 at 15:58

answered Jan 04 '10 at 19:51

Rob Wells

36,220
13
81
146

18

Why use `-v` if you're just piping the output to `/dev/null`? – zrajm Dec 29 '13 at 13:38
5

The `-z` option is also unneeded. It does nothing in extract or list mode. – asmeurer Jan 14 '14 at 22:07
How about matching the SHA-1 hash of the source file with the transferred file. – Asad Hasan Feb 16 '14 at 17:31
1

@asmeurer Re: `-z` That's definitely the case with GNU tar -- do you know if this is this true elsewhere (BSD, etc.)? – belacqua May 13 '14 at 17:59
1

@belacqua For others' information, as in the manpage of BSD TAR, `-z`: "... In extract or list modes, this option is ignored." – Frozen Flame Jul 09 '15 at 07:48
2

@bobwells But does successful uncompressing or content files listing imply data integrity of the `tar.gz`? Any support information? – Frozen Flame Jul 09 '15 at 07:53
@FrozenFlame it doesn't. In fact, because tar is a utility originally designed for tape archive you can create a tar archive with multiple copies of the one file! Though some modern variants will optimise this away if you want. :-) – Rob Wells Jul 23 '15 at 15:56
1

How are we suppose to get the `.tar.gz` list of files if we are piping the output to `/dev/null` ? – pkaramol Aug 21 '18 at 10:41
@pkaramol This is for when the objective is just to verify that the file is valid. If you need to get the file names, don't throw away the contents of the listing. :-) – Rob Wells Oct 09 '18 at 14:56
@Frozen Flame I doubt it: I had a bad copy (partial) of a tar file where file listing and decompression worked flowlessly, but some files were missing. – Kiruahxh Oct 21 '21 at 10:16

John Boker · Answer 2 · 2020-06-01T14:25:56.443

119

you could probably use the gzip -t option to test the files integrity

http://linux.about.com/od/commands/l/blcmdl1_gzip.htm

from: http://unix.ittoolbox.com/groups/technical-functional/shellscript-l/how-to-test-file-integrity-of-targz-1138880

To test the gzip file is not corrupt:

gunzip -t file.tar.gz

To test the tar file inside is not corrupt:

gunzip -c file.tar.gz | tar -t > /dev/null

As part of the backup you could probably just run the latter command and check the value of $? afterwards for a 0 (success) value. If either the tar or the gzip has an issue, $? will have a non zero value.

edited Jun 01 '20 at 14:25

answered Jan 04 '10 at 19:51

John Boker

82,559
17
97
130

3

And `bzip2 -t file.bz2` for bz2 files. – asmeurer Jan 14 '14 at 22:08
6

Isn't this just using two commands to do what "tar -tzf my_tar.tar.gz >/dev/null" does? – Rob Wells Apr 16 '18 at 18:11
Isn't it supposed to be `tar -t > /dev/null` (note: `t` vs `-t`)? – Intrastellar Explorer May 30 '20 at 03:29
1

@IntrastellarExplorer that is a typo on my part, although it should still work without the hyphen. I'm just used to the old style options on tar. https://unix.stackexchange.com/questions/394060/command-options-with-hyphen-or-not – John Boker Jun 01 '20 at 14:25
At first glance I feel this is the most correct answer here, as at least the integrity of the zipped 'file,tar' is checked against the checksum contained in .gz file. However for individual files tared in 'files.tar' relying on tar -t is riskier because random byte errors may very likely not be detected. Anyhow I suppose there is nothing better than 'tar -t" you can get here. – grenix Feb 14 '22 at 12:43

score 38 · Answer 3 · edited Jan 31 '15 at 23:33

38

If you want to do a real test extract of a tar file without extracting to disk, use the -O option. This spews the extract to standard output instead of the filesystem. If the tar file is corrupt, the process will abort with an error.

Example of failed tar ball test...

$ echo "this will not pass the test" > hello.tgz
$ tar -xvzf hello.tgz -O > /dev/null
gzip: stdin: not in gzip format
tar: Child returned status 1
tar: Error exit delayed from previous errors
$ rm hello.*

Working Example...

$ ls hello*
ls: hello*: No such file or directory
$ echo "hello1" > hello1.txt
$ echo "hello2" > hello2.txt
$ tar -cvzf hello.tgz hello[12].txt
hello1.txt
hello2.txt
$ rm hello[12].txt
$ ls hello*
hello.tgz
$ tar -xvzf hello.tgz -O
hello1.txt
hello1
hello2.txt
hello2
$ ls hello*
hello.tgz
$ tar -xvzf hello.tgz
hello1.txt
hello2.txt
$ ls hello*
hello1.txt  hello2.txt  hello.tgz
$ rm hello*

edited Jan 31 '15 at 23:33

Giacomo1968

25,759
11
71
103

answered Jan 17 '12 at 19:41

Ossie Moore

391
3
3

2

It seems to me that the best test is this one. It truly extracts each file and makes sure there are no errors. – sleeves Jan 23 '12 at 20:36
Really useful. I've made a shell script, add an argument hook to pass the path of the file and put it in my path :) [ tar -xvzf $1 -O > /dev/null ] – smonff Aug 22 '12 at 19:19
@sleeves Why do you think that it is better than the accepted answer? tar -tvzf hello.tgz > /dev/null also gives the same error. – dash17291 Mar 04 '13 at 15:48
7

@dash17291 I say this because I expect it to be a tough problem to prove that for all cases, a -tvf will catch all errors or corruptions that a -xvf. In other words, -xvf will catch all that -tvf, but I cannot say the converse is true. – sleeves May 25 '13 at 02:20
Maybe you should use `> /dev/null` for the working example, too. – moi Jul 23 '16 at 15:34

score 12 · Answer 4 · edited Aug 20 '15 at 19:10

12

You can also check contents of *.tag.gz file using pigz (parallel gzip) to speedup the archive check:

pigz -cvdp number_of_threads /[...]path[...]/archive_name.tar.gz | tar -tv > /dev/null

edited Aug 20 '15 at 19:10

KalEl

8,978
13
47
56

answered Dec 09 '12 at 13:09

ein

121
1
2

1

Mini benchmark: running it with pigz -cvd: 80s, while running with accepted answer tar -tzv: 143s on a 22G archive. – Wadih M. Aug 30 '18 at 13:18
how do u remove the notification from cronjob about "tar: Removing leading `/' from member names" when using this method ? – MaXi32 Jun 01 '20 at 14:42

score 5 · Answer 5 · answered Aug 01 '13 at 17:29

A nice option is to use tar -tvvf <filePath> which adds a line that reports the kind of file.

Example in a valid .tar file:

> tar -tvvf filename.tar 
drwxr-xr-x  0 diegoreymendez staff       0 Jul 31 12:46 ./testfolder2/
-rw-r--r--  0 diegoreymendez staff      82 Jul 31 12:46 ./testfolder2/._.DS_Store
-rw-r--r--  0 diegoreymendez staff    6148 Jul 31 12:46 ./testfolder2/.DS_Store
drwxr-xr-x  0 diegoreymendez staff       0 Jul 31 12:42 ./testfolder2/testfolder/
-rw-r--r--  0 diegoreymendez staff      82 Jul 31 12:42 ./testfolder2/testfolder/._.DS_Store
-rw-r--r--  0 diegoreymendez staff    6148 Jul 31 12:42 ./testfolder2/testfolder/.DS_Store
-rw-r--r--  0 diegoreymendez staff  325377 Jul  5 09:50 ./testfolder2/testfolder/Scala.pages
Archive Format: POSIX ustar format,  Compression: none

Corrupted .tar file:

> tar -tvvf corrupted.tar 
tar: Unrecognized archive format
Archive Format: (null),  Compression: none
tar: Error exit delayed from previous errors.

This occurs on BSD tar but not on GNU tar. – mcallister May 20 '14 at 16:48 — mcallister, May 20 '14 at 16:48

score 3 · Answer 6 · answered Feb 18 '16 at 21:32

3

I have tried the following command and they work well.

bzip2 -t file.bz2
gunzip -t file.gz

However, we can found these two command are time-consuming. Maybe we need some more quick way to determine the intact of the compress files.

answered Feb 18 '16 at 21:32

Shicheng Guo

1,233
16
19

score 1 · Answer 7 · edited Oct 07 '21 at 11:19

1

These are all very sub-optimal solutions. From the GZIP spec

ID2 (IDentification 2)
These have the fixed values ID1 = 31 (0x1f, \037), ID2 = 139 (0x8b, \213), to identify the file as being in gzip format.

Has to be coded into whatever language you're using.

edited Oct 07 '21 at 11:19

Community

1
1

answered May 12 '17 at 23:18

Kevin Liu

11
3

if the file has been truncated, that would seem to require full decompression to detect – philwalk Feb 19 '20 at 16:06

score 0 · Answer 8 · edited Jan 31 '15 at 23:33

0

> use the -O option. [...] If the tar file is corrupt, the process will abort with an error.

Sometimes yes, but sometimes not. Let's see an example of a corrupted file:

echo Pete > my_name
tar -cf my_data.tar my_name 

# // Simulate a corruption
sed < my_data.tar 's/Pete/Fool/' > my_data_now.tar
# // "my_data_now.tar" is the corrupted file

tar -xvf my_data_now.tar -O

It shows:

my_name
Fool

Even if you execute

echo $?

tar said that there was no error:

but the file was corrupted, it has now "Fool" instead of "Pete".

edited Jan 31 '15 at 23:33

Giacomo1968

25,759
11
71
103

answered Oct 16 '12 at 15:58

Sys

33
2

People rarely use tar files without any compression. I guess your remark regards only uncompressed files. – Jarekczek Nov 18 '12 at 12:47
8

You are confusing integrity with corruption. Your file has lost it's integrity but it is still an acceptable archive format. – Phil Oct 19 '15 at 08:34

How to check if a Unix .tar.gz file is a valid file without uncompressing?

8 Answers8

Linked

Related