0

I've started to get some problems with corrupt gzip files. I'm not sure exactly when it happens and the colleague how set our storage have quit etc. so I'm not an expert in cron jobs etc. but this is how it looks today:

/var/spool/new_files/*.csv
{
        daily
        rotate 12
        missingok
        notifempty
        delaycompress
        compress
        sharedscripts
        postrotate
                service capture_data restart >/dev/null 2>&1 || true
        endscript
}

In principle at midnight the script restart all csv files in var/spool/new_files/, change the name to them (increment them by 1) and gzip the one which is then named "2" and moves that to our long time storage.

I don't know if the files are corrupt just after they have been gzip or if this happens during the "transfer" to the storage. If I run zcat file_name | tail I get an invalid compressed data--length error. This error happens randomly 1-3 times per month.

So the first thing I want to do is to run gzip -k and keep the original,

  • Check if the files are corrupt after they have been gziped
    • Retry once
    • If this also fails add an error in logs
    • Stop the cron job
  • If the gzip file is ok after creation move it to long time storage
  • Test if they are ok, if not:
    • Retry once
    • If this also fails add an error in logs
    • Stop the cron job
  • Throw away the original file

Does the logic that I suggest make sense? Any suggestions how to add it to the cron job?

axel_ande
  • 359
  • 1
  • 4
  • 20

1 Answers1

0

It seems ok... for the integrity check you could see something here How to check if a Unix .tar.gz file is a valid file without uncompressing?

You can make a .sh where you add all the commands you need and after that you can add the script .sh in the crontab in:

/var/spool/cron/

if you want to run .sh script with root just add or modify /var/spool/cron/root file... in a similar way you can add cron runned by other users.

the cron would be something like:

0 0 * * * sh <path to your sh>
Gicu Aftene
  • 502
  • 2
  • 9
  • I'm not so good with linux and bash scripts etc. but I assume that I'll add the file to the list mentioned above between `sharedscripts` and `postrotate`? And the how do I write the logic? Can i write something like: `if (tar -tzf file.gz >/dev/null) { rm file} else { gzip file -k if (tar -tzf file.gz >/dev/null) { rm file} else {write error?}} ` (How do I run the `compress` command and keep the original?) – axel_ande Jul 14 '22 at 11:50
  • Yeah you can use conditional logics, regex, run every command that you can run on your command line ecc... you can find a lot of examples and a lot of forums in indexed pages. For the compress comand look [HERE](https://www.cyberciti.biz/faq/how-to-gzip-and-keep-original-file-on-unix-or-linux-command-line/) – Gicu Aftene Jul 14 '22 at 12:43