9

What's the proper way to log any errors or warnings when performing a quiet rsync?

This is what I currently run from my crontab:

gsutil -m -q rsync -r -C /mount1/share/folder gs://my-bucket-1/folder/ > /mount2/share/folder/gsutil.log

Since the log file is always completely empty and I'm uploading terabytes of data I'm starting to think that maybe even errors and warnings are being supressed.

fredrik
  • 9,631
  • 16
  • 72
  • 132
  • 1
    Errors are still reported with `-q` but they should get sent to stderr instead of stdout. Can you redirect stderr as well? In general though, you shouldn't receive many errors, if any. – jterrace Dec 09 '14 at 16:34
  • Isn't it just too good to be true not to get any errors? ;) How about `command 2>&1 >> gsutil.log`. Is that what you mean? It's hard to know if it is really working since there is absolutely nothing being logged to the file. – fredrik Dec 10 '14 at 13:03
  • If you want progress information, why not remove -q? – jterrace Dec 10 '14 at 15:06
  • If possible, I only want errors and warnings logged since I'm transferring a large amount of files. The log would get enormous if everything (INFO level) would get logged. – fredrik Dec 10 '14 at 15:08
  • I simply removed the -q option, and noticed that nothing gets logged to that file I'm piping output to. All INFO level messages are printed in my terminal. I wonder what I'm doing wrong... – fredrik Dec 10 '14 at 15:50
  • I think you've got your redirect set up wrong. You want `command >> gsutil.log 2>&1`. – jterrace Dec 10 '14 at 16:38
  • 1
    Ah, yes – you are right about `command >> gsutil.log 2>&1`. Thank you. Still not getting anything in the log when using -q though... but I guess that's a good thing! :) – fredrik Dec 11 '14 at 14:55

1 Answers1

10

After having realized that this is related to how you pipe stdout and/or stderr to files in general, the answer really lies within this existing thread: How to redirect both stdout and stderr to a file

So a simple solution to log as much as possible into one single log file could be something like:

gsutil -d rsync [src] [dst] &> [logfile]

...where -d enables debug output. I found this to be the only way to show files which were affected by an error such as CommandException: 3 files/objects could not be copied. Please note that -d exposes authentication credentials.

Community
  • 1
  • 1
fredrik
  • 9,631
  • 16
  • 72
  • 132
  • is this command allows the command to execute even if SSH connection breaks ? – arvindwill Jun 17 '18 at 14:09
  • 1
    working eg: gsutil -m rsync -r  /mnt/disks/stutzen.co/ gs://stutzen.me/sbkp/vm10-vm/ &> /tmp/ry1.txt & – arvindwill Jun 17 '18 at 14:27
  • 2
    -d is for Delete extra files under dst_url not found under src_url, use -D instead. – Khalid K Feb 24 '20 at 03:05
  • This is debugging. Not logging. It's really just not useful. -d after the rsync is for delete. -d before the rsync is debug. Just confusing and not useful. It should just actually log the relevant/important information. – nroose Mar 29 '23 at 04:26