7-zip an in-use file

Question

We are running a number of Apache Tomcat servers. A good portion of the logs rollover as they should based on the hour or at the end of the day. However, there are still a number that remain locked by Apache Tomcat, or web services.

Have actually logged this as a bug at - http://sourceforge.net/p/sevenzip/bugs/1488/

Have seen dbenham's awesome post (NOTE: it is under the second "EDIT" header, to what I am referring to) about how to deal with an in-use file. However, I am still running into contention with in-use files. I know WinZip Command-line maybe an option, but I really don't want to use that.

We have a nightly process to come in, which is much has more functions than this, but for illustration we are going to use the following:

    FOR /r "C:\tomcat\logs" %%X IN (*) DO (
    7za a -y D:\17\%%~nxX.zip %%X
    )
    PAUSE

So, what we see as a result are a number or 22-byte, empty zip files. Yes, we know not to zip an "in-use" file, but if it comes in does a directory list, we see this issue.

Is there something that we can do?

Volume in drive D is New Volume
Volume Serial Number is A476-BD38

Directory of D:\17

03/07/2015 02:05 PM <DIR> .
03/07/2015 02:05 PM <DIR> ..
03/07/2015 02:05 PM 403 catalina.2015-03-06.log.zip
03/07/2015 02:05 PM 22 catalina.2015-03-07.log.zip
03/07/2015 02:05 PM 22 host-manager.2015-02-23.log.zip
03/07/2015 02:05 PM 22 localhost.2015-02-23.log.zip
03/07/2015 02:05 PM 22 manager.2015-02-24.log.zip
03/07/2015 02:05 PM 22 probe.log.zip
6 File(s) 513 bytes
2 Dir(s) 51,783,634,944 bytes free

Normally, I would not really care about 22-byte files, but we have to watch them as they are being sent nightly to a vendor for report processing.

Looking at the script in how we are using it, we have:

REM SET VARIABLES
FOR /f "tokens=2-4 delims=.:/ " %%a IN ("%date%") DO SET CurrentDate=%%c-%%a-%%b

SETLOCAL ENABLEDELAYEDEXPANSION
FOR /r "D:\17" %%X IN (*) DO (
    SET FileName=%%~nxX
    FOR /f "tokens=2-4 delims=.:/ " %%a IN ("%%~tX") DO SET FDate=%%c-%%a-%%b
    REM CHECK FOR UNLOCKED FILE
    REM https://stackoverflow.com/a/10520609/1333331
    2>nul (   >>"D:\17\!FileName!" (call )) && (
    REM DO STUFF HERE
    IF "%CurrentDate%" NEQ "!FDate!" (
        IF %%~zX GTR 0 (
            ECHO ARCHIVING "D:\logs\!FileName!" >>%logfile%
            7za.exe -tzip -y a "D:\Archive\some.zip" "D:\17\!FileName!" && DEL "D:\17\!FileName!"
        )
    )
    REM END OF UNLOCKED ZONE
    ) || (
    ECHO FILE IS LOCKED
    )
)
ENDLOCAL

Thanks!

Answer 1

I've had to deal with something similar for a recent project and opted to clone the source directory with robocopy first in order to escape the in-use file problem. Robocopy offers /z, /b, and /zb modes to copy files that in Restartable, Backup, and Backup+Restartable modes, respectively. Using /zb allows it to copy in-use files to the destination directory of your choice. I also add "/r:1 /w:1" to robocopy calls to improve performance when it chokes on a file or runs out of space (/r:1 = retry only once, /w:1 = wait only one second between retries). Better safe than sorry.

This will occupy more space as it duplicates the logs, but will save you some trouble with the in-use files. Changing your original code to the following would get the results you're after (and delete the file copies as they are compressed):

robocopy "c:\tomcat\logs" "c:\temp\tomcat\logs" /zb /s /e /mir /r:1 /w:1
FOR /r "C:\temp\tomcat\logs" %%X IN (*) DO (
    7za a -sdel -y D:\17\%%~nxX.zip %%X
)
robocopy "C:\temp\tomcat\logs" "C:\temp\tomcat\logs" /s /move

The first line duplicates the logs folder to temp. The 7za change (-sdel) deletes the logs in the temp folder as they're compressed. The last line removes any empty sub-directories from the temporary logs folder.

If you don't mind using the extra space rather than using the additional CPU+RAM+time each pass you can leave a permanent cache in the temp folder and overwrite only the newer files using:

robocopy "c:\tomcat\logs" "c:\temp\tomcat\logs" /zb /s /e /mir /r:1 /w:1
FOR /r "C:\temp\tomcat\logs" %%X IN (*) DO (
    7za a -y D:\17\%%~nxX.zip %%X
)

Answer 2

I know this thread is old but I had the same issue recently and solved it by using the '-sse' switch on 7z.exe. This will stop an empty archive being created if a file is in use.

Answer 3

I have been able to figure out a pretty major part of this issue.

Problem we have been facing is the use of Touch.exe from cygwin and Unxutils.

If you read on how to create a zero-byte file, you will see the following article when doing a search for this..

http://www.computerhope.com/issues/ch001314.htm#linux

And, as a result, this got me thinking that this is exactly my issue.. If the file does not exist or if it is in use by Tomcat or other web services, touch is going to create a zero-byte file anyway (Note: I did think that I move off the files from previous days processing into working folders and archive from there). As a result, with a zero-byte file, we were getting 22-byte "air-zip" files. Doh!!!

The reason we are using touch is that we wanted to re-timestamp the file with the date when the file was generated, not necessarily the date the files were archived as we have about 600GB of web log data we have archived off. We were able to work around this by using a small PowerShell script and that way, we remove any risk of incorrect files being generated. Yes, this can be accomplished using a VBScript, but PowerShell we don't have to codesign our scripts.

REM RE-DATE THE FILES
PowerShell -file D:\work\scripts\redate.ps1

and the content of the re-date PowerShell script is (Your milage may vary as we have the date variables as part of the file name):

$1dir='D:\logs\Archives\SubLog'
$2dir='D:\logs\Archives\Tomcat'
$3dir='D:\logs\Archives'

if (Test-Path "$1dir\*.zip")
{
$files = Get-ChildItem $1dir *.zip
    foreach ($file in $files)
    {
        $filename = $file.name
        $fullname = $file.FullName
        $DT = $file.name.substring(0, 8)
        $mm = $file.name.substring(4, 2)
        $dd = $file.name.substring(6, 2)
        $yy = $file.name.substring(0, 4)
        $DTNew = [datetime]"$mm/$dd/$yy"
        (dir $fullname).lastwritetime = $DTNew
    }
}

if (Test-Path "$2dir\*.zip")
{
$files = Get-ChildItem $2dir *.zip
foreach ($file in $files)
{
        $filename = $file.name
        $fullname = $file.FullName
        $mm = $filename.Split("-")[2]
        $dd = $filename.Split("-")[3]
        $yy = $filename.Split("-")[4]
        $DateVar = [datetime]"$mm/$dd/$yy"
        (dir $fullname).lastwritetime = $DateVar
}
}

if (Test-Path "$3dir\*.zip")
{
$files = Get-ChildItem $3dir *.zip
foreach ($file in $files)
{
        $filename = $file.name
        $fullname = $file.FullName
        $yy = $filename.Split("-")[1]
        $mm = $filename.Split("-")[2]
        $dd = $filename.Split("-")[3]
        $DateVar = [datetime]"$mm/$dd/$yy"
        (dir $fullname).lastwritetime = $DateVar
}
}