JVM crashes at libzip.so

Question

My JVM keeps on crashing continuously and unexpectedly at libzip.so all the time. I've filed the bug with Oracle, but decided to see if anyone here has experienced the problem and if so, how did you deal with it? This is a web app running off

Linux 2.6.34-gentoo-r6 #1 SMP Fri Sep 24 00:15:06 EDT 2010 i686 Intel(R) Xeon(R) CPU X5460 @ 3.16GHz GenuineIntel GNU/Linux
Tomcat 7.0.14 with jsvc.

I've included a snapshot of the bug report below. Its a standalone server, nobody is accessing any of tomcat's jar or any other jars while running and its not hosted from NFS.

 SIGSEGV (0xb) at pc=0xb6a72295, pid=19470, tid=241171312

 JRE version: 6.0_29-b11  Java VM: Java HotSpot(TM) Server VM (20.4-b02 mixed mode linux-x86 )

 Problematic frame:  C  [libzip.so+0x5295]  double+0x45

 If you would like to submit a bug report, please visit:    http://java.sun.com/webapps/bugreport/crash.jsp  The crash happened outside the Java Virtual Machine in native code.  See problematic frame for where to report the bug.


---------------  T H R E A D  ---------------

Current thread (0x1044dc00):  JavaThread "catalina-exec-177" daemon [_thread_in_native, id=21423, stack(0x0e5af000,0x0e600000)]

siginfo:si_signo=SIGSEGV: si_errno=0, si_code=2 (SEGV_ACCERR), si_addr=0x10dfc000

Registers: EAX=0x10d00018, EBX=0xb6a7dabc, ECX=0x000fbfe6, EDX=0x00000000 ESP=0x0e5febf0, EBP=0x0e5fec18, ESI=0x0eadc098, EDI=0x10458800 EIP=0xb6a72295, EFLAGS=0x00010246, CR2=0x10dfc000

Top of Stack: (sp=0x0e5febf0) 0x0e5febf0:   b7869118 00000000 0ef2e648 b6a71216 0x0e5fec00:   13d1f690 10c492b0 0000bfe1 b6a7dabc 0x0e5fec10: 10458800 0ef2e648 0e5fec48 b6a7134d 0x0e5fec20:   10458800 00000004 0e5fec48 b77b727c 0x0e5fec30:   00000007 3d4af570 ffffffff b6a7dabc 0x0e5fec40:   31fe9ad0 1044dd20 0e5feca8 b6a6f585 0x0e5fec50:   0ef2e648 00000004 00000002 00000000 0x0e5fec60:   10b59810 3d5cff58 00000002 00004114

Instructions: (pc=0xb6a72295) 0xb6a72275:   05 01 00 00 0f 86 79 03 00 00 83 fa 02 76 41 8b 0xb6a72285:   57 40 8b 4f 50 8b 47 30 8b 77 38 d3 e2 8b 4f 64 0xb6a72295:   0f b6 44 01 02 31 c2 8b 47 4c 21 c2 8b 47 2c 89 0xb6a722a5:   57 40 21 c1 8b 47 3c 0f b7 04 50 89 45 f0 66 89

Register to memory mapping:

EAX=0x10d00018 is an unknown value EBX=0xb6a7dabc: <offset 0x10abc> in /usr/local/jdk1.6.0_29/jre/lib/i386/libzip.so at 0xb6a6d000 ECX=0x000fbfe6 is an unknown value EDX=0x00000000 is an unknown value ESP=0x0e5febf0 is pointing into the stack for thread: 0x1044dc00 EBP=0x0e5fec18 is pointing into the stack for thread: 0x1044dc00 ESI=0x0eadc098 is an unknown value EDI=0x10458800 is an unknown value


Stack: [0x0e5af000,0x0e600000],  sp=0x0e5febf0,  free space=318k Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code) C  [libzip.so+0x5295]  double+0x45 C  [libzip.so+0x434d]  double+0x10d C  [libzip.so+0x2585]  Java_java_util_zip_Deflater_deflateBytes+0x355 J  java.util.zip.Deflater.deflateBytes(J[BII)I

Java frames: (J=compiled Java code, j=interpreted, Vv=VM code) J  java.util.zip.Deflater.deflateBytes(J[BII)I J  java.util.zip.GZIPOutputStream.finish()V J  org.apache.coyote.http11.Http11NioProcessor.actionInternal(Lorg/apache/coyote/ActionCode;Ljava/lang/Object;)V J  org.apache.coyote.http11.AbstractHttp11Processor.action(Lorg/apache/coyote/ActionCode;Ljava/lang/Object;)V J  org.apache.catalina.connector.Response.finishResponse()V J  org.apache.catalina.connector.CoyoteAdapter.service(Lorg/apache/coyote/Request;Lorg/apache/coyote/Response;)V J  org.apache.coyote.http11.Http11NioProcessor.process(Lorg/apache/tomcat/util/net/NioChannel;)Lorg/apache/tomcat/util/net/AbstractEndpoint$Handler$SocketState; J  org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run()V J  java.util.concurrent.ThreadPoolExecutor$Worker.run()V j  java.lang.Thread.run()V+11 v  ~StubRoutines::call_stub

Any hints appreciated.

Answer 1

This is the patched class, sort of nightly built and need to test it. The JGZipOutputStream class can be found here: Creating gzip file using jzlib

JZlib can be found at http://www.jcraft.com/jzlib/

You can patch it either by adding a small jar in the bootstrap path ( -Xbootclasspath/a: ) or unzip/jar the tomcat itself.

Extra bonus is that java impl. is actually faster than the native code.

/*
 *  Licensed to the Apache Software Foundation (ASF) under one or more
 *  contributor license agreements.  See the NOTICE file distributed with
 *  this work for additional information regarding copyright ownership.
 *  The ASF licenses this file to You under the Apache License, Version 2.0
 *  (the "License"); you may not use this file except in compliance with
 *  the License.  You may obtain a copy of the License at
 *
 *      http://www.apache.org/licenses/LICENSE-2.0
 *
 *  Unless required by applicable law or agreed to in writing, software
 *  distributed under the License is distributed on an "AS IS" BASIS,
 *  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 *  See the License for the specific language governing permissions and
 *  limitations under the License.
 */

package org.apache.coyote.http11.filters;

import java.io.IOException;
import java.io.OutputStream;
import java.util.zip.GZIPOutputStream;

import org.apache.coyote.OutputBuffer;
import org.apache.coyote.Response;
import org.apache.coyote.http11.OutputFilter;
import org.apache.tomcat.util.buf.ByteChunk;

import bestsss.util.JGZipOutputStream;

import com.jcraft.jzlib.JZlib;

/**
 * Gzip output filter.
 * 
 * @author Remy Maucherat
 */
public class GzipOutputFilter implements OutputFilter {


    /**
     * Logger.
     */
    protected static org.apache.juli.logging.Log log =
        org.apache.juli.logging.LogFactory.getLog(GzipOutputFilter.class);


    // ----------------------------------------------------- Instance Variables


    /**
     * Next buffer in the pipeline.
     */
    protected OutputBuffer buffer;


    /**
     * Compression output stream.
     */
    protected OutputStream compressionStream = null;


    /**
     * Fake internal output stream.
     */
    protected OutputStream fakeOutputStream = new FakeOutputStream();


    // --------------------------------------------------- OutputBuffer Methods


    /**
     * Write some bytes.
     * 
     * @return number of bytes written by the filter
     */
    @Override
    public int doWrite(ByteChunk chunk, Response res)
        throws IOException {
        if (compressionStream == null) {
            compressionStream = new JGZipOutputStream(fakeOutputStream, new byte[Math.min(32768, Math.max(2048, Integer.highestOneBit(chunk.getLength()-1)<<1))]);
        }
        compressionStream.write(chunk.getBytes(), chunk.getStart(), 
                                chunk.getLength());
        return chunk.getLength();
    }


    @Override
    public long getBytesWritten() {
        return buffer.getBytesWritten();
    }


    // --------------------------------------------------- OutputFilter Methods

    /**
     * Added to allow flushing to happen for the gzip'ed outputstream
     */
    public void flush() {
        if (compressionStream != null) {
            try {
                if (log.isDebugEnabled()) {
                    log.debug("Flushing the compression stream!");
                }
                compressionStream.flush();
            } catch (IOException e) {
                if (log.isDebugEnabled()) {
                    log.debug("Ignored exception while flushing gzip filter", e);
                }
            }
        }
    }

    /**
     * Some filters need additional parameters from the response. All the 
     * necessary reading can occur in that method, as this method is called
     * after the response header processing is complete.
     */
    @Override
    public void setResponse(Response response) {
        // NOOP: No need for parameters from response in this filter
    }


    /**
     * Set the next buffer in the filter pipeline.
     */
    @Override
    public void setBuffer(OutputBuffer buffer) {
        this.buffer = buffer;
    }


    /**
     * End the current request. It is acceptable to write extra bytes using
     * buffer.doWrite during the execution of this method.
     */
    @Override
    public long end()
        throws IOException {
        if (compressionStream == null) {
            compressionStream = new JGZipOutputStream(fakeOutputStream, new byte[128]);
        }        
        compressionStream.close();
        return ((OutputFilter) buffer).end();
    }


    /**
     * Make the filter ready to process the next request.
     */
    @Override
    public void recycle() {
        // Set compression stream to null
        compressionStream = null;
    }


    // ------------------------------------------- FakeOutputStream Inner Class


    protected class FakeOutputStream
        extends OutputStream {
        protected ByteChunk outputChunk = new ByteChunk();
        protected byte[] singleByteBuffer = new byte[1];
        @Override
        public void write(int b)
            throws IOException {
            // Shouldn't get used for good performance, but is needed for 
            // compatibility with Sun JDK 1.4.0
            singleByteBuffer[0] = (byte) (b & 0xff);
            outputChunk.setBytes(singleByteBuffer, 0, 1);
            buffer.doWrite(outputChunk, null);
        }
        @Override
        public void write(byte[] b, int off, int len)
            throws IOException {
            outputChunk.setBytes(b, off, len);
            buffer.doWrite(outputChunk, null);
        }
        @Override
        public void flush() throws IOException {/*NOOP*/}
        @Override
        public void close() throws IOException {/*NOOP*/}
    }


}