1

I use protostuff-runtime to serialize object graphs. Some of these objects have reference to Guava immutable collections, such as ImmutableList and ImmutableSet. Protostuff is unable to deserialize these collections out of the box, because it tries to construct an instance and then "add" elements to it from the inputStream (which fails, since the collections are immutable).

Do you know of any library / protostuff plugin that does that out of the box? If not, is there a best practice to do this myself?

I've investigated, and found that protostuff has a concept of "delegate", that lets you take control of the serialization for specific types. It seems to be the answer to my problem, but I can't seem to get it working.

Here is what I have right now:

import com.dyuproject.protostuff.Schema;
import com.dyuproject.protostuff.runtime.RuntimeSchema;
import com.google.common.base.Preconditions;
import com.google.common.collect.ImmutableList;

import javax.annotation.Nonnull;
import javax.annotation.concurrent.Immutable;

/**
 * This is the POJO I want to serialize. Note that the {@code strings} field refers to an {@link ImmutableList}.
 */
@Immutable
public class Foo {

    public static final Schema<Foo> SCHEMA = RuntimeSchema.getSchema(Foo.class);

    @Nonnull
    private final ImmutableList<String> strings;

    public Foo(ImmutableList<String> strings) {
        this.strings = Preconditions.checkNotNull(strings);
    }

    @Nonnull
    public ImmutableList<String> getStrings() {
        return strings;
    }

    @Override
    public boolean equals(Object obj) {
        if (obj instanceof Foo) {
            Foo that = (Foo) obj;
            return this.strings.equals(that.strings);
        }
        return false;
    }

    @Override
    public int hashCode() {
        return strings.hashCode();
    }

}

import com.dyuproject.protostuff.*;
import com.dyuproject.protostuff.runtime.Delegate;
import com.dyuproject.protostuff.runtime.RuntimeSchema;
import com.google.common.collect.ImmutableList;
import com.google.common.collect.Lists;

import java.io.IOException;
import java.util.ArrayList;

public class ImmutableListDelegate implements Delegate<ImmutableList<?>> {

    private static final Schema<ArrayList> LIST_SCHEMA = RuntimeSchema.getSchema(ArrayList.class);

    @Override
    public WireFormat.FieldType getFieldType() {
        return WireFormat.FieldType.MESSAGE;
    }

    @Override
    public ImmutableList<?> readFrom(Input input) throws IOException {
        ArrayList<?> list = LIST_SCHEMA.newMessage();
        input.mergeObject(list, LIST_SCHEMA);
        return ImmutableList.copyOf(list);
    }

    @Override
    public void writeTo(Output output, int number, ImmutableList<?> value, boolean repeated) throws IOException {
        ArrayList<?> list = Lists.newArrayList(value);
        output.writeObject(number, list, LIST_SCHEMA, repeated);
        LIST_SCHEMA.writeTo(output, list);
    }

    @Override
    public void transfer(Pipe pipe, Input input, Output output, int number, boolean repeated) throws IOException {
        throw new UnsupportedOperationException("TODO");
    }

    @Override
    public Class<?> typeClass() {
        return ImmutableList.class;
    }
}

import com.dyuproject.protostuff.LinkedBuffer;
import com.dyuproject.protostuff.ProtostuffIOUtil;
import com.dyuproject.protostuff.runtime.DefaultIdStrategy;
import com.dyuproject.protostuff.runtime.RuntimeEnv;
import com.google.common.collect.ImmutableList;
import org.junit.Assert;
import org.junit.Before;
import org.junit.Test;

import java.io.ByteArrayInputStream;
import java.io.ByteArrayOutputStream;
import java.io.IOException;

public class ImmutableListDelegateTest {

    @Before
    public void before() {
        // registers the delegate
        if (RuntimeEnv.ID_STRATEGY instanceof DefaultIdStrategy) {
            ((DefaultIdStrategy) RuntimeEnv.ID_STRATEGY).registerDelegate(new ImmutableListDelegate());
        }
    }

    @Test
    public void testDelegate() throws IOException {
        Foo foo = new Foo(ImmutableList.of("foo"));

        Assert.assertEquals(foo, serializeThenDeserialize(foo));
    }

    private Foo serializeThenDeserialize(Foo fooToSerialize) throws IOException {
        ByteArrayOutputStream out = new ByteArrayOutputStream();
        ProtostuffIOUtil.writeDelimitedTo(out, fooToSerialize, Foo.SCHEMA, buffer());
        ByteArrayInputStream in = new ByteArrayInputStream(out.toByteArray());
        Foo fooDeserialized = Foo.SCHEMA.newMessage();
        ProtostuffIOUtil.mergeDelimitedFrom(in, fooDeserialized, Foo.SCHEMA, buffer());
        return fooDeserialized;
    }

    private LinkedBuffer buffer() {
        return LinkedBuffer.allocate(LinkedBuffer.DEFAULT_BUFFER_SIZE);
    }
}

The test fails with the following exception, which seems to mean that my delegate only deserializes null values:

java.lang.NullPointerException
    at com.google.common.base.Preconditions.checkNotNull(Preconditions.java:191)
    at com.google.common.collect.SingletonImmutableList.<init>(SingletonImmutableList.java:40)
    at com.google.common.collect.ImmutableList.asImmutableList(ImmutableList.java:305)
    at com.google.common.collect.ImmutableList.copyFromCollection(ImmutableList.java:314)
    at com.google.common.collect.ImmutableList.copyOf(ImmutableList.java:253)
    at test.ImmutableListDelegate.readFrom(ImmutableListDelegate.java:25)
    at test.ImmutableListDelegate.readFrom(ImmutableListDelegate.java:12)
    at com.dyuproject.protostuff.runtime.RuntimeUnsafeFieldFactory$19$1.mergeFrom(RuntimeUnsafeFieldFactory.java:1111)
    at com.dyuproject.protostuff.runtime.MappedSchema.mergeFrom(MappedSchema.java:188)
    at com.dyuproject.protostuff.IOUtil.mergeDelimitedFrom(IOUtil.java:109)
    at com.dyuproject.protostuff.ProtostuffIOUtil.mergeDelimitedFrom(ProtostuffIOUtil.java:151)
    at test.ImmutableListDelegateTest.serializeThenDeserialize(ImmutableListDelegateTest.java:38)
    at test.ImmutableListDelegateTest.testDelegate(ImmutableListDelegateTest.java:30)

Is this the right approach? What am I missing?

This is not a duplicate of the What is a Null Pointer Exception, and how do I fix it? question, which makes no sense. The fact that I mentioned that an NPE is thrown when trying to use a Protostuff delegate to de-serialize immutable collections, doesn't mean that this duplicates the "What is a NPE?" question in any way, shape, or form.

Community
  • 1
  • 1
Etienne Neveu
  • 12,604
  • 9
  • 36
  • 59
  • Hi, dont know much about protostuff, but I did found this bug once. And it was in Guava implementation... I changed to a copy list (that would work fine for me)... – Plínio Pantaleão Sep 10 '12 at 13:24
  • @PlínioPantaleão My workaround for now is to store an ArrayList instead (constructing a new ArrayList in the constructor, and constructing an ImmutableList in the getter). But I'd like to avoid doing that :/ – Etienne Neveu Sep 10 '12 at 13:29
  • I found this bug on version 11 of guava. You should see if it still happens in current version – Plínio Pantaleão Sep 10 '12 at 13:31
  • 1
    I'm using Guava 13.0.1, but I don't think it's a Guava bug. It's more that protostuff expects collections to be mutable during deserialization / merging, which is not the case of Guava's immutable collections. – Etienne Neveu Sep 10 '12 at 13:33
  • I don't thing that. If that is true, your Delegate approach should had solved the problem right? – Plínio Pantaleão Sep 10 '12 at 14:00
  • @Plínio Pantaleão: What Guava bug? Here it looks like you forgot that `ImmutableList` can't contain `null`, but this is no bug and well documented. – maaartinus Sep 10 '12 at 15:27
  • I know ImmutableList cannot contain null values. That's not the problem. My ImmutableList contains non-null values before serialization. The problem is that the delegate seems to be filled with null values. Maybe I implemented the delegate incorrectly, or maybe there is a better approach than a delegate to serialize Guava's immutable collections using Protostuff... – Etienne Neveu Feb 02 '16 at 00:39

1 Answers1

1

Everything looks fine and the

java.lang.NullPointerException
    at com.google.common.base.Preconditions.checkNotNull(Preconditions.java:191)
    at com.google.common.collect.SingletonImmutableList.<init>(SingletonImmutableList.java:40)

says that you're trying to put null into an ImmutableList, which is forbidden. To be sure, inspect you list just before the failing line. Make sure your input json doesn't look like [null].

maaartinus
  • 44,714
  • 32
  • 161
  • 320
  • The problem is indeed that I'm trying to construct an `ImmutableList` with a `null` element. But I don't understand why protostuff returns a list containing a `null` element when I do `ArrayList> list = LIST_SCHEMA.newMessage(); input.mergeObject(list, LIST_SCHEMA);` in the Delegate's `readFrom()` method. My `writeTo()` and `readFrom()` methods may be wrong, or I might be using the RuntimeSchema incorrectly. I just want a Delegate to tell protostuff "ok, instead of serializing this ImmutableList, why don't you take this ArrayList instead and serialize it, since you know how to do that well". – Etienne Neveu Sep 10 '12 at 15:53
  • I hoped you could solve this part yourself... I did it [with Gson](http://stackoverflow.com/a/7773201/581205) and it'd trivial if there were no generics. Can you deserialize an arbitrary `ArrayList` without knowing it's member type? – maaartinus Sep 10 '12 at 16:34
  • That's actually where I'm stuck with my approach. I don't really understand how I can ask protostuff to serialize a generic List. I could have worded my question to ask specifically for that bit of information, but I wanted to know if there were other better approaches than my Delegate. – Etienne Neveu Sep 20 '12 at 08:51
  • @eneveu: No idea... maybe start another question here or look for a mailing list. This one concerns the `ImmutableList` problem, assuming the `List` problem has been solved. – maaartinus Sep 20 '12 at 11:45