Creating a JSON with Gson makes the app crash

Question

I'm using Retrofit and Gson in order to upload a custom objects list to the server. I didn't have any problem to do it: tested with Mororola, Asus, and many other devices. Never a problem! Now I'm working with a Zebra smartphone, an industrial one, and I'm getting my app almost always crash during the creation of the JSON (I've logged that the app was writing the JSON before crashing). I'm checking the data with TypeAdapters so it shouldn't be a problem of data values. This is the log:

D/dalvikvm: JIT unchain all for threadid=13
W/dalvikvm: threadid=15: spin on suspend #1 threadid=13 (pcf=0)
W/dalvikvm: threadid=15: spin on suspend #2 threadid=13 (pcf=0)
I/dalvikvm: "Retrofit-Idle" prio=5 tid=15 RUNNABLE JIT
I/dalvikvm:   | group="main" sCount=0 dsCount=0 obj=0x41d52cc8 self=0x6065ec38
I/dalvikvm:   | sysTid=14452 nice=10 sched=0/0 cgrp=apps/bg_non_interactive handle=1599766536
I/dalvikvm:   | state=R schedstat=( 0 0 0 ) utm=9 stm=2 core=0
I/dalvikvm:     at java.lang.AbstractStringBuilder.enlargeBuffer(AbstractStringBuilder.java:~94)
I/dalvikvm:     at java.lang.AbstractStringBuilder.append0(AbstractStringBuilder.java:145)
I/dalvikvm:     at java.lang.StringBuffer.append(StringBuffer.java:219)
I/dalvikvm:     at java.io.StringWriter.write(StringWriter.java:167)
I/dalvikvm:     at com.google.gson.stream.JsonWriter.string(JsonWriter.java:570)
I/dalvikvm:     at com.google.gson.stream.JsonWriter.value(JsonWriter.java:419)
I/dalvikvm:     at my.company.app.rest.RestClient$1.write(RestClient.java:197)
I/dalvikvm:     at my.company.app.rest.RestClient$1.write(RestClient.java:165)
I/dalvikvm:     at com.google.gson.internal.bind.TypeAdapterRuntimeTypeWrapper.write(TypeAdapterRuntimeTypeWrapper.java:68)
I/dalvikvm:     at com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$1.write(ReflectiveTypeAdapterFactory.java:113)
I/dalvikvm:     at com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$Adapter.write(ReflectiveTypeAdapterFactory.java:240)
I/dalvikvm:     at com.google.gson.internal.bind.ObjectTypeAdapter.write(ObjectTypeAdapter.java:107)
I/dalvikvm:     at com.google.gson.internal.bind.TypeAdapterRuntimeTypeWrapper.write(TypeAdapterRuntimeTypeWrapper.java:68)
I/dalvikvm:     at com.google.gson.internal.bind.CollectionTypeAdapterFactory$Adapter.write(CollectionTypeAdapterFactory.java:96)
I/dalvikvm:     at com.google.gson.internal.bind.CollectionTypeAdapterFactory$Adapter.write(CollectionTypeAdapterFactory.java:60)
I/dalvikvm:     at com.google.gson.internal.bind.TypeAdapterRuntimeTypeWrapper.write(TypeAdapterRuntimeTypeWrapper.java:68)
I/dalvikvm:     at com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$1.write(ReflectiveTypeAdapterFactory.java:113)
I/dalvikvm:     at com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$Adapter.write(ReflectiveTypeAdapterFactory.java:240)
I/dalvikvm:     at com.google.gson.Gson.toJson(Gson.java:605)
I/dalvikvm:     at com.google.gson.Gson.toJson(Gson.java:584)
I/dalvikvm:     at com.google.gson.Gson.toJson(Gson.java:539)
I/dalvikvm:     at com.google.gson.Gson.toJson(Gson.java:519)
I/dalvikvm:     at retrofit.converter.GsonConverter.toBody(GsonConverter.java:80)
I/dalvikvm:     at retrofit.RequestBuilder.setArguments(RequestBuilder.java:375)
I/dalvikvm:     at retrofit.RestAdapter$RestHandler.invokeRequest(RestAdapter.java:298)
I/dalvikvm:     at retrofit.RestAdapter$RestHandler.access$100(RestAdapter.java:220)
I/dalvikvm:     at retrofit.RestAdapter$RestHandler$2.obtainResponse(RestAdapter.java:278)
I/dalvikvm:     at retrofit.CallbackRunnable.run(CallbackRunnable.java:42)
I/dalvikvm:     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1112)
I/dalvikvm:     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:587)
I/dalvikvm:     at retrofit.Platform$Android$2$1.run(Platform.java:142)
I/dalvikvm:     at java.lang.Thread.run(Thread.java:841)

I worked on it for days with no concrete result: could it be a memory problem? (I tested the app with older devices with less memory and I hadn't the problem.)

Thank you very much!

UPDATE

Can it be the Dalvik? I make the upload of a list of objects, splitting it in multiple json and post call using a while(boolean) to wait that the transmission completes before make the next one. Looking on the net I've found that the vendor-modified dalvik could be the "killer", not letting the thread wait in loop.....see this answer https://stackoverflow.com/a/20804679/2306946

Yeah I presume its a modified dalvik issue or maybe the phone is rooted? — Mike, Feb 02 '17 at 17:19
the phone isn't rooted.....aaaaaaargh if i'ts really the dalvik the problem I will kill zebra ! — gauss, Feb 02 '17 at 17:22
no, I'm not creating it in ui thread! but i tried also to do it in ui thread....same crash — gauss, Feb 03 '17 at 08:28

score 1 · Answer 1 · edited Apr 13 '17 at 12:36

Almost an exact duplicate can be reproduced at a "desktop" JVM. For example, this is what I get if I flood my test echo HTTP server:

retrofit.RetrofitError: Java heap space
    at retrofit.RetrofitError.unexpectedError(RetrofitError.java:44)
    at retrofit.RestAdapter$RestHandler.invokeRequest(RestAdapter.java:400)
    at retrofit.RestAdapter$RestHandler.access$100(RestAdapter.java:220)
    at retrofit.RestAdapter$RestHandler$2.obtainResponse(RestAdapter.java:278)
    at retrofit.CallbackRunnable.run(CallbackRunnable.java:42)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at retrofit.Platform$Base$2$1.run(Platform.java:94)
    at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.OutOfMemoryError: Java heap space
    at java.util.Arrays.copyOf(Arrays.java:3332)
    at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:137)
    at java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:121)
    at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:421)
    at java.lang.StringBuffer.append(StringBuffer.java:272)
    at java.io.StringWriter.write(StringWriter.java:101)
    at com.google.gson.stream.JsonWriter.open(JsonWriter.java:327)
    at com.google.gson.stream.JsonWriter.beginObject(JsonWriter.java:308)
    at q1.Q1$1$1.write(Q1.java:39)
    at com.google.gson.Gson.toJson(Gson.java:669)
    at com.google.gson.Gson.toJson(Gson.java:648)
    at com.google.gson.Gson.toJson(Gson.java:603)
    at com.google.gson.Gson.toJson(Gson.java:583)
    at retrofit.converter.GsonConverter.toBody(GsonConverter.java:80)
    at retrofit.RequestBuilder.setArguments(RequestBuilder.java:375)
    at retrofit.RestAdapter$RestHandler.invokeRequest(RestAdapter.java:298)
    ... 7 more

Configuration:

JVM: -Xms4m -Xmx8m
web server: npm install http-echo-server -g && export PORT=8080 && http-echo-server (see more at Generic HTTP server that just dumps POST requests?)

Why does this happen? Your application on the device most likely is running out of memory, because you're writing to StringWriter that essentially accumulates a whole string in memory (also note that memory penalties are paid in StringBuffer due to its internal buffers). Take a look at this code:

interface IService {

    @POST("/")
    void query(@Body String s, Callback<String> callback);

}

    final int floodLength = 1024 * 1024;
    // This flooding type adapter factory substitutes any type, and this is OK for this case
    final TypeAdapterFactory flooder = new TypeAdapterFactory() {
        @Override
        public <T> TypeAdapter<T> create(final Gson gson, final TypeToken<T> typeToken) {
            return new TypeAdapter<T>() {
                @Override
                public void write(final JsonWriter out, final T value)
                        throws IOException {
                    // Just flood with `[{},{},{},...]`
                    out.beginArray();
                    for ( int i = 0; i < floodLength; i++ ) {
                        out.beginObject();
                        out.endObject();
                    }
                    out.endArray();
                }

                @Override
                @SuppressWarnings("unchecked")
                public T read(final JsonReader in)
                        throws IOException {
                    // Parse JSON response, however it does not work for this server giving war "POST "
                    // (the echo HTTP server dumps requests back as response bodies -- maybe another HTTP echo server is more appropriate for the experiment?)
                    loop:
                    for ( ; ; ) {
                        final JsonToken token = in.peek();
                        // @formatter:off
                        switch ( token ) {
                        case BEGIN_ARRAY: in.beginArray(); break;
                        case END_ARRAY: in.endArray(); break;
                        case BEGIN_OBJECT: in.beginObject(); break;
                        case END_OBJECT: in.endObject(); break;
                        case NAME: in.nextName(); break;
                        case STRING: in.nextString(); break;
                        case NUMBER: in.nextDouble(); break;
                        case BOOLEAN: in.nextBoolean(); break;
                        case NULL: in.nextNull(); break;
                        case END_DOCUMENT: break loop;
                        default: throw new AssertionError("no case for " + token);
                        }
                        // @formatter:on
                    }
                    return (T) "<mock>";
                }
            };
        }
    };
    final Gson gson = new GsonBuilder()
            .registerTypeAdapterFactory(flooder) // registering the evil flooder for the testing purposes
            .create();
    final RestAdapter retrofit = new Builder()
            .setEndpoint("http://localhost:8080/")
            .setConverter(new GsonConverter(gson))
            .build();
    final IService service = retrofit.create(IService.class);
    service.query("foo", new Callback<String>() {
        @Override
        public void success(final String string, final Response response) {
            System.out.println("OK: " + string);
            System.exit(0);
        }

        @Override
        public void failure(final RetrofitError retrofitError) {
            retrofitError.printStackTrace(System.err);
            System.exit(1);
        }
    });

Writer instances, generally speaking, should write data to somewhere not accumulating the whole data internally (therefore small internal buffers used for bufferization are fine). All you need to do to fix your issue is avoid using StringWriter and replace it with something else that can write to something with larger capacity: say, OutputStreamWriter with bound OutputStream that writes to a device storage, network connections, etc. GsonConverter provided by Retrofit v1 (as well as Retrofit v2) does write to the output stream, but rather just accumulates the request into memory first. I'm not sure if the following custom Gson Converter implementation is well-written, but at least it uses streaming:

final class StreamingGsonConverter
        implements Converter {

    private static final Charset charset = StandardCharsets.UTF_8;
    private static final String MIME_TYPE = "application/json; charset=" + charset;

    private final Gson gson;

    StreamingGsonConverter(final Gson gson) {
        this.gson = gson;
    }

    @Override
    public Object fromBody(final TypedInput body, final Type type)
            throws ConversionException {
        try {
            // Reading a stream can be much cheaper
            final Reader reader = new InputStreamReader(body.in(), charset);
            return gson.fromJson(reader, type);
            // No need to close the underlying InputStreamReader, let Retrofit do it
        } catch ( final IOException ex ) {
            throw new ConversionException(ex);
        }
    }

    @Override
    public TypedOutput toBody(final Object object) {
        return new BodyTypedOutput(object);
    }

    private final class BodyTypedOutput
            implements TypedOutput {

        private final Object object;

        private BodyTypedOutput(final Object object) {
            this.object = object;
        }

        @Override
        public String fileName() {
            return null;
        }

        @Override
        public String mimeType() {
            return MIME_TYPE;
        }

        @Override
        public long length() {
            // No one can know the length in advance for such cases...
            return -1;
        }

        @Override
        public void writeTo(final OutputStream out) {
            final Appendable appendable = new OutputStreamWriter(out, charset);
            // This is where the original GsonConverter fails for you: don't collect all the data in memory, but rather write it directly to the output stream
            gson.toJson(object, appendable);
            // No need to close the underlying OutputStreamWriter prematurely -- let Retrofit manage it itself because it's the owner of the resource
        }
    }

}

And then just replace GsonConverter with:

.setConverter(new StreamingGsonConverter(gson))

For me, the client now fails with a JSON parsing exception (because of the HTTP server echoing nature with POST), but does not waste memory now after a response is received (however the test HTTP server also gets down due to massive flooding). The solution above may work around your issue, but I think it has to be fine-tuned first, or another streaming workaround should be found over the internet. Good luck!

Creating a JSON with Gson makes the app crash

1 Answers1