Connection reset when downloading to Android from Rails

Question

I have an android application I am trying to sync with a rails app.

In the android app I download and load a json object from the server (using an OAuth token for authentication):

private JSONArray getJSON(URL url, String authToken) throws IOException, JSONException {
    URLConnection con = url.openConnection();
    con.setRequestProperty("Content-Type", "application/json");
    con.setRequestProperty("Authorization", "Bearer " + authToken);
    InputStream is = new BufferedInputStream(con.getInputStream());
    Scanner s = new Scanner(is).useDelimiter("\\A");
    String text = s.hasNext() ? s.next() : "";
    return new JSONArray(text);
}

I'm trying to reload the existing data and it is ~380k. When I run the code I get the following in the rails server log:

Started GET "/events.json?created_since=0" for 192.168.1.111 at 2014-01-22 20:15:16 -0500
Processing by EventsController#index as JSON
  Parameters: {"created_since"=>"0", "event"=>{}}
  Doorkeeper::AccessToken Load (0.6ms)  SELECT "oauth_access_tokens".* FROM "oauth_access_tokens" WHERE "oauth_access_tokens"."token" = [:filtered:] ORDER BY "oauth_access_tokens"."id" ASC LIMIT 1
  User Load (0.3ms)  SELECT "users".* FROM "users" WHERE "users"."id" = ? LIMIT 1  [["id", 501661262]]
  Habit Load (0.7ms)  SELECT "habits".* FROM "habits" WHERE "habits"."user_id" = ? ORDER BY "habits"."id" ASC LIMIT 1000  [["user_id", 501661262]]
  Event Load (26.7ms)  SELECT "events".* FROM "events" WHERE "events"."habit_id" = ?  [["habit_id", 1]]
  ⋮
  Rendered events/index.json.jbuilder (3422.5ms)
Completed 200 OK in 4491ms (Views: 3436.2ms | ActiveRecord: 73.4ms)
[2014-01-22 20:15:21] ERROR Errno::ECONNRESET: Connection reset by peer
    /home/will/.rvm/rubies/ruby-2.0.0-p247/lib/ruby/2.0.0/webrick/httpserver.rb:80:in `eof?'
    /home/will/.rvm/rubies/ruby-2.0.0-p247/lib/ruby/2.0.0/webrick/httpserver.rb:80:in `run'
    /home/will/.rvm/rubies/ruby-2.0.0-p247/lib/ruby/2.0.0/webrick/server.rb:295:in `block in start_thread'

The connection reset is repeated seven times. The client receives about 260k of data. app/views/events/index.json.jbuilder is:

json.array!(@events) do |event|
  json.extract! event, :id, :habit_id, :time, :description
end

The same method is used to load a different model with only a few entries and it loads correctly. Is there a limit to how big a file can be downloaded? In any case pagination seems like a good idea. Anyone know of any guidelines on what size chunks I ought to break it up into?

score 0 · Answer 1 · answered Jan 23 '14 at 02:51

0

Instead using Scanner, maybe you can use Reader to read the result InputStream

Reader reader = new InputStreamReader(is);
JsonParser parser = new JsonParser();
JsonArray jsonArray = parser.parse(reader).getAsJsonArray();

answered Jan 23 '14 at 02:51

Fate Chang

197
3

That may be more efficient, but my fundamental problem is I'm not getting all the data. – dysbulic Jan 24 '14 at 17:21

score 0 · Answer 2 · edited May 23 '17 at 10:25

I ended up paginating the data and it now loads correctly:

JSONArray events;
int page = 1;
SimpleDateFormat timeFormat = new SimpleDateFormat("yyyy-MM-dd'T'HH:mm:ss.SSS'Z'");
batch = new ArrayList<ContentProviderOperation>();

do {
    events = getJSON(new URL(EVENT_READ_URL + "?page=" + (page++)), authToken);
    for(int i = 0; i < events.length(); i++) {
        JSONObject event = events.getJSONObject(i);
        batch.add(ContentProviderOperation.newInsert(HabitContentProvider.EVENTS_URI)
         .withValue(EventTable.COLUMN_ID, event.getInt("id"))
         .withValue(EventTable.COLUMN_HABIT_ID, event.getInt(EventTable.COLUMN_HABIT_ID))
         .withValue(EventTable.COLUMN_TIME, timeFormat.parse(event.getString(EventTable.COLUMN_TIME)).getTime() / 1000)
         .withValue(EventTable.COLUMN_DESCRIPTION, event.getString(EventTable.COLUMN_DESCRIPTION))
         .build());
    }
} while(events.length() > 0);

try {
    mContentResolver.applyBatch(HabitContentProvider.AUTHORITY, batch);
} catch(SQLiteConstraintException e) {
    Log.e(TAG, "SQLiteConstraintException: " + e.getMessage());
}

In my rails code I added the will_paginate gem, included array support, and added the following to my code:

if params[:page]
  @events = @events.paginate(page: params[:page], per_page: params[:per_page] || 300)
end

Connection reset when downloading to Android from Rails

2 Answers2