9

We are using a REST service that returns huge data. In production, the server hardware can handle it. I need to test the REST service by getting the data on my local machine, which cannot handle the huge data. My local machine is a 4G i5. I am getting out of memory exception every time I hit the service.

response.getStatus() returns 200 status. But while collecting the data using input stream reader, I get an out of memory exception. 

BufferedReader br = new BufferedReader(new
InputStreamReader(newByteArrayInputStream(response.getEntity().getBytes())));

Is there any other way to collect the data without hitting the memory exception?

I tried pushing my VM size to 2G but, it still doesn't work.

DwB
  • 37,124
  • 11
  • 56
  • 82
om39a
  • 1,406
  • 4
  • 20
  • 39
  • How about returning only part of the response? –  Feb 06 '14 at 17:17
  • write to disk or to /dev/null – PeterMmm Feb 06 '14 at 17:23
  • @Petey I need the byet's array for writing into file, but while getting the bytes it fails 'response.getEntity().getBytes()' – om39a Feb 06 '14 at 17:28
  • @LutzHorn, You mean altering the web service to return in parts or while receiving the response we can split it into parts? – om39a Feb 06 '14 at 17:30
  • 2
    Consider adding a "test mode" to your REST service that either writes the response data to a file or returns only a fraction of the data. – DwB Feb 06 '14 at 17:52

2 Answers2

5

If the service supports it, use the HTTP standard approach of requesting chunks within the range of the entire response content payload.

Another approach (again, assuming the API supports it) is pagination.

Community
  • 1
  • 1
Brian Kelly
  • 19,067
  • 4
  • 53
  • 55
2

You can implement a threshold setting in the REST service configuration that limits the amount of data returned. In production, the threshold would either not be set or would be set to a large value. In your test environment, you can set the threshold to a small, managable value.

DwB
  • 37,124
  • 11
  • 56
  • 82