4

I am creating a java process to download WebEx recordings using their NBR API's NBRFileOpenService call. It returns a multipart response with the recording file contents attached. I have it somewhat working with the code below. However, when the recording file is large enough, I get OutOfMemoryError exception.

It is quite common for the recordings to be large and if the API only returned the file alone, I could just stream the download, however I'm not so sure how I can safely handle the multipart response. So I'm wondering if there is any way to read the file metadata as well as save the binary content to a file without holding the entire response in memory.

API Response Format:

------=_Part_674_458057647.1593732813745
Content-Type: text/xml; charset=UTF-8
Content-Transfer-Encoding: binary
Content-Id: <AD79B5747EFC01CDDA9A281BA8CDEF0C>

[SOAP RESPONSE]
------=_Part_674_458057647.1593732813745
Content-Type: application/octet-stream
Content-Transfer-Encoding: binary
Content-Id: <C498AB4664B57130F869695A1C5B584E>

[FILE METADATA]
------=_Part_674_458057647.1593732813745
Content-Type: application/octet-stream
Content-Transfer-Encoding: binary
Content-Id: <003D9EBA1E491CE2E9E5903C996EFD4C>

[BINARY FILE CONTENT]
------=_Part_674_458057647.1593732813745--

My Code:

public void retrieveRecordingFile(String uri, String recordId, String serviceType) throws Exception {
    HttpClient httpClient = generateHttpClient();
    
    HttpPost httpPost = new HttpPost(uri);
    httpPost.addHeader("Content-Type", ContentType.APPLICATION_XML.getMimeType());
    httpPost.addHeader("SOAPAction", "NBRFileOpenService");
    
    String requestXml = buildNBRDownloadFileXml(recordId, serviceType);
    HttpEntity httpEntity = new ByteArrayEntity(requestXml.getBytes(Charset.forName("UTF-8")));
    
    httpPost.setEntity(httpEntity);
    HttpResponse httpResponse = httpClient.execute(httpPost);
    if (httpResponse.getStatusLine().getStatusCode() == 200) {
        MimeMultipart mimeMultipart = new MimeMultipart(new ByteArrayDataSource(httpResponse.getEntity().getContent(), "multipart/form-data"));
        String filename = null;
        File targetFile = null;
        for (int i = 0; i < mimeMultipart.getCount(); i++) {
            if (i == 1) {
                filename = retrieveFileName(mimeMultipart.getBodyPart(i).getInputStream());
            } else if (i == 2) {
                targetFile = new File(DOWNLOAD_DIR + filename);
                FileUtils.copyInputStreamToFile(mimeMultipart.getBodyPart(i).getInputStream(), targetFile);
            }
        }
    }
}

Any help is truly appreciated.

rohant
  • 41
  • 3
  • You can add the situation in which the error occurs, I think that if you simply increase the memory for that app / microservice the problem would be solved, I do not think you need much more memory. – Adrian Lagartera Jul 18 '20 at 00:37
  • The same code works for smaller files without a problem, but fails when retrieving larger files. My concern is that the current code is holding the complete response in memory and while increasing memory may solve the issue for now, we may have an even larger recording file in the future that it may fail for; A constant cat and mouse game. I really think if there was a way to stream the file content part of the response, the file size becomes irrelevant to the retrieval process. I haven't been successful in finding any libraries that can do that as well. – rohant Jul 19 '20 at 20:25
  • 1
    I imagined that your code would work with small files, in general it should work with everyone, but I think that with a little more memory it should be managed, I don't know if you have little memory or you are dealing with just enough. I had a similar problem with an app, it downloaded large files, and the problem was that my microservice only had 500mb of ram, I changed it to 700mb and my problem was solved. This probably won't fix it but just in case – Adrian Lagartera Jul 20 '20 at 07:17

0 Answers0