I have an application that needs to save large files (up to a few GB) from clients to a server. The problem I have is that the server has max of 2 GB heap size and i get a lot of times "OutOfMemoryError".
Currently I have this code (and I know this isn't a good approach,and that the file is loaded completely to the server heap):
Angular component ts:
public void uploadFile() {
let body = new FormData();
body.append("file", this.file, this.fileName);
this.http.post("/uploadFile", body).subscribe(res => {
alert("success");
});
}
Spring controller:
@RequestMapping(value = "/uploadFile", method = RequestMethod.POST)
@ResponseBody
public void uploadFile(@RequestParam("file") MultipartFile file) {
SimpleDDateFormat formatter = new SimpleDDateFormat("dd-mm-yyyy-hh-mm-ss");
String filename = FILE_DIR_LOCATION + file.getOriginalFileName() + formater.format(new Date());
InputStream in = new BufferedInputStream(file.getInputStream());
FileOutputStream fos = new FileOutputStream(filename);
byte[] buffer = new byte[4096];
int readedBytes = 0;
while ((readedBytes = in.read(buffer)) != -1) {
fos.write(buffer, 0, readedBytes);
fos.flush;
}
fos.close();
}
I would like to divide the files to smaller parts on the client side and send them like that to the server, that would save them as one file again, without taking too much of the heap. Since there could be multiple clients at the same time and one file can be bigger than the heap. I couldn't find the best practice to do so, and I thought of something like this:
Angular component ts:
static MAX_PART_SIZE = 4096;
public void uploadFile() {
this.fileReader = new FileReader();
this.fileReader.readAsArrayBuffer(this.file);
}
public void startSendingFile() {
this.bytesArray = this.fileReader.result;
this.http("uploadFirstPart", //send only the MAX_PART_SIZE to the server, size and name).subscribe(res => {
if (this.bytesArray.length > MAX_PART_SIZE) {
this.keepSendingToServer(res, this.bytesArray.length - MAX_PART_SIZE);
}
});
}
public void keepSendingToServer(fileName, remainBytes) {
this.http("uploadPart", //send the next part of the file and if this is the end of the file).subscribe(res => {
if (remainBytes - MAX_PART_SIZE > 0) {
this.keepSendingToServer(fileName, remainBytes - MAX_PART_SIZE);
}
});
}
Spring controller:
Static HashMap<String, FileOutputStream> files = new HashMap<String, FileOutputStream>();
@RequestMapping(value = "/uploadFirstPart", method = RequestMethod.POST)
@ResponseBody
public String uploadFile(@RequestParam("file") byte[] file, @RequestParam("filename") String name, @RequestParam("size") int size) {
SimpleDDateFormat formatter = new SimpleDDateFormat("dd-mm-yyyy-hh-mm-ss");
String filename = FILE_DIR_LOCATION + name + formater.format(new Date());
FileOutputStream fos = new FileOutputStream(filename);
files.put(filename, fos);
fos.write(file, 0, size);
fos.flush;
return filename;
}
@RequestMapping(value = "/uploadPart", method = RequestMethod.POST)
@ResponseBody
public void uploadPart(@RequestParam("file") byte[] file, @RequestParam("filename") String name, @RequestParam("size") int size, @RequestParam("isEnd") boolean isEnd) {
FileOutputStream fos = files.get(name);
fos.write(file, 0, size);
fos.flush;
if (isEnd) {
fos.close();
files.remove(name);
}
}
Some questions: 1. Is this the best practice? are there another ways? 2. Can you help me fix the comments in the code for the things I couldn't find how to do. 3. Is there a way to get only the wanted bytes without loading the whole large file on the client side (like inputstream in java)? 4. In my server there are places that a file is sent as base64 string and another as byte[], is there a preferable way to send files (the files can be images,audio, video, binary, etc.)?
