0

I want to do :

Text File (logs) --> String in JSON --> upload to a server. Do I even stand a chance for file of 11Mo ?

My code : 

ArrayList<String> listInfoLog= new ArrayList<String>();
listInfoLog = Tools.readFile(strPathFile); // load my logs from the file as ArrayList

StringBuilder sb = new StringBuilder();
for (int i = 0; i < listInfoLog.size(); i++){
   sb.append(listInfoLog.get(i) + "\n");
}
// code en base64
byte[] byteContenu = sb.toString().getBytes("UTF-8"); // OUTOFMEMORY
String byteContenuEncoded64 = Base64.encodeToString(byteContenu, Base64.DEFAULT);
synchroFile.setContenu(byteContenuEncoded64);
JSONObject jsonFile = synchroFile.convertFileToJson();

The call to getBytes("UTF8") generates an outofmemory exception. 

12-11 13:49:42.780: E/dalvikvm-heap(13868): Out of memory on a 23040392-byte allocation.
12-11 13:49:42.780: I/dalvikvm(13868): "AsyncTask #3" prio=5 tid=15 RUNNABLE
12-11 13:49:42.780: I/dalvikvm(13868):   | group="main" sCount=0 dsCount=0 obj=0x62e4bc60 self=0xab360220
12-11 13:49:42.780: I/dalvikvm(13868):   | sysTid=13957 nice=10 sched=0/0 cgrp=apps/bg_non_interactive handle=-1422168792
12-11 13:49:42.780: I/dalvikvm(13868):   | state=R schedstat=( 6042297526 1923573007 5457 ) utm=578 stm=26 core=1
12-11 13:49:42.780: I/dalvikvm(13868):   at java.nio.charset.Charsets.toUtf8Bytes(Native Method)
12-11 13:49:42.780: I/dalvikvm(13868):   at java.lang.String.getBytes(String.java:856)
12-11 13:49:42.780: I/dalvikvm(13868):   at java.lang.String.getBytes(String.java:840)
12-11 13:49:42.780: I/dalvikvm(13868):   at com.ier.tepv.base.SynchroReceiptAsync.sendLogs(SynchroReceiptAsync.java:951)
12-11 13:49:42.780: I/dalvikvm(13868):   at com.ier.tepv.base.SynchroReceiptAsync.doInBackground(SynchroReceiptAsync.java:600)
12-11 13:49:42.780: I/dalvikvm(13868):   at com.ier.tepv.base.SynchroReceiptAsync.doInBackground(SynchroReceiptAsync.java:1)
12-11 13:49:42.780: I/dalvikvm(13868):   at android.os.AsyncTask$2.call(AsyncTask.java:288)
12-11 13:49:42.780: I/dalvikvm(13868):   at java.util.concurrent.FutureTask.run(FutureTask.java:237)
12-11 13:49:42.780: I/dalvikvm(13868):   at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1112)
12-11 13:49:42.780: I/dalvikvm(13868):   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:587)
12-11 13:49:42.780: I/dalvikvm(13868):   at java.lang.Thread.run(Thread.java:841)
12-11 13:49:42.780: W/dalvikvm(13868): threadid=15: thread exiting with uncaught exception (group=0x61817bc0)
12-11 13:49:42.780: E/AndroidRuntime(13868): FATAL EXCEPTION: AsyncTask #3
12-11 13:49:42.780: E/AndroidRuntime(13868): Process: com.ier.tepv.base, PID: 13868
12-11 13:49:42.780: E/AndroidRuntime(13868): java.lang.RuntimeException: An error occured while executing doInBackground()
12-11 13:49:42.780: E/AndroidRuntime(13868):    at android.os.AsyncTask$3.done(AsyncTask.java:300)
12-11 13:49:42.780: E/AndroidRuntime(13868):    at java.util.concurrent.FutureTask.finishCompletion(FutureTask.java:355)
12-11 13:49:42.780: E/AndroidRuntime(13868):    at java.util.concurrent.FutureTask.setException(FutureTask.java:222)
12-11 13:49:42.780: E/AndroidRuntime(13868):    at java.util.concurrent.FutureTask.run(FutureTask.java:242)
12-11 13:49:42.780: E/AndroidRuntime(13868):    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1112)
12-11 13:49:42.780: E/AndroidRuntime(13868):    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:587)
12-11 13:49:42.780: E/AndroidRuntime(13868):    at java.lang.Thread.run(Thread.java:841)
12-11 13:49:42.780: E/AndroidRuntime(13868): Caused by: java.lang.OutOfMemoryError
12-11 13:49:42.780: E/AndroidRuntime(13868):    at java.nio.charset.Charsets.toUtf8Bytes(Native Method)
12-11 13:49:42.780: E/AndroidRuntime(13868):    at java.lang.String.getBytes(String.java:856)
12-11 13:49:42.780: E/AndroidRuntime(13868):    at java.lang.String.getBytes(String.java:840)
12-11 13:49:42.780: E/AndroidRuntime(13868):    at com.ier.tepv.base.SynchroReceiptAsync.sendLogs(SynchroReceiptAsync.java:951)
12-11 13:49:42.780: E/AndroidRuntime(13868):    at com.ier.tepv.base.SynchroReceiptAsync.doInBackground(SynchroReceiptAsync.java:600)
12-11 13:49:42.780: E/AndroidRuntime(13868):    at com.ier.tepv.base.SynchroReceiptAsync.doInBackground(SynchroReceiptAsync.java:1)
12-11 13:49:42.780: E/AndroidRuntime(13868):    at android.os.AsyncTask$2.call(AsyncTask.java:288)
12-11 13:49:42.780: E/AndroidRuntime(13868):    at java.util.concurrent.FutureTask.run(FutureTask.java:237)
12-11 13:49:42.780: E/AndroidRuntime(13868):    ... 3 more

I don't get how I should procede ? It works fine except for smaller files (150ko) but crashes for this one at 11Mo.

Practically, I have never met this giant log file since the system starts running but for the sake of it, how can I do it ?

I tried :

  • divide the loop in parts but the stringbuilder outofmemory too

  • streaming with bytes but I don't know how to set it in UTF8.

    FileInputStream fis = new FileInputStream(fileLog); ByteArrayOutputStream bos = new ByteArrayOutputStream(); byte[] buf = new byte[1024]; try { for (int readNum; (readNum = fis.read(buf)) != -1;) { bos.write(buf, 0, readNum); //no doubt here is 0 } } catch (IOException ex) { new TePVException(tagLocal, "sendLog", "Streaming Log File ex = "+ex.getLocalizedMessage()); } byte[] bytes = bos.toByteArray(); byteContenuEncoded64 = Base64.encodeToString(bytes, Base64.DEFAULT);

This works, which is a good step forward, but then it outofmemory here : String str = json.toString(); and I am back at square one :(

Poutrathor
  • 1,990
  • 2
  • 20
  • 44
  • Not the actual problem, but note that concatenating in a call to `StringBuilder.append` (`sb.append(listInfoLog.get(i) + "\n");`) somewhat defeats the point of a `StringBuilder`. – Andy Turner Dec 11 '15 at 14:42
  • 1
    Why read the whole file in one go? You could read up to a configurable maximum number of bytes/lines/chars and send the data to the server in smaller chunks. The network interface would buffer the information to some degree in any case. – Dennis Dec 11 '15 at 14:43
  • How about using a stream to upload a file? Example -> http://stackoverflow.com/questions/12623532/java-upload-outputstream-as-http-file-upload – Darshan Mehta Dec 11 '15 at 14:43
  • I can't send chunk by chunk because the serveur is expecting one and only one file in one request. So each chunk would replace the previous one and the server will end with only logs from the last hours of the day. – Poutrathor Dec 11 '15 at 16:58
  • I cannot use a stream because the server expect a JSON, asaik ? – Poutrathor Dec 11 '15 at 17:00
  • Why are you trying to base 64 encode UTF-8 encoded text? If you are processing 23 MB of text I suggest you either use more memory or send the data progressively instead of all in one go. – Peter Lawrey Dec 12 '15 at 00:03
  • UTF-8 encoding is to convert characters into bytes. If you already have bytes, you don't need to convert them again. – Peter Lawrey Dec 12 '15 at 00:04

0 Answers0