1

I have a large csv file in sdcard and i want to know the best way to send the data from my csv file to my hosted server through my android app. My app writes to the csv file for an hour and then at the end of every hour i run a service which reads this csv file and sends it to the server.

I was initially not writing anything to the csv file and sending each data as http post request. But it turned out that my app was showing too much of data usage for uploading.

So i switched to writing the data to a file for an hour and at the end of the hour read each line and convert each tuple to json objects , put it into a json array and then send that array as a json object via name value pair.

Problem 1:

My CSV file is too large so I get Out Of Memory Exception when i try to add more than 2000 json objects to the array and send like this (Problem occurs on line marked with **)

nameValuePairs.add(new BasicNameValuePair("myjson", json_array_obj.toString()));

Thread server = new Thread(){
    String line = null;
    public void run()
    {
        try{

            HttpClient httpclient1 = new DefaultHttpClient();
            HttpPost httppost1 = new HttpPost("http://www.example.com/filename.php");


            **httppost1.setEntity(new UrlEncodedFormEntity(nameValuePairs));**
            HttpResponse httpResponse = httpclient1.execute(httppost1);

            HttpEntity httpEntity = httpResponse.getEntity();
            line = EntityUtils.toString(httpEntity);

            Log.e("line" , line);
          }
          catch(Exception e)
          {
            Log.v("catch", "executed");
          } 

    }
};
server.start();

Logcat showed :

`12-28 13:01:18.828: E/AndroidRuntime(2690FATAL EXCEPTION: Thread-16132
 12-28 13:01:18.828: E/AndroidRuntime(26908): java.lang.OutOfMemoryError
 12-28 13:01:18.828: E/AndroidRuntime(26908):   at java.lang.AbstractStringBuilder.enlargeBuffer(AbstractStringBuilder.java:94)
 12-28 13:01:18.828: E/AndroidRuntime(26908):   at java.lang.AbstractStringBuilder.append0(AbstractStringBuilder.java:145)
 12-28 13:01:18.828: E/AndroidRuntime(26908):   at java.lang.StringBuilder.append(StringBuilder.java:216)
 12-28 13:01:18.828: E/AndroidRuntime(26908):   at libcore.net.UriCodec.appendHex(UriCodec.java:212)
 12-28 13:01:18.828: E/AndroidRuntime(26908):   at libcore.net.UriCodec.appendHex(UriCodec.java:206)
 12-28 13:01:18.828: E/AndroidRuntime(26908):   at libcore.net.UriCodec.appendEncoded(UriCodec.java:109)
 12-28 13:01:18.828: E/AndroidRuntime(26908):   at libcore.net.UriCodec.encode(UriCodec.java:133)
 12-28 13:01:18.828: E/AndroidRuntime(26908):   at java.net.URLEncoder.encode(URLEncoder.java:57)
 12-28 13:01:18.828: E/AndroidRuntime(26908):   at org.apache.http.client.utils.URLEncodedUtils.encode(URLEncodedUtils.java:184)
 12-28 13:01:18.828: E/AndroidRuntime(26908):   at org.apache.http.client.utils.URLEncodedUtils.format(URLEncodedUtils.java:163)
 12-28 13:01:18.828: E/AndroidRuntime(26908):   at org.apache.http.client.entity.UrlEncodedFormEntity.<init>(UrlEncodedFormEntity.java:71)
 12-28 13:01:18.828: E/AndroidRuntime(26908):   at com.example.acgylo.CSVtoServer$1.run(CSVtoServer.java:168)8): `

I then found out that json array with large number of objects cannot be sent as namevaluepair as it will cause memory overflow.

So i want to know what is the best way to do this :

option 1 : Send data in batches of 500(i.e 500 lines of csv together).

Is there any other efficient way to achieve sending of 2k tuples(or more) from CSV via HTTP in android ?

Problem 2:

a) I wanted to know how to reduce the data usage of android application ?

b) What uses less data : 

One http request with large json object sent as string OR Multiple http requests with smaller json objects?

Please help me find an efficient solution to the above problems. Any help is appreciated.

Thanks in advance.

Android
  • 25
  • 5
  • Can you modify the server side to accept a zip file? – Morrison Chang Dec 28 '12 at 17:37
  • There is a small amount of overhead in each request, along with the (slight) protocol overhead of sending JSON, which means that simply breaking up the larger request into smaller ones will likely increase your bandwidth use, not reduce it. The OOM is on your client side - can you not increase the memory available to it? – Perception Dec 28 '12 at 17:45
  • No, i cant as it is a php file which inserts each tuple into the database – Android Dec 28 '12 at 17:47
  • I dont know how to increase the memory on client side .Will it not depend on the phone memory? This is my first android app so i am not very familiar with it – Android Dec 28 '12 at 17:51
  • I read about this json issue here :http://code.google.com/p/json-simple/issues/detail?id=59 – Android Dec 28 '12 at 17:55
  • You can solve the problem in 2 ways. 1) Send data as is, in CSV; then server-side can parse it without problems; 2) if an OOM exception occurs, you can return specific code from the function that implies you shoud split data set in 2 halves, and try the same function on these new sets, if they fail - split again, and so on, so you will finally send a data when it will fit into memory. This is like an autoadjustment of json size. – Stan Dec 28 '12 at 18:26
  • Do you know what is the max file size that can be sent like the way you said ?Can you provide me with some example code to send the csv file as is? And is this normal practice to handle such problems ? – Android Dec 28 '12 at 18:34
  • @Android, please don't forget to add at-user in the beginning of your messages, otherwise nobody will be notified (I'm here just in case ;-)not by norification). Look at this answer on how to [send files from Android](http://stackoverflow.com/questions/4126625/how-to-send-a-file-in-android-from-mobile-to-server-using-http). I admit this is not a most reliable way to send big data. The reliable way would be to send in by chunks. I mentioned above how you can empirically deduce size of the chunk. Every method is good if it solves your problem, but there can be found a better one. This is endless – Stan Dec 28 '12 at 19:08
  • @Stan Sorry, yes i will use the tagging from now. I will try your solution and get back to you. Thanks a lot – Android Dec 28 '12 at 19:27
  • @Android, the Henry's answer is great BTW. – Stan Dec 28 '12 at 19:28
  • @Stan, i could not understand henry's solution. Can you explain me what exactly "send" means in his steps ? – Android Dec 28 '12 at 19:43
  • @Android, you should possibly search for existing solutions on streaming and posting by chunks. Here is [one the them](http://stackoverflow.com/questions/12363084/how-to-upload-video-file-into-webserver-in-android). I hope you can adapt it for your needs. – Stan Dec 28 '12 at 19:49
  • @Stan, Thanks for the link. I will learn how to receive it in php and then try this method of sending the file. – Android Dec 28 '12 at 19:58
  • @MorrisonChang : Thanks a lot.I ended up following your way by sending the whole file. – Android Jan 11 '13 at 11:19

2 Answers2

2

Streaming is the magic word here. Instead of reading all the data into memory and then sending it out you can do that record by record.

  1. open HTTP post connection
  2. write the opening "["
  3. read a record from CSV file and convert it to JSON
  4. send the JSON string over the line
  5. send the "," separating the objects in the JSON array
  6. repeat steps 3-5 until all data is sent (leave out the "," after the last record).
  7. send the "]" ending the array
  8. finish the HTTP request
Henry
  • 42,982
  • 7
  • 68
  • 84
  • the word "send" in your sentences mean send it via http post ? Or you meant append the objects and then send it together ? – Android Dec 28 '12 at 19:29
  • In step 1 just open the connection and get an output stream from it (see for example http://developer.android.com/reference/java/net/HttpURLConnection.html). Send means to write the character to the stream. So the whole array is sent in just one HTTP request. – Henry Dec 28 '12 at 21:01
  • Thanks a lot! i am just working on that..Hopefully this will work. – Android Dec 29 '12 at 14:43
  • I could not implement this method. I ended up sending the file itself and its working fine now. Thanks a lot! – Android Jan 11 '13 at 11:18
0

As you've mentioned if you are sending large amounts of data from a mobile device to a server you can either

a) break up the data into pieces and if there is no dependencies between the pieces that could work

or

b) just compress the data into a zip file and send that over to the server. In most cases its probably easier just to modify the server to accept zip files and then do the regular processing.

Morrison Chang
  • 11,691
  • 3
  • 41
  • 77