2

I have and android app which sends live data to a PHP server. The data is in JSON format and is sent in HTTP POST request from the android app to PHP. The data sent is very large, to the tune of 11KB. I need to send 11 KB of such information every 2 seconds for 1 week. This has led to two problems:

  1. The data is very large and sent over a cellular network. Hence, it is turning out to be quite expensive operation.
  2. It is taking a long time to transfer this 11 KB of information from app to server. Thus delaying the subsequent POST requests, thereby making the entire data transfer very very slow.

We have considered using web sockets and protocol buffers as alternatives. The problem with these options is that we will need to change the entire structure of our code.

Are there any other options which will help in reducing the data (like compressing it significantly) so that the data transfer over the cellular network will consume less space and can be done faster? I have also read about gson, bson etc. but I am not convinced about their efficiency over the current json structure.

abhogu
  • 393
  • 4
  • 18
  • What http client are you using? You could enable `gzip` compression, if it's supported (some clients use it by default afaik) – wasyl Sep 25 '15 at 17:17
  • Is this 11K of totally new data, or mainly just the same stuff repeated with only a few changes – RiggsFolly Sep 25 '15 at 17:26
  • I am using HTTPURLConnection in Android. It is said to compress data by default, however this is not good enough to meet our requirements. – abhogu Sep 25 '15 at 17:27
  • @RiggsFolly: Its totally new data. – abhogu Sep 25 '15 at 17:28
  • What could you possibly need to send from a phone every 2 seconds that would be 11K of new data every 2 seconds? Unless you are ripping off someones address book etc – RiggsFolly Sep 25 '15 at 17:28
  • @RiggsFolly: This is an IoT application which collects information from several sensors and sends them to the server. – abhogu Sep 25 '15 at 17:32
  • Have you considered storing data on the device and then when you see a active WiFi, connect and send it over the wifi link? – RiggsFolly Sep 25 '15 at 17:36
  • @RiggsFolly: Having wifi connectivity is not guaranteed. Cellular network availability is guaranteed. Hence, considering only cellular network for our application. – abhogu Sep 25 '15 at 17:39
  • 1
    how about using sockets instead of http? – Daniel Mendel Sep 25 '15 at 17:48

1 Answers1

4

It is possible to compress post data using the HTTP protocol:

  1. Depending on the HTTP client you are using, you will probably have to compress the content yourself. This is quite easily done with the GZIPOutputStream provided by the JDK 7.

  2. On the client-side you will need to provide the following HTTP request headers:

    • Content-Encoding: gzip
    • Content-Length: X, where X is the size in bytes of the entity body after compression.

  3. The data has to be decoded on the server, this is normally done by using mod_deflate.

However, I do not think that compression will help much. Transferring 11 KB of information over a cellular network does not take much longer than transferring 11 bytes of information. It is the ping (and the HTTP protocol) what is the problem.

I'd suggest using a persistent HTTP connection, which might improve transfer times by keeping the connection open.

But you will definitely be better of using web-sockets and binary serialization. Web-sockets use a persistent connection with much less overhead, which results in a very low latency connection.

Community
  • 1
  • 1
Tim
  • 5,521
  • 8
  • 36
  • 69
  • Thank you Tim. I still do not have a clear understanding of how websockets will help my problem, though I have come across this suggestion a few times. Can you point me to resources which help me understand how web sockets will help me? – abhogu Sep 25 '15 at 17:45
  • Websockets keep an open connection with the server, an Http request, will first open a socket, send the data, and close the socket again, the act of opening a socket and closing takes time, with Websockets you can keep an open socket to a server and send data continuously(just like opening an input or output stream when you write to/ or read from a file) – Daniel Mendel Sep 25 '15 at 17:50
  • @user3855126 Web-sockets use a persistent connection with much less overhead, which results in a very low latency connection. Information can be send in a binary format, which is much smaller than JSON for example. – Tim Sep 25 '15 at 17:51
  • Thank you Daniel and Tim for the explanation. Is using web sockets from an android phone considered secure? – abhogu Sep 25 '15 at 17:52
  • @user3855126 How do you define safe? It could be as safe as you want, depending on how you would secure it. Furthermore, HTTP isn't safe at all. – Tim Sep 25 '15 at 17:54
  • 1
    @Tim: Thank you so much for clearing up my doubts. Accepting your answer for suggesting web sockets and binary serialization. – abhogu Sep 25 '15 at 17:58