Java client doesn't send whole message to python server?

Question

I am writing a program in android studio that communicates with a python server. I tried to send a long message (mp3 file encoded in base64 - about 10k bytes). The problem is that when I check what I received in the server, I get a lot less than 10k bytes.

Anyone knows how to fix it?

Thanks in advance.

Recording message and putting it in base64:

// Record audio from user in mp3 format
                    MediaRecorder recorder = new MediaRecorder();
                    recorder.setAudioSource(MediaRecorder.AudioSource.MIC);
                    recorder.setOutputFormat(MediaRecorder.OutputFormat.MPEG_4);
                    recorder.setOutputFile(Environment.getExternalStorageDirectory()
                                           .getAbsolutePath() + "/messageRecord.mp3");
                    recorder.setAudioEncoder(MediaRecorder.AudioEncoder.AAC);
                    recorder.prepare();
                    recorder.start();

                    TimeUnit.SECONDS.sleep(5);
                    recorder.stop();

                    File file = new File(Environment.getExternalStorageDirectory()
                                         .getAbsolutePath() + "/messageRecord.mp3");
                    int size = (int) file.length();
                    byte[] bytes = new byte[size];
                    BufferedInputStream buf = new BufferedInputStream(new FileInputStream(file));
                    buf.read(bytes, 0, bytes.length);
                    buf.close();
                    String content = Base64.encodeToString(bytes, Base64.DEFAULT);

                    // Prepare audio message request
                    JSONObject sendRecordReq = new JSONObject();
                    sendRecordReq.put("code", Codes.SPEECH_TO_TEXT_CODE);
                    sendRecordReq.put("src_phone", ChatScreen.this.srcPhone);
                    sendRecordReq.put("dst_phone", ChatScreen.this.dstPhone);
                    sendRecordReq.put("content", content);

                    // Send message request
                    ChatScreen.this.client.send(sendRecordReq);

How I send it:

//In class client
    public void send(JSONObject request) {
        this.outgoingMessages.addConversationFlow(request); //Send request
    }
//In class OutgoingMessages
    public void run() {
        try {
            while (true) {
                while(!this.conversationFlow.isEmpty()) {
                    JSONObject msgToSend = this.conversationFlow.remove();
                    String strRequest = msgToSend.toString();
                    this.out.write(Integer.toString(strRequest.length()).getBytes()); //Sends message size
                    this.out.write(strRequest.getBytes()); //Sends message
                }
            }
        }
        catch (Exception e) {
            System.out.println(e);
        }
    }

Server:

while True:
            # Receiving data size from client
            message_size = int(client_socket.recv(MAX_SIZE_LEN))

            # Receiving data from the client
            client_message = client_socket.recv(message_size)
            print client_message
            # Add message to messages queue with client's socket
            MESSAGES_QUEUE.append((client_socket, client_message))

EDIT: the "message_size" value is right (14806 - the size of the message that should b e received in the next line) but it still doesn't receive it all.

EDIT2: I figured it out, ill post the solution in the answers

score 0 · Answer 1 · answered Jan 30 '19 at 11:43

Your java code is sending data in a protocol that couldn't possibly work.

If the input JSON object is, say, the string 'a', then you'd send this:

3"a"

as in, 4 bytes: 51, 34, 97, 34.

The python side has no idea when the 'this is how long the data' is part ends.

I assume what you intended to send is something along the lines of:

00, 00, 00, 03, 34, 97, 34.

In other words: 4 bytes containing a network-endian sent integer value with the length, and then that many bytes.

Separately, don't call .getBytes(); when converting strings to bytes, you should always explicitly specify encoding. JSON is more or less by definition UTF-8, so, call .getBytes("UTF-8") instead.

NB: To replace your 'send the length' code, see Convert integer into byte array (Java)

I changed the sending of the length to: this.out.write(String.format("%010d", strRequest.length()).getBytes()); The server reads 10 bytes when it gets the length, is it ok now? — Netanel, Jan 30 '19 at 12:03
Another question, when I do getBytes("UTF-8") it should be when I turn the file into byte array or before the sending? or both? — Netanel, Jan 30 '19 at 12:05

score 0 · Answer 2 · answered Jan 30 '19 at 13:30

So the problem wasn't with the length of the file or anything else. The problem was that the recv function in the python would get some of the bytes but the code continued for some reason, so it didn't get the whole message.

The solution is to add a function that doesn't continue till the whole length that is specified is received. I found the solution in the next post: Python Socket Receive Large Amount of Data

I'll add the code as well:

Replace the recv function with recvall(socket, size). This is not my code, it was posted by Adam Rosenfield and edited by Hedde van der Heide.

def recvall(sock, n):
    # Helper function to recv n bytes or return None if EOF is hit
    data = b''
    while len(data) < n:
        packet = sock.recv(n - len(data))
        if not packet:
            return None
        data += packet
    return data

Java client doesn't send whole message to python server?

2 Answers2