UDP Packet received okay in Java but corrupted in Python

Question

I am trying to record audio from an Android tablet and send it to a python server. At the start of the byte packet, I include some relevant information about the state of the Android app (A byte array called "actives" -- but considering it's receiving fine by a Java server, this should not be relevant). The android code is as follows:

                   int read = recorder.read(buffer, 0, buffer.length);

                    for (int a = 0; a < actives.length; a++) {
                        outBuffer[a+1] = (byte)actives[a];
                        logger = logger + Byte.toString(actives[a]) + ",";
                    }
                    int furthest=0;
                    for(int a =0; a < buffer.length; a++){
                        outBuffer[actives.length+1+a]=buffer[a];
                        if(buffer[a]!=0)furthest=a;
                    }
                    packet = new DatagramPacket(outBuffer, read,
                            serverAddress, PORT);
                    Log.d("writing", logger+Byte.toString(outBuffer[7])+".length"+Integer.toString(1+furthest+actives.length+1));
                    Log.d("streamer","Packet length "+outBuffer.length);
                    try {
                        socket.send(packet);
                    }catch (IOException e){
                        Log.e("streamer", "Exception: " + e);
                    }
                    Log.d("streamer","packetSent");

I receive a clean signal on the other end using a Java server. Image of received java output: !(https://i.stack.imgur.com/OCFmP.png) This is my Java server:

DatagramSocket serverSocket = new DatagramSocket(3001);
int byteSize=970;
byte[] receiveData = new byte[byteSize];
DatagramPacket receivePacket = new DatagramPacket(receiveData,
        receiveData.length);
while(true){        // recieve data until timeout
                        try {
                            serverSocket.receive(receivePacket);
                            String rcvd = "rcvd from " + receivePacket.getAddress();
                           System.out.println("receiver"+"Received a packet!" +rcvd);

                            break;
                        }
                        catch (Exception e) {
                            // timeout exception.
                            System.out.println("Timeout reached without packet!!! " + e);
            timeoutReached=true;
                            break;

                        }
                 }
if(timeoutReached)continue;
    currTime = System.currentTimeMillis();

    data = receivePacket.getData();

Here is my Python server's output: !(https://i.stack.imgur.com/VXjLX.png) And here is the code:

import socket
ip="192.ip.address"
port=3001;
sock=socket.socket(socket.AF_INET,socket.SOCK_DGRAM);
sock.bind(('',port));
while(True):
    data,addr=sock.recvfrom(970);
    print("address",addr);
    print("received a data!");
    print(data);

In the last line of the python script, I have tried to change "print(data)" to "print(data.decode())", in which case I get this error:

UnicodeDecodeError: 'ascii' codec can't decode byte 0xff in position 0: ordinal not in range(128)

I am not running these servers at the same time My guess is that it has to do something with Java using unsigned ints and python not doing that. Is there a way in Python that I can convert this data, because data.decode() is not working? Alternatively I should be able to convert the data in Java somehow? None of the answers on stackoverflow that I have tried have worked.

Answer 1

Decoding is the right approach. In your android app explicitly mention the character encoding. UTF-8 is the standard Charset that is used.

Your log is pretty clear. You are trying to decode the data packet as ASCII (which is the default encoding of the decode() function) but I'm guessing its ISO_8859_1 or UTF-8 (more likely).

Next try data.decode('utf8', 'ignore') in your android app. Note: 'ignore' is an optional argument and to be used only in case of debugging as it will ignore malformed(corrupted) data and try to convert individual characters. If you want to use decode() in production use 'strict' or no second argument ('strict' is the default).

In place of 'utf8' try other options from other Python Encodings.

Answer 2

This was pretty brutal to attack head-on. I tried specifying the encoding in Java (before sending) like another SO post suggested, but that didn't help. So I side-stepped the problem by converting my Android byte array into a comma-separated string, then converting the string back into UTF-8 bytes.

sendString="";
for(int a =0; a < buffer.length; a++){
           sendString=sendString+Byte.toString(buffer[a])+",";
}
byte[] outBuffer = sendString.getBytes("UTF-8");

Make sure you reset your string to null ("") each time you go through the while loop, or your ish will get very slow af.

Then in Python,right after receiving:

data=data.decode("utf8");

Although I am stringifying 980 characters, it does not appear to add much to the processing time... although I do wish that I could send the raw bytes, as speed is very important to me here. I'll leave the question open in case someone can come up with a better solution.