0

I need to pass some strings of data between a Java application and a python script on a local Windows machine. So, I decided to do so with a Java socket server that is communicating with TCP to a python client. The Java creates two threads to handle two socket connections over the ports 9998 and 9999 of the localhost. I use port 9998 to handle incoming messages, while I use port 9999 to handle sending messages. My two applications run smoothly for the first few messages sent/received, and at some point, it stops on the call that sends the string from Java to Python. Here is part of my code:

This Java class handles the creation of the socket server and the communication

    public class ServerSocketConnection {

    private int port;
    private Socket socket;
    private ServerSocket serverSocket;
    private Logger logger;
    private BufferedWriter out;
    private BufferedReader in;

    public ServerSocketConnection(int port) {
        this.port = port;
        logger = App.getLogger();
    }

    // Create a server for a socket connection
    public void createServer() {
        try {
            // Create a server socket
            serverSocket = new ServerSocket(port);
            // Socket creation
            socket = serverSocket.accept();
            // Create a print writer
            out = new BufferedWriter(new OutputStreamWriter(socket.getOutputStream()));
            // Create a buffered reader
            in = new BufferedReader(new InputStreamReader(socket.getInputStream()));
        } catch (IOException e) {
            logger.severe("Error creating server socket");
        }
    }

    // Close the server socket
    public void closeServer() {
        try {
            serverSocket.close();
        } catch (IOException e) {
            logger.severe("Error closing server socket");
        }
    }

    public void sendMessage(String message) {
        try {
            // Sending the byte lenght of the message
            byte[] ptext = message.getBytes("UTF-8");
            send(String.valueOf(ptext.length));
            // Sending the message
            send(message);
        } catch (IOException e) {
            logger.severe("Error sending message:" + e.getMessage());
        }
    }

    private void send(String message) throws IOException {
        out.write(message);
        out.newLine();
        out.flush();
    }

    public String receiveMessage() {
        try {
            return in.readLine();
        } catch (IOException e) {
            logger.severe("Error receiving message");
            return null;
        }
    }

This is the Java Thread handling the sending of the messages. It gets the message to send from a Queue that is shared between other threads.

public class SendToPlatform implements Runnable {

    private static final int PORT = 9999;
    private Thread worker;
    private AtomicBoolean running;
    private AtomicBoolean stopped = new AtomicBoolean(false);
    private BlockingQueue<String> queueOut;
    private Logger logger;
    private ServerSocketConnection serverSocketConnection;

    public SendToPlatform(BlockingQueue<String> queueOut, AtomicBoolean running) {
        this.queueOut = queueOut;
        this.running = running;
        this.logger = App.getLogger();
        serverSocketConnection = new ServerSocketConnection(PORT);
    }

    public void run() {
        stopped.set(false);
        serverSocketConnection.createServer();
        while (running.get()) {
            socketSender();
        }
        stopped.set(true);
    }

    private void socketSender() {
        if (!queueOut.isEmpty()) {
            String element = null;
            try {
                element = queueOut.poll(1000, TimeUnit.MILLISECONDS);
            } catch (InterruptedException e) {
                logger.severe("SendToPlatform: InterruptedException: " + e.getMessage());
            }
            serverSocketConnection.sendMessage(element);
        }
    }
}

This is the python thread that is used to receive the message from the Java socket server:

    def __socket_reading_no_server(self, queue_input : queue.Queue, is_running : bool):
        HOST = "localhost"
        PORT = 9999
        s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
        s.connect((HOST, PORT))

        while is_running:
            data = s.recv(4)
            message_size = int(data.decode('UTF-8').strip())
            data = s.recv(min(message_size + 2, 1024))
            message = data.decode('UTF-8').strip()
            queue_input.put(message)
        s.close()

And this method is lanched as a thread with these instructions:

input_thread = threading.Thread(target=self.__socket_reading_no_server , args =(self.__queue_input, self.__running, ), daemon=True)
input_thread.start()

By debugging, logging, and using Wireshark to understand the problem in my code, I concluded that I have a recurrent problem with the out.write instruction that blocks while sending the message after around 10 messages are sent correctly. The pending message gets released when I close the socket connection. I tried using PrintWriter and DataOutputStream instead of BufferedWriter, but the same problem occurred. I tried not sending the length of the message before sending the string to adapt the s.recv() size, but the same problem occurred. I'm new to socket programming, and probably I did something incredibly wrong, but I cannot find where the problem is. Maybe is there a better way to pass data between processes that I'm unaware of that I can use for my needs instead of sockets?

Edits after @absuu answer

After applying the corrections suggested in the answer I still get the same problem of out.write in the send method blocking while trying to write into the socket. I edited my code as follows:

public class ServerSocketConnection {
    [...]
    public void sendMessage(String message) {
        try {
            send(message);
        } catch (IOException e) {
            logger.severe("Error sending message:" + e.getMessage());
        }
    }

    private void send(String message) throws IOException {
        message += "\r\n";
        byte[] ptext = message.getBytes("UTF-8");
        out.write(String.format("%2d",ptext.length));
        out.write("\r\n");
        out.flush();
        out.write(new String(ptext));
        logger.info("Data sent");
        out.flush();
    }
}

I also increased the s.recv size but nothing changed

liogiu2
  • 3
  • 3
  • Too much to read here, but it looks to me that your 'messages' are inconsistent. You send the length as the text representation of the length of the message as it would be in UTF-16, though there's nothing I can see to say that you actually send the message in UTF-16. Meanwhile, on the Python side I see talk of UTF-8. – user16632363 Nov 11 '21 at 12:22
  • 1
    The way you're sending the message length is flawed. You're sending a string representation of the length. If your string is 3 characters then you'll send "3". If it's 10 characters you'll send "10". However, on the server side you always read 4 bytes. In other words, the potential for this working reliably is constrained by the message length being >=1000 and <=9999. Take a look at [this answer](https://stackoverflow.com/questions/69894610/java-client-and-python-server-socket-connection-end-of-file-error-when-receivi/69896395#69896395) I wrote a couple of days ago relating to a similar issue –  Nov 11 '21 at 12:45
  • @BrutusForcus I changed my code following the answer you linked, and I still have the same problem of the socket blocking the stream after some messages were sent correctly. I used in python the `from multiprocessing.connection import Listener` with the method `read_bytes` and changed the Java as you suggested, but nothing changed. – liogiu2 Nov 11 '21 at 15:06
  • You're still sending the message length inappropriately. Take a look at the Java client code in the Answer I alluded to previously –  Nov 11 '21 at 15:20

2 Answers2

0

Here's a "cut down" implementation of a Java client and Python server that demonstrates an efficient mechanism for sending messages (in this case strings) of arbitrary length:

import java.io.DataOutputStream;
import java.io.IOException;
import java.net.Socket;

public class Client {

    public static void main(String[] args) throws IOException {
        String message = "Talking isn't doing. It is a kind of good deed to say well; and yet words are not deeds.";
        try (Socket socket = new Socket("localhost", 7070)) {
            try (DataOutputStream out = new DataOutputStream(socket.getOutputStream())) {
                byte[] dts = message.getBytes();
                out.writeInt(dts.length);
                out.write(dts);
            }
        }
    }
}

Note how the client sends the length of the upcoming message as a 32-bit integer before sending the actual message.

from multiprocessing.connection import Listener

def main():
    listener = Listener(address=('0.0.0.0', 7070), family='AF_INET', authkey=None)
    with listener.accept() as conn:
        print(conn.recv_bytes().decode())
            

if __name__ == '__main__':
    main()

The connection class expects to receive messages using the protocol implemented on the client side (Java in this case) - i.e., a 32-bit integer that gives the amount of data to follow.

I hope this clarifies the matter

  • I am unable to find documentation that shows that `recv_bytes()` expects a 32-bit integer length followed by that many bytes. Can you provide a link? – President James K. Polk Nov 11 '21 at 17:28
  • @PresidentJamesK.Polk I couldn't find any documentation either. Not everything Python-related is well documented. The protocol that I describe is well-known and I just guessed that it might be using that strategy. I was able to determine empirically that I was right. Also, if you think about it, the 2 code samples in my answer simply wouldn't work if that was not the case. –  Nov 11 '21 at 18:06
0

TL;DR , see Codes Corrections below.


Before any of your questions, there are some points that need to be concerned with:

  1. Change your server's data encoding from UTF-16 to UTF-8. WHY? Data transmission between any endpoints relies on the consistency, that is, data encoding for server/client applications. While your server (ServerSocketConnection) is sending messages encoded with UTF-16, your client (__socket_reading_no_server) is receiving messages with UTF-8. Even though your client is capable of receiving all the messages from the server, it cannot recognize them at all. For example, UTF-16 encodes string "5" into bytes [0,53], for UTF-8 the result is [53] (assuming big-endian byte ordering). Refer to Wikipedia for more details.
  2. Do not use out.newLine(). Use out.write("\r\n") instead. WHY? newline()'s behavior is platform-dependent, which results in returning one or two characters, for Unix-like OS or Windows OS, respectively. It relies on a line.separator system property, you can refer to Java doc for more details.
  3. Your data = s.recv(4) put a constraint that you client reads at most 4 bytes at a time, which is dangerous. WHY? Since according to Python doc, your 4 is not the actual number of bytes that client received, but the maximum amount of data to be received. Furthermore, the client would be theoretically only able to receive up to 9999 bytes (byte1~4: '9') of the next incoming message.

For your question "...instruction that blocks while sending the...": Unfortunately, there are no error messages provided here, and we cannot precisely deduce which part of the whole thing went wrong. However, we can infer that, it is more likely the consequence of Java's implementation for network socket, since the situation you encountered is probably rather uncommon in C network programming (i.e., raw sockets), that is, blocking won't happen during the continuous transmission of bytes, according to POSIX definitions for write system call (note that most high-level languages eventually invoke write system call to send bytes) :

Upon successful completion, write() and pwrite() shall return the number of bytes actually written to the file associated with fildes. This number shall never be greater than nbyte. Otherwise, -1 shall be returned and errno set to indicate the error.

That is, after invoking write to send bytes to stream buffer, it would just return something, instead of blocking.

The Java implementation for network socket is rather sophisticated, and this is definitely not Java's fault. In fact, if we could use sockets properly, then the obscure errors will disappear. For example, according to my tests, after applying the following corrections, your application works just fine:


Codes Corrections:

  1. ServerSocketConnection / byte[] ptext = message.getBytes("UTF-16"); -> byte[] ptext = message.getBytes("UTF-8");
  2. ServerSocketConnection / send(String.valueOf(ptext.length)); -> send(String.format("%2d",ptext.length));
  3. ServerSocketConnection / out.newLine() -> out.write("\r\n")

Tests:

server:

BlockingQueue<String> q = new ArrayBlockingQueue<String>(20);
q.add("str 1");
q.add("str 2");
q.add("str 3");
serverSocketConnection.sendMessage(element);
logger.info("element:"+element);  // debug
########################################################## Server Outputs  
Nov 11, 2021 11:05:51 PM SendToPlatform socketSender
INFO: element:str 1
Nov 11, 2021 11:05:51 PM SendToPlatform socketSender
INFO: element:str 2
Nov 11, 2021 11:05:51 PM SendToPlatform socketSender
INFO: element:str 3

client:

print("message: %s" % message)   # debug
# queue_input.put(message)
########################################################## Client Outputs
message: str 1
message: str 2
message: str 3

EDITS :

One more thing I want to emphasis is that although we don't actually know what your application gonna do, the simple length-based message transmission is possible through network sockets. Maybe it is not much practical and robust, but it is definitely possible. Here are some more detailed corrections for your codes:

ServerSocketConnection :

// ............. other parts stay unchanged
public void sendMessage(String message) {
    try {
        int msgLen = message.getBytes("UTF-8").length;
        send(String.format("%3d", msgLen));     // tell client the message size
        send(message);      // send actual message
    } catch (IOException e) {
        logger.severe("Error sending message:" + e.getMessage());
    }
}

private void send(String message) throws IOException {
    out.write(message);
    out.flush();
}
// ............. other parts stay unchanged

__socket_reading_no_server :

# ............. other parts stay unchanged
while is_running:
    data = s.recv(3)
    message_size = int(data.decode('UTF-8').strip())
    data = s.recv(min(message_size, 1024))
    message = data.decode('UTF-8').strip()
    print("incoming message:[%s]" % message)
# ............. other parts stay unchanged

VERY VERY VERY IMPORTANT :

Here we have M=3, which put a constraint that any of your messages should be a string with 999 character at most ! That is, if you want your application works, each of your message, say String msg, should satisfy msg.length <= 999.

absuu
  • 340
  • 1
  • 11
  • I tried your corrections and I read carefully your answer. Thank you for the clarifications and I'll edit my question to address your concerns. However, after implementing your corrections I still have the same problem of the java blocking while sending. I don't have any error messages to show, because it is not giving me one. It just blocks on the `out.write` call until I close the stream. – liogiu2 Nov 12 '21 at 10:07
  • @liogiu2 Have you tried the test cases I've posted? If those tests works fine, the problem might be related to the length of strings in your message queue. The suggestions I've provided are just responsible for **My Tests**, which use some of relatively short strings as inputs. `str 1` or `str 2` takes 5 characters, this is why I suggested `String.format("%2d",len)`, that is, len(" 5\r\n")=4. Similarly, if your string has size of number N, and N takes M digits at most, then you should use `String.format("%{M}d",len)` instead, plus `s.recv({M})`, plus deleting line 4 `out.write("\r\n");` – absuu Nov 12 '21 at 14:23
  • @liogiu2 see EDITS. Hope that part could finally remove the obstacles. – absuu Nov 13 '21 at 13:30