How to properly decode and encode characters from JTextArea

Question

I have a program that works on a console, and i want to make a custom console for it. Current command line interface can be started with a method that takes an InputStream and PrintStream as arguments.

I have two text areas (JTextArea), one of which i want to use for input and the other one for output. I've extended InputStream and OutputStreams to provide streams to my starting method:

    public class ConsoleInputStream extends InputStream implements KeyListener {

    private BlockingDeque<Integer> mBuffer = new LinkedBlockingDeque<>();
    private JTextArea mJTextArea;

    public ConsoleInputStream(JTextArea JTextArea) {
        mJTextArea = JTextArea;
        mJTextArea.addKeyListener(this);
    }

    @Override
    public void keyTyped(KeyEvent e) {}

    @Override
    public void keyPressed(KeyEvent e) {}

    @Override
    public void keyReleased(KeyEvent e) {
        int key = e.getKeyChar();
        char c = (char) key;
        mBuffer.add(key);
    }

    @Override
    public int read() {
        try {
            char c = (char) (int) mBuffer.take();

            if(c == '\n')
                mJTextArea.setText("");

            return c;
        } catch (InterruptedException e) {
            e.printStackTrace();
        }

        return 0;
    }

    @Override
    public int read(byte[] b, int off, int len) {
        if (b == null) {
            throw new NullPointerException();
        } else if (off < 0 || len < 0 || len > b.length - off) {
            throw new IndexOutOfBoundsException();
        } else if (len == 0) {
            return 0;
        }
        int c = read();
        if (c == -1) {
            return -1;
        }
        b[off] = (byte)c;

        int i = 1;
        try {
            for (; i < len && available() > 0 ; i++) {
                c = read();
                if (c == -1) {
                    break;
                }
                b[off + i] = (byte)c;
            }
    } catch (IOException e) {
    }
        return i;

    }
}

And for the output:

    public class ConsoleOutputStream extends OutputStream {

    private JTextArea mJTextArea;

    public ConsoleOutputStream(JTextArea JTextArea) {
        mJTextArea = JTextArea;
    }

    @Override
    public void write(int b) throws IOException {
        mJTextArea.append(String.valueOf((char) b));
    }
}

Start the program:

 CommandInterface.get().start(ui.getConsoleIn(), new PrintStream(ui.getConsoleOut()));

(ui is an instance of a class that extends JFrame, the getConsoleIn() and getConsoleOut() return an instance of ConsoleInputStream and ConsoleOutputStream)

Inside of which i use scanner to read the input stream:

public void start(InputStream inputStream, PrintStream outputStream){
    Scanner scanner = new Scanner(inputStream, "UTF-8");

    while (true){
        String[] input = scanner.nextLine().split(" ");

        if(input[0].equals("exit"))
            break;

        Command command = mCommands.get(input[0]);
        if(command == null){
            displayErrorMessage("No such command", outputStream);
            continue;
        }

        List<String> flags = new LinkedList<>();
        List<String> params = new LinkedList<>();

        for(String s : Arrays.copyOfRange(input, 1, input.length)){
            if(s.charAt(0) == '/')
                flags.add(s.substring(1));
            else
                params.add(s);
        }

        command.execute(outputStream, flags, params);
    }

}

And this works fine, untill I try to use the local characters: ś ć ó ż ź etc.

I have tried many diffrent solutions, none worked for me. Then I tried to figure it out myself. Every time I read a char I also printed it to standard output (my IDE), which I know can display those characters correctly. I found out that they are being read correctly, but thre are characters (UTF-8 65535) inbetween them (not in a regular pattern but in pairs), for reasons that are unclear to me. I also tried:

Scanner scanner = new Scanner(System.in);
        while (true){
          ui.getConsoleOut().write(scanner.nextLine().getBytes(StandardCharsets.UTF_8));
        }

with diffrent charsets, but couldn't get them do display correctly.

What is the proper way to display those (and other UTF-8) characters?

Answer 1

I'm not sure whether you have done anything else wrong, but I know you at least need to fix this:

read and write methods don't work with characters, they work with bytes. One character != one byte.

I'm talking about these:

public int read() {
    try {
        char c = (char) (int) mBuffer.take();

        if(c == '\n')
            mJTextArea.setText("");

        return c;
    } catch (InterruptedException e) {
        e.printStackTrace();
    }

    return 0;
}

public void write(int b) throws IOException {
    mJTextArea.append(String.valueOf((char) b));
}

You need to turn the char into a byte array using an encoding that the Scanner can understand. Then turn each of those bytes to unsigned ints, as opposed to treating each character as a single byte.

public void keyReleased(KeyEvent e) {
    int key = e.getKeyChar();
    char c = (char) key;

    if(c == '\n')
        mJTextArea.setText("");

    byte[] byteArray = Character.toString(c).getBytes(StandardCharset.UTF_8);
    for (byte b : byteArray) {
        mBuffer.add(Byte.toUnsignedInt(b));
    }
}

public int read() {
    try {
        byte b = (int) mBuffer.take();
        return b;
    } catch (InterruptedException e) {
        e.printStackTrace();
    }

    return -1;
}

For write, you can't treat each byte as a single character either. One way to handle this is to subclass PrintStream directly. See solution 2 in this answer for an example.

Answer 2

The thing i needed to do, besides what Sweeper said, is to ignore the undefined characters (some keys like ALT or CTRL do not have a char associated with them, so the result is char 65535 - char undefined - )

@Override
    public void keyReleased(KeyEvent e) {

        char c = e.getKeyChar();


        if(c == '\n')
            mJTextArea.setText("");

        if(c == KeyEvent.CHAR_UNDEFINED)
            return;

        byte[] byteArray = Character.toString(c).getBytes(StandardCharsets.UTF_8);
        for (byte b : byteArray) {
            mBuffer.add(Byte.toUnsignedInt(b));
        }
    }