Graylog GELF max field size

Question

I'm trying to insert web service input/output as xml into Graylog. To do this I used "GELFJ - A GELF Appender for Log4j and a GELF Handler for JDK Logging". Here is the sample code:

public static void main(String[] args) throws IOException {
    GelfSender gelfSender = new GelfTCPSender("172.21.120.139", 12201);

    String xmlMessage = readFile("c:\\temp\\xml.xml");

    GelfMessage message = new GelfMessage("short message", "long message", System.currentTimeMillis(), "1");
    message.setHost("localhost");

    message.addField("XML", xmlMessage);
    message.addField("LEN", xmlMessage.length());

    if (message.isValid()) {
        GelfSenderResult result = gelfSender.sendMessage(message);
        Exception exception = result.getException();
        if (exception != null) {
            exception.printStackTrace();
        }
    } else {
        System.err.println("Message is not valid!");
    }
}

And this is the GELF TCP input properties.

I can't insert a message field bigger than 20k (characters). and a message total size bigger than 1.6 MB.

My question is what are the limits of a message field and the message total size in bytes?

Answer 1

Graylog and the GELF protocol itself don't have any size restrictions but it should be noted that Lucene (the library underlying Elasticsearch) has a limit of 32 KiB per field for analyzed fields (e. g. the "message" and "full_message" fields of a GELF message).

Given that the default encoding in Elasticsearch and Graylog is UTF-8, the 20.000 characters could approximately match the maximum field size of 32 KiB (for analyzed fields). Non-analyzed fields can be (almost) arbitrarily big.

See https://github.com/Graylog2/graylog2-server/issues/873 for a related issue on GitHub.