4

I am implementing a monitoring tool on servers of my company service. To do that, I am using logstash. Our applications send their logs via a log4net udp appender to logstash (input udp) and then logstash grok them, and send them to elasticsearch. When I display my logs in kibana, I see that some logs are truncated, the last main part is missing (for big logs). So my question is, does Logstash has a size limit for each message-event received. If yes, is it possible to increase the size. I need all my logs and none of them truncated.

user2443476
  • 1,935
  • 9
  • 37
  • 66

4 Answers4

4

Update for 2021:

The maximum size is set by the buffer_size parameter in the UDP input. https://www.elastic.co/guide/en/logstash/current/plugins-inputs-udp.html#plugins-inputs-udp-buffer_size

The default buffer size is:

  • 65536 bytes on Logstash >= 5.1
  • 8192 bytes on Logstash >= 2.0
  • 4096 bytes on older versions.

A UDP datagram is limited to 65535 bytes, the length header being 16 bits.

user5994461
  • 5,301
  • 1
  • 36
  • 57
3

For the udp case, I think that I have found the solution : -increase the buffer_size parameter in udp.rb file.

I cannot test it now, but I will tell you if it works.

user2443476
  • 1,935
  • 9
  • 37
  • 66
  • You can just specific at the configuration. No need to modify the udp.rb. Please check the buffer_size in http://logstash.net/docs/1.4.2/inputs/udp – Ban-Chuan Lim Jul 08 '14 at 01:21
  • Setting the buffer_size in the input filter does NOT work in Logstash v5.2. – Chris F Mar 10 '17 at 14:39
2

I have test it with Logstash 1.4.0 and Logstash 1.3.3. I found that the maximum size of an event is 4095!

So, If your logs have larger than this size, maybe you have to split it to multiple event at the time you send the logs to logstash.

Ban-Chuan Lim
  • 7,840
  • 4
  • 35
  • 52
  • ok, thanks for your answer. But I have some basic questions : What is the unit of 4095? How to calculate the size of my message to know if it reaches 4095(unit)? – user2443476 Jul 07 '14 at 09:10
  • I think it is probably 4096, 4KB. I try it with input over 4096 character and then the message only output 4095 character. Maybe the last byte is '\n' – Ban-Chuan Lim Jul 07 '14 at 10:31
  • 2
    ok, for my case, all the messages with caracters count less than 8551 appears in integrality. For the others, messages are truncated at the 8551th caracter. – user2443476 Jul 07 '14 at 11:05
  • Can you tell how/where you found this information ? – vdolez Mar 26 '15 at 16:28
2

Logstash's property buffer_size is by default set to 8192. That's why messages sent over UDP to Logstash are truncated at 8192th symbol.

Try increasing UDP buffer_size in Logstash.

References:

Tony
  • 1,433
  • 1
  • 15
  • 18