I cannot find the cause of this WARN (logstash): "Invalid version of beats protocol"

Question

Dear Stackoverflow commmunity,

I would like to know why I'm staked at this problem with logstash:

2021-01-20T01:02:33,444][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}
[2021-01-20T01:02:41,603][INFO ][org.logstash.beats.BeatsHandler][synlite_suricata][input_beats] [local: 10.0.100.12:5044, remote: 10.0.100.1:39666] Handling exception: org.logstash.beats.InvalidFrameProtocolException: Invalid version of beats protocol: 22
[2021-01-20T01:02:41,614][WARN ][io.netty.channel.DefaultChannelPipeline][synlite_suricata][input_beats] An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception.
io.netty.handler.codec.DecoderException: org.logstash.beats.InvalidFrameProtocolException: Invalid version of beats protocol: 22
    at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:472) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
    at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
    at io.netty.channel.AbstractChannelHandlerContext.access$600(AbstractChannelHandlerContext.java:38) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
    at io.netty.channel.AbstractChannelHandlerContext$7.run(AbstractChannelHandlerContext.java:353) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
    at io.netty.util.concurrent.DefaultEventExecutor.run(DefaultEventExecutor.java:66) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
    at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:897) [netty-all-4.1.30.Final.jar:4.1.30.Final]
    at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [netty-all-4.1.30.Final.jar:4.1.30.Final]
    at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: org.logstash.beats.InvalidFrameProtocolException: Invalid version of beats protocol: 22
    at org.logstash.beats.Protocol.version(Protocol.java:22) ~[logstash-input-beats-6.0.9.jar:?]
    at org.logstash.beats.BeatsParser.decode(BeatsParser.java:62) ~[logstash-input-beats-6.0.9.jar:?]
    at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
    at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
    ... 8 more
[2021-01-20T01:02:41,637][INFO ][org.logstash.beats.BeatsHandler][synlite_suricata][input_beats] [local: 10.0.100.12:5044, remote: 10.0.100.1:39666] Handling exception: org.logstash.beats.InvalidFrameProtocolException: Invalid version of beats protocol: 3
[2021-01-20T01:02:41,639][WARN ][io.netty.channel.DefaultChannelPipeline][synlite_suricata][input_beats] An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception.

I found similar topics, but I cannot find a clear explaination... Logstash beats input "invalid version of beats protocol" Mock an ELK Beat output to Logstash with Postman

I can share my config:

pipelines.yml:

- pipeline.id: synlite_suricata
  path.config: "/etc/logstash/synlite_suricata/conf.d/*.conf"

- pipeline.id: fallback
  path.config: "/etc/logstash/fallback/conf.d/*.conf"

synlite_suricata input:

input {
  # Beats
  beats {
    id => "input_beats"
    host => "${SYNLITE_SURICATA_BEATS_HOST}"
    port => "${SYNLITE_SURICATA_BEATS_PORT}"
    client_inactivity_timeout => 180
    ssl => false
    ssl_certificate_authorities => "${SYNLITE_SURICATA_CACERT}"
    ssl_certificate => "${SYNLITE_SURICATA_BEATS_CERT}"
    ssl_key => "${SYNLITE_SURICATA_BEATS_KEY}"
    ssl_verify_mode => "peer"
    ssl_peer_metadata => true
    tls_min_version => 1.2
    cipher_suites => [ "TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256", "TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384" ]

  }
}

and the systemd variables in /etc/systemd/system/logstash.service.d/synlite_suricata.conf:

[Service]
# Synesis Lite for Suricata global configuration
Environment="SYNLITE_SURICATA_DICT_PATH=/etc/logstash/synlite_suricata/dictionaries"
Environment="SYNLITE_SURICATA_TEMPLATE_PATH=/etc/logstash/synlite_suricata/templates"
Environment="SYNLITE_SURICATA_GEOIP_DB_PATH=/etc/logstash/synlite_suricata/geoipdbs"
Environment="SYNLITE_SURICATA_GEOIP_CACHE_SIZE=8192"
Environment="SYNLITE_SURICATA_GEOIP_LOOKUP=true"
Environment="SYNLITE_SURICATA_ASN_LOOKUP=true"
Environment="SYNLITE_SURICATA_CLEANUP_SIGS=false"

# Name resolution option
Environment="SYNLITE_SURICATA_RESOLVE_IP2HOST=false"
Environment="SYNLITE_SURICATA_NAMESERVER=127.0.0.1"
Environment="SYNLITE_SURICATA_DNS_HIT_CACHE_SIZE=25000"
Environment="SYNLITE_SURICATA_DNS_HIT_CACHE_TTL=900"
Environment="SYNLITE_SURICATA_DNS_FAILED_CACHE_SIZE=75000"
Environment="SYNLITE_SURICATA_DNS_FAILED_CACHE_TTL=3600"

# Elasticsearch connection settings
Environment="SYNLITE_SURICATA_ES_HOST=10.0.100.11"
Environment="SYNLITE_SURICATA_ES_USER=logstash"
Environment="SYNLITE_SURICATA_ES_PASSWD=password"

# Beats input
Environment="SYNLITE_SURICATA_BEATS_HOST=10.0.100.12"
Environment="SYNLITE_SURICATA_BEATS_PORT=5044"

# Certs config
Environment="SYNLITE_SURICATA_CACERT=/etc/logstash/tls/root-ca.crt"
Environment="SYNLITE_SURICATA_BEATS_CERT=/etc/logstash/tls/logstash-input-server.crt"
Environment="SYNLITE_SURICATA_BEATS_KEY=/etc/logstash/tls/logstash-input-server.pk8"
Environment="SYNLITE_SURICATA_ES_KEYSTORE=/etc/logstash/tls/logstash-elasticsearch-output-client.p12"
Environment="SYNLITE_SURICATA_ES_KEYSTORE_PASSWORD=password"

I don't know how to find and information related with "beat version"... Logstash-oss 7.8.0

Thanks, waiting for your attention

Badger · Answer 1 · 2022-01-05T04:09:20.700

Typically this is caused by something connecting to the beats input that is not talking the beats (lumberjack) protocol. The input is basically saying that a byte in certain position in the byte stream has a value it cannot understand. There can be many reasons for this.

To identify the cause you will need to find the program that has

remote: 10.0.100.1:39666

open and examine what it is sending you. tcpdump might help. Possible causes include (but are certainly not limited to):

A beat being configured to use SSL but the input is not
The beat input is expecting SSL but filebeat does not have it configured
If you are on a corporate network the corporate security folks may be running port scans against every internal IP address and trying to connect to every TCP port (possibly hundreds of millions of ports each week) and checking if it responds to HTTP or a number of other protocols that might be security weaknesses if not properly secured.
If that address is on the internet (unlikely, since it is a private address, but you might have obfuscated the message) then I can guarantee that Shodan and many others are port scanning you.
Another possibility is commenting out the elasticsearch hosts: entry in filebeat.yml (but not the elasticsearch: entry) and then uncommenting the logstash hosts: entry (but not the logstash: entry). This results in the beat trying to talk HTTP to the beats input, which will throw this exception.

score 0 · Answer 2 · answered Jan 20 '21 at 13:48

0

Very helpful, thanks! I will explore with tcpdump. I am under a private network using Wireguard tunel.

answered Jan 20 '21 at 13:48

Ecofintech

323
2
5
15

I cannot find the cause of this WARN (logstash): "Invalid version of beats protocol"

2 Answers2