Increase the maximum size of log messages

Question

I would like to increase the maximum size of message saved in graylog with elasticsearch.

The maximum message size is 32 kb

So I update the mapping to remove the index on full_message:

graylog-custom-mapping.json:

{
  "template": "graylog_*",
  "mappings": {
    "message": {
      "properties": {
        "full_message": {
          "index": "no",
          "doc_values": false,
          "type": "string"
        }
      }
    }
  }
}

curl -X PUT -d @'graylog-custom-mapping.json' 'http://localhost:9200/_template/graylog-custom-mapping?pretty'

{
  "acknowledged" : true
}

I created a new Graylog index (graylog_5) from Graylog interface (System > Indices > Maintenance > Manually cycle deflector)

But my mapping seems to not be concidered:

curl -X GET 'http://localhost:9200/graylog_5/_mapping/message'

...
"full_message": {
    "type": "string",
    "analyzer": "standard"
},
...

My active template:

{
  "graylog-internal": {
    "order": -2147483648,
    "template": "graylog_*",
    "settings": {
      "index": {
        "analysis": {
          "analyzer": {
            "analyzer_keyword": {
              "filter": "lowercase",
              "tokenizer": "keyword"
            }
          }
        }
      }
    },
    "mappings": {
      "message": {
        "_source": {
          "enabled": true
        },
        "dynamic_templates": [
          {
            "internal_fields": {
              "mapping": {
                "index": "not_analyzed",
                "type": "string"
              },
              "match": "gl2_*"
            }
          },
          {
            "store_generic": {
              "mapping": {
                "index": "not_analyzed"
              },
              "match": "*"
            }
          }
        ],
        "properties": {
          "full_message": {
            "analyzer": "standard",
            "index": "analyzed",
            "type": "string"
          },
          "streams": {
            "index": "not_analyzed",
            "type": "string"
          },
          "source": {
            "analyzer": "analyzer_keyword",
            "index": "analyzed",
            "type": "string"
          },
          "message": {
            "analyzer": "standard",
            "index": "analyzed",
            "type": "string"
          },
          "timestamp": {
            "format": "yyyy-MM-dd HH:mm:ss.SSS",
            "type": "date"
          }
        }
      }
    },
    "aliases": {}
  },
  "graylog-custom-mapping": {
    "order": 0,
    "template": "graylog_*",
    "settings": {},
    "mappings": {
      "message": {
        "properties": {
          "full_message": {
            "index": "no",
            "type": "string",
            "doc_values": false
          }
        }
      }
    },
    "aliases": {}
  }
}

What's wrong with my configuration ?

Graylog 2.1.2 + ES 2.4.2

I have the following logs:

[2018-02-16 16:26:36,598][INFO ][cluster.metadata         ] [Zero] [graylog_5] creating index, cause [api], templates [graylog-internal, graylog-custom-mapping], shards [4]/[0], mappings [message]
[2018-02-16 16:26:37,091][INFO ][cluster.routing.allocation] [Zero] Cluster health status changed from [RED] to [GREEN] (reason: [shards started [[graylog_5][1], [graylog_5][2], [graylog_5][0], [graylog_5][2], [graylog_5][0]] ...]).
[2018-02-16 16:27:03,665][INFO ][cluster.metadata         ] [Zero] [graylog_5] update_mapping [message]
[2018-02-16 16:27:03,816][INFO ][cluster.metadata         ] [Zero] [graylog_5] update_mapping [message]

Thx

How did you create graylog_5 index? What command did you use? — Andrei Stefan, Feb 21 '18 at 09:27
@AndreiStefan Graylog > System > Indices > Maintenance > Manually cycle deflector — Paul, Feb 21 '18 at 09:37
:-) I am no Graylog expert, ES one wannabe. No idea what that does. Sorry. Getting back to your original question... what is the connection between your template with "index": "no" and "doc_values":"false" and increasing the message size? — Andrei Stefan, Feb 21 '18 at 09:51
@AndreiStefan it's to try the solution explained on https://github.com/Graylog2/graylog2-server/issues/873 — Paul, Feb 21 '18 at 09:57
After installing your template, have you deleted the grayloag index and recreated it again? — Val, Feb 21 '18 at 10:26
@Val yes from the Graylog interface, System > Indices > Maintenance > Manually cycle deflector — Paul, Feb 21 '18 at 10:28
When I perform all your steps above manually, I get the properly merged mapping, i.e. `"full_message" : { "type" : "string", "index" : "no", "analyzer" : "standard"},` — Val, Feb 21 '18 at 11:56
Can you see the logs generated from ES? If yes, what do you see when the index is created? Do you see something like this: `[graylog_5] creating index, cause [auto(index api)], templates [graylog-custom-mapping, graylog-internal]` ? — Val, Feb 21 '18 at 11:59
@Val `[2018-02-16 16:26:36,598][INFO ][cluster.metadata] [Zero] [graylog_5] creating index, cause [api], templates [graylog-internal, graylog-custom-mapping], shards [4]/[0], mappings [message] [2018-02-16 16:26:37,091][INFO ][cluster.routing.allocation] [Zero] Cluster health status changed from [RED] to [GREEN] (reason: [shards started [[graylog_5][1], [graylog_5][2], [graylog_5][0], [graylog_5][2], [graylog_5][0]] ...]). [2018-02-16 16:27:03,665][INFO ][cluster.metadata] [Zero] [graylog_5] update_mapping [message]` — Paul, Feb 21 '18 at 12:04
So we can see that both templates are being applied `graylog-internal, graylog-custom-mapping`. Can you try to delete your graylog_5 index manually via curl (i.e. not use the cycle deflector) and try again? — Val, Feb 21 '18 at 12:05
simply run `curl -XDELETE localhost:9200/graylog*` to delete all graylog indices and then restart over. — Val, Feb 23 '18 at 12:08
I installed a new configuration from scratch, I receive the long message as multiple messages, do you know why ? — Paul, Feb 23 '18 at 15:18
Can you update your question with the news logs you're seeing, please? — Val, Feb 27 '18 at 11:29
@Val https://serverfault.com/questions/899071/rsyslog-upload-limitation — Paul, Feb 27 '18 at 11:30

score 1 · Accepted Answer · answered Feb 28 '18 at 11:41

1

The problem was not elasticsearch indexing but the maximum message size uploaded from rsyslog udp protocol.

To fix it:

/etc/rsyslog.conf

and define

$MaxMessageSize 256k

at the first line.

answered Feb 28 '18 at 11:41

Paul

1,290
6
24
46

score 0 · Answer 2 · answered Jun 05 '20 at 15:33

0

I had the same problem with nxlog. Changing ShortMessageLength value in etc/nxlog/nxlog.conf fixed my issue. I hope it maybe be useful for someone.

answered Jun 05 '20 at 15:33

LugiHaue

2,702
1
15
13

Increase the maximum size of log messages

2 Answers2