We're running a Graylog2 server in a Docker container in our development environment. It works like a charm apart from the fact that we have to re-create the UDP input every time we launch the container.
Has anyone figured out a convenient way to automatically create Graylog2 inputs?
Use a auto-loaded content pack in a newly created docker container.
Dockerfile (since Graylog 3.2 - thanks to T. van den Berg):
FROM graylog2/server:latest
COPY udp-input-graylog.json /usr/share/graylog/data/contentpacks
ENV GRAYLOG_CONTENT_PACKS_AUTO_INSTALL udp-input-graylog.json
ENV GRAYLOG_CONTENT_PACKS_LOADER_ENABLED true
ENV GRAYLOG_CONTENT_PACKS_DIR data/contentpacks
Dockerfile (pre 3.0, see this pull request ). :
FROM graylog2/server:latest
COPY udp-input-graylog.json /usr/share/graylog/data/contentpacks
ENV GRAYLOG_CONTENT_PACKS_AUTO_LOAD udp-input-graylog.json
ENV GRAYLOG_CONTENT_PACKS_LOADER_ENABLED true
ENV GRAYLOG_CONTENT_PACKS_DIR data/contentpacks
udp-input-graylog.json (Pre 3.0):
{
"name":"UDP GELF input on 12201",
"description":"Adds a global UDP GELF input on port 12201",
"category":"Inputs",
"inputs":[
{
"title":"udp input",
"configuration":{
"override_source":null,
"recv_buffer_size":262144,
"bind_address":"0.0.0.0",
"port":12201,
"decompress_size_limit":8388608
},
"static_fields":{},
"type":"org.graylog2.inputs.gelf.udp.GELFUDPInput",
"global":true,
"extractors":[]
}
],
"streams":[],
"outputs":[],
"dashboards":[],
"grok_patterns":[]
}
To get a contentpack json compatible with 3.0, just create the input via the GUI and then create and download the contentpack via the GUI as well.
Steps to create multiple inputs using a contentpack:
Write them into a file with json format (e.g.)
{"id" : null,
"name":" Inputs",
"description":"Contentpack that adds global inputs",
"category":"Inputs",
"inputs":[
{
"title":"udp input",
"configuration":{
"override_source":null,
"recv_buffer_size":262144,
"bind_address":"0.0.0.0",
"port":12201,
"decompress_size_limit":8388608
},
"static_fields":{},
"type":"org.graylog2.inputs.gelf.udp.GELFUDPInput",
"global":true,
"extractors":[]
},
{
"title":"tcp input",
"configuration":{
"override_source":null,
"recv_buffer_size":262144,
"bind_address":"0.0.0.0",
"port":12202,
"decompress_size_limit":8388608
},
"static_fields":{},
"type":"org.graylog2.inputs.gelf.tcp.GELFTCPInput",
"global":true,
"extractors":[]
}]
}
copy the contentpack to the contentpacks directory in graylog using ansible
- name: create graylog inputs for receiving logs
shell: cp .templates/inputs.json /usr/share/graylog-server/contentpacks/inputs.json
Set contentpacks autoload to True in graylog.conf or via ansible
graylog_content_packs_loader_enabled: true
Set contentpacks autoload to load inputs.json (e.g. via ansible)
graylog_content_packs_auto_load: inputs.json
Hope this helps!
I use ansible for starting and preparing graylog2 in containers. And I just create global udp input via calling graylog2 rest api (after graylog2 auto configuration has been finished):
- name: create graylog global udp input for receiving logs
uri:
url: http://{{ ipv4_address }}:9000/api/system/inputs
method: POST
user: "{{ graylog_admin }}"
password: "{{ graylog_pwd }}"
body: '{"title":"xxx global input","type":"org.graylog2.inputs.gelf.udp.GELFUDPInput","configuration":{"bind_address":"0.0.0.0","port":12201,"recv_buffer_size":262144,"override_source":null,"decompress_size_limit":8388608},"global":true}'
force_basic_auth: yes
status_code: 201
body_format: json
[ansible] [docker] [graylog2]
If that helps, used graylog3 REST API to create arbitrary input. I guess it might be the case to use sample bash script below to invoke once graylog-api is available.
#!/bin/bash
if [ `curl -s -u admin:admin -H 'Content-Type: application/json' -X GET 'http://graylog:9000/api/system/inputs' | grep -c 'Standard GELF UDP input'` == 0 ]
then
curl -u admin:admin -H 'Content-Type: application/json' -X POST 'http://graylog:9000/api/system/inputs' -d '{
"title": "Standard GELF UDP input",
"type": "org.graylog2.inputs.gelf.udp.GELFUDPInput",
"global": true,
"configuration": {
"recv_buffer_size": 1048576,
"tcp_keepalive": false,
"use_null_delimiter": true,
"number_worker_threads": 2,
"tls_client_auth_cert_file": "",
"bind_address": "0.0.0.0",
"tls_cert_file": "",
"decompress_size_limit": 8388608,
"port": 12201,
"tls_key_file": "",
"tls_enable": false,
"tls_key_password": "",
"max_message_size": 2097152,
"tls_client_auth": "disabled",
"override_source": null
},
"node": null
}' -H 'X-Requested-By: cli'
else
echo "Standard GELF UDP input exists already"
fi
In the end this did the trick for me. I ended up inserting the relevant configuration directly into MongoDB.
We have a puppet solution for this (graylog2 v2.2.2). Basically enable content-packs in the server.conf, and list the relevant file(s) that will be your json content (see above for UDP inputs as a nice example). Simply a file resource in puppet placed on the graylog server in the configured dir (default is /usr/share/graylog-server/contentpacks)
This will be loaded on 1st run of graylog.
It a nice way of getting in a lot of config.
After messing around with this for some time, I've come up with a kludge. It's probably not the best way to do this, but documentation is sparse and it seems to work (however, see the caveats at the end of this post).
The problem is that every time you restart your Graylog2 container, the server generates a new unique node ID. The inputs that you define through the web interface are tied to a specific node ID. Thus, every time the node ID changes, the inputs that you've defined become useless.
The solution has two parts:
Ensure that the MongoDB is storing its data somewhere persistent - either a data container or in a directory mounted from the host file system.
Force the container to use the same node ID every time. I did this by extending the sjoerdmulder/graylog2 image:
Here's the Dockerfile:
FROM sjoerdmulder/graylog2
# set a Graylog2 node ID
#
RUN echo "mynodeid" > /opt/graylog2-server/server-node-id
RUN chmod 0444 /opt/graylog2-server/server-node-id
This will write "mynodeid" to the appropriate file and write-protect it by changing the permissions. The server then starts up normally and loads the appropriate node ID. Now you can go into the web interface, create your input, and be confident that the next time you start up your container, it will still be there.
I'm not using a cluster. I'm just running a single instance of Graylog2 accepting input from a single web application. I have no idea what this will do when part of a cluster.
Another option is to create your inputs as "global" - that way regardless of the node id that is generated for your current instance, you will get back all of your configured global inputs.
The content pack json provided in the answer didn't work for me. I got an error:
ERROR: org.graylog2.periodical.ContentPackLoaderPeriodical - Uncaught exception in periodical java.lang.IllegalArgumentException: Unsupported content pack version: 0
However, it turned out that creating of a content pack itself is easy and straightforward process.
