2

I'm trying to automate the Prometheus node_exporter and my Prometheus Server. For the node_exporter I've written a module to install all the needed packages, set the $::ipaddress based on facter and some more..

Now I'd like to make sure that the collected informations ($hostname, $job_name, [...]) from the applying node are exported into the respective remote Prometheus configfile, but I want to have this step done asynchronously, so for example with a puppet agent run afterwards on the Prometheus Server.

I've tried to orientate the classes towards the puppetlabs/logrotate module, which is basically doing the following:

logrotate/init.pp

class logrotate (
  String $ensure              = present,
  Boolean $hieramerge         = false,
  Boolean $manage_cron_daily  = true,
  Boolean $create_base_rules  = true,
  Boolean $purge_configdir    = false,
  String $package             = 'logrotate',
  Hash $rules                 = {},
) {
  do some stuff
}

logrotate/rules.pp

class logrotate::rules ($rules = $::logrotate::rules){
  #assert_private()
  create_resources('logrotate::rule', $rules)
}

logrotate/rule.pp

define logrotate::rule(
  Pattern[/^[a-zA-Z0-9\._-]+$/] $rulename           = $title,
  Enum['present','absent'] $ensure                  = 'present',
  Optional[Logrotate::Path] $path                   = undef,
  (...)
  ) {
    do some stuff
  }

Shortened my ni_trending (node_exporter) & ni_prometheus modules currently look very similar to logrotate:

ni_trending/init.pp

class ni_trending (
  $hostname       = $::fqdn,
  $listen_address = $::ipaddress,
  $listen_port    = 51118,
) { 

) inherits ni_trending::params {

anchor { 'ni_trending::start': }
  ->class { 'ni_trending::package': }
  ->class { 'ni_trending::config':
    (...)
    listen_address => $listen_address,
    listen_port    => $listen_port,
    (...)
    }
  ->class { 'ni_trending::service': }
  ->class { ' ni_trending::prometheus':
    (...)
    hostname     => $hostname,
    listen_port  => $listen_port,
    (...)
    }
    ->anchor { 'ni_trending::end': }
}

ni_trending/prometheus.pp

class ni_trending::prometheus (
  Hash $options        = {},
) {

  ni_prometheus::nodeexporterrule { 'node_exporter' :
    ensure      => pick_default($options['ensure'], 'present'),
    hostname    => pick_default($options['hostname'], $ni_trending::hostname),
    listen_port => pick_default($options['hostname'], $ni_trending::listen_port),
    }
}

ni_prometheus/nodeexporterrules.pp

class ni_prometheus::nodeexporterrules ($rules = $::ni_prometheus::nodeexporterrules) {

  create_resources('ni_prometheus::nodeexporterrule', $nodeexporterrules)

}

ni_prometheus/nodeexporterrule.pp

define ni_prometheus::nodeexporterrule (
  $job_name                         = $title,
  Enum['present','absent'] $ensure  = 'present',
  $hostname                         = $hostname,
  $listen_port                      = $listen_port,
) {

  file_line { "prometheus-${job_name}" :
    path  => "/etc/prometheus/${job_name}.list",
    after => 'hosts:',
    line  => "${hostname}:${listen_port}",
  }
}

But this will just work when I apply the node_exporter locally on the Prometheus Master - not in the case that an external machine has the ni_trending::prometheus class included, which makes sense to me - because it clearly feels that something is missing. :-) How can I get this working?

Thanks!

Yannick Dixken
  • 37
  • 7
  • I'm not quite following what you want to do. It may simply arise from unfamiliarity with Prometheus jargon. How is "*I want to have this step done asynchronously, so for example with a puppet agent run afterwards on the Prometheus Server*" consistent with being dissatisfied that "*this will just work when I apply the `node_exporter` locally on the Prometheus Master*"? – John Bollinger Nov 17 '18 at 12:53
  • Sorry - wording :-) Let's name the VM with the `ni_trending` (the actual `node_exporter`) applied "Publisher" and the Prometheus Server "Consumer". I don't want the (Publisher) module `ni_trending` to do the actual change within the central Prometheus configuration file on the Prometheus Server (Consumer) - instead calling the `ni_prometheus::nodeexporterrule` class with variables to create a new entry in the config file - this `ni_prometheus` class is only applied on the Server (Consumer). – Yannick Dixken Nov 17 '18 at 13:03
  • Another simplified usecase would be to setup a simple Icinga TCP Portcheck on a remote Icinga Server, when you deploy a certain service (Let's take Nginx as an example). In this case you already know all needed informations required to create a check on the serverside additionally to the check on the nginx node => The $::ipaddress out of facter and $http(s)_port out of the module or Hiera. For me it's just not clear how to do this properly. – Yannick Dixken Nov 17 '18 at 13:14
  • Is the configuration file that you want to provide to the Consumer generated on the Publisher node itself? Could it be (or is it already) generated on the Puppet master, on behalf of the Publisher? – John Bollinger Nov 17 '18 at 13:15
  • No, I think that's an overhead because the configuration file on the Consumer does only need to hold the FQDNs & Ports of the publishers. `dev01.invaliddomain.com:51181` - as an example. I already get these two values (and work with them, e.g. modifying a `node_exporter-systemd-service` with environment variables) when I run the node_exporter (Publisher) Class and now would like to set them on the Server (Consumer) as well, by calling another module and class (ni_prometheus). – Yannick Dixken Nov 17 '18 at 13:27
  • *Could it be (or is it already) generated on the Puppet master, on behalf of the Publisher?* Sorry, I did not understand this sentence correctly. If you mean Puppet Master-generated like `content => template('ni_trending/node_exporter-systemd-service')` - then yes. – Yannick Dixken Nov 17 '18 at 13:39

1 Answers1

2

This sounds like a job for exported resources (that makes two in one day!). This is a facility for one node's catalog building to generate resources that can be applied to other nodes (and also, optionally, to the exporting node itself). I'm still not tracking the details of what you want to manage where, so here's a more generic example: maintaining a local hosts file.

Generic example

Suppose we want to automatically manage a hosts file listing all our nodes under management. Puppet has a built-in resource, Host, representing one entry in a hosts file. We make use of that by having every node under management export an appropriate host resource. Something like this would go inside a class included on every node:

@@host { "$hostname": ip => $ipaddress; }

The @@ prefix marks the resource as exported. It is not applied to the current target node, unless by the mechanism I will describe in a moment. the $hostname and $ipaddress are just facts presented by the target node, and they are resolved in that context. Note, too, that the resource title is globally unique: each target node has a different hostname, therefore all the exported Host resources that apply to different target nodes will have distinct titles.

Then, separately, every node that wants all those Host entries applied to it will import them in its own catalog by using an exported resource collector:

<<|Host|>>

The nodes that export those resources can also collect some or all of them. Additionally, there are ways to be more selective about which resources are collected; see the link above.

Community
  • 1
  • 1
John Bollinger
  • 160,171
  • 8
  • 81
  • 157
  • 1
    In general, exported resources always seem to be necessary when using Puppet to manage monitoring software, so unsurprising it fits Prometheus here. I think the example in the documentation is even for Nagios. – Matthew Schuchard Nov 17 '18 at 13:55
  • Yeees, thanks! Already looked into the documentation, matches my requirement! – Yannick Dixken Nov 17 '18 at 13:55