4

The problem: Servers running puppet agent in my environment are receiving empty [classes] (without parameters), instead of the expected parameters stored in their Hiera document. This causes puppet modules to run with null parameters which in turn causes them to successfully run with default values instead of the actual expected values (which is obviously unwanted and destructive behavior).

What causes the problem to trigger? Our Hiera are based on CouchDB document database (as I will further expand on later). When the CouchDB service is down the puppet agents (upon asking the puppet-masters for a new catalog) receive empty [classes] (without parameters), instead of the expected parameters stored in their Hiera document.

My environment architecture:

  • 4 Puppet master servers under NetworkLoadBalancer (Cisco Ace)
  • 1 puppet ca server
  • 2 Hiera servers (couchDB 1.6.0)behind NetworkLoadBalancer (Cisco Ace)
  • all the servers OS are RedHat 6.3
  • Puppet version 3.7.4
  • Puppet masters communicate with hiera servers with Http_Backend v1.0.1
  • Using puppetDB with postgress sql to save servers inventory

How are we able to simulate the problem?

  • By stopping the CouchDB service on the Hiera server - hiera01 (just one of the two) we are able to trigger the problem.

  • puppet-masters logs show "connection refused ..." error for the sessions that were already open to the hiera01 server for more than 20 minutes

  • sessions not being closed when the couchdb services is stopped.

  • new requests are routed to hiera02.

  • client servers that their session to get catalog from master through hiera01 - got default params for their classes !!!

Puppet master`s main configuration file

[main]
logdir = /var/log/puppet
rundir = /var/run/puppet
ssldir = %vardir/ssl {group = service, mode = 640}
ca = false
certname = master_server_01.domain
dns_alt_names = puppet-master-ace.domain, puppet-master-ace
use_srv_records = true
pluginsource = puppet:///plugins
pluginfactsource = puppet:///pluginfacts
reports = log, foreman
enviromentpath = $confdir/enviroments
basemodulepath = $confdir/modules

[agent]
classfile = $vardir/classes.txt
localconfig = $vardir/localconfig

[master]
storeconfigs = true
storeconfigs_backend = puppetdb
always_cache_features = true

Hiera.yaml (master server)

---
:backends:
 - http
 - yaml
:hierarchy:
 - "%{fqdn}"
 - "%{enviroment}"
 - common
:http:
 :host: hieraserverace.domain
 :use_auth: true
 :auth_user: admin
 :auth_pass: Passowrd
 :api_user: apiUser
 :api_pass: apipassword
 :merge_behavior: deeper
 :port: 5984
 :output: json
 :failure: graceful
 :path:
  - "/%{environment}/%{fqdn}"
  - "/%{environment}/%{osfamily}"
  - "/%{environment}/%{enviroment}"
  - "/%{environment}/common"

:yaml:
 :datadir: /etc/puppet/hieradata

Notes

  • environment is productions and updating versions is almost impossible.
  • hieradata directory is empty (not using the yaml backend)

Thank you !

PS: Since we are running a classified environment disconnected from the internet uploading log files is a very complicated process.

Oz Bar-Shalom
  • 1,747
  • 1
  • 18
  • 33
  • 2
    Any reason you have `failure: graceful`? Apparently, this will make the hiera http backend ignore http errors: https://github.com/crayfishx/lookup_http/blob/master/lib/lookup_http.rb#L83 . Call [here](https://github.com/crayfishx/hiera-http/blob/master/lib/hiera/backend/http_backend.rb#L84) – Artefacto Feb 23 '16 at 14:28
  • There is no reason for `failure: graceful` parameter. After deleting this parameter, puppet agents raised an exeption `bad http response` and did not run successfuly even with both Hiera services (01 & 02) running. Adding another parameter ` :ignore_404: true` (without the failure parameter) fixed puppet agents exeptions. But still when one of the Hiera services stops (Hiera01), The sessions are not being closed as expected ( the expected behavior is closing the session and trying again to Hiera02). – Oz Bar-Shalom Feb 25 '16 at 09:20

0 Answers0