Sensu remediation - restart failing monitored process -

we using sensu monitoring of processes on remote servers have installed sensu clients.

is there way initiate restart of monitored process when sensu information failing. have found information online remediation handler

http://thesoftjaguar.com/posts/2015/06/14/sensu-remediation/

http://dev.nuclearrooster.com/2013/07/27/remediation-with-sensu/

but not seems work me, remediation never triggered:

also there 2 ruby scripts should not sure 1 use:

https://github.com/sensu-plugins/sensu-plugins-sensu/blob/master/bin/handler-sensu.rb

https://github.com/nstielau/sensu-community-plugins/blob/remediation/handlers/remediation/sensu.rb

update 20th april:

we use sensu enterprise. in meantime manage remediator.rb script called, not working properly, since can’t read json response client , there following exceptions:

{"timestamp":"2017-04-20t03:06:41.733000-0700","level":"error","message":"handler output","handler":{"command":"/etc/sensu/plugins/remediator.rb","type":"pipe","timeout":10,"severities":["critical","warning","unknown"],"name":"remediator"},"event":{"id":"f38cd413-575a-46f6-8845-09d713a29815"},"output":["/opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/sensu-plugin-1.4.2/lib/sensu-plugin/utils.rb:54:in `[]': no implicit conversion of string integer (typeerror)\n\tfrom /opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/sensu-plugin-1.4.2/lib/sensu-plugin/utils.rb:54:in `block in deep_merge'\n\tfrom /opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/sensu-plugin-1.4.2/lib/sensu-plugin/utils.rb:52:in `each'\n\tfrom /opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/sensu-plugin-1.4.2/lib/sensu-plugin/utils.rb:52:in `deep_merge'\n\tfrom /opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/sensu-plugin-1.4.2/lib/sensu-plugin/utils.rb:22:in `block in settings'\n\tfrom /opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/sensu-plugin-1.4.2/lib/sensu-plugin/utils.rb:22:in `each'\n\tfrom /opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/sensu-plugin-1.4.2/lib/sensu-plugin/utils.rb:22:in `reduce'\n\tfrom /opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/sensu-plugin-1.4.2/lib/sensu-plugin/utils.rb:22:in `settings'\n\tfrom /opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/sensu-plugin-1.4.2/lib/sensu-handler.rb:123:in `api_settings'\n\tfrom /opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/sensu-plugin-1.4.2/lib/sensu-handler.rb:131:in `api_request'\n\tfrom /opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/sensu-plugin-1.4.2/lib/sensu-handler.rb:179:in `stash_exists?'\n\tfrom /opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/sensu-plugin-1.4.2/lib/sensu-handler.rb:191:in `block (2 levels) in filter_silenced'\n\tfrom /opt/sensu/embedded/lib/ruby/2.3.0/timeout.rb:91:in `block in timeout'\n\tfrom /opt/sensu/embedded/lib/ruby/2.3.0/timeout.rb:33:in `block in catch'\n\tfrom /opt/sensu/embedded/lib/ruby/2.3.0/timeout.rb:33:in `catch'\n\tfrom /opt/sensu/embedded/lib/ruby/2.3.0/timeout.rb:33:in `catch'\n\tfrom /opt/sensu/embedded/lib/ruby/2.3.0/timeout.rb:106:in `timeout'\n\tfrom /opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/sensu-plugin-1.4.2/lib/sensu-handler.rb:190:in `block in filter_silenced'\n\tfrom /opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/sensu-plugin-1.4.2/lib/sensu-handler.rb:188:in `each'\n\tfrom /opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/sensu-plugin-1.4.2/lib/sensu-handler.rb:188:in `filter_silenced'\n\tfrom /opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/sensu-plugin-1.4.2/lib/sensu-handler.rb:36:in `filter'\n\tfrom /opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/sensu-plugin-1.4.2/lib/sensu-handler.rb:80:in `block in <class:handler>'\nwarning: event filtering in sensu-plugin deprecated, see http:// bit.ly/sensu-plugin\n"]} 

we used following script remediator.rb https://github.com/sensu-plugins/sensu-plugins-sensu/blob/master/bin/handler-sensu.rb

check performed , response sensu client server, looks remediator.rb failing process it.

here configuration:

remediator.json {   "handlers": {     "remediator": {       "command": "/etc/sensu/plugins/remediator.rb",       "type": "pipe",       "timeout": 10,       "severities": ["critical", "warning", "unknown"]     }   } } 

made simple possible check purpose:

/etc/sensu/conf.d/checks {   "checks": {     "seyren_check": {       "command": "/opt/sensu/embedded/bin/ruby /etc/sensu/plugins/check-procs.rb -p unexisent_process",       "interval": 30,       "subscribers": ["trep"],       "handlers": ["remediator","default","file"],       "occurrences": 1,       "refresh": 10,       "remediation": {         "first_remediation": {           "occurrences": [1, 2],           "severities": [1]         },         "medium_remediation": {           "occurrences": ["3-10"],           "severities": [1]         },         "heavy_remediation": {           "occurrences": ["1+"],           "severities": [2]         }       }     },     "first_remediation": {       "command": "touch /etc/sensu/plugins/test_lr",       "subscribers": ["my.machine.local"],       "handlers": ["default"],       "interval": 10,       "publish": false     },     "medium_remediation": {       "command": "touch /etc/sensu/plugins/test_mr",       "subscribers": ["my.machine.local"],       "handlers": ["default"],       "interval": 10,       "publish": false     },     "heavy_remediation": {       "command": "touch /etc/sensu/plugins/test_hr",       "subscribers": ["my.machine.local"],       "handlers": ["default"],       "interval": 10,       "publish": false     }   } } 

• we prefer here use cm tool called ansible. here handler , basic idea:

#

 #! /usr/bin/env ruby  require 'sensu-handler'     require 'json'      class ansible < sensu::handler   def handle     ansible = settings['ansible']['command'] || 'ansible-playbook'     playbook = settings['ansible']['playbook'] || nil     extra_vars = json.generate(@event)      unless @event['check']['ansible'].nil?       playbook = @event['check']['ansible']['playbook'] || playbook     end      command = ansible.to_s playbook.to_s     output = `#{command}`      if $?.exitstatus > 0       puts output       exit 1     else       puts "success: #{command}"     end   end end 

• then create, handler config:

  { "handlers": { "ansible": { "type": "pipe", "command": "/etc/sensu/handlers/handler-ansible.rb" } } }

and settings remediation playbook:

cat conf.d/handler/config_ansible.json  {     "ansible": {         "command": "/etc/sensu/scripts/provision",         "playbook": " --tags checksum"     } } 

• and in check configuration, add name of remediation handler:

  { ... 

  "handlers": ["email","ansible", "logstash"] .... }

• where command here:

  ssh root@my_ansible_server.comp.com -o batchmode=yes -o "stricthostkeychecking no" -o connecttimeout=10 ansible-playbook -i /etc/ansible/generic.hosts /etc/ansible/remediation.yaml ${@}