Sensu remediation - restart failing monitored process -


we using sensu monitoring of processes on remote servers have installed sensu clients.

is there way initiate restart of monitored process when sensu information failing. have found information online remediation handler

http://thesoftjaguar.com/posts/2015/06/14/sensu-remediation/

http://dev.nuclearrooster.com/2013/07/27/remediation-with-sensu/

but not seems work me, remediation never triggered:

also there 2 ruby scripts should not sure 1 use:

https://github.com/sensu-plugins/sensu-plugins-sensu/blob/master/bin/handler-sensu.rb

https://github.com/nstielau/sensu-community-plugins/blob/remediation/handlers/remediation/sensu.rb

update 20th april:

we use sensu enterprise. in meantime manage remediator.rb script called, not working properly, since can’t read json response client , there following exceptions:

{"timestamp":"2017-04-20t03:06:41.733000-0700","level":"error","message":"handler output","handler":{"command":"/etc/sensu/plugins/remediator.rb","type":"pipe","timeout":10,"severities":["critical","warning","unknown"],"name":"remediator"},"event":{"id":"f38cd413-575a-46f6-8845-09d713a29815"},"output":["/opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/sensu-plugin-1.4.2/lib/sensu-plugin/utils.rb:54:in `[]': no implicit conversion of string integer (typeerror)\n\tfrom /opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/sensu-plugin-1.4.2/lib/sensu-plugin/utils.rb:54:in `block in deep_merge'\n\tfrom /opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/sensu-plugin-1.4.2/lib/sensu-plugin/utils.rb:52:in `each'\n\tfrom /opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/sensu-plugin-1.4.2/lib/sensu-plugin/utils.rb:52:in `deep_merge'\n\tfrom /opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/sensu-plugin-1.4.2/lib/sensu-plugin/utils.rb:22:in `block in settings'\n\tfrom /opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/sensu-plugin-1.4.2/lib/sensu-plugin/utils.rb:22:in `each'\n\tfrom /opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/sensu-plugin-1.4.2/lib/sensu-plugin/utils.rb:22:in `reduce'\n\tfrom /opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/sensu-plugin-1.4.2/lib/sensu-plugin/utils.rb:22:in `settings'\n\tfrom /opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/sensu-plugin-1.4.2/lib/sensu-handler.rb:123:in `api_settings'\n\tfrom /opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/sensu-plugin-1.4.2/lib/sensu-handler.rb:131:in `api_request'\n\tfrom /opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/sensu-plugin-1.4.2/lib/sensu-handler.rb:179:in `stash_exists?'\n\tfrom /opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/sensu-plugin-1.4.2/lib/sensu-handler.rb:191:in `block (2 levels) in filter_silenced'\n\tfrom /opt/sensu/embedded/lib/ruby/2.3.0/timeout.rb:91:in `block in timeout'\n\tfrom /opt/sensu/embedded/lib/ruby/2.3.0/timeout.rb:33:in `block in catch'\n\tfrom /opt/sensu/embedded/lib/ruby/2.3.0/timeout.rb:33:in `catch'\n\tfrom /opt/sensu/embedded/lib/ruby/2.3.0/timeout.rb:33:in `catch'\n\tfrom /opt/sensu/embedded/lib/ruby/2.3.0/timeout.rb:106:in `timeout'\n\tfrom /opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/sensu-plugin-1.4.2/lib/sensu-handler.rb:190:in `block in filter_silenced'\n\tfrom /opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/sensu-plugin-1.4.2/lib/sensu-handler.rb:188:in `each'\n\tfrom /opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/sensu-plugin-1.4.2/lib/sensu-handler.rb:188:in `filter_silenced'\n\tfrom /opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/sensu-plugin-1.4.2/lib/sensu-handler.rb:36:in `filter'\n\tfrom /opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/sensu-plugin-1.4.2/lib/sensu-handler.rb:80:in `block in <class:handler>'\nwarning: event filtering in sensu-plugin deprecated, see http:// bit.ly/sensu-plugin\n"]}

we used following script remediator.rb https://github.com/sensu-plugins/sensu-plugins-sensu/blob/master/bin/handler-sensu.rb

check performed , response sensu client server, looks remediator.rb failing process it.

here configuration:

remediator.json {   "handlers": {     "remediator": {       "command": "/etc/sensu/plugins/remediator.rb",       "type": "pipe",       "timeout": 10,       "severities": ["critical", "warning", "unknown"]     }   } }

made simple possible check purpose:

/etc/sensu/conf.d/checks {   "checks": {     "seyren_check": {       "command": "/opt/sensu/embedded/bin/ruby /etc/sensu/plugins/check-procs.rb -p unexisent_process",       "interval": 30,       "subscribers": ["trep"],       "handlers": ["remediator","default","file"],       "occurrences": 1,       "refresh": 10,       "remediation": {         "first_remediation": {           "occurrences": [1, 2],           "severities": [1]         },         "medium_remediation": {           "occurrences": ["3-10"],           "severities": [1]         },         "heavy_remediation": {           "occurrences": ["1+"],           "severities": [2]         }       }     },     "first_remediation": {       "command": "touch /etc/sensu/plugins/test_lr",       "subscribers": ["my.machine.local"],       "handlers": ["default"],       "interval": 10,       "publish": false     },     "medium_remediation": {       "command": "touch /etc/sensu/plugins/test_mr",       "subscribers": ["my.machine.local"],       "handlers": ["default"],       "interval": 10,       "publish": false     },     "heavy_remediation": {       "command": "touch /etc/sensu/plugins/test_hr",       "subscribers": ["my.machine.local"],       "handlers": ["default"],       "interval": 10,       "publish": false     }   } }

  • we prefer here use cm tool called ansible. here handler , basic idea:

#

 #! /usr/bin/env ruby  require 'sensu-handler'     require 'json'      class ansible < sensu::handler   def handle     ansible = settings['ansible']['command'] || 'ansible-playbook'     playbook = settings['ansible']['playbook'] || nil     extra_vars = json.generate(@event)      unless @event['check']['ansible'].nil?       playbook = @event['check']['ansible']['playbook'] || playbook     end      command = ansible.to_s playbook.to_s     output = `#{command}`      if $?.exitstatus > 0       puts output       exit 1     else       puts "success: #{command}"     end   end end

  • then create, handler config:

    { "handlers": { "ansible": { "type": "pipe", "command": "/etc/sensu/handlers/handler-ansible.rb" } } }

and settings remediation playbook:

cat conf.d/handler/config_ansible.json  {     "ansible": {         "command": "/etc/sensu/scripts/provision",         "playbook": " --tags checksum"     } }

  • and in check configuration, add name of remediation handler:

    { ...

    "handlers": ["email","ansible", "logstash"] .... }

  • where command here:

    ssh root@my_ansible_server.comp.com -o batchmode=yes -o "stricthostkeychecking no" -o connecttimeout=10 ansible-playbook -i /etc/ansible/generic.hosts /etc/ansible/remediation.yaml ${@}


Comments

Post a Comment

Popular posts from this blog

commonjs - How to write a typescript definition file for a node module that exports a function? -

consider, toml node module can use: // toml.d.ts declare module toml { export function parse(value:string):any; } declare module "toml" { export = toml; } then: /// <reference path="../../../../../defs/toml/toml.d.ts"/> import toml = require('toml'); toml.parse(...); however, node module export single function, such 'glob' ( https://github.com/isaacs/node-glob ). the node usage of module be: var glob = require("glob") glob("*.js", {}, callback(err, files) { ... }); you'd naively expect this: // glob.d.ts declare function globs(paths:string, options:any, callback:{(err:any, files:string[]):void; ...but because typescripts 'import' semantics bit strange, seems can use 'import .. = require()' statement alias modules . trying call: /// <reference path="../../../../../defs/glob/glob.d.ts"/> import blog = require('glob'); results in: error ts2072: mod...
Read more

openid - Okta: Failed to get authorization code through API call -

i'm integrating okta own idp server using okta's api. i'm implementing authorization code flow following steps below: in own server, use /api/v1/authn endpoint sessiontoken. use sessiontoken obtain authorization calling endpoint: /oauth2/v1/authorize?client_id=" + clientid + "&sessiontoken=" + sessiontoken + "&response_type=code&response_mode=query&scope=openid&redirect_uri=" + redirecturl + "&state=evanyang&nonce=" it's supposed return response status code 302 , location header containing redirect url code value. however, keep getting response status code 200 , without location header, html body saying "you using unsupported browser." , "javascript disabled on browser." according api documentation: http://developer.okta.com/docs/api/resources/oidc.html#authentication-request , sessiontoken parameter sufficient this: an okta one-time sessiontoken. allows api-based ...
Read more

ios - Change Storyboard View using Seague -

i have following code checks connection server, , depending on want change views if no connection found. nsstring *connect = [nsstring stringwithcontentsofurl:[nsurl urlwithstring:@"http://myserver.com"] encoding:nsutf8stringencoding error:nil]; if (connect == null) { [self performseguewithidentifier:@"network_failure" sender:self]; } else { // other code here } my problem not change views. have added storyboard segue between 2 views identifier of network_failure . not have navigation controller or that, problem? thanks help! no not problem. first of all, in objectivec exists nil object oriented null . must use that. so: if(connect == nil) or better: if(!connect) so due fact nil not same of null , probably code does't enter in if scope . you have been able discover breakpoint. anyway: be sure set trigger segue (line) 2 view controller , , name network_failure (that not best name. should instead view1toview2 ) as, m...
Read more