部署 使用 Monit+Mina 监控服务器

rubyu2 · 2014年12月13日 · 最后由 ericguo 回复于 2016年06月08日 · 11677 次阅读
本帖已被管理员设置为精华贴

服务器不上个监控总是没有安全感,偶尔出点问题,追踪起原因也不是很方便。经过在monitgodeye之间选择,最终选择了 monit,最主要的优点就是占用内存很小,而且和 ruby 无关,所以也不用考虑 root 用户下 ruby 环境的问题。
monit的安装和简单使用就不做简单的介绍,大家可以在官网上查看。需要注意的就是建议使用 root 用户来安装 monit,这样就比较方便的使用 monit 启动其他用户下的服务,不用考虑权限问题。
mina,部署利器,具体使用方法可以到官网查看,不再介绍。
这里主要分享,使用 monit+mina 对服务器进行监控的一些监控和部署脚本,以及踩过的坑。
部分代码如下:
config/deploy.rb

Dir[File.dirname(__FILE__) + '/mina/*.rb'].each {|file| require file }

environments = {
  'production1' => {
    domain: '115.115.115.115',
    branch: 'master',
    env: 'production1',
    path: "/var/app/wedding",
    user: "root",
    user_home: "/root",
    rvm_home: "/usr/local/rvm",
    thin_ports: 3000..3003
  },
  'production2' => {
    domain: '222.222.222.222',
    branch: 'master',
    env: 'production2',
    path: "/var/app/wedding",
    user: "cloud",
    user_home: "/home/cloud",
    rvm_home: "/usr/local/rvm",
    thin_ports: 2000..2007
  }
}

rails_env = environments.keys.include?(ENV['RAILS_ENV']) ? ENV['RAILS_ENV'] : 'develop'

env_attrs = environments[rails_env]
branch = env_attrs[:branch]
domain = env_attrs[:domain]
rvm_home = env_attrs[:rvm_home]
rvm_path = rvm_home + "/bin/rvm"
env = env_attrs[:env]
path = env_attrs[:path]
user = env_attrs[:user]
rvm = "ruby-2.1.2"
thin_ports = env_attrs[:thin_ports]
user_home = env_attrs[:user_home]

set :rails_env, env
set :domain, domain
set :deploy_to, path
set :repository, '[email protected]:huoshaoyun/wedding-memo-rails.git'
set :branch, branch
set :ssh_options, '-A'
set :port, '29168' if rails_env == 'production2'
set :user, user
set :term_mode, :nil
set :rvm_path, rvm_path

## script templates
## 根据template文件生成脚本,需要设置必要的变量
set :rvm_home, rvm_home
set :user_home, user_home
set :rvm, rvm
set :path, env_attrs[:path]
set :thin_ports, thin_ports

config/templates/monit/monitrc.erb

#这里是monitrc的配置,省略1000行代码。
include /etc/monit/conf.d/*

config/templates/monit/mysql.erb
记得加上as uid user and gid usergroup,这样就可以以非 root 用户启动服务。

check process mysqld with pidfile /var/run/mysqld/mysqld.pid
        group mysql
        start program = "/etc/init.d/mysql start" as uid <%= user %> and gid <%= user %>
        stop program = "/etc/init.d/mysql stop" as uid <%= user %> and gid <%= user %>
        if failed host 127.0.0.1 port 3306 then restart
        if 5 restarts within 5 cycles then timeout

config/templates/monit/nginx.erb

check process nginx with pidfile /run/nginx.pid
        start program = "/etc/init.d/nginx start" as uid <%= user %> and gid <%= user %>
        stop  program = "/etc/init.d/nginx stop" as uid <%= user %> and gid <%= user %>

config/templates/monit/redis.erb

# redis
check process redis with pidfile <%= user_home %>/pids/redis.pid
        start = "/bin/bash -c '/bin/sh <%= user_home %>/mysh/startRedis.sh'" as uid <%= user %> and gid <%= user %>
        stop program = "/etc/init.d/redis-server stop" as uid <%= user %> and gid <%= user %>
        group redis

config/templates/monit/sidekiq.erb

check process sidekiq
        with pidfile /var/app/wedding/shared/pids/sidekiq.pid
        start program = "/etc/init.d/sidekiq start" with timeout 90 seconds
        stop program = "/etc/init.d/sidekiq stop" with timeout 90 seconds
        group sidekiq

config/templates/monit/thin.erb

<% thin_ports.each do |thin_port| %>
check process thin-<%= thin_port %> with pidfile /var/app/wedding/tmp/pids/thin.<%= thin_port %>.pid
        start = "/bin/bash -c '/bin/sh <%= user_home %>/mysh/thin.<%= thin_port %>.sh start'" as uid <%= user %> and gid <%= user %>
        stop = "/bin/bash -c '/bin/sh <%= user_home %>/mysh/thin.<%= thin_port %>.sh stop'"  as uid <%= user %> and gid <%= user %>

        if 3 restarts within 5 cycles then timeout
        if failed port <%= thin_port %> protocol http with timeout 30 seconds for 2 cycles then restart
        group thin
<% end %>

config/templates/mysh/redis-server.sh.erb

#!/bin/sh

# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY 
# EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES 
# OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT 
# SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, 
# INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED 
# TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR 
# BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN 
# CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN 
# ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH  
# DAMAGE. 

# 存在bug不能正常start,只能stop

### BEGIN INIT INFO
# Provides:             redis-server
# Required-Start:       $syslog
# Required-Stop:        $syslog
# Should-Start:         $local_fs
# Should-Stop:          $local_fs
# Default-Start:        2 3 4 5
# Default-Stop:         0 1 6
# Short-Description:    redis-server - Persistent key-value db
# Description:          redis-server - Persistent key-value db
### END INIT INFO


PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin
DAEMON=/usr/local/bin/redis-server
DAEMON_ARGS=/etc/redis.conf
NAME=redis-server
DESC=redis-server
PIDFILE=<%= user_home %>/pids/redis.pid

test -x $DAEMON || exit 0
test -x $DAEMONBOOTSTRAP || exit 0

set -e

case "$1" in
  start)
        echo -n "Starting $DESC: "
        touch $PIDFILE
        chown <%= user %>:<%= user %> $PIDFILE
        if start-stop-daemon --start --quiet --umask 007 --pidfile $PIDFILE --chuid <%= user %>:<%= user %> --exec $DAEMON -- $DAEMON_ARGS
        then
                echo "$NAME."
        else
                echo "failed"
        fi
        ;;
  stop)
        echo -n "Stopping $DESC: "
        if start-stop-daemon --stop --retry 10 --quiet --oknodo --pidfile $PIDFILE --exec $DAEMON
        then
                echo "$NAME."
        else
                echo "failed"
        fi
        rm -f $PIDFILE
        ;;

  restart|force-reload)
        ${0} stop
        ${0} start
        ;;
  *)
        echo "Usage: /etc/init.d/$NAME {start|stop|restart|force-reload}" >&2
        exit 1
        ;;
esac

exit 0

这里存在一个问题不能通过这个脚本来启动 redis,只能停止 redis,所以又单独写了一个 startRedis。
config/templates/mysh/startRedis.sh.erb

#!/bin/sh
cd $HOME && /usr/local/bin/redis-server $HOME/redis-2.8.16/redis.conf 

config/templates/mysh/thin.sh.erb
需要 export ruby 环境,否则 root 用户下没有办法来启动 thin,而且export PATH="${PATH}:<%= rvm_home %>/gems/<%= rvm %>/bin:/usr/bin:<%= rvm_home %>/bin:<%= rvm_home %>/gems/<%= rvm %>@global/bin:<%= rvm_home %>/rubies/<%= rvm %>/bin/"要将${PATH}放到前面,这样就不会被系统默认的 ruby 环境覆盖。

#!/bin/bash

# Set the environment, as required by Monit
export PATH="${PATH}:<%= rvm_home %>/gems/<%= rvm %>/bin:/usr/bin:<%= rvm_home %>/bin:<%= rvm_home %>/gems/<%= rvm %>@global/bin:<%= rvm_home %>/rubies/<%= rvm %>/bin/"
export GEM_PATH="<%= rvm_home %>/gems/<%= rvm %>:<%= rvm_home %>/gems/<%= rvm %>@global"
export GEM_HOME="<%= rvm_home %>/gems/<%= rvm %>"

start () {
  cd <%= path %>/current
  BUNDLE_GEMFILE=<%= path %>/current/Gemfile bundle exec thin -C /etc/thin/wedding.<%= thin_port %>.yml -d start
}

stop () {
  kill -s QUIT $(cat <%= deploy_to %>/tmp/pids/thin.<%= thin_port %>.pid)
}

case $1 in
  start)
    start
  ;;
  stop)
    stop
  ;;
  *)
  echo $"Usage: $0 {start|stop}"
  exit 1
  ;;
esac

exit 0

config/templates/thin.yml.erb

---
chdir: /var/app/wedding/current
environment: <%= rails_env %>
address: 0.0.0.0
port: <%= thin_port %>
timeout: 30
log: /var/app/wedding/shared/log/thin.log
pid: /var/app/wedding/tmp/pids/thin.pid
max_conns: 1024
max_persistent_conns: 100
require: []
wait: 30
servers: <%= servers_number %>
daemonize: true

config/mina/base.rb

require 'erb'

def template(from, to)
  queue %{echo "-----> Put #{from} file to #{to}"}
  erb = File.read(File.expand_path("../../templates/#{from}", __FILE__))
  put ERB.new(erb).result(binding), to
end

def put(content, file)
  queue %[echo #{escape content} > "#{file}"]
end

def escape(str)
  Shellwords.escape(str)
end

config/mina/mysh.rb

namespace :mysh do

  task :setup do
    invoke :'mysh:thin'
    invoke :'mysh:sidekiq'
    invoke :'mysh:redis-server' if rails_env == 'production2'
    %w[startRedis.sh].each do |name|
      destination = "$HOME/mysh/#{name}"
      template "mysh/#{name}.erb", destination
      queue "chmod a+x #{destination}"
    end
  end

  task :thin do
    thin_ports.each do |thin_port|
      set :thin_port, thin_port
      destination = "$HOME/mysh/wedding.#{thin_port}.sh"
      template "mysh/thin.sh.erb", destination
      queue "chmod a+x #{destination}"
    end
  end

  task :sidekiq do
    destination = "$HOME/mysh/sidekiq.sh"
    template "mysh/sidekiq.sh.erb", destination
    queue "chmod a+x #{destination}"
    queue "sudo cp #{destination} /etc/init.d/sidekiq"
  end

  task "redis-server" do
    destination = "$HOME/mysh/redis-server.sh"
    template "mysh/redis-server.sh.erb", destination
    queue "chmod a+x #{destination}"
    queue "sudo cp #{destination} /etc/init.d/redis-server"
  end

end

通过mina mysh:setup来生成服务器相关的启动脚本。

config/mina/thin.rb

namespace :thin do

  task :setup do
    thin_ports.each do |thin_port|
      destination = "/etc/thin/wedding.#{thin_port}.yml"
      set :thin_port, thin_port
      set :servers_number, 1
      template "thin.yml.erb", destination
    end
    invoke :'thin:railsapp'
  end

  task "railsapp" do
    destination = "/etc/thin/railsapp.yml"
    set :thin_port, thin_ports.first
    set :servers_number, thin_ports.size
    template "thin.yml.erb", destination
  end

end

通过mina thin:setup来生成 thin 的配置文件。 config/mina/monit.rb

namespace :monit do

  desc "Setup all Monit configuration"
  task :setup do
    queue %{echo "-----> Setting up Monit..."}
    monit_config "monitrc", "/etc/monit/monitrc"
    monit_config "monitrc", "/etc/monitrc"
    invoke :'monit:mysql'
    invoke :'monit:redis' if rails_env == 'production2'
    invoke :'monit:nginx'
    invoke :'monit:thin'
    invoke :'monit:sidekiq'
  end
  after "deploy:setup", "monit:setup"

  task(:nginx) { monit_config "nginx" }
  task(:mysql) { monit_config "mysql" }
  task(:redis) { monit_config "redis" }
  task(:thin) { monit_config "thin" }
  task(:sidekiq) { monit_config "sidekiq" }

  %w[start stop syntax reload].each do |command|
    desc "Run Monit #{command} script"
    task command do
      queue %{echo "-----> Monit #{command}"}
      queue "sudo service monit #{command}"
    end
  end

  task :status do
    queue "sudo monit status"
  end

end

def monit_config(name, destination = nil)
  destination ||= "/etc/monit/conf.d/#{name}"
  template "monit/#{name}.erb", "$HOME/monit/#{name}"
  queue "sudo cp $HOME/monit/#{name} #{destination}"
  queue "sudo chown root #{destination}"
  queue "sudo chmod 600 #{destination}"
end

通过mina monit:setup来生成 monit 的配置文件。这里需要注意,一定要将 monit 的一些配置文件的 owner 修改为 root,即和 monit 安装用户是同一用户,否则是无法启动的。另外需要注意的是,需要考虑各项服务的 pid 文件,某些 service 是默认关闭 pid 文件的,需要修改相应的配置文件打开。 将配置文件和脚本模版化,可以在配置完一台服务器后,非常迅速的配置完成其他服务器。

一切搞定,这样就可以随时查看服务器的状态,来点效果图。 试着 kill 掉一个 thin

😃

monit 好像是趋势哦,毕竟做监控更专业一点。就是使用比 eye 还是略麻烦一点。

#2 楼 @as181920 还没有用过 eye。monit 比较明显的一个优点就是占用内存很少,我们的服务器才占 1.8M 内存,可以忽略不计。

monit +1

#3 楼 @rubyu2 eye 资源消耗(cpu)挺多的。

如果自动部署用 cap,其中用 monit 来重启?那有一些操作需要 root 权限来进行?

#5 楼 @as181920 cap 部署的时候,正常重启一些服务就好了,不用 monit 的。monit 只是做监控用的,如果某些服务意外崩溃 monit 就会帮你来启动。需要 root 权限的很多啊,比如 nginx。

railscast 有一期讲的 monit 与 cap 的也不错 👏

mina 还有 generator 的功能啊,不错~

过去我是用 Rails 的 Generator 实现的动态生成配置 https://github.com/jasl/a_rails_start_up_omakase/tree/master/lib/generators/conf/templates

monit 有个问题是调试命令会比较坑爹,比方说默认执行用的 shell 是 sh,然后有可能因为环境变量的原因在不知情的情况下脚本会失败,然后也不会留下任何日志做记录 用https://github.com/jasl/a_rails_start_up_omakase/blob/master/script/debug_monit 包装命令就可以啦~

学习了

原来可以用 as user 的方式切换 uid,学习了。

怎么发提问帖子啊????

#11 楼 @zhulin0504 上传一个头像。 #6 楼 @rubyu2 有时候比如 sidekiq,部署启动时正确的,但是没启动起来,这时候 monit 就有作用了。

需要 登录 后方可回复, 如果你还没有账号请 注册新账号