Sidekiq process in production with Systemd and Monit

Recently, we have upgraded our Sidekiq version from 5.2 to 6.5. Before Sidekiq 6.0 we were managing the Sidekiq process directly using Monit. With the release of Sidekiq 6, the team has removed the daemonization, logfile, and pidfile command line arguments and sidekiqctl binary. Managing services manually is more error-prone, let our operating system do it for us. We have three options to go with systemd, upstart, and foreman. We decided to go ahead with the systemd.

Systemd

Systemd is a system and service manager for linux. Systemd tasks are organized as units. Most common units are services(.service), mount points(.mount), devices(.device), sockets(.socket), or timers(.timer)

Systemctl

The systemctl command is a utility which is responsible for examining and controlling the systemd system and service manager.

Sidekiq

Simple, efficient background processing for Ruby.

Sidekiq running as Systemd Service

1. To manage Sidekiq we need to create a service file for Sidekiq which can be used to start, stop or restart the Sidekiq process.

   Sudo nano /lib/systemd/system/sidekiq.service

2. Content in the Sidekiq.service. Sidekiq has provided us with the template for the service file here Sidekiq.service. We modified it according to our use case

   [Unit]
   Description=sidekiq
   After=syslog.target network.target
   
   [Service]
   
   Type=simple
   # If your Sidekiq process locks up, systemd's watchdog will restart it within seconds.
   #WatchdogSec=10
   
   WorkingDirectory=/opt/myapp/current
   
   ExecStart=/usr/local/bin/bundle exec sidekiq -C /opt/myapp/shared/config/sidekiq.yml -e production
   ExecStop=/bin/kill -TSTP $MAINPID
   ExecStartPost=/bin/sh -c '/bin/echo $MAINPID > /opt/myapp/shared/pids/sidekiq.pid'
   ExecStopPost=/bin/sh -c 'rm /opt/myapp/shared/pids/sidekiq.pid'
   
   User=deploy
   Group=deploy
   UMask=0002
   
   # Greatly reduce Ruby memory fragmentation and heap usage
   # https://www.mikeperham.com/2018/04/25/taming-rails-memory-bloat/
   Environment=MALLOC_ARENA_MAX=2
   
   # if we crash, restart
   RestartSec=10
   Restart=on-failure
   
   # output goes to /var/log/syslog (Ubuntu) or /var/log/messages (CentOS)
   StandardOutput=syslog
   StandardError=syslog
   
   # This will default to "bundler" if we don't specify it
   SyslogIdentifier=sidekiq
   
   [Install]
   WantedBy=multi-user.target

3. Our Modified Configurations:
   • As we were using system ruby and using Sidekiq with some custom configurations. To start Sidekiq we used.

     ExecStart=/usr/local/bin/bundle exec sidekiq -C /opt/myapp/shared/config/sidekiq.yml -e production

   • To stop Sidekiq we need to send a TSTP signal to process all the busy jobs before terminating Sidekiq.

     ExecStop=/bin/kill -TSTP $MAINPID

   • For Managing with Monit we need the process id, After starting or stopping the service we were maintaining the process id file.

     ExecStartPost=/bin/sh -c '/bin/echo $MAINPID > /opt/myapp/shared/pids/sidekiq.pid'
     ExecStopPost=/bin/sh -c 'rm /opt/myapp/shared/pids/sidekiq.pid'

   • As we want to use our app user to run this service.

     User=deploy
     Group=deploy
     UMask=0002

   • And we want to restart only when there is a failure.

     # if we crash, restart
     RestartSec=10
     Restart=on-failure

4. Reload the systemctl daemon for the new created service

   Sudo systemctl daemon-reload

5. Now we can start the Sidekiq service:

   sudo systemctl start|stop|restart sidekiq

Monitor Sidekiq process with Monit

Now we have systemd to start, stop and restart the Sidekiq process. Now we will look at how to monitor the Sidekiq process with the help of monit.

Monit

Monit is a utility for managing and monitoring processes, programs, files, directories and filesystems on a Unix system.

1. Modified monitrc

   check process sidekiq with pidfile "/opt/myapp/shared/pids/sidekiq.pid"
     start program = "/bin/bash -l -c  'sudo systemctl start sidekiq' as uid deploy and gid deploy"
       with timeout 20 seconds
     stop program  = "/bin/bash -l -c  'sudo systemctl stop sidekiq' as uid deploy and gid deploy"
       with timeout 20 seconds
     if totalmem is greater than 800 MB for 3 cycles then restart
     if changed pid then exec "/etc/monit/slack_notifier.sh"
     if cpu is greater than 65% for 2 cycles then exec "/etc/monit/slack_notifier.sh" else if succeeded then exec "/etc/monit/slack_notifier.sh"

2. We can check if sidekiq is up and running:

   sudo monit summary sidekiq

Monit will check the Sidekiq process and it will automatically start in case of the unexpected kill of the Sidekiq process.

We have successfully completed the Sidekiq process monitoring with the help of Monit and Systemd.