Mastering Multi Tenant setup with rails - background jobs
Welcome back to the Rails multi-tenant architecture series! If you’re just joining in, be sure to check out Part 1, where you’ll find an introduction to multi-tenancy and a detailed walkthrough on setting up a multi-tenant Rails application.
Quick Recap
In the previous blog post, the focus was on delving into the concept of multi-tenancy in software design, with a specific emphasis on managing separate databases for each tenant. After exploring three types of multi-tenant application architectures, a step-by-step guide was provided for setting up a multi-tenant Rails blog application. This included configuring databases for each tenant, implementing automatic connection switching in Rails 6/7, and using Nginx to run multiple databases simultaneously on different ports.
Introduction
In this blog post, the focus is on background job processing within a multi-tenant Rails environment. Specifically, it addresses the challenges of running background jobs across multiple databases and proposes solutions to ensure seamless execution of jobs.
Sidekiq
First we will setup Sidekiq, A popular background job processing library for Ruby. Here’s a quick guide on how to set it up:
- Add sidekiq( use > 6 version) in Gemfile. Follow This Guide for setup.
- Create a sidekiq job
rails generate sidekiq:job multi_db_testing
# app/sidekiq/multi_db_testing_job.rb
class MultiDbTestingJob < ApplicationJob
def perform
p "Number of articles is #{Article.count}"
end
end
Running up application along with sidekiq
To start both the Rails server and Sidekiq, follow these steps:
- Install foreman gem to start both rails server and sidekiq.
- In Gemfile add
foreman
gem & run bundle install. - Create a Procfile to define the processes:
# procfile
web: bin/rails server --binding=0.0.0.0 --port=3000 --environment=development
sidekiq: bundle exec sidekiq
Triggering Background jobs
- Create a route and controller action to trigger the Sidekiq job:
# config/routes.rb
resources :articles do
collection do
get :run_background_job
end
end
# app/controllers/articles_controller.rb def run_background_job
MultiDbTestingJob.perform_later
redirect_to root_path
end
# app/views/articles/index.html.erb
<%= link_to "Run sidekiq job", run_background_job_articles_path %>
- Start the server using
foreman start
- Navigate to http://localhost:3000, and trigger the job.
- You’ll notice that the job is executed, but it retrieves data only from the default database. why? Continue reading to find out the reason.
Problem?
When a Sidekiq server initializes, it establishes a connection pool to manage database queries. During job execution, it retrieves a connection from this pool. If a specific database is not specified for the job, it defaults to the primary database (default - db 1).
Addressing the Database Connection Issue
To ensure that background jobs access the correct database, we need to pass the database name as a parameter to each job and modify the job accordingly:
# /app/controllers/articles_controller.rb
def run_background_job
MultiDbTestingJob.perform_later(shard_name)
redirect_to root_path
end
# /app/sidekiq/multi_db_testing.rb
class MultiDbTestingJob < ApplicationJob
def perform(shard)
ActiveRecord::Base.connected_to(shard: shard) do
p "Number of articles in DB is #{Article.count}"
end
end
end
Now, you’ll get the desired result for both databases.
However, this approach has its drawbacks:
- For each background job, we need to pass an additional parameter.
- We need to write additional code to connect to the correct database for each background job.
To address these issues, we can create a Sidekiq adapter that will decide which database to connect to based on the database that initiated the background job. But before creating the adapter, we need a global attribute to remember which database we are connected to. To achieve this, Rails CurrentAttributes
and Sidekiq Middleware
will be utilized.
Current Attributes
From the definition of Current Attributes, Abstract super class that provides a thread-isolated attributes singleton, which resets automatically before and after each request. This allows you to keep all the per-request attributes easily available to the whole system.
# app/models/current.rb
class Current < ActiveSupport::CurrentAttributes
attribute :tenant
end
# app/controllers/application_controller.rb
before_action :setup_tenant
def setup_tenant
tenants = Rails.application.config_for(:settings)[:tenants]
current_tenant = tenants.keys.find { |key| tenants[key][:hosts].include?(request.env['HTTP_HOST']) } || :app1_shard
Current.tenant = current_tenant.to_sym
end
Note - Sidekiq also introduced the cattr
feature, this will help in persisting the value of current attributes when sidekiq job runs.
Read More
Sidekiq Middleware
It is a set of customizable modules that intercept and augment the behavior of Sidekiq job processing in Ruby on Rails applications. Sidekiq Middleware
- Create file
config/initializers/sidekiq.rb
and paste following code.
# config/initializers/sidekiq.rb
require 'sidekiq'
require 'sidekiq/web'
require 'sidekiq/middleware/current_attributes'
require_relative '../../app/middleware/sidekiq_adapter'
Sidekiq::CurrentAttributes.persist('Current')
Sidekiq.configure_server do |config|
config.server_middleware do |chain|
chain.add Middleware::SidekiqAdapter
end
end
- Create file
app/middleware/sidekiq_adapter.rb
and paste following code.
module Middleware
class SidekiqAdapter
include Sidekiq::ServerMiddleware
def call(job_instance, job_payload, queue)
shard = current_shard(job_payload)
ApplicationRecord.connected_to(shard: shard, role: :writing) do
yield
end
rescue StandardError => e
p "Error occured #{e}"
end
def current_shard(job_payload)
job_payload.try(:[], 'cattr').try(:[], 'tenant')&.to_sym
end
end
end
- With the middleware in place, we can simplify our Sidekiq job and remove the shard logic from it. The middleware will handle connecting to the correct shard.
# multi_db_testing_job.rb
class MultiDbTestingJob < ApplicationJob
def perform
p "Number of articles in DB is #{Article.count}"
end
end
# /app/controllers/articles_controller.rb
def run_background_job
MultiDbTestingJob.perform_later(shard_name)
redirect_to root_path
end
- Run the project again and subsqeuently run the sidekiq job to test it out.
![](/blog/images/multi-tenant/sidekiqbg.jpg)
- You will notice that with the middleware in place, when executing a background job, it connects to the correct database.
Code - Github Link
Summary
In this blog post, we solved database issue with background job processing in a multi-tenant Rails application. We introduced a custom Sidekiq middleware adapter, that fixes the issue of running background jobs across multiple databases. This approach provides a robust & scalable framework for managing background job execution in complex multi-tenant environments.
![Nikhil Bhatt photo](/blog/images/nikhil.jpg)