User:Nitin oodd/sidekiq

 Sidekiq  is a Ruby framework for processing background jobs. It is simple to integrate with any modern Rails application and provide much higher performance than other existing solutions like Resque or Delayed Job. Sidekiq scales the web application by moving long-running jobs into the background. It handles multiple jobs using threads.

Introduction
Background jobs are key to building truly scalable web applications as they transfer both time and computationally intensive tasks from the web layer to a background process outside the user request/response lifecycle. This ensures that web requests can always return immediately and reduces compounding performance issues that occur when requests become backlogged. Examples of common tasks that could be done in the background include fetching large chunks of data from remote APIs, reading RSS feeds, resizing images etc.

Here's how a web application could read an RSS feed in the background.

Sidekiq performs the aforementioned tasks thereby processing background jobs.

Installation and Setup
1. Sidekiq can be integrated into the Rails application by adding the sidekiq gem in the gemfile with the following command.

gem 'sidekiq'

2. A worker class that includes the Sidekiq::Worker module and responds to the perform method can be created in app/workers to process jobs asynchronously.

class SampleWorker include Sidekiq::Worker def perform(name, count) # do something end end

3. The job to be performed in the background can be queued to the worker by invoking the perform_async method with any required arguments.

SomeWorker.perform_async('oodd', 4)

The arguments to perform_async should be simple JSON datatypes: string, integer, float, boolean, null, array and hash. The Sidekiq client API uses JSON.dump to send the data to Redis. The Sidekiq server pulls that JSON data from Redis and uses JSON.load to convert the data back into Ruby types to pass to the perform method. Symbols, named parameters or complex Ruby objects (Date, Time, ActiveRecords etc.) should not be passed as arguments as those will not survive the dump/load round trip correctly.

For example, if you pass an object as an argument, it might change in the client by the time sidekiq fetches it from the database. In this case, the sidekiq server won't have the updated object. Instead, it is better to pass the object ID as an argument so that the sidekiq server can fetch the latest object using the ID.

4. To process the background jobs, sidekiq needs to be started from the root directory of the Rails application:

bundle exec sidekiq

Components
There are three components in the Sidekiq framework :


 * 1) Client
 * 2) Redis database
 * 3) Server

Client
The Sidekiq client runs in the web application process (typically a Rails unicorn or passenger process) and allows jobs to be pushed into the background for processing. There are three ways to create a job in the application code:

MyWorker.perform_async(1, 2, 3) SomeClass.delay.some_class_method(1, 2, 3)                       # See Delayed Extensions wiki page Sidekiq::Client.push('class' => MyWorker, 'args' => [1, 2, 3])   # Lower-level generic API Sidekiq::Client.push('class' => 'MyWorker', 'args' => [1, 2, 3]) # Pass as a string.

All three ways create a Hash which represents the job, serializes that Hash to a JSON string and pushes that String into a queue in Redis.

Redis
Redis provides data storage for Sidekiq. It holds all the job data along with runtime and historical data to power Sidekiq's Web UI.

Server
Each Sidekiq server process pulls jobs from the queue in Redis and processes them. Like the web processes, Sidekiq boots Rails so the jobs and workers have the full Rails API, including Active Record, available for use. The server will instantiate the worker and pass the arguments.

Queues
By default, a single queue called "default" is used by sidekiq in Redis. Multiple queues can be used either by specifying them as arguments to the sidekiq command using the -q option, or by setting them in the Sidekiq configuration file. Queues can be also assigned weights to signify priority. The weight of a queue by default is 1. A higher weight implies that higher priority and a weight of say, 5 implies that the new queue will be checked for jobs 5 times more often than the default queue.

Threads
Sidekiq uses threads to handle multiple jobs. To avoid problems with multi-threading, ensure that the worker code and any library the worker uses is thread-safe. The number of concurrent threads can be set using the pool attribute in the database.yml file. Sidekiq uses Celluloid to handle it's multi-threading concurrent environment.

Job scheduling
Instead of using perform_async, jobs can be scheduled to be processed later using perform_in and perform_at.

SomeWorker.perform_in(1.hour, 'oodd', 4) SomeWorker.perform_at(1.hour.from_now, 'oodd', 4) SomeWorker.perform_at(1.day.from_now, 'oodd', 4)

This can be useful in situations like clearing caches.

Exception Handling
Gems can attach to Sidekiq's global error handlers to inform users any time there is an error inside Sidekiq. Error services should all provide integration automatically by including their gem within the application's Gemfile.

Sidekiq reports errors using the following services
 * Airbrake

Sample code

def send_to_airbrake(msg, ex) ::Airbrake.notify(ex, :parameters => msg) end


 * Exceptional

Sample code

def send_to_exceptional(msg, ex) if ::Exceptional::Config.should_send_to_api? ::Exceptional.context(msg) ::Exceptional::Remote.error(::Exceptional::ExceptionData.new(ex)) end end


 * ExceptionNotifier

Sample code

def send_to_exception_notifier(msg, ex) ::ExceptionNotifier::Notifier.background_exception_notification(ex, :data => { :message => msg }) end

Backtrace Logging
Enabling backtrace logging for a job will cause the backtrace to be persisted throughout the lifetime of the job. This can cause Redis memory usage to grow without new jobs being added if a large quantity of jobs are failing repeatedly and being requeued.

Automatic job retry
Sidekiq will retry all jobs assigned to it repeatedly (default value is 25 times, it can be changed). If a bug fix is deployed within that time, the job will get retried and successfully processed. After 25 times, Sidekiq will move that job to the Dead Job queue, assuming that it will need manual intervention to work. The number of retries for a particular worker can be specified by setting the sidekiq_options :retry field in the worker class. Retry can also be disabled.

Checking for new jobs
Sidekiq checks for scheduled jobs every 15 seconds by default. This interval can be modified in the config file:

Sidekiq.configure_server do |config| config.average_scheduled_poll_interval = 15 end

Other gems
 Resque  and  Delayed Job  are other gems that are used for processing background jobs. Resque uses Redis for storage as well but it is single-threaded and hence takes up a lot of memory as compared to Sidekiq. However, the advantage of Resque over Sidekiq is that it does not require thread-safety. Delayed Job uses SQL database for storage. However, it is single-threaded and performance and scalability aren't as great as Sidekiq.