gem-hunter
Language: Ruby
Multithreaded, Postgres-based, Active Job backend for Ruby on Rails.
GoodJob is a multithreaded, Postgres-based, Active Job backend for Ruby on Rails.
Inspired by Delayed::Job and Que, GoodJob is designed for maximum compatibility with Ruby on Rails, Active Job, and Postgres to be simple and performant for most workloads.
schema.rb, and LISTEN/NOTIFY to reduce queuing latency.For more of the story of GoodJob, read the introductory blog post.
📊 Comparison of GoodJob with other job queue backends (click to expand)| Queues, priority, retries | Database | Concurrency | Reliability/Integrity | Latency | |
|---|---|---|---|---|---|
| GoodJob | ✅ Yes | ✅ Postgres | ✅ Multithreaded | ✅ ACID, Advisory Locks | ✅ Postgres LISTEN/NOTIFY |
| Solid Queue | ✅ Yes | ✅ Postgres and other databases ✨ | 🔶 Multithreaded in forked process | ✅ ACID, Advisory Locks | 🔶 Polling |
| Que | ✅ Yes | 🔶️ Postgres, requires structure.sql |
✅ Multithreaded | ✅ ACID, Advisory Locks | ✅ Postgres LISTEN/NOTIFY |
| Delayed Job | ✅ Yes | ✅ Postgres | 🔴 Single-threaded | ✅ ACID, record-based | 🔶 Polling |
| Sidekiq | ✅ Yes | 🔴 Redis | ✅ Multithreaded | 🔴 Crashes lose jobs | ✅ Redis BRPOP |
| Sidekiq Pro | ✅ Yes | 🔴 Redis | ✅ Multithreaded | ✅ Redis RPOPLPUSH | ✅ Redis RPOPLPUSH |
Add good_job to your application's Gemfile and install the gem:
bundle add good_job
Run the GoodJob install generator. This will generate a database migration to create a table for GoodJob's job records:
bin/rails g good_job:install
Run the migration:
bin/rails db:migrate
Optional: If using Rails' multiple databases with the migrations_paths configuration option, use the --database option:
bin/rails g good_job:install --database animals bin/rails db:migrate:animals
Configure the Active Job adapter:
# config/application.rb or config/environments/{RAILS_ENV}.rb config.active_job.queue_adapter = :good_job
Inside of your application, queue your job 🎉:
YourJob.perform_later
GoodJob supports all Active Job features:
YourJob.set(queue: :some_queue, wait: 5.minutes, priority: 10).perform_later
In Rails' development environment, by default, GoodJob's Adapter executes jobs async in a background thread pool in rails server.
Because of Rails deferred autoloading, jobs enqueued via the rails console may not begin executing on a separate server process until the Rails application is fully initialized by loading a web page once. To force early initialization (so pre-existing jobs run immediately on boot), add the following to an initializer:
# config/initializers/good_job.rb Rails.configuration.after_initialize do ActiveJob::Base && ActiveRecord::Base end
Remember, only Active Job's perform_later sends jobs to the queue adapter; Active Job's perform_now executes the job immediately and does not invoke the queue adapter. GoodJob is not involved in perform_now jobs.
In Rails' test environment, by default, GoodJob's Adapter executes jobs inline immediately in the current thread.
GoodJob.perform_inline using a tool like Timecop or ActiveSupport::Testing::TimeHelpers.assert_enqueued_with), may override GoodJob's Adapter in some configurations.In Rails' production environment, by default, GoodJob's Adapter enqueues jobs in external mode to be executed by a separate execution process:
By default, GoodJob separates job enqueuing from job execution so that jobs can be scaled independently of the web server. Use the GoodJob command-line tool to execute jobs:
bundle exec good_job startIdeally the command-line tool should be run on a separate machine or container from the web process. For example, on Heroku:
web: rails server worker: bundle exec good_job start
The command-line tool supports a variety of options, see the reference below for command-line configuration.
GoodJob can also be configured to execute jobs within the web server process to save on resources. This is useful for low-workloads when economy is paramount.
GOOD_JOB_EXECUTION_MODE=async rails server
Additional configuration is likely necessary, see the reference below for configuration.
There are several top-level commands available through the good_job command-line tool.
Configuration options are available with help.
good_job start executes queued jobs.
$ bundle exec good_job help start Usage: good_job start Options: [--queues=QUEUE_LIST] # Queues or pools to work from. (env var: GOOD_JOB_QUEUES, default: *) [--threads=COUNT] # Default number of threads per pool to use for working jobs. (env var: GOOD_JOB_THREADS, default: 5) [--max-threads=COUNT] # (DEPRECATED: use --threads) (env var: GOOD_JOB_MAX_THREADS, default: 5) [--poll-interval=SECONDS] # Interval between polls for available jobs in seconds (env var: GOOD_JOB_POLL_INTERVAL, default: 10) [--max-cache=COUNT] # Maximum number of scheduled jobs to cache in memory (env var: GOOD_JOB_MAX_CACHE, default: 10000) [--shutdown-timeout=SECONDS] # Number of seconds to wait for jobs to finish when shutting down before stopping the thread. (env var: GOOD_JOB_SHUTDOWN_TIMEOUT, default: -1 (forever)) [--enable-cron] # Whether to run cron process (default: false) [--enable-listen-notify] # Whether to enqueue and read jobs with Postgres LISTEN/NOTIFY (default: true) [--idle-timeout=SECONDS] # Exit process when no jobs have been performed for this many seconds (env var: GOOD_JOB_IDLE_TIMEOUT, default: nil) [--daemonize] # Run as a background daemon (default: false) [--pidfile=PIDFILE] # Path to write daemonized Process ID (env var: GOOD_JOB_PIDFILE, default: tmp/pids/good_job.pid) [--probe-port=PORT] # Port for http health check (env var: GOOD_JOB_PROBE_PORT, default: nil) [--probe-handler=PROBE_HANDLER] # Use 'webrick' to use WEBrick to handle probe server requests which is Rack compliant, otherwise default server that is not Rack compliant is used. [--queue-select-limit=COUNT] # The number of queued jobs to select when polling for a job to run. (env var: GOOD_JOB_QUEUE_SELECT_LIMIT, default: nil)" Executes queued jobs. All options can be configured with environment variables. See option descriptions for the matching environment variable name. == Configuring queues Separate multiple queues with commas; exclude queues with a leading minus; separate isolated execution pools with semicolons and threads with colons.
good_job cleanup_preserved_jobs destroys preserved job records. See GoodJob.preserve_job_records for when this command is useful.
$ bundle exec good_job help cleanup_preserved_jobs Usage: good_job cleanup_preserved_jobs Options: [--before-seconds-ago=SECONDS] # Destroy records finished more than this many seconds ago (env var: GOOD_JOB_CLEANUP_PRESERVED_JOBS_BEFORE_SECONDS_AGO, default: 1209600 (14 days)) [--discarded-before-seconds-ago=SECONDS] # Destroy discarded records finished more than this many seconds ago (env var: GOOD_JOB_CLEANUP_DISCARDED_JOBS_BEFORE_SECONDS_AGO, default: same as --before-seconds-ago) Manually destroys preserved job records. By default, GoodJob automatically destroys job records when the job is performed and this command is not required to be used.
Active Job configuration depends on where the code is placed:
config.active_job.queue_adapter = :good_job within config/application.rb or config/environments/*.rb.ActiveJob::Base.queue_adapter = :good_job within an initializer (e.g. config/initializers/active_job.rb).GoodJob configuration can be placed within Rails config directory for all environments (config/application.rb), within a particular environment (e.g. config/environments/development.rb), or within an initializer (e.g. config/initializers/good_job.rb).
Configuration examples:
# config/initializers/good_job.rb OR config/application.rb OR config/environments/{RAILS_ENV}.rb Rails.application.configure do # Configure options individually... config.good_job.preserve_job_records = true config.good_job.retry_on_unhandled_error = false config.good_job.on_thread_error = -> (exception) { Rails.error.report(exception) } config.good_job.execution_mode = :async config.good_job.queues = '*' config.good_job.threads = 5 config.good_job.poll_interval = 30 # seconds config.good_job.shutdown_timeout = 25 # seconds config.good_job.enable_cron = true config.good_job.cron = { example: { cron: '0 * * * *', class: 'ExampleJob' } } config.good_job.cron_graceful_restart_period = 5.minutes config.good_job.dashboard_default_locale = :en # ...or all at once. config.good_job = { preserve_job_records: true, retry_on_unhandled_error: false, on_thread_error: -> (exception) { Rails.error.report(exception) }, execution_mode: :async, queues: '*', threads: 5, poll_interval: 30, shutdown_timeout: 25, enable_cron: true, cron: { example: { cron: '0 * * * *', class: 'ExampleJob' }, }, dashboard_default_locale: :en, } end
Available configuration options are:
execution_mode (symbol) specifies how and where jobs should be executed. You can also set this with the environment variable GOOD_JOB_EXECUTION_MODE. It can be any one of:
:inline executes jobs immediately in whatever process queued them (usually the web server process). This should only be used in test and development environments.:external causes the adapter to enqueue jobs, but not execute them. When using this option (the default for production environments), you’ll need to use the command-line tool to actually execute your jobs.:async (or :async_server) executes jobs in separate threads within the Rails web server process (bundle exec rails server). It can be more economical for small workloads because you don’t need a separate machine or environment for running your jobs, but if your web server is under heavy load or your jobs require a lot of resources, you should choose :external instead. When not in the Rails web server, jobs will execute in :external mode to ensure jobs are not executed within rails console, rails db:migrate, rails assets:prepare, etc.:async_all executes jobs in separate threads in any Rails process.queues (string) sets queues or pools to execute jobs. You can also set this with the environment variable GOOD_JOB_QUEUES.
threads (integer) sets the default number of threads per pool to use for working jobs. You can also set this with the environment variable GOOD_JOB_THREADS. (Formerly max_threads / GOOD_JOB_MAX_THREADS, which are deprecated but still honored.)
poll_interval (integer) sets the number of seconds between polls for jobs when execution_mode is set to :async. You can also set this with the environment variable GOOD_JOB_POLL_INTERVAL. A poll interval of -1 disables polling completely.
poll_interval.max_cache (integer) sets the maximum number of scheduled jobs that will be stored in memory to reduce execution latency when also polling for scheduled jobs. Caching 10,000 scheduled jobs uses approximately 20MB of memory. You can also set this with the environment variable GOOD_JOB_MAX_CACHE.
shutdown_timeout (integer) number of seconds to wait for jobs to finish when shutting down before stopping the thread. Defaults to forever: -1. You can also set this with the environment variable GOOD_JOB_SHUTDOWN_TIMEOUT.
enable_cron (boolean) whether to run cron process. Defaults to false. You can also set this with the environment variable GOOD_JOB_ENABLE_CRON.
cron_graceful_restart_period (integer seconds or ActiveSupport::Duration) when cron starts, attempt to re-enqueue jobs that would have been enqueued by cron within this time period (e.g. 1.minute). This should match the expected downtime during deploys. Like enabling cron on multiple processes, this relies on cron-created job records being preserved to avoid re-enqueuing jobs that already ran, so it is ignored when preserve_job_records is false or :on_unhandled_error. It does not apply to a cron proc that returns a time, which already enqueues the runs it missed since its last job. Defaults to nil (disabled). You can also set this with the environment variable GOOD_JOB_CRON_GRACEFUL_RESTART_PERIOD.
enable_listen_notify (boolean) whether to enqueue and read jobs with Postgres LISTEN/NOTIFY. Defaults to true. You can also set this with the environment variable GOOD_JOB_ENABLE_LISTEN_NOTIFY.
cron (hash) cron configuration. Defaults to {}. You can also set this as a JSON string with the environment variable GOOD_JOB_CRON
cleanup_discarded_jobs (boolean) whether to destroy discarded jobs when cleaning up preserved jobs using the $ good_job cleanup_preserved_jobs CLI command or calling GoodJob.cleanup_preserved_jobs. Defaults to true. Can also be set with the environment variable GOOD_JOB_CLEANUP_DISCARDED_JOBS.
cleanup_preserved_jobs_before_seconds_ago (integer) number of seconds to preserve jobs when using the $ good_job cleanup_preserved_jobs CLI command or calling GoodJob.cleanup_preserved_jobs. Defaults to 1209600 (14 days). Can also be set with the environment variable GOOD_JOB_CLEANUP_PRESERVED_JOBS_BEFORE_SECONDS_AGO.
cleanup_discarded_jobs_before_seconds_ago (integer) number of seconds to preserve discarded jobs, when they should be kept longer (or shorter) than other jobs. Has no effect when cleanup_discarded_jobs is false. Defaults to nil, which uses cleanup_preserved_jobs_before_seconds_ago. Can also be set with the environment variable GOOD_JOB_CLEANUP_DISCARDED_JOBS_BEFORE_SECONDS_AGO.
cleanup_interval_jobs (integer) Number of jobs a Scheduler will execute before cleaning up preserved jobs. Defaults to 1000. Disable with false. Can also be set with the environment variable GOOD_JOB_CLEANUP_INTERVAL_JOBS and disabled with 0).
cleanup_interval_seconds (integer) Number of seconds a Scheduler will wait before cleaning up preserved jobs. Defaults to 600 (10 minutes). Disable with false. Can also be set with the environment variable GOOD_JOB_CLEANUP_INTERVAL_SECONDS and disabled with 0).
logger (Rails Logger) lets you set a custom logger for GoodJob. It should be an instance of a Rails Logger (Default: Rails.logger).
preserve_job_records (boolean, symbol, or lambda) keeps job records in your database even after jobs are completed. If set to true, all job records are preserved. If set to :on_unhandled_error, only jobs that finished with an unhandled error are preserved. If set to a lambda, the lambda will be called with the error_event (e.g., :discarded, :retry_stopped, or :unhandled) and should return a boolean indicating whether to preserve the job. (Default: true)
advisory_lock_heartbeat (boolean) whether to use an advisory lock for the purpose of determining whether an execeution process is active. (Default true in Development; false in other environments)
retry_on_unhandled_error (boolean) causes jobs to be re-queued and retried if they raise an instance of StandardError. Be advised this may lead to jobs being repeated infinitely (see below for more on retries). Instances of Exception, like SIGINT, will always be retried, regardless of this attribute’s value. (Default: false)
on_thread_error (proc, lambda, or callable) will be called when there is an Exception. It can be useful for logging errors to bug tracking services, like Sentry or Airbrake. Example:
config.good_job.on_thread_error = -> (exception) { Rails.error.report(exception) }
probe_server_app (Rack application) allows you to specify a Rack application to be used for the probe server. Defaults to nil which uses the default probe server. Example:
config.good_job.probe_app = -> (env) { [200, {}, ["OK"]] }
probe_handler (string) allows you to use WEBrick, a fully Rack compliant webserver instead of the simple default server. Note: You'll need to ensure WEBrick is in your load path as GoodJob doesn't have WEBrick as a dependency. Example:
config.good_job.probe_handler = 'webrick'
enable_pauses (boolean) whether job processing can be paused. Defaults to false. You can also set this with the environment variable GOOD_JOB_ENABLE_PAUSES.
dequeue_query_sort (symbol) sets the sort order used when querying for the next job(s) to execute. Defaults to :created_at. It can be any one of:
:created_at sorts by priority, then job creation order. This is the default, but no longer best matches GoodJob's contemporary data model.:scheduled_at sorts by priority, then scheduled_at, then id, which better matches GoodJob's contemporary data model. This will become the default/only behavior in a future major release.By default, GoodJob configures the following execution modes per environment:
# config/environments/development.rb config.active_job.queue_adapter = :good_job config.good_job.execution_mode = :async # config/environments/test.rb config.active_job.queue_adapter = :good_job config.good_job.execution_mode = :inline # config/environments/production.rb config.active_job.queue_adapter = :good_job config.good_job.execution_mode = :external
Good Job’s general behavior can also be configured via attributes directly on the GoodJob module:
GoodJob.configure_active_record { ... } Inject Active Record configuration into GoodJob's base model, for example, when using multiple databases with Active Record or when other custom configuration is necessary for the Active Record model to connect to the Postgres database. Example:
# config/initializers/good_job.rb GoodJob.configure_active_record do connects_to database: { writing: :special_database } self.table_name_prefix = "special_application_" end
GoodJob.active_record_parent_class (string) Alternatively, modify the Active Record parent class inherited by GoodJob's Active Record model GoodJob::Job (defaults to "ActiveRecord::Base"). Configure this The value must be a String to avoid premature initialization of Active Record.
You’ll generally want to configure these in config/initializers/good_job.rb, like so:
# config/initializers/good_job.rb GoodJob.active_record_parent_class = "ApplicationRecord"
The following options are also configurable via accessors, but you are encouraged to use the configuration attributes instead because these may be deprecated and removed in the future:
GoodJob.logger (Rails Logger) lets you set a custom logger for GoodJob. It should be an instance of a Rails Logger.GoodJob.preserve_job_records (boolean, symbol, or lambda) keeps job records in your database even after jobs are completed. If set to true, all job records are preserved. If set to :on_unhandled_error, only jobs that finished with an unhandled error are preserved. If set to a lambda, the lambda will be called with Active Job instance, and if it exists, the exception the error_event (e.g., :discarded, :retry_stopped, or :unhandled) and should return a boolean indicating whether to preserve the job (e.g. -> (active_job, error, error_event) { !(active_job.is_a(Turbo::Streams::BroadcastStreamJob) || error_event == :retry_stopped). (Default: true)GoodJob.retry_on_unhandled_error (boolean) causes jobs to be re-queued and retried if they raise an instance of StandardError. Be advised this may lead to jobs being repeated infinitely (see below for more on retries). Instances of Exception, like SIGINT, will always be retried, regardless of this attribute’s value. (Default: false)GoodJob.on_thread_error (proc, lambda, or callable) will be called when there is an Exception. It can be useful for logging errors to bug tracking services, like Sentry or Airbrake.
🚧 GoodJob's dashboard is a work in progress. Please contribute ideas and code on Github.
GoodJob includes a Dashboard as a mountable Rails::Engine.
Mount the engine in your config/routes.rb file. The following will mount it at http://example.com/good_job.
# config/routes.rb # ... mount GoodJob::Engine => 'good_job'
Configure authentication. Because jobs can potentially contain sensitive information, you should authorize access. For example, using Devise's authenticate helper, that might look like:
# config/routes.rb # ... authenticate :user, ->(user) { user.admin? } do mount GoodJob::Engine => 'good_job' end
Another option is using basic auth like this:
# config/initializers/good_job.rb GoodJob::Engine.middleware.use(Rack::Auth::Basic) do |username, password| ActiveSupport::SecurityUtils.secure_compare(Rails.application.credentials.good_job_username, username) & ActiveSupport::SecurityUtils.secure_compare(Rails.application.credentials.good_job_password, password) end
To support custom authentication, you can extend GoodJob's ApplicationController using the following hook:
# config/initializers/good_job.rb ActiveSupport.on_load(:good_job_application_controller) do # context here is GoodJob::ApplicationController before_action do raise ActionController::RoutingError.new('Not Found') unless current_user&.admin? end def current_user # load current user end end
To view finished jobs (succeeded and discarded) on the Dashboard, GoodJob must be configured to preserve job records. Preservation is enabled by default.
Troubleshooting the Dashboard: Some applications are unable to autoload the Goodjob Engine. To work around this, explicitly require the Engine at the top of your config/application.rb file, immediately after Rails is required and before Bundler requires the Rails' groups.
# config/application.rb require_relative 'boot' require 'rails/all' require 'good_job/engine' # <= Add this line # ...
API-only Rails applications may not have all of the required Rack middleware for the GoodJob Dashboard to function. To re-add the middleware:
# config/application.rb module MyApp class Application < Rails::Application #... config.middleware.use Rack::MethodOverride config.middleware.use ActionDispatch::Flash config.middleware.use ActionDispatch::Cookies config.middleware.use ActionDispatch::Session::CookieStore end end
The Dashboard can be set to automatically refresh by checking "Live Poll" in the Dashboard header, or by setting ?poll=10 with the interval in seconds (default 30 seconds).
The Dashboard hides job arguments the same way Rails hides them from logs:
Rails.application.config.filter_parameters (e.g. password:) are displayed as [FILTERED].[FILTERED] for jobs with log_arguments disabled, either for every job with config.active_job.log_arguments = false or per job class with self.log_arguments = false.Filtering only applies to how arguments are displayed; arguments are still stored unfiltered in the database and are used when the job is performed or retried.
GoodJob exposes some views that are intended to be overridden by placing views in your application:
app/views/good_job/_custom_head.html.erb: content added to this partial will be added at the end of the <head> tag in all GoodJob views. This is ideal for injecting custom scripts or styles.app/views/good_job/_custom_job_details.html.erb: content added to this partial will be displayed above the argument list on the good_job/jobs#show page.app/views/good_job/_custom_execution_details.html.erb: content added to this partial will be displayed above each execution on the good_job/jobs#show page.Warning: these partials expose classes (such as GoodJob::Job) that are considered internal implementation details of GoodJob. You should always test your custom partials after upgrading GoodJob.
For example, if your app deals with widgets and you want to show a link to the widget a job acted on, you can add the following to app/views/good_job/_custom_job_details.html.erb:
<%# file: app/views/good_job/_custom_job_details.html.erb %> <% arguments = job.active_job.arguments rescue [] %> <% widgets = arguments.select { |arg| arg.is_a?(Widget) } %> <% if widgets.any? %> <div class="my-4"> <h5>Widgets</h5> <ul> <% widgets.each do |widget| %> <li><%= link_to widget.name, main_app.widget_url(widget) %></li> <% end %> </ul> </div> <% end %>
As a second example, you may wish to show a link to a log aggregator next to each job execution. You can do this by adding the following to app/views/good_job/_custom_execution_details.html.erb:
<%# file: app/views/good_job/_custom_execution_details.html.erb %> <div class="py-3"> <%= link_to "Logs", main_app.logs_url(filter: { job_id: job.id }, start_time: execution.performed_at, end_time: execution.finished_at + 1.minute) %> </div>
Smaller priority values have higher priority and run first (default: 0), in accordance with Active Job's definition of priority.
Prior to GoodJob v4, this was reversed: higher priority numbers ran first in all versions of GoodJob v3.x and below. When migrating from v3 to v4, new behavior can be opted into by setting config.good_job.smaller_number_is_higher_priority = true in your GoodJob initializer or application.rb.
Labels are the recommended way to add context or metadata to specific jobs. For example, all jobs that have a dependency on an email service could be labeled email. Using labels requires adding the Active Job extension GoodJob::ActiveJobExtensions::Labels to your job class.
class ApplicationJob < ActiveJob::Base include GoodJob::ActiveJobExtensions::Labels end # Add a default label to every job within the class class WelcomeJob < ApplicationJob self.good_job_labels = ["email"] def perform # Labels can be inspected from within the job puts good_job_labels # => ["email"] end end # Or add to individual jobs when enqueued WelcomeJob.set(good_job_labels: ["email"]).perform_later
Labels can be used to search jobs in the Dashboard. For example, to find all jobs labeled email, search for email.
GoodJob can extend Active Job to provide limits on concurrently running jobs, either at time of enqueue or at perform. Limiting concurrency can help prevent duplicate, double or unnecessary jobs from being enqueued, or race conditions when performing, for example when interacting with 3rd-party APIs.
class MyJob < ApplicationJob include GoodJob::ActiveJobExtensions::Concurrency # Define one or more concurrency rules. Each rule is scoped to a label, # which is a value derived from the job's arguments at enqueue time and # stored on the job record. Jobs must be enqueued with the matching label # via `good_job_labels:` for the rule to apply. # # Multiple rules can be defined; they are evaluated in order and the first # exceeded rule short-circuits the rest. good_job_concurrency_rule( # A label that scopes this rule. Can be a static String or a Lambda/Proc # invoked in the context of the job instance (see "Dynamic labels" below). # The rule only applies to jobs that were enque