riverqueue/river

README

River

River is a fast, transactional job queue backed by Postgres or SQLite. It includes automatic retries, scheduling, unique jobs, and a web UI, with support for Go, JavaScript/TypeScript, Ruby, and Rust. All implementations are fully cross-compatible -- enqueue in one language and process in another.

See homepage, docs, and godoc, as well as the River UI and its live demo.

River encourages using the same database for application data and the job queue. By enqueueing jobs transactionally along with other database changes, whole classes of distributed systems problems are avoided. Jobs are guaranteed to be enqueued if their transaction commits, are removed if their transaction rolls back, and aren't visible for work until commit. See transactional enqueueing for more background on this philosophy.

Job args and workers

In Go, jobs are defined in struct pairs, with an implementation of JobArgs and one of Worker.

Job args contain json annotations and define how jobs are serialized to and from the database, along with a "kind", a stable string that uniquely identifies the job.

type SortArgs struct {
    // Strings is a slice of strings to sort.
    Strings []string `json:"strings"`
}

func (SortArgs) Kind() string { return "sort" }

Workers expose a Work function that dictates how jobs run.

type SortWorker struct {
    // An embedded WorkerDefaults sets up default methods to fulfill the rest of
    // the Worker interface:
    river.WorkerDefaults[SortArgs]
}

func (w *SortWorker) Work(ctx context.Context, job *river.Job[SortArgs]) error {
    sort.Strings(job.Args.Strings)
    fmt.Printf("Sorted strings: %+v\n", job.Args.Strings)
    return nil
}

Registering workers

Jobs are uniquely identified by their "kind" string. Workers are registered on start up so that River knows how to assign jobs to workers:

workers := river.NewWorkers()
// AddWorker panics if the worker is already registered or invalid:
river.AddWorker(workers, &SortWorker{})

Starting a client

A River Client provides an interface for job insertion and manages job processing and maintenance services. A client's created with a database pool, driver, and config struct containing a Workers bundle and other settings. Here's a Client working one queue ("default") with up to 100 worker goroutines at a time:

riverClient, err := river.NewClient(riverpgxv5.New(dbPool), &river.Config{
    Queues: map[string]river.QueueConfig{
        river.QueueDefault: {MaxWorkers: 100},
    },
    Workers: workers,
})
if err != nil {
    panic(err)
}

// Run the client inline. All executed jobs will inherit from ctx:
if err := riverClient.Start(ctx); err != nil {
    panic(err)
}

Workers can also be omitted, but it's better to include it so River can check that inserted job kinds have a worker that can run them.

Stopping

The client should also be stopped on program shutdown. There's a number of ways to go about this (see graceful shutdown), but the shortest is to cancel the context send to Start when the program's ready to stop. For example, to stop on SIGINT/SIGTERM:

riverClient, err := river.NewClient(riverpgxv5.New(dbPool), &river.Config{
    SoftStopTimeout: 10 * time.Second,
    ...
})
if err != nil {
    panic(err)
}

signalCtx, stop := signal.NotifyContext(ctx, syscall.SIGINT, syscall.SIGTERM)
defer stop()

// Stop fetching new work and wait for active jobs to finish. Cancel jobs after
// SoftStopTimeout elapses.
if err := riverClient.Start(signalCtx); err != nil {
    panic(err)
}

<-riverClient.Stopped()

Alternatively, use an explicit call to Stop:

if err := riverClient.Stop(ctx); err != nil {
    panic(err)
}

Insert-only clients will insert jobs, but not work them, and don't need to be started or stopped.

Inserting jobs

Client.InsertTx is used in conjunction with an instance of job args to insert a job to work on a transaction:

_, err = riverClient.InsertTx(ctx, tx, SortArgs{
    Strings: []string{
        "whale", "tiger", "bear",
    },
}, nil)

if err != nil {
    panic(err)
}

See the InsertAndWork example for complete code.

Other features

Batch job insertion for efficiently inserting many jobs at once using Postgres COPY FROM.
Cancelling jobs from inside a work function.
Error and panic handling.
Multiple queues to better guarantee job throughput, worker availability, and isolation between components.
Periodic and cron jobs.
Scheduled jobs that run automatically at their scheduled time in the future.
Snoozing jobs from inside a work function.
Subscriptions to queue activity and statistics, providing easy hooks for telemetry like logging and metrics.
Test helpers to verify that jobs are inserted as expected.
Transactional job completion to guarantee job completion commits with other changes in a transaction.
Unique jobs by args, period, queue, and state.
Web UI for inspecting and interacting with jobs and queues.
Work functions for simplified worker implementation.

Cross language enqueueing

Jobs inserted by any of River's language implementations can be worked by another, so services in different languages can share a queue. See the source links above for each implementation's client and worker APIs.

Jobs can also be inserted from Python.

Development

See developing River.