sequinstream/sequin

README

Documentation · Website · Discord · Slack · Blog

Sequin

Sequin is the fastest change data capture (CDC) platform for Postgres. Sequin makes it easy to stream Postgres to streaming platforms, queues, search indexes, and more (e.g. Kafka, GCP Pub/Sub, another Postgres, etc.). You can backfill existing rows and stream new changes in real-time.

Sequin is a standalone Docker container that you can deploy next to your Postgres database.

For production deployments on AWS, see the deployment/ directory for Terraform infrastructure-as-code templates.

Sequin is open source. To help us make this project great, tell us what you're building in our Discord server or Slack community.

Getting started

See our quickstart to get up and running in your dev environment in about 5 minutes.

Performance

Sequin delivers industry-leading performance for change data capture (CDC), sustaining 50k operations per second (40 MB/s) with 55ms average latency.

Tool / Deployment	Sustained throughput	Avg latency	99p latency
Sequin	>50 k ops / s	55 ms	253 ms
Debezium · MSK	6 k ops / s	258 ms	499 ms
Debezium · Server	23 k ops / s	210 ms	440 ms
Fivetran	-	5+ minutes	-
Airbyte	-	1+ hours	-

Read more about our benchmarking methodology and performance in our docs.

Use cases

Sequin works great for change data capture use cases like:

Streaming events: Stream database changes as events for other services to consume using Kafka, SQS, Pub/Sub and more.
Audit logging: Track, enrich, and record every change in your database for compliance or feature development.
Maintaining a cache: Keep application caches in sync with your database by streaming changes to Redis compliant caches.
Refreshing search indexes: End-to-end tooling to warm search indexes like Typesense and Elasticsearch and keep them in sync with sub-second latency.

Sinks

Sink	Docs	Description
Azure EventHubs	Reference	Publish messages to Azure EventHubs
GCP Pub/Sub	Reference	Publish messages to Google Cloud Pub/Sub topics
Elasticsearch	Reference	Index database changes with Elasticsearch
Sequin Stream (HTTP `GET`)	Reference	Consume changes directly from Sequin with exactly-once processing
Kafka	Reference	Stream changes to Apache Kafka topics
Kinesis	Reference	Send messages to Amazon Kinesis streams queues
Meilisearch	Reference	Index database changes with Meilisearch
NATS	Reference	Stream changes to NATS subjects
RabbitMQ	Reference	Publish messages to RabbitMQ exchanges
Redis Stream	Reference	`XADD` to Redis Streams
Redis String	Reference	`SET` to Redis keys
S2	Reference	Send messages to S2 streams
SQS	Reference	Send messages to Amazon SQS queues
SNS	Reference	Send messages to Amazon SNS
Typesense	Reference	Index database changes with Typesense
Webhooks (HTTP `POST`)	Reference	Send changes to any HTTP endpoint

Killer features

Bring your database: Sequin is not an extension. It works with any Postgres database version 14+.
Never miss a change: Sequin ensures 100% delivery of database changes to sinks with strict ordering.
Filters: Write custom filters to exclude or include changes to sink.
Transforms: Transform message payloads by writing functions in low-latency Elixir.
Routing: Route messages to specific topics, endpoints, or indexes.
Exactly-once processing: Sequin ensures exactly-once processing of database changes using idempotency keys.
Backfills: Backfill sinks at any point in time. Run partial backfills to only replay specific rows.
Infrastructure as code: Manage Sequin as code using a YAML file paired with the Sequin CLI. Or use our Management API.
Rich web console: Your team can configure and monitor sinks from a full feature web console.
Observability: Sequin ships a Prometheus metrics endpoint.

How Sequin works

Sequin connects to any Postgres database. Specify the schemas and tables you want to stream, as well as optional filters and transformations. Route changes to sinks like Kafka, Google Pub/Sub, Redis, etc.

When you setup a sink, you can opt to backfill data from the source table to the sink.

After setup, Sequin will stream new changes to the sink as they occur in real-time. If there are any issues with delivery, Sequin will automatically retry delivery with exponential backoff.

Sequin comes with a web console/UI for configuration and monitoring. You can also configure Sequin as code using YAML config files and our Management API.

Why Sequin

We all know Postgres is great for storing and querying data. But what about when you need to stream changes to other systems?

Existing tools like Debezium require significant infrastructure to operate (Kafka), endless guess-and-check configuration, and limited tooling. Other ETL tools (e.g. Fivetran) run in batches and are expensive.

This leads many teams to build brittle, homegrown solutions that are difficult to maintain and lack essential features.

Fast, simple, reliable Postgres CDC

Sequin provides fastest, simple Postgres CDC. It's tuned to Postgres to stream changes in real-time, provides strict guarantees, and doesn't require complex infrastructure to operate.

Contributing

See CONTRIBUTING.md for detailed instructions on how to contribute to Sequin.