gem-hunter
主要言語: Rust
A high-performance Postgres replication engine written in Rust. Embed it in your Rust application or run it as a standalone binary.
High-performance Postgres replication, written in Rust.
Embed it as a library or run it as a standalone binary.
A project by Supabase.
Documentation · First Pipeline · Standalone Replicator · Examples · Issues
Note
Supabase ETL is under active development. APIs and setup steps may change before the first stable release.
Supabase ETL is a high-performance Postgres replication engine written in Rust. Embed it in your Rust application or run it as a standalone binary. For each published table, it performs an initial sync of existing rows, then replicates changes to a built-in or custom destination.
This repository contains the framework, destination modules, standalone replicator, examples, and documentation. For the managed product in the Supabase Dashboard—including availability, pricing, and operations—use the Supabase Pipelines documentation.
flowchart LR
Postgres["Postgres publication"] --> Sync["Initial sync"]
Sync --> Replication["Ongoing replication"]
Replication --> Destination["Destination"]
Supabase ETL replicates each table in two phases:
Across both phases, a store persists checkpoints, schemas, destination metadata, and table state so replication can recover safely after a restart. Copy and change data capture (CDC) are replication paths, not customer-visible phases. Initial sync uses the copy path followed by CDC catch-up; ongoing replication uses the CDC path after the table is ready.
Each table's initial copy holds ACCESS SHARE locks from before snapshot
creation until its source copy workers finish. On a primary, ordinary DML can
continue while ACCESS EXCLUSIVE DDL waits; queued DDL can delay later queries.
Long copies can also retain WAL and old row versions. Review
source impact and migration planning
before starting a large initial sync.
Schema-change handling is in public beta, with known catalog-visibility, concurrent-DDL,
and relation-metadata risks. Read Schema Changes
before planning migrations alongside a running pipeline.
| Goal | Documentation |
|---|---|
| Build a working pipeline | First Pipeline |
| Run the standalone process | Standalone Replicator |
| Prepare a source database | Configure Postgres |
| Implement a store or destination | Custom Implementations |
| Understand the runtime | Architecture |
| Browse runnable destinations | etl-examples |
ETL is installed from Git while we prepare for a crates.io release:
[dependencies] etl = { git = "https://github.com/supabase/etl" } tokio = { version = "1", features = ["full"] }
~33× lower tail latency. ~125× smaller measured runtime memory footprint. Compared with Debezium Server 3.1.3.Final in Hardbyte's independent CDC benchmark: 26 ms vs. 848 ms and 13 MB vs. 1,629 MB, at 200 source changes/s across four fast HTTP consumers.
Tested ETL d206d66e (July 2026) with a custom HttpDestination and
MemoryStore. Results compare specific configurations: versions, polling,
batching, destination, store, workload, and deployment all affect performance.
See benchmark details
for versions, measurement methodology, and configuration differences.
| Feature | Destination | Status |
|---|---|---|
clickhouse |
ClickHouse | Public alpha |
bigquery |
Google BigQuery | Public alpha |
ducklake |
DuckLake | Public alpha |
snowflake |
Snowflake | Public alpha |
iceberg |
Apache Iceberg | Deprecated |
Public alpha describes the managed-service release stage. Local and self-hosted use of the open-source destinations is unrestricted.
ClickHouse is the fastest way to start locally: cargo x init runs it, and
cargo x setup replicator configures it by default. BigQuery is the most
mature cloud destination. See the
Destinations reference
for release status and limitations, and the
etl-examples guide for runnable examples.
Supabase ETL supports PostgreSQL 14 through 18. PostgreSQL 15 or newer is recommended for column and row publication filters. PostgreSQL 16 or newer is required when the replication connection points at a physical read replica.
The source must use wal_level = logical, and the replication user needs the
REPLICATION role. See Configure Postgres
for the complete setup and production guidance.
The standalone replicator is publicly available; no private registry credentials
are required. Set IMAGE_TAG to a published full commit SHA or release tag:
docker pull "public.ecr.aws/supabase/etl-replicator:$IMAGE_TAG"Each newly published tag contains Linux AMD64 and ARM64 images. Docker selects
the host architecture automatically; use --platform linux/amd64 or
--platform linux/arm64 to select it explicitly. Pin deployments to a commit,
release tag, or digest. latest follows the current main tip and is mutable.
Manual builds publish only <full-commit-sha>-experimental, even for commits on main.
The image includes the replicator, the DuckLake maintenance binary, and DuckDB extensions. Supply configuration at runtime; see the replicator guide.
cargo x init # Docker, databases, migrations cargo x setup replicator && cargo x seed && cargo x run replicator
See DEVELOPMENT.md to start the replicator, pick a
destination, and run tests. The workspace uses Rust 1.95.0 from
rust-toolchain.toml.
Contributions are welcome. Before proposing a new destination, start a discussion or issue so maintainership and long-term demand can be evaluated first.
Report suspected vulnerabilities privately according to SECURITY.md.
Apache-2.0. See LICENSE.