ClickHouse/walshadow

README

walshadow replicates PostgreSQL rows into ClickHouse from physical WAL, including initial load, continuous changes, schema evolution, restarts, and planned source switchover

Get Started

Follow Quickstart to run PostgreSQL, ClickHouse, and walshadow locally with Docker Compose
Follow Getting started to connect existing PostgreSQL and ClickHouse databases

PostgreSQL 16 or newer is required. Source and walshadow shadow must use same PostgreSQL major version

Documentation

System design and diagrams live in architecture, unfinished work in plans, and build and test guidance in development. Read source for implementation details

Build from source

Clone repository with recursive submodules, then build Rust workspace and PostgreSQL module

# Clone source plus required pg-clickhouse-c and clickhouse-c submodules
git clone --recurse-submodules https://github.com/ClickHouse/walshadow.git
cd walshadow

# Build release binaries
cargo build --release

# Build and install PostgreSQL module with PGXS
make -C pgext install

Default features include LZ4. Use --no-default-features for uncompressed builds or enable zstd for Zstandard support

Built binaries:

walshadow-stream, replication daemon
walshadow-filter, segment-level filter for offline WAL files
walshadow-classify, record-level classifier for diagnostics

walshadow PostgreSQL module is not an SQL extension. Managed shadow loads it through shared_preload_libraries

Test

# Build PostgreSQL module before integration tests
make -C pgext

# Run test suite and lints
cargo nextest run --workspace --all-targets
cargo clippy --all-targets -- -D warnings

Integration tests require initdb and pg_ctl on PATH