gem-hunter

pg_mooncake is a Postgres extension that creates a columnstore mirror of your Postgres tables in Iceberg, enabling fast analytics queries with sub-second freshness:

• Real-time ingestion powered by moonlink for streaming and batched INSERT/UPDATE/DELETE.
• Fast analytics accelerated by DuckDB, ranking top 10 on ClickBench.
• Postgres-native allowing you to query a columnstore table just like a regular Postgres table.
• Iceberg-native making your data readily accessible by other query engines.

For new users, we recommend using the Docker image to get started quickly:

docker run --name mooncake --rm -e POSTGRES_PASSWORD=password mooncakelabs/pg_mooncake

This will start Postgres with pg_mooncake preinstalled. You can then connect to it using psql with the default user postgres:

docker exec -it mooncake psql -U postgres

To build pg_mooncake, first install Rust, pgrx, and the build tools for DuckDB.

Then, clone the repository:

git clone --recurse-submodules https://github.com/Mooncake-Labs/pg_mooncake.git

To build and install for Postgres versions 14-18, run:

cargo pgrx init --pg18=$(which pg_config)   # Replace with your Postgres version
make pg_duckdb                              # Skip if pg_duckdb is already installed
make install PG_VERSION=pg18

Finally, add pg_mooncake to shared_preload_libraries in your postgresql.conf file and enable logical replication:

duckdb.allow_community_extensions = true
shared_preload_libraries = 'pg_duckdb,pg_mooncake'
wal_level = logical

For a complete walkthrough, refer to our Dockerfile.

First, create the pg_mooncake extension:

CREATE EXTENSION pg_mooncake CASCADE;

Next, create a regular Postgres table trades:

CREATE TABLE trades(
  id bigint PRIMARY KEY,
  symbol text,
  time timestamp,
  price real
);

Then, create a columnstore mirror trades_iceberg that stays in sync with trades:

CALL mooncake.create_table('trades_iceberg', 'trades');

Now, insert some data into trades:

INSERT INTO trades VALUES
  (1,  'AMD', '2024-06-05 10:00:00', 119),
  (2, 'AMZN', '2024-06-05 10:05:00', 207),
  (3, 'AAPL', '2024-06-05 10:10:00', 203),
  (4, 'AMZN', '2024-06-05 10:15:00', 210);

Finally, query trades_iceberg to see that it reflects the up-to-date state of trades:

SELECT avg(price) FROM trades_iceberg WHERE symbol = 'AMZN';

pg_mooncake is an open-source project maintained by Mooncake Labs and licensed under the MIT License. We'd love your help to make it even better! Join our Slack, participate in discussions, open issues to report bugs or suggest features, contribute code and documentation, or help us improve the project in any way. All contributions are welcome! 🥮