gem-hunter
主要言語: JavaScript
Open-source data contract testing for Snowflake, Databricks, BigQuery, Kafka & 15 more. Built on the Open Data Contract Standard (ODCS). MIT.
Turn your schemas into data contracts and test your data against them, on Snowflake, Databricks, BigQuery, Kafka, Postgres & 15 more. Open source, built on the Open Data Contract Standard.
The datacontract CLI is an open-source command-line tool for working with data contracts.
It natively supports the Open Data Contract Standard to lint data contracts, connect to data sources and execute schema and quality tests, and export to different formats.
The tool is written in Python.
It can be used as a standalone CLI tool, in a CI/CD pipeline, or directly as a Python library.
📖 Full documentation: cli.datacontract.com
Quick links: Quickstart · Commands · Best Practices · Custom Export and Import · Release Notes · Development Setup
For LLMs: cli.datacontract.com/llms.txt · cli.datacontract.com/llms-full.txt · Data Contract Skills for coding agents
Let's look at this data contract: https://datacontract.com/orders-v1.odcs.yaml
We have a servers section with endpoint details to a Postgres database, schema for the structure and semantics of the data, service levels and quality attributes that describe the expected freshness and number of rows.
This data contract contains all information to connect to the database and check that the actual data meets the defined schema specification and quality expectations. We can use this information to test if the actual data product is compliant to the data contract.
Let's use uv to install the CLI (or use the Docker image),
$ uv tool install --python python3.11 --upgrade 'datacontract-cli[all]'Now, let's run the tests:
$ export DATACONTRACT_POSTGRES_USERNAME=datacontract_cli.egzhawjonpfweuutedfy $ export DATACONTRACT_POSTGRES_PASSWORD=jio10JuQfDfl9JCCPdaCCpuZ1YO $ datacontract test https://datacontract.com/orders-v1.odcs.yaml # returns: Testing https://datacontract.com/orders-v1.odcs.yaml Server: production (type=postgres, host=aws-1-eu-central-2.pooler.supabase.com, port=6543, database=postgres, schema=dp_orders_v1) ╭────────┬───────────────────────────────────────────────────────────────────┬─────────────────────────┬─────────╮ │ Result │ Check │ Field │ Details │ ├────────┼───────────────────────────────────────────────────────────────────┼─────────────────────────┼─────────┤ │ passed │ Check that field 'line_item_id' is present │ line_items.line_item_id │ │ │ passed │ Check that field line_item_id has physical type UUID │ line_items.line_item_id │ │ │ passed │ Check that field line_item_id has no missing values │ line_items.line_item_id │ │ │ passed │ Check that primary key field line_item_id has no duplicate values │ line_items.line_item_id │ │ │ passed │ Check that field 'order_id' is present │ line_items.order_id │ │ │ passed │ Check that field order_id has physical type UUID │ line_items.order_id │ │ │ passed │ Check that field 'price' is present │ line_items.price │ │ │ passed │ Check that field price has physical type INTEGER │ line_items.price │ │ │ passed │ Check that field price has no missing values │ line_items.price │ │ │ passed │ Check that field 'sku' is present │ line_items.sku │ │ │ passed │ Check that field sku has physical type TEXT │ line_items.sku │ │ │ passed │ Check that field sku has no missing values │ line_items.sku │ │ │ passed │ Check that model orders has row_count > 100000 │ orders │ │ │ passed │ Check that field 'customer_id' is present │ orders.customer_id │ │ │ passed │ Check that field customer_id has physical type TEXT │ orders.customer_id │ │ │ passed │ Check that field customer_id has no missing values │ orders.customer_id │ │ │ passed │ Check that field customer_id has a min length of 10 │ orders.customer_id │ │ │ passed │ Check that field customer_id has a max length of 10 │ orders.customer_id │ │ │ passed │ Check that field 'order_id' is present │ orders.order_id │ │ │ passed │ Check that field order_id has physical type UUID │ orders.order_id │ │ │ passed │ Check that field order_id has no missing values │ orders.order_id │ │ │ passed │ Check that unique field order_id has no duplicate values │ orders.order_id │ │ │ passed │ Check that field 'order_status' is present │ orders.order_status │ │ │ passed │ Check that field order_status has physical type TEXT │ orders.order_status │ │ │ passed │ Check that field order_status has invalid_count = 0 │ orders.order_status │ │ │ passed │ Check that field 'order_timestamp' is present │ orders.order_timestamp │ │ │ passed │ Check that field order_timestamp has physical type TIMESTAMPTZ │ orders.order_timestamp │ │ │ passed │ Check that field 'order_total' is present │ orders.order_total │ │ │ passed │ Check that field order_total has physical type INTEGER │ orders.order_total │ │ │ passed │ Check that field order_total has no missing values │ orders.order_total │ │ ╰────────┴───────────────────────────────────────────────────────────────────┴─────────────────────────┴─────────╯ 🟢 Data contract is valid. Ran 30 checks. Took 2.504495 seconds.
Voilà, the CLI tested that the YAML itself is valid, all records comply with the schema, and all quality attributes are met.
To test your own data, import a contract straight from an existing table (this also generates the servers block) and run the tests against the actual data:
$ datacontract import snowflake --source <account> --database ORDER_DB --schema PUBLIC --output datacontract.yaml $ datacontract test datacontract.yaml
Copy-paste guides with credentials setup are available for Snowflake, BigQuery, Databricks, and 15+ other sources.
We can also use the data contract metadata to export in many formats, e.g., to generate a SQL DDL:
$ datacontract export sql https://datacontract.com/orders-v1.odcs.yaml # returns: -- Data Contract: orders -- SQL Dialect: postgres CREATE TABLE orders ( order_id UUID not null primary key, customer_id text not null, order_total integer not null, order_timestamp TIMESTAMPTZ, order_status text ); CREATE TABLE line_items ( line_item_id UUID not null primary key, sku text not null, price integer not null, order_id UUID );
Or generate an HTML export:
$ datacontract export html --output orders-v1.odcs.html https://datacontract.com/orders-v1.odcs.yaml# create a new data contract from example and write it to odcs.yaml $ datacontract init odcs.yaml # edit the data contract in the Data Contract Editor (web UI) $ datacontract edit odcs.yaml # lint the odcs.yaml and stop after the first validation error (default). $ datacontract lint odcs.yaml # show a changelog between two data contracts $ datacontract changelog v1.odcs.yaml v2.odcs.yaml # fail when a contract change breaks a guarantee consumers rely on $ datacontract breaking v1.odcs.yaml v2.odcs.yaml # execute schema and quality checks (define credentials as environment variables) $ datacontract test odcs.yaml # generate dbt tests from a contract into your dbt project, then run them # (omit the contract to sync/test every *.odcs.yaml in the project) $ datacontract dbt sync orders.odcs.yaml --project-dir ./warehouse $ datacontract dbt test orders.odcs.yaml --project-dir ./warehouse # export data contract as html (other formats: avro, dbt-models, dbt-sources, dbt-staging-sql, jsonschema, odcs, rdf, sql, sodacl, terraform, ...) $ datacontract export html datacontract.yaml --output odcs.html # import sql (other formats: avro, glue, bigquery, jsonschema, excel ...) $ datacontract import sql --source my-ddl.sql --dialect postgres --output odcs.yaml # import from Excel template $ datacontract import excel --source odcs.xlsx --output odcs.yaml # export to Excel template $ datacontract export excel --output odcs.xlsx odcs.yaml
from datacontract.data_contract import DataContract data_contract = DataContract(data_contract_file="odcs.yaml") run = data_contract.test() if not run.has_passed(): print("Data quality validation failed.") # Abort pipeline, alert, or take corrective actions...
Choose the most appropriate installation method for your needs:
The preferred way to install is uv:
uv tool install --python python3.11 --upgrade 'datacontract-cli[all]'
If you have uv installed, you can run datacontract-cli directly without installing:
uvx --from 'datacontract-cli[all]' datacontract --version
Python 3.10–3.14 are supported. We recommend using Python 3.11.
python3 -m pip install 'datacontract-cli[all]'
datacontract --versionTypically it is better to install the application in a virtual environment for your projects:
cd my-project python3.11 -m venv venv source venv/bin/activate pip install 'datacontract-cli[all]' datacontract --version
pipx installs into an isolated environment.
pipx install 'datacontract-cli[all]'
datacontract --versionYou can also use our Docker image to run the CLI tool. It is also convenient for CI/CD pipelines.
docker pull datacontract/cli
docker run --rm -v ${PWD}:/home/datacontract datacontract/cliYou can create an alias for the Docker command to make it easier to use:
alias datacontract='docker run --rm -v "${PWD}:/home/datacontract" datacontract/cli:latest'
Note: The output of Docker command line messages is limited to 80 columns and may include line breaks. Don't pipe docker output to files if you want to export code. Use the --output option instead.
Released images are signed with cosign keyless signing. See Verifying the image to check a signature before you run it.
The CLI tool defines several optional dependencies (also known as extras) that can be installed for using with specific servers types.
With all, all server dependencies except hana are included.
uv tool install --python python3.11 --upgrade 'datacontract-cli[all]'A list of available extras:
| Dependency | Installation command |
|---|---|
| API (run as web server) | pip install datacontract-cli[api] |
| Amazon Athena | pip install datacontract-cli[athena] |
| Avro | pip install datacontract-cli[avro] |
| Azure | pip install datacontract-cli[azure] |
| Google BigQuery | pip install datacontract-cli[bigquery] |
| Databricks | pip install datacontract-cli[databricks] (also inside a Databricks Runtime, using the cluster's own Spark session) |
| DataFrame (Spark) | pip install datacontract-cli[dataframe] (PySpark not included — you supply the Spark session) |
| DBML | pip install datacontract-cli[dbml] |
| DuckDB (local file, CSV import and API response testing) | pip install datacontract-cli[duckdb] |
| Exasol | pip install datacontract-cli[exasol] |
| Excel | pip install datacontract-cli[excel] |
| Excel workbooks (testing) | pip install datacontract-cli[xlsx] |
| GCS | pip install datacontract-cli[gcs] |
| SAP HANA / Datasphere | pip install datacontract-cli[hana] (not part of all: hdbcli is proprietary) |
| Apache Iceberg (schema import and export, REST catalog testing) | pip install datacontract-cli[iceberg] |
| Impala | pip install datacontract-cli[impala] |
| Kafka | pip install datacontract-cli[kafka] |
| MySQL | pip install datacontract-cli[mysql] |
| Oracle | pip install datacontract-cli[oracle] |
| Parquet | pip install datacontract-cli[parquet] |
| PostgreSQL | pip install datacontract-cli[postgres] |
| Protobuf | pip install datacontract-cli[protobuf] |
| RDF | pip install datacontract-cli[rdf] |
| Amazon Redshift | pip install datacontract-cli[redshift] |
| S3 | pip install datacontract-cli[s3] |
| Snowflake | pip install datacontract-cli[snowflake] |
| Microsoft SQL Server | pip install datacontract-cli[sqlserver] |
| Trino | pip install datacontract-cli[trino] |
| XML Schema (import) | pip install datacontract-cli[xml] |
📖 The full documentation is at cli.datacontract.com.
It covers everything in depth, including the complete command reference:
init, lint, test, export, import, dbt, ci, catalog, publish, api, and moreuv python pin 3.11
uv venv
uv pip install -e '.[dev]'
uv run ruff check
uv run pytestWe are happy to receive your contributions. Propose your change in an issue or directly create a pull request with your improvements.
Before creating a pull request, please make sure that all tests are passing (uv run pytest) and
your code is properly formatted (ruff format). Create a changelog entry and reference fixed
issues (if any).
Run in WSL. (We need to fix the paths in the tests so that normal Windows will work, contributions are appreciated)
This uv issue might be relevant.
Try to sync all groups:
uv sync --all-groups --all-extras
docker build -t datacontract/cli . docker run --rm -v ${PWD}:/home/datacontract datacontract/cli
We've included a docker-compose.yml configuration to simplify the build, test, and deployment of the image.
To build the Docker image using Docker Compose, run the following command:
docker compose build
This command utilizes the docker-compose.yml to build the image, leveraging predefined settings such as the build context and Dockerfile location. This approach streamlines the image creation process, avoiding the need for manual build specifications each time.
After building the image, you can test it directly with Docker Compose:
docker compose run --rm datacontract --version
This command runs the container momentarily to check the version of the datacontract CLI. The --rm flag ensures that the container is automatically removed after the command executes, keeping your environment clean.
Created by Stefan Negele, Jochen Christ, and Simon Harrer.
<style>.github-corner:hover .octo-arm{animation:octocat-wave 560ms ease-in-out}@keyframes octocat-wave{0%,100%{transform:rotate(0)}20%,60%{transform:rotate(-25deg)}40%,80%{transform:rotate(10deg)}}@media (max-width:500px){.github-corner:hover .octo-arm{animation:none}.github-corner .octo-arm{animation:octocat-wave 560ms ease-in-out}}</style>