gem-hunter
Language: JavaScript
A set of UDFs and Procedures to extend BigQuery, Snowflake, Redshift, Postgres and Databricks with Spatial Analytics capabilities
The CARTO Analytics Toolbox is a set of UDFs and Stored Procedures to unlock Spatial Analytics. It is organized into modules based on the functionality they offer. This toolbox is cloud-native, which means it is available for different data warehouses: BigQuery, Snowflake, Redshift, Postgres, and Databricks. It is built on top of the data warehouse's GIS features, extending and complementing this functionality.
| BigQuery | Snowflake | Redshift | Postgres | Databricks |
|---|---|---|---|---|
 |
 |
 |
 |
 |
This repo contains the core open-source modules of the toolbox. CARTO offers a set of premium modules that are available for CARTO users.
The Analytics Toolbox has two parallel components:
Gateway (gateway/): Lambda-based Python functions deployed to AWS Lambda
gateway/logic/gateway/functions/Clouds (clouds/{cloud}/): Native SQL UDFs specific to each cloud platform
Using the functions on this project depends on the Datawarehouse you are using. In BigQuery and Snowflake you can access them directly as a shared resources without having to install them, for the rest you will have to install them locally on your database.
You can use the functions directly as they're globally shared in the US region:
SELECT `carto-os.carto.H3_CENTER`('84390cbffffffff')
If you need to use them from the Europe region use:
SELECT `carto-os-eu.carto.H3_CENTER`('84390cbffffffff')
If you need to install them in your own VPC or a different region, see the cloud-specific documentation below.
The easiest way to start using these functions is to add them to your Datawarehouse through the Snowflake Marketplace. Go there and install it using the regular methods. After that you should be able to use them on the location you have installed them. For example try:
SELECT carto_os.carto.H3_FROMGEOGPOINT(ST_POINT(-3.7038, 40.4168), 4)
If you need to install them directly, not through the data share process, follow the instructions later on.
Right now the only way to get access the Analytics toolbox is by installing it directly on your database. Follow the instructions later on.
| Cloud | Documentation |
|---|---|
| BigQuery | https://docs.carto.com/analytics-toolbox-bigquery |
| Snowflake | https://docs.carto.com/analytics-toolbox-snowflake |
| Redshift | https://docs.carto.com/analytics-toolbox-redshift |
| Postgres | https://docs.carto.com/analytics-toolbox-postgres |
| Databricks | https://docs.carto.com/analytics-toolbox-databricks |
For developing and deploying Lambda-based Python functions:
Documentation:
Quick Start:
cd gateway # Setup make venv cp .env.template .env # Edit .env with your credentials # Build and test make build cloud=redshift make test-unit cloud=redshift # Deploy make deploy cloud=redshift
All deployment logic lives in gateway/logic/:
logic/common/engine/ - Catalog, validators, packagerslogic/clouds/redshift/ - Redshift CLI and SQL generationlogic/platforms/aws-lambda/ - Lambda deploymentFor cloud-specific native SQL UDFs, see the cloud-specific READMEs:
| Cloud | Development Guide |
|---|---|
| BigQuery | README.md |
| Snowflake | README.md |
| Redshift | README.md |
| Postgres | README.md |
| Databricks | README.md |
Gateway functions:
cd gateway # Build functions (required before testing) make build cloud=redshift # Run tests make test-unit cloud=redshift make test-integration cloud=redshift # Deploy make deploy cloud=redshift # Lint make lint make lint-fix # Create distribution package make create-package cloud=redshift
Cloud functions:
Switch to a specific cloud directory (e.g., cd clouds/snowflake):
# All tests make test # Specific module(s) make test modules=h3 make test modules=h3,transformations # Specific function(s) make test functions=H3_POLYFILL make test functions=H3_POLYFILL,ST_BUFFER # Deploy make deploy
This project is open source. We're happy to receive feedback and contributions! Feel free to:
For technical details and architecture, see CLAUDE.md.