gem-hunter
Language: Go
Sling is a CLI tool that extracts data from a source storage/database and loads it in a target storage/database.
Slings from a data source to a data target.
See docs.slingdata.io for more details.
Sling is a passion project turned into a free CLI Product which offers an easy solution to create and maintain small to medium volume data pipelines using the Extract & Load (EL) approach. It focuses on data movement between:
Some key features:
--src-stream='select * from my_table where col1 > 10'sling conns sub-commandexport MY_PG='postgres//...)'my_schema.*).select * from my_table where date = '{date}')flatten option, which auto-creates columns from your nested fields.Example Replication:
Available Connectors:
Databases: adbc azuredwh azuresql azuretable bigquery bigtable clickhouse d1 databricks dbase duckdb dynamodb elasticsearch opensearch exasol fabric firebolt mariadb mongodb motherduck mysql odbc oracle postgres prometheus proton redshift scylladb snowflake sqlite sqlserver starrocks trino
File Systems: azure b2 dospaces gs local minio r2 s3 sftp wasabi
File Formats: csv, parquet, xlsx, json, geojson, avro, xml, sas7bday
Here are some additional links:
Ever wanted to quickly pipe in a CSV or JSON file into your database? Use sling to do so:
cat my_file.csv | sling run --tgt-conn MYDB --tgt-object my_schema.my_tableOr want to copy data between two databases? Do it with sling:
sling run --src-conn PG_DB --src-stream public.transactions \ --tgt-conn MYSQL_DB --tgt-object mysql.bank_transactions \ --mode full-refresh
Sling can also easily manage our local connections with the sling conns command:
$ sling conns set MY_PG url='postgresql://postgres:myPassword@pghost:5432/postgres' $ sling conns list +--------------------------+-----------------+-------------------+ | CONN NAME | CONN TYPE | SOURCE | +--------------------------+-----------------+-------------------+ | AWS_S3 | FileSys - S3 | sling env yaml | | FINANCE_BQ | DB - BigQuery | sling env yaml | | DO_SPACES | FileSys - S3 | sling env yaml | | LOCALHOST_DEV | DB - PostgreSQL | dbt profiles yaml | | MSSQL | DB - SQLServer | sling env yaml | | MYSQL | DB - MySQL | sling env yaml | | ORACLE_DB | DB - Oracle | env variable | | MY_PG | DB - PostgreSQL | sling env yaml | +--------------------------+-----------------+-------------------+ $ sling conns discover LOCALHOST_DEV 9:05AM INF Found 344 streams: - "public"."accounts" - "public"."bills" - "public"."connections" ...
Databricks: Unity Catalog Volumes are a file connection (type: databricks-volume). Direct Delta ingest uses copy_method: zerobus on type: databricks. See https://docs.slingdata.io/connections/database-connections/databricks
curl -fsSL https://slingdata.io/install.sh | bash # You're good to go! sling -h
irm https://slingdata.io/install.ps1 | iex # You're good to go! sling -h
brew install slingdata-io/sling/sling
scoop bucket add sling https://github.com/slingdata-io/scoop-sling.git scoop install sling
Pre-built binaries for macOS, Linux, and Windows are available on the releases page.
Requirements:
CGO_ENABLED=1 (needed for SQLite and Databricks Zerobus; this repo's build scripts already set it)git clone https://github.com/slingdata-io/sling-cli.git
cd sling-cli
bash scripts/build.sh
./sling --helpgit clone https://github.com/slingdata-io/sling-cli.git cd sling-cli .\scripts\build.ps1 .\sling --help
pip install sling
Then you should be able to run sling --help from command line.
Repository link: https://github.com/slingdata-io/sling-python
Here are the links of the official development builds, which are the latest builds of the upcoming release.
If you're using the python wrapper, decompress the downloaded binary and put the path in the environment variable SLING_BINARY before running your python program.
We welcome contributions to improve Sling! Here are some guidelines to help you get started.
Sling has three main test suites: Database, File and CLI. When contributing, please ensure that your changes pass the relevant tests.
To run the full test suite, run below. However you'd need to define all the needed connections as shown here, so it's recommended to target specific tests instead.
./scripts/build.sh ./scripts/test.sh
You can target specific tests or suites using environment variables:
Database Suite:
cd cmd/sling go test -v -run TestSuiteDatabasePostgres # run all Postgres tests go test -v -run TestSuiteDatabasePostgres -- 1-3 # run Postgres tests 1, 2, 3
File Suite:
cd cmd/sling go test -v -run TestSuiteFileS3 # run all S3 tests go test -v -run TestSuiteFileS3 -- 1,2,3 # run S3 tests 1, 2, 3
CLI Suite:
cd cmd/sling export SLING_BIN=./sling go test -v -run TestCLI # run all CLI tests go test -v -run TestCLI -- 31+ # run CLI tests 31 and all subsequent tests
You can specify individual test numbers, ranges, or use the '+' suffix to run all tests from a certain number:
go test -v -run TestCLI -- "1,2,3": Run tests 1, 2, and 3go test -v -run TestSuiteFileS3 -- "1-5": Run tests 1 through 5go test -v -run TestCLI -- "3+": Run test 3 and all subsequent testsDatabase Suite: Tests database-related functionality.
cmd/sling/sling_test.gotests/suite.db.template.tsvFile Suite: Tests file system operations.
cmd/sling/sling_test.gotests/suite.file.template.tsvCLI Suite: Tests command-line interface functionality.
cmd/sling/sling_cli_test.gotests/suite.cli.yamlWhen introducing new features or addressing bugs, it's essential to incorporate relevant tests, focusing mainly on the CLI suite file located at cmd/sling/suite.cli.yaml. The database and file suites serve as templates applicable across all connectors, making them more sensitive to modifications. Therefore, any changes to these suites will be managed internally.
When adding new test entries in the CLI suite file, feel free to create a new replication file in folder tests/replications, or a corresponding source file in the tests/files directory. Also include the expected output or the number of expected rows/streams in the new test entry.
Thank you for contributing to Sling!