gem-hunter

A scalable, lightweight object storage service.

Read this post on why we decided to build a new object storage service.

• Multi-protocol support (HTTP, TUS, S3, Iceberg)
• Uses Postgres as its datastore for storing metadata
• Authorization rules are written as Postgres Row Level Security policies
• Integrates with S3 Compatible Storages
• Extremely lightweight and performant

Supported Protocols

• HTTP/REST
• TUS Resumable Upload
• S3 Compatible API
• Iceberg REST Catalog

• OpenAPI Spec
• Storage Guides
• Client library

• Copy .env.sample to .env file.
• Copy .env.test.sample to .env.test.

cp .env.sample .env && cp .env.test.sample .env.test

Your root directory should now have both .env and .env.test files.

• Then run the following:

# this sets up a postgres database and postgrest locally via docker
npm run infra:restart
# Start the storage server
npm run dev

The server should now be running at http://localhost:5000/

The following request should insert and return the list of buckets.

# insert a bucket named avatars
curl --location --request POST 'http://localhost:5000/bucket' \
--header 'Authorization: Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJyb2xlIjoic2VydmljZV9yb2xlIiwiaWF0IjoxNjEzNTMxOTg1LCJleHAiOjE5MjkxMDc5ODV9.th84OKK0Iz8QchDyXZRrojmKSEZ-OuitQm_5DvLiSIc' \
--header 'Content-Type: application/json' \
--data-raw '{
    "name": "avatars"
}'

# get buckets
curl --location --request GET 'http://localhost:5000/bucket' \
--header 'Authorization: Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJyb2xlIjoic2VydmljZV9yb2xlIiwiaWF0IjoxNjEzNTMxOTg1LCJleHAiOjE5MjkxMDc5ODV9.th84OKK0Iz8QchDyXZRrojmKSEZ-OuitQm_5DvLiSIc'

To perform your tests you can run the following command: npm test

Biome formats and lints code, Markdown, and YAML.

• Check: npm run lint
• Automatic fix if possible: npm run lint:fix