gem-hunter
Language: Go
Tool for easy backup and restore for ClickHouse® using object storage for backup files.
A tool for easy backup and restore utility for ClickHouse databases with support for many cloud and non-cloud storage types.
During backup and restore data, clickhouse-backup requires access to the same files as clickhouse-server in /var/lib/clickhouse folders.
For that reason, it's required to run clickhouse-backup on the same host or same Kubernetes Pod or the neighbor container on the same host where clickhouse-server ran.
WARNING You can backup and restore only schema when connect to remote clickhouse-server hosts.
rclone, kopia, restic, rsync etcSETTINGS disk = disk(...) (ClickHouse 23.2+, tested from 24.8)clickhouse-server 22.7+ and USE_EMBEDDED_BACKUP_RESTORE=true)Altinity Backup for ClickHouse is a community effort sponsored by Altinity. The best way to reach us or ask questions is:
Altinity is the primary maintainer of clickhouse-backup. We offer a range of software and services related to ClickHouse.
Download the latest binary from the releases page and decompress with:
tar -zxvf clickhouse-backup.tar.gz
Use the official tiny Docker image and run it on a host with clickhouse-server installed:
docker run -u $(id -u clickhouse) --rm -it --network host -v "/var/lib/clickhouse:/var/lib/clickhouse" \ -e CLICKHOUSE_PASSWORD="password" \ -e S3_BUCKET="clickhouse-backup" \ -e S3_ACCESS_KEY="access_key" \ -e S3_SECRET_KEY="secret" \ altinity/clickhouse-backup --help
Build from the sources (required go 1.21+):
GO111MODULE=on go install github.com/Altinity/clickhouse-backup/v2/cmd/clickhouse-backup@latest
Data files are immutable in the clickhouse-server.
During a backup operation, clickhouse-backup creates file system hard links to existing clickhouse-server data parts via executing the ALTER TABLE ... FREEZE query.
During the restore operation, clickhouse-backup copies the hard links to the detached folder and executes the ALTER TABLE ... ATTACH PART query for each data part and each table in the backup.
A more detailed description is available here: https://www.youtube.com/watch?v=megsNh9Q-dw
One-shot CLI commands (create, upload, download, restore, delete, create_remote, restore_remote, watch, ...):
SIGINT (Ctrl+C) or SIGTERM cancels the running command: it unwinds, removes the shadow directories it froze (FREEZE ... WITH NAME <uuid>) and exits with a non-zero code; a create interrupted this way keeps its incomplete local backup directory, clean_local_broken or backups_to_keep_local retention removes itSIGINT/SIGTERM exits immediately without waiting for the cleanup to finishSIGKILL (including OOM kill and pod eviction) can't be handled, the frozen shadow of the table processed at that moment stays behind; every FREEZE is recorded in <backup_name>/freezes.tmp before it is executed, so the next clean, delete local, clean_local_broken or retention run unfreezes it, see clean for detailsserver mode:
SIGTERM cancels all running commands (same as POST /backup/kill for each of them), removes their pid files and stops the API server; in Kubernetes make sure terminationGracePeriodSeconds covers the shadow cleanup of a big table, otherwise the following SIGKILL leaves it for the next cleanSIGHUP reloads the config and restarts the API server, running commands are canceled the same way as via POST /restartAll log messages go to stderr. stdout only carries the command output (list, tables, print-config, ...), so it can be piped to other tools, e.g. clickhouse-backup list remote --format json | jq.
To keep the log in a file, redirect stderr as well:
clickhouse-backup create_remote my_backup >> clickhouse-backup.log 2>&1
docker logs and kubectl logs show only the output of the container's main process, e.g. clickhouse-backup server in a sidecar container.
A command started with docker exec or kubectl exec writes its log to the exec session, so it doesn't appear in the container log.
To get it there, run the command through the API, e.g. POST /backup/create_remote or INSERT INTO system.backup_actions: the server executes it in its own process.
Use log_level / LOG_LEVEL to change the verbosity.
By default, the config file is located at /etc/clickhouse-backup/config.yml, but it can be redefined via the CLICKHOUSE_BACKUP_CONFIG environment variable or via --config command line parameter.
All options can be overwritten via environment variables.
Use clickhouse-backup default-config to print the default config.
Use clickhouse-backup print-config to print the current config.
Environment variables can override each config parameter defined in the config file. Their names should be UPPERCASE, and exact names are provided after the comment character #.
The following values are not defaults; they explain what each config parameter with an example.
general: remote_storage: none # REMOTE_STORAGE, choice from: `azblob`,`gcs`,`s3`, etc; if `none` then `upload` and `download` commands will fail. max_file_size: 1073741824 # MAX_FILE_SIZE, 1G by default, useless when upload_by_part is true, use to split data parts files by archives backups_to_keep_local: 0 # BACKUPS_TO_KEEP_LOCAL, how many latest local backup should be kept, 0 means all created backups will be stored on local disk # -1 means backup will keep after `create` but will delete after `create_remote` command # You can run `clickhouse-backup delete local <backup_name>` command to remove temporary backup files from the local disk backups_to_keep_remote: 0 # BACKUPS_TO_KEEP_REMOTE, how many latest backup should be kept on remote storage, 0 means all uploaded backups will be stored on remote storage. # If old backups are required for newer incremental backup then it won't be deleted. Be careful with long incremental backup sequences. # Backups are ordered by `creation_date` from backup metadata (upload date is used only for backups with unreadable metadata), so `rebase` doesn't change the backup position in the retention order. rebase_before_remove_old_remote: false # REBASE_BEFORE_REMOVE_OLD_REMOTE, makes `backups_to_keep_remote` a strict limit: when deletion is blocked by `required_backup` links from kept backups, # the oldest kept increment is rebased first (same as the `rebase` command, requires `upload_by_part: true` and the same `compression_format` for the whole chain), # so the whole out-of-window chain becomes deletable; rebase failure is not fatal and falls back to the legacy keep-required behavior. rebase_during_delete: false # REBASE_DURING_DELETE, `delete remote <backup_name>` refuses to delete a backup which other backups require via `required_backup`, # when `true` every dependent increment is rebased first (same as the `rebase` command, requires `upload_by_part: true` and the same `compression_format` for the whole chain), # so the chain stays restorable and the backup becomes deletable; rebase copies data, so deletion time grows with the copied size, rebase failure is fatal and the backup is not deleted. # `delete local|remote --force` skips both the check and the rebase. log_level: info # LOG_LEVEL, a choice from `debug`, `info`, `warning`, `error` disable_environment_override: false # can be set ONLY in the config file (has no environment variable name on purpose); when `true` config values come only from the config file, # all environment variables and the `--env` CLI flag are ignored during config loading; # protects against accidental overrides such as Kubernetes service-discovery variables (a `clickhouse` Service in the same namespace exports `CLICKHOUSE_PORT=tcp://...`) allow_empty_backups: false # ALLOW_EMPTY_BACKUPS # Concurrency means parallel tables and parallel parts inside tables # For example, 4 means max 4 parallel tables and 4 parallel parts inside one table, so equals 16 concurrent streams download_concurrency: 1 # DOWNLOAD_CONCURRENCY, max 255, by default, the value is floor(AVAILABLE_CPU_CORES / 2). If result is < 1, then 1. download_transfer_concurrency: 0 # DOWNLOAD_TRANSFER_CONCURRENCY, global limit of concurrent part/archive transfers of one `download`, shared by all tables, so the last remaining table can use the slots released by finished tables instead of draining with only `download_concurrency` streams, 0 means download_concurrency^2 (the same maximum number of connections as the nested table/part limits), a negative value restores the legacy per-table limit download_table_order: largest_first # DOWNLOAD_TABLE_ORDER, `largest_first` or `metadata`, order in which `download` dispatches tables; `largest_first` starts big tables first, so they are not left to drain alone at the end of the download, only the dispatch order changes, restore order is not affected upload_concurrency: 1 # UPLOAD_CONCURRENCY, max 255, by default, the value is round(sqrt(AVAILABLE_CPU_CORES / 2)). If result is < 1, then 1. file_transfer_concurrency: 1 # FILE_TRANSFER_CONCURRENCY, max concurrent file streams inside one data part, applies only to `compression_format: none` (with archive formats one part is one stream), 1 means sequential transfer of part files. Extra streams only take free slots of the global transfer budget (`download_transfer_concurrency` for download, upload_concurrency^2 for upload) and never block, so the total number of connections never grows, they use slots left idle by other parts, mostly at the tail of a transfer with high-latency remote storage. Disabled for download when `download_transfer_concurrency` is negative. Try raising `upload_concurrency` / `download_concurrency` first, values above 8 rarely help rebase_concurrency: 1 # REBASE_CONCURRENCY, max 255, how many tables process in parallel during `rebase` command and how many CopyObject requests run in parallel inside each table, so up to rebase_concurrency^2 copies in flight, by default the same as download_concurrency. # Throttling speed for upload and download, enforced inline via a token-bucket rate limiter shared across all concurrent workers, so the value is an aggregate cap with smooth (non-bursty) throughput. # Throttling does NOT apply to server-side object disk copy (`CopyObject`, e.g. S3 server-side copy / Azure copy-blob), because those bytes move inside the cloud provider and never pass through clickhouse-backup. When server-side copy is unavailable (incompatible src/dst `remote_storage`, or a failed `CopyObject` falling back to streaming through local memory), the streaming copy IS throttled. # Throttling does NOT apply to `remote_storage: custom`, because data transfer happens inside your external upload/download command, not through clickhouse-backup. # When `clickhouse->use_embedded_backup_restore: true`, throttling is delegated to the ClickHouse server via the `max_backup_bandwidth` query setting passed in the BACKUP/RESTORE SETTINGS clause (requires ClickHouse 25.1+); upload_max_bytes_per_second applies to BACKUP, download_max_bytes_per_second to RESTORE. On older ClickHouse versions embedded transfers are not throttled. download_max_bytes_per_second: 0 # DOWNLOAD_MAX_BYTES_PER_SECOND, 0 means no throttling upload_max_bytes_per_second: 0 # UPLOAD_MAX_BYTES_PER_SECOND, 0 means no throttling allow_missing_files_on_download: false # ALLOW_MISSING_FILES_ON_DOWNLOAD, salvage mode for partially corrupted remote backups: `download` and `restore_remote` skip data parts whose files are missing on remote storage (404/NoSuchKey/BlobNotFound) with an `error` log and drop them from local table metadata instead of failing, metadata files are never skipped, requires `upload_by_part: true`, `--allow-missing-files` CLI argument or `allow_missing_files` API parameter overrides it per command, see https://github.com/Altinity/clickhouse-backup/issues/1456 download_disk_limit: 0 # DOWNLOAD_DISK_LIMIT, refuse `download` and `restore_remote` when usage of any local disk would exceed this percent (1..100) after download, 0 means no limit, `--disk-limit` CLI argument or `disk_limit` API parameter overrides it per command, see https://github.com/Altinity/clickhouse-backup/issues/1458 # MAX_BROKEN_PART_RATIO, maximum allowed fraction (0..1) of broken data parts (e.g. caused by S3-disk or filesystem failures) that still produces a successful but partial backup during backup creation (`create`, and the create stage of `create_remote`). # 0 (default) preserves legacy behavior where any broken part stops the backup completely. When >0 and the broken/total part ratio stays at or below this value, creation skips the broken parts, logs a warning, and the backup is marked successful. # Skipped parts are recorded in the `broken_parts` section of the table metadata json (next to `parts`) and counted in the `clickhouse_backup_failed_parts_count` prometheus metric in server mode, see https://github.com/Altinity/clickhouse-backup/issues/1418 max_broken_part_ratio: 0 # MAX_BROKEN_PART_RATIO # Buffer tuning for high-bandwidth (10Gbit+) networks, see https://github.com/Altinity/clickhouse-backup/issues/1376 and Examples.md#tuning-for-high-bandwidth-10gbit-networks pipe_buffer_size: 131072 # PIPE_BUFFER_SIZE, size in bytes of the in-memory ring buffer between the compression and the upload/download stream handlers, default 128KB; raise (e.g. 8388608 = 8MB) to let compression run ahead of uploads on fast networks download_copy_buffer_size: 0 # DOWNLOAD_COPY_BUFFER_SIZE, explicit buffer size in bytes for io.CopyBuffer during download/extract, 0 means use the Go default (32KB); raise (e.g. 1048576 = 1MB) to reduce syscalls per file on fast networks # zstd/gzip compression tuning, see https://github.com/Altinity/clickhouse-backup/issues/1378 and Examples.md#multi-threaded-zstdgzip-compression compression_use_multi_thread: true # COMPRESSION_USE_MULTI_THREAD, enable per-stream multi-threaded zstd/gzip compression and decompression; default true to match pre-1378 behavior (gzip always used pgzip). A single large table dominating a backup is gated by per-stream compression speed and gets no benefit from upload_concurrency/download_concurrency. Set false to save CPU when many tables upload in parallel. Only affects compression_format: zstd and gzip (silently ignored for other formats) compression_threads: 0 # COMPRESSION_THREADS, number of per-stream compression threads when compression_use_multi_thread is enabled (zstd concurrency / pgzip block workers), 0 means auto (GOMAXPROCS); must be unset/0 when compression_use_multi_thread is false compression_buffer_size: 0 # COMPRESSION_BUFFER_SIZE, compression buffer size in bytes, 0 keeps library defaults. Meaning and valid range depend on compression_format and compression_use_multi_thread: zstd = encoder window (power of two, 1024..536870912, e.g. 4194304 = 4MB); single-threaded gzip = DEFLATE window (32..32768); multi-threaded gzip = pgzip block size (>16384). Other formats reject it # when table data contains in system.disks with type=ObjectStorage, then we need execute remote copy object in object storage service provider, this parameter can restrict how many files will copied in parallel for each table object_disk_server_side_copy_concurrency: 32 # when CopyObject failure or object disk storage and backup destination have incompatible, will warning about possible high network traffic allow_object_disk_streaming: false # RESTORE_SCHEMA_ON_CLUSTER, execute all schema related SQL queries with `ON CLUSTER` clause as Distributed DDL. # Check `system.clusters` table for the correct cluster name, also `system.macros` can be used. # This isn't applicable when `use_embedded_backup_restore: true` restore_schema_on_cluster: "" upload_by_part: true # UPLOAD_BY_PART download_by_part: true # DOWNLOAD_BY_PART use_resumable_state: true # USE_RESUMABLE_STATE, allow resume upload and download according to the <backup_name>.resumable file. Resumable state is not supported for custom method in remote storage. # RESTORE_DATABASE_MAPPING, restore rules from backup databases to target databases, which is useful when changing destination database, all atomic tables will be created with new UUIDs. # The format for this env variable is "src_db1:target_db1,src_db2:target_db2". For YAML please continue using map syntax restore_database_mapping: {} # RESTORE_TABLE_MAPPING, restore rules from backup tables to target tables, which is useful when changing destination tables. # The format for this env variable is "src_table1:target_table1,src_table2:target_table2". For YAML please continue using map syntax restore_table_mapping: {} retries_on_failure: 3 # RETRIES_ON_FAILURE, how many times to retry after a failure during upload or download retries_pause: 5s # RETRIES_PAUSE, duration time to pause after each download or upload failure retries_jitter: 30 # RETRIES_JITTER, percent of RETRIES_PAUSE for jitter to avoid same time retries from parallel operations delete_batch_size: 1000 # DELETE_BATCH_SIZE, default batch size for bulk DeleteObjects() requests in remote storages that support batch delete (e.g. S3); upper bound for one API call, must be between 1 and 1000 for s3 # callback_url - CALLBACK_URL, optional HTTP endpoint notified with POST application/json when a backup command completes # (API, one-shot CLI commands, and each watch-loop iteration). API `?callback=` overrides this when non-empty. # Payload always includes status (success|error|cancel), error (empty string on success), operation_id, command and duration. # Read-only commands (list, tables, status, ...) never send a callback. # Callbacks are sent asynchronously; failures are logged and never change the backup command exit code / result. callback_url: "" callback_timeout: 5s # CALLBACK_TIMEOUT, max wait for the completion callback HTTP POST # status_history_size - STATUS_HISTORY_SIZE, how many finished operations are kept in the in-memory # status list exposed by `/backup/status` and `system.backup_actions`. `watch` records one operation # per iteration, so the history needs an upper bound. Operations still running are never dropped. status_history_size: 1000 watch_interval: 1h # WATCH_INTERVAL, use only for `watch` command, backup will create every 1h full_interval: 24h # FULL_INTERVAL, use only for `watch` command, full backup will create every 24h watch_backup_name_template: "shard{shard}-{type}-{time:20060102150405}" # WATCH_BACKUP_NAME_TEMPLATE, used only for `watch` command, macros values will apply from `system.macros` for time:XXX, look format in https://go.dev/src/time/format.go # watch_schedules - WATCH_SCHEDULES, use only for `watch` command, named cron driven backup chains, mutually exclusive with watch_interval/full_interval # `name` added as prefix to watch_backup_name_template to isolate backup chains, `full` and `increment` are cron expressions (standard 5 fields, optional leading seconds field, @every/@daily descriptors) # `full_type: rebase` creates scheduled full backup as increment + `rebase` command (server-side copy of previous chain instead of full re-upload) # `delete_previous_cycle: true` deletes all older backups of the chain after successful full backup # in WATCH_SCHEDULES env variable use `name=daily,full=0 0 * * *,increment=0 * * * *;name=weekly,full=@weekly` format, `;` as separator between schedules # watch_schedules: # - name: daily # full: "0 0 * * *" # increment: "0 * * * *" # full_type: create # delete_previous_cycle: false watch_schedules: [] sharded_operation_mode: none # SHARDED_OPERATION_MODE, how different replicas will shard backing up data for tables. Options are: none (no sharding), table (table granularity), database (database granularity), first-replica (on the lexicographically sorted first active replica). If left empty, then the "none" option will be set as default. cpu_nice_priority: 15 # CPU niceness priority, to allow throttling CPU intensive operation, more details https://manpages.ubuntu.com/manpages/xenial/man1/nice.1.html io_nice_priority: "idle" # IO niceness priority, to allow throttling DISK intensive operation, more details https://manpages.ubuntu.com/manpages/xenial/man1/ionice.1.html, on macOS mapped to setiopolicy_np(3): none=IOPOL_DEFAULT, realtime=IOPOL_IMPORTANT, best-effort=IOPOL_STANDARD, idle=IOPOL_THROTTLE rbac_backup_always: true # always backup RBAC objects rbac_restore_always: false # RBAC_RESTORE_ALWAYS, always restore RBAC objects during `restore` and `restore_remote` like `--rbac`, disabled by default cause it overwrites existing RBAC objects and restarts clickhouse-server rbac_conflict_resolution: "recreate" # RBAC_CONFLICT_RESOLUTION, action when RBAC object with the same name already exists during restore, allowed values "recreate", "ignore", "fail" config_backup_always: false # always backup CONFIGS, disabled by default cause configuration shall be manage via Infrastructure as Code approach config_restore_always: false # CONFIG_RESTORE_ALWAYS, always restore CONFIGS during `restore` and `restore_remote` like `--configs`, disabled by default cause it overwrites existing configuration files and restarts clickhouse-server named_collections_backup_always: false # always backup Named Collections, disabled by default cause configuration shall be manage via Infrastructure as Code approach named_collections_restore_always: false # NAMED_COLLECTIONS_RESTORE_ALWAYS, always restore Named Collections during `restore` and `restore_remote` like `--named-collections`, disabled by default cause it drops and re-creates existing SQL named collections with the same names clickhouse: username: default # CLICKHOUSE_USERNAME password: "" # CLICKHOUSE_PASSWORD host: localhost # CLICKHOUSE_HOST, To make backup data `clickhouse-backup` requires access to the same file system as clickhouse-server, so `host` should localhost or address of another docker container on the same machine, or IP address bound to some network interface on the same host. port: 9000 # CLICKHOUSE_PORT, don't use 8123, clickhouse-backup doesn't support HTTP protocol # CLICKHOUSE_DISK_MAPPING, use this mapping when your `system.disks` are different between the source and destination clusters during backup and restore process. # The format for this env variable is "disk_name1:disk_path1,disk_name2:disk_path2". For YAML please continue using map syntax. # If destination disk is different from source backup disk then you need to specify the destination disk in the config file: # disk_mapping: # disk_destination: /var/lib/clickhouse/disks/destination # `disk_destination` needs to be referenced in backup (source config), and all names from this map (`disk:path`) shall exist in `system.disks` on destination server. # During download of the backup from remote location (s3), if `name` is not present in `disk_mapping` (on the destination server config too) then `default` disk path will used for download. # `disk_mapping` is used to understand during download where downloaded parts shall be unpacked (which disk) on destination server and where to search for data parts directories during restore. # `disk_mapping` is also required when clickhouse-server is started with a relative `<path>./</path>` (ClickHouse 25.x and older then report relative paths in `system.disks`): # set `default` to the absolute clickhouse-server data path, it is used as the root to resolve the relative paths of all other disks, otherwise clickhouse-backup fails with an explicit error. disk_mapping: {} # CLICKHOUSE_SKIP_TABLES, the list of tables (pattern are allowed) which are ignored during backup and restore process # The format for this env variable is "pattern1,pattern2,pattern3". For YAML please continue using list syntax skip_tables: - system.* - INFORMATION_SCHEMA.* - information_schema.* # CLICKHOUSE_SKIP_TABLE_ENGINES, the list of tables engines which are ignored during backup, upload, download, restore process # The format for this env variable is "Engine1,Engine2,engine3". For YAML please continue using array syntax skip_table_engines: [] # CLICKHOUSE_SKIP_DISKS, list of disk names which are ignored during create, upload, download and restore command # The format for this env variable is "Disk1,disk2,disk3". For YAML please continue using array syntax skip_disks: [] # CLICKHOUSE_SKIP_DISK_TYPES, list of disk types which are ignored during create, upload, download and restore command # The format for this env variable is "Engine1,Engine2,engine3". For YAML please continue using array syntax skip_disk_types: [] timeout: 5m # CLICKHOUSE_TIMEOUT freeze_by_part: false # CLICKHOUSE_FREEZE_BY_PART, allow freezing by part instead of freezing the whole table freeze_by_part_where: "" # CLICKHOUSE_FREEZE_BY_PART_WHERE, allow parts filtering during freezing when freeze_by_part: true secure: false # CLICKHOUSE_SECURE, use TLS encryption for connection skip_verify: false # CLICKHOUSE_SKIP_VERIFY, skip certificate verification and allow potential certificate warnings sync_replicated_tables: true # CLICKHOUSE_SYNC_REPLICATED_TABLES tls_key: "" # CLICKHOUSE_TLS_KEY, filename with TLS key file tls_cert: "" # CLICKHOUSE_TLS_CERT, filename with TLS certificate file tls_ca: "" # CLICKHOUSE_TLS_CA, filename with TLS custom authority file log_sql_queries: true # CLICKHOUSE_LOG_SQL_QUERIES, logging `clickhouse-backup` SQL queries on `info` level, when true, `debug` level when false debug: false # CLICKHOUSE_DEBUG # CLICKHOUSE_FORCE_REBALANCE, when true, forces disk rebalancing during download even when the backup's disk # name exists on the target server. This is useful when restoring a backup taken on a single-disk machine # to a target with multiple JBOD disks under the same storage policy (e.g. both named "default"). # Without this option, all data lands on the built-in "default" disk because the rebalancing logic # only triggers when the disk name is missing. With force_rebalance, parts are distributed across # all disks in the matching storage policy using the "least_used" strategy. force_rebalance: false # CLICKHOUSE_FORCE_REBALANCE config_dir: "/etc/clickhouse-server" # CLICKHOUSE_CONFIG_DIR # CLICKHOUSE_KEEPER_IDENTITY, `user:password` for ClickHouse Keeper / ZooKeeper digest auth, used when backing up or restoring # `user_directories.replicated` RBAC objects and keeper-stored named collections. By default it is read from `<zookeeper><identity>` # in `preprocessed_configs/config.xml`; set it here when the server config hides the value (`hide_in_preprocessed="1"`, `from_env`) keeper_identity: "" # CLICKHOUSE_RESTART_COMMAND, use this command when restoring with --rbac, --rbac-only or --configs, --configs-only options # will split command by ; and execute one by one, all errors will logged and ignore # available prefixes # - sql: will execute SQL query # - exec: will execute command via shell restart_command: "exec:systemctl restart clickhouse-server" ignore_not_exists_error_during_freeze: true # CLICKHOUSE_IGNORE_NOT_EXISTS_ERROR_DURING_FREEZE, helps to avoid backup failures when running frequent CREATE / DROP tables and databases during backup, `clickhouse-backup` will ignore `code: 60` and `code: 81` errors during execution of `ALTER TABLE ... FREEZE` check_replicas_before_attach: true # CLICKHOUSE_CHECK_REPLICAS_BEFORE_ATTACH, helps avoiding concurrent ATTACH PART execution when restoring ReplicatedMergeTree tables default_replica_path: "/clickhouse/tables/{cluster}/{shard}/{database}/{table}" # CLICKHOUSE_DEFAULT_REPLICA_PATH, will use during restore Replicated tables without macros in replication_path if replica already exists,