gem-hunter

edb-deployment tool is an easy way to provision Cloud resources and deploy PostgreSQL, EDB Postgres Advanced Server and tools (high availability, backup/recovery, monitoring, connection poolers). edb-deployment can also be used to deploy Postgres architectures on existing infrastructure like physical servers (baremetal) or local Virtual Machines.

Supported Cloud providers are AWS, Azure and Google Cloud.

edb-deployment helps user to deploy Postgres Reference Architectures. List and details of the Reference Architecture can be find here.

edb-deployment is an open source tool and is not officially supported by EDB Support. It is maintained and supported by the GitHub members of this repository. Please provide feedback by posting issues and providing pull requests.

Before starting to delve into this repository, it is best to get familiar with the steps in the deployment process of a specific cloud (AWS, Azure and Google Cloud).

edb-deployment is dependent on following components. Install the following components before using it.

1. Python 3
2. pip3

Third party pre-requisites:

1. Latest vendor Cloud CLI or SDK ( AWS, Azure or Google Cloud )

   Depending on the cloud provider, install the latest version for: AWS CLI, Azure CLI or Google Cloud SDK on the system.

2. Terraform >= 1.3.6

3. Ansible >= 2.10.8

4. AWS CLI >= 2.0.45

5. Azure CLI >= 2.23.0

6. Google Cloud CLI >= 329.0.0

To help the installation of the third party pre-requisites listed above, edb-deployment provides the setup sub-command working for Linux and Darwin (macOS). Please refer to section Pre-Requisites installation.

Installation is done using the pip3 command. Once the code has been downloaded, either by cloning the repository or downloading a release, go to the created folder and run the command pip3 install:

$ cd postgres-deployment
$ sudo pip3 install . --upgrade

$ sudo pip3 install edb-deployment

Make sure the tool is well installed by running the command:

$ edb-deployment --version

edb-deployment supports command line auto-completion with the tab key.

Supported shells are bash ans zsh.

To enable auto-completion in current session, the following command must be ran:

$ eval "$(register-python-argcomplete edb-deployment)"

To enable auto-completion for all the sessions, the command above must be added at the end of your ~/.bashrc file or ~/.zshrc file, depending on the shell you use.

To ease installation of the third party pre-requisites tools like aws, terraform, ansible, etc.. edb-deployment provides the setup sub-command.

The following packages are required in order to execute the setup sub-command: gcc (Linux only), python3-devel (Linux only), unzip, wget, tar. These packages should be installed through usual package manager (dnf, apt, brew, etc..).

Note for Debian users: the package libffi-dev must be present.

Finally, Python virtualenv must be installed with root privileges:

$ sudo pip3 install virtualenv

Pre-requisites automated installation:

$ edb-deployment <CLOUD_VENDOR> setup

Each new deployment will be done under a dedicated name space, this the <PROJECT_NAME>.

edb-deployment CLI features are implemented through sub-commands. Each sub-command can be executed like this:

$ edb-deployment <CLOUD_VENDOR> <SUB_COMMAND> [<PROJECT_NAME>]

• aws: Amazon Web Services
• aws-pot: EDB POT (Proof Of Technology) on AWS Cloud
• aws-rds: Amazon Web Services RDS for PostgreSQL
• aws-rds-aurora: Amazon Aurora
• azure: Microsoft Azure Cloud
• azure-pot: EDB POT (Proof Of Technology) on Azure Cloud
• azure-db: Microsoft Azure Database
• gcloud: Google Cloud
• gcloud-pot: EDB POT (Proof Of Technology) on Google Cloud
• gcloud-sql: Google Cloud SQL for PostgreSQL

• baremetal: Baremetal servers and VMs
• vmware: VMWare Workstation
• virtualbox: Virtualbox
  • Windows Support through WSL2

• configure: New project initialization and configuration
• provision: Cloud resources provisioning
• destroy: Cloud resources destruction
• deploy: Postgres and tools deployment
• show: Show configuration
• display: Display project inventory
• passwords: Display project passwords
• list: List projects
• specs: Show Cloud Vendor default specifications
• logs: Show project logs
• remove: Remove project

Deployment of new project should follow the work flow below:

1. Configure Cloud credentials
2. Project configuration
3. Cloud resources provisioning
4. Postgres and tools deployment

This step depends on the target Cloud vendor.

If the Cloud tools have been installed with the help of the setup sub-command, it's recommended to update the value of the PATH environment variable to include tools binary location:

$ export PATH=$PATH:$HOME/.edb-cloud-tools/bin

AWS credentials configuration will be done through aws tool. For this step, we need to get your AWS Access Key ID and AWS Secret Access Key. For more information about Amazon Access Key management, please go to official documentation page.

Run the following command and enter the Access Key ID and Secret Access Key:

$ aws configure

Azure Cloud credentials configuration can be achieved using the az tool with the following command:

$ az login --use-device-code

GCloud credentials configuration includes more steps than the other Cloud vendors. GCloud project ID is required.

1. Login with your email address:

$ gcloud auth login <LOGIN_EMAIL> --no-launch-browser

2. Open the link in your browser and copy the given verification code.
3. Project configuration

$ gcloud config set project <PROJECT_ID>

4. To find the IAM account of the service, please enter the following command to list service accounts:

$ gcloud iam service-accounts list

5. Finally, to create and download a new service account key:

$ gcloud iam service-accounts keys create ~/accounts.json --iam-account=<IAM_ACCOUNT>

The JSON file $HOME/accounts.json must be kept and will be required by edb-deployment.

Once Cloud vendor credentials have been configured, you can proceed with project configuration step.

edb-deployment brings default configuration values for the Cloud resources to be provisioned, like instance type, disk size, OS image, additional volumes, etc..

To change these configuration values, you need first to dump the default values with:

$ edb-deployment <CLOUD_VENDOR> specs > my_configuration.json

When deploying on baremetal servers, IP address and SSH user configuration must be done through the specifications:

$ edb-deployment baremetal specs --reference-architecture EDB-RA-1 > baremetal-edb-ra-1.json

The baremetal-edb-ra-1.json file will contain:

{
  "ssh_user": null,
  "pg_data": null,
  "pg_wal": null,
  "postgres_server_1": {
    "name": "pg1",
    "public_ip": null,
    "private_ip": null
  },
  "pem_server_1": {
    "name": "pem1",
    "public_ip": null,
    "private_ip": null
  },
  "backup_server_1": {
    "name": "backup1",
    "public_ip": null,
    "private_ip": null
  }
}

Then, you can edit and update resources configuration stored in the JSON file.

Project initialialization will done using the configure sub-command:

$ edb-deployment <CLOUD_VENDOR> configure <PROJECT_NAME> \
  -a <REFERENCE_ARCHITECTURE_CODE> \
  -o <OPERATING_SYSTEM> \
  -t <PG_ENGINE_TYPE> \
  -v <PG_VERSION> \
  -u "<EDB_REPO_USERNAME>:<EDB_REPO_PASSWORD>" \
  -r <CLOUD_REGION> \
  -s my_configuration.json

Notes:

• REFERENCE_ARCHITECTURE_CODE

  Reference architecture code name. Allowed values are: EDB-RA-1 for a single Postgres node deployment with one backup server and one PEM monitoring server, EDB-RA-2 for a 3 Postgres nodes deployment with quorum base synchronous replication and automatic failover, one backup server and one PEM monitoring server, EDB-RA-3 for extending EDB-RA-2 with 3 PgPoolII nodes, and HammerDB-TPROC-C for setting up a 2-tier configuration for benchmarking with an OLTP workload. Default: EDB-RA-1

• OPERATING_SYSTEM

  Operating system. Allowed values are: CentOS7, RockyLinux8, RedHat7 and RedHat8. Default: RockyLinux8

• PG_ENGINE_TYPE

  Postgres engine type. Allowed values are: PG for PostgreSQL, EPAS for EDB Postgres Advanced Server. Default: PG

• PG_VERSION

  PostgreSQL or EPAS version. Allowed values are: 11, 12, 13 and 14. Default: 14

• "EDB_REPO_USERNAME:EDB_REPO_PASSWORD"

  EDB Packages repository credentials. Required.

• CLOUD_REGION

  Cloud vendor region. Default value depends on Cloud vendor.

For more details, please use:

$ edb-deployment <CLOUD_VENDOR> configure --help

After project configuration, we can proceed to Cloud resources provisioning:

$ edb-deployment <CLOUD_VENDOR> provision <PROJECT_NAME>

Finally, we can deploy the components with the deploy sub-command:

$ edb-deployment <CLOUD_VENDOR> deploy <PROJECT_NAME>

List of projects:

$ edb-deployment <CLOUD_VENDOR> list

Execute Ansible pre deployment playbook

$ edb-deployment <CLOUD_VENDOR> deploy --pre-deploy-ansible pre_deploy_playbook.yml <PROJECT_NAME>

Execute Ansible post deployment playbook

$ edb-deployment <CLOUD_VENDOR> deploy --post-deploy-ansible post_deploy_playbook.yml <PROJECT_NAME>

Display of projects inventory:

$ edb-deployment <CLOUD_VENDOR> display <PROJECT_NAME>

Display of projects passwords:

$ edb-deployment <CLOUD_VENDOR> passwords <PROJECT_NAME>

Cloud resources destruction:

$ edb-deployment <CLOUD_VENDOR> destroy <PROJECT_NAME>

Project tree deletion:

$ edb-deployment <CLOUD_VENDOR> remove <PROJECT_NAME>

Open SSH connection to one host of the project:

$ edb-deployment <CLOUD_VENDOR> ssh <PROJECT_NAME> <NODE_NAME>

Note: the node name list can be found by executing the display sub-command. It can be: epas1, epas2, pemserver1, etc...

Get a copy of the SSH keys and the ssh_config file. The files are copied into the current directory:

$ edb-deployment <CLOUD_VENDOR> get_ssh_keys <PROJECT_NAME>

Note: This sub-command is only available for aws-pot, azure-pot and gcloud-pot.

Original work Copyright 2019-2020, EnterpriseDB Corporation

All rights reserved.

Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:

1. Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.

2. Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.

3. Neither the name of EnterpriseDB nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission.

THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE.