gem-hunter
Language: Python
A Django/Postgres/Redis Kubernetes Template
This is a tutorial repo associated with a serise of blog posts on Medium.com. It demonstrates how to run Django, Postgres, and Redis all on a Kubernetes cluster using either Minikube (locally) or Google Container Engine.
If you're following along part 1 or part 2 of the tutorial, check out the part1 git branch
(viewable here ) which is an older
branch before the Minikube changes.
Since this project demonstrates deploying Postgres and Redis to the cluster, it's slightly involved. For a simpler example of Django on Container Engine/Kubernetes, try
https://github.com/GoogleCloudPlatform/python-docs-samples/tree/master/container_engine/django_tutorial
which also deploys Django to Kubernetes but uses a CloudSQL managed MySQL database, no cache, no secrets, and does not demonstrate autoscaling.
While this project is written for Google Container Engine and Minikube, these instructions should work on other Kubernetes platforms with some adjustments and should also deploy on other Kubernetes providers besides Google Container. Specifically, cluster creation steps, disks, load balancers, cloud storage options, and node autoscalers should get replaced by their equivalents on your platform.
While this repo focus on Google Container Engine and Minikube, these instructions should work on other Kubernetes platforms with some adjustments. Specifically, cluster creation steps, disks, load balancers, cloud storage options, and node autoscalers should get replaced by their equivalents on your platform.
Please submit any code or doc issues to the issue tracker!
Originally, this project focused only on how to run the project on Container Engine. As a followup, instructions for how to run the project on a local Kubernetes cluster using minikube have been added. Minikube has several advantages. It's free, it works offline, but it still emulates a local Kubernetes cluster.
Since Minikube fully emulates a Kubernetes cluster, only a few small changes need to be made when deploying the project. Notably:
The PostgreSQL Deployment requires a Persistent Volume Claim. The Persistent Volume that is bound to is different. For Container Engine, it's bound to a GCE Persistent Disk. For Minikube, it's bound to a directory on your local machine (hostMount)
Images for Container Engine are stored on Google Container Registry. Minikube
is not easily able to authenticate and pull these images, so instead, you use
eval $(minikube docker-env) to share the Docker daemon with Minikube. Images
you build with docker build will then be available to Minkube. Note that
the default imagePullPolicy will be 'Always' for any images without tags,
so imagePullPolicy has been explicitly set to IfNotPresent for all the images.
We use jinja2 CLI to template the configs, allowing Minikube or GKE specific parts to be included conditionally.
will generate the GKE configs, see gke_jinja.json for example input parameters.
Several commands listed below are provided in simpler form via the Makefile. Many of them use the GCLOUD_PROJECT environment variable, which will be picked up from your gcloud config. Make sure you set this to the correct project,
gcloud config set project <your-project-id>
There are more Makefiles in sub-directories to help build and push specific images.
This project uses the jinja 2 CLI to share templates between the GKE config and Minikube config.
Run:
pip install -r requirements-dev.txt
to install the CLI. At that point, you can see minikube_jinja.json and
gke_jinja.json as examples of variables you need to poplate to generate the
templates.
make template
will use the json variables to create the templates.
Install Docker.
Create a project in the Google Cloud Platform Console.
Enable billing for your project.
Enable APIs for your project. The provided link will enable all necessary APIs, but if you wish to do so manually you will need Compute, Datastore, Pub/Sub, Storage, and Logging. Note: enabling the APIs can take a few minutes.
If on OSX or Linux then install the Google Cloud SDK:
curl https://sdk.cloud.google.com | bash
or if on Windows then use the Google Cloud Shell (because kubectl doesn't work yet on Windows) which you can start from the Google Cloud Platform Console.
(Re-)Initialise the settings to set the compute zone:
gcloud init
Authenticate the CLI:
gcloud auth application-default login
Create a cluster for the bookshelf application
gcloud container clusters create guestbook --scopes "https://www.googleapis.com/auth/userinfo.email","cloud-platform" --num-nodes 2
gcloud container clusters get-credentials guestbook
The get-credentials commands initializes the kubectl CLI tool with the cluster you just created.
Alternatively, you can use the Makefile:
make create-cluster
Please see the minikube project for installation instructions. Note that if you're using Docker for Mac, you should specify the correct driver when you start Minikube.
minikube start --vm-driver=xhyve
Or set this permamently with:
minikube config set vm-driver xhyve
The --num-nodes flag in the cluster create specifies how many instances are created. Container Engine orchestrator is free up to 5 instances, but you will pay for the instances themselves, so to minimize costs just create 1 node.
At the end of the tutorial, run
gcloud container clusters delete guestbook
gcloud compute disks delete pg-data
or
make delete
To delete the cluster and not get charged for continued use. Deleting resources you are not using is especially important if you run the autoscaling example to create many instances.
The Django app depends on a PostgreSQL and Redis service. While this README explains how to deploy those services within the Kubernetes cluster. Looking in mysite/settings.py, you can see the app looks for several environment variables.
The first environment variable NODB, if set to 1, uses a SQLite database and an in-memory cache, allowing the app to
run without connecting to a real database. This is useful to test the app locally without Postgres/Redis, or deploy it
to Kubernetes without Postgres/Redis. Instead, it will use a local SQLite file and in-memory cache. In the Kubernetes
cluster, each container will have its own SQLite database and memory-cache, so the persistence and cache storage of the
values will not be shared between containers so will not be right. The NODB setting is just to help debug
incrementally and should be turned off.
The Jinja templates also contain a "has_db" variable which optionally attaches the database password secrets to the guestbook pod. If NODB is eanbled, make sure your templates are generated with "has_db" set to false, and set it to true otherwise.
Enter the guestbook directory
cd guestbook
First make sure you have Django installed. It's recommended you do so in a virtualenv. The requirements.txt contains just the Django dependency.
pip install -r requirements.txt
The app can be run locally the same way as any other Django app.
# disable Postgres and Redis until we set them up
export NODB=1
python manage.py runserver
Within the Dockerfile, NODB is turned on or off. Once you have deployed PostgreSQL or Redis, you can disable this
flag. If you deploy those services within Kubernetes, the environment variables will be automatically populated.
Otherwise you should set the environment variables manually using ENV in the Dockerfile.
Before the application can be deployed to Container Engine, you will need to build the image:
cd guestbook
# Make sure NODB is enabled and set to 1 in the Dockerfile if you still haven't setup Postgres and REDIS.
export GOOGLE_CLOUD_PROJECT=$(gcloud config list project --format="value(core.project)")
docker build -t gcr.io/$GOOGLE_CLOUD_PROJECT/guestbook .
For GKE, you would then push the image to Google Container Registry.
gcloud docker push gcr.io/$GCLOUD_PROJECT/guestbook
or alternatively:
make push
For Minikube, you don't push the image. Instead, simply make sure that when you build the image, you are using the Minikube docker daemon:
$ eval $(minikube docker-env)
# docker build -t gcr.io/$GOOGLE_CLOUD_PROJECT/guestbook
Once the image is built, it can be deployed in a Kubernetes pod.
kubernetes_configs/guestbook/ contains the
Deployment templates to spin up Pods with this image, as well as a Service with
an external load balancer. However,
the frontend depends on secret passwords for the database, so before it's
deployed, a Kubernetes Secret resource with your database passwords must be created.
Remember to follow the above instructions in "Jinja Templates" to generate the following configs:
kubectl apply -f kubernetes_config/guestbook/guestbook_gke.yaml
or on Minikube:
kubectl apply -f kubernetes_config/guestbook/guestbook_minikube.yaml
Even if NODB is enabled, the frontend replication controller is configured to use the Secret volume, so it must
be created in your cluster first.
Kubernetes Secrets are used to store the database password.
They must be base64 encoded and store in kubernetes_configs/db.password.yaml. A template containing an example config
is created, but your actual Secrets should not be added to source control.
In order to get the base64 encoded password, use the base64 tool
echo mysecretpassword | base64
Then copy and paste that value into the appropriate part of the Secret config
in kubernetes_configs/postgres/postgres.yaml.jinja
In the Postgres and Guestbook deployments, these Secrets are mounted onto their pods if "has_db" is set to true in the Jinja variables. In their Dockerfile they are read into environment variables.
Once the resources are created, there should be 3 frontend pods on the cluster. To see the pods and ensure that
they are running:
kubectl get pods
To get more information about a pod, or dig into why it's having problem starting, try
kubectl describe pod <pod-id>
If the pods are not ready or if you see restarts, you can get the logs for a particular pod to figure out the issue:
kubectl logs <pod-id>
Once the pods are ready, you can get the public IP address of the load balancer:
kubectl get services frontend
You can then browse to the external IP address in your browser to see the bookshelf application.
Alternatively on Minikube, there is no external IP, so instead run:
minikube service
When you are ready to update the replication controller with a new image you built, the following command will do a rolling update
export GCLOUD_PROJECT=$(gcloud config list project --format="value(core.project)")
kubectl rolling-update frontend --image=gcr.io/${GCLOUD_PROJECT}/guestbook:latest
which can also be done with the make update command. If you encounter problems with the rolling-update, then
check the events:
kubectl get events
It can happen that the nodes don't have enough resources left. A manual scaling down of the replication controller can help (see above) or you can resize the cluster to have an additional node:
gcloud container clusters resize guestbook --size 3
Give it a few minutes to provision the node, then try the rolling update again.
Since the Redis cluster has no volumes or secrets, it's pretty easy to setup:
kubectl apply -f kubernetes_configs/redis/redis.yaml
This creates a redis-master read/write service and redis-slave service. The images used are configured to properly replicate from the master to the slaves.
There should only be one redis-master pod, so the replication controller configures 1 replicas. There can be many redis-slave pods, so if you want more you can do:
kubectl scale deployment redis-slave --replicas=5
PostgresSQL will need a disk backed by a Volume to run in Kubernetes. For this example, we create a persistent disk using a GCE persistent disk:
gcloud compute disks create pg-data --size 200GB
or
make disk
Edit kubernetes_configs/postgres/postgres.yaml.jinja volume name to match the
name of the disk you just created, if different.
For Postgres, the secrets need to get populated and a script to initialize the database needs to be added, so a image should be built:
cd kubernetes_configs/postgres/postgres_image
make build
make push
Create the directory to mount:
sudo mkdir /data/pv0001/
sudo chown $(whoami) /data/pv001
cd kubernetes_config/postgres/postgres_image
make build
make push
Finally, you should be able to create the PostgreSQL service and pod.
Again, remember to follow the above instructions in "Jinja Templates" to generate the following configs:
kubectl apply -f kubernetes_configs/postgres/postgres_gke.yaml
or on Minikube:
kubectl apply -f kubernetes_configs/postgres/postgres_minikube.yaml
Only one pod can read and write to a GCE disk, so the PostgreSQL replication controller is set to control 1 pod. Don't scale more instances of the Postgres pod.
Now that the database and redis service are created, you should rebuild the frontend guestbook image with NODB
disabled.
cd guestbook
# edit Dockerfile to comment out NODB
make build
make push
With this new image, the replication controller should be updated. One way is to do a rolling update
make update
This will safely spin down the old images and replace it with the new image. However, for development purposes it can be quicker to scale the controller to 0 and then back up.
kubectl scale --replicas=0 rc/frontend
kubectl scale --replicas=3 rc/frontend
Finally, the Django migrations must be run to create the table. This can also be accomplished with kubectl-exec, this
time with the frontend pod. However, make sure your frontend-pod is actually talking to the database. If you earlier
built the image with NODB, rebuild it with NODB commented out. Then you can run the migrations:
export FRONTEND_POD_NAME=$(kubectl get pods | grep frontend -m 1 | awk '{print $1}')
kubectl exec ${FRONTEND_POD_NAME} -- python /app/manage.py makemigrations
kubectl exec ${FRONTEND_POD_NAME} -- python /app/manage.py migrate
or:
make migrations
When DEBUG is enabled, Django can serve the files directly from that folder. Currently, the app is configured to serve static content from the files that way to simplify development. For production purposes, you should disable the DEBUG flag and serve static assets from a CDN.
The application uses Google Cloud Storage to store static content. You can alternatively use a CDN of your choice. Create a bucket for your project:
gsutil mb gs://<your-project-id>
gsutil defacl set public-read gs://<your-project-id>
or
make create-bucket
Collect all the static assets into the static/ folder.
python manage.py collectstatic
Upload it to CloudStorage using the gsutil rsync command
gsutil rsync -R static/ gs://<your-gcs-bucket>/static
Now your static content can be served from the following URL:
http://storage.googleapis.com/<your-gcs-bucket/static/
Change the STATIC_URL in mysite/settings.py to reflect this new URL by uncommenting
the appropriate line and replacing <your-cloud-bucket>
Please note that autoscaling is a beta feature.
load_test_image provides a super minimal load generator - it hits a CPU intensive endpoint in a loop with curl. First build the image:
cd load_testing_image
make build
make push
Then create the Replication Controller and scale some clients:
kubectl apply -f kubernetes_configs/load_tester.yaml
kubectl scale rc load --replicas=20
to generate load. Then
make autoscaling
will create both Node autoscaling (gcloud) and Pod autoscaling (Kubernetes Horizontal Pod Autoscaling).
Again, by default this will scale to 10 nodes whcih you will be charged for, so pleaes disable autoscaling and scale the nodes back down or delete the cluster if you don't want to pay for sustained use of the 10 instances.
For more sophisticated load testing with Kubernetes, see:
https://cloud.google.com/solutions/distributed-load-testing-using-kubernetes
Please use the Issue Tracker for any issues or questions.
For another popular Django/Kubernets project + blog post, see:
https://harishnarayanan.org/writing/kubernetes-django/
This project was originally made for a talk/blog post series.
https://speakerdeck.com/waprin/deploying-django-on-kubernetes
and you can watch the talk here:
https://www.youtube.com/watch?v=HKKUgWuIZro
This project is owned by Google Copyright 2016 but is not an official Google-product nor officially maintained or supported by Google.