The GitOps workflow to manage Kubernetes applications at any scale

Preface

  • Two-way synchronization (docker registry, config-repository)
  • Out-of-sync detection
  • You don’t need to run additional software on your cluster.
  • You can easily reproduce the state on your local machine.
  • The CI/CD lifecycle is sequential. (Push based pipeline)
  • No coupling to specific CD solutions. Tools can be replaced like lego.

Project structure

We consider each directory as a separate repository. That should reflect a real-world scenario with multiple applications.

├── config-repository (contains all kubernetes manifests)
│ ├── .release
│ │ ├── umbrella-chart
│ │ └── state.yaml
│ ├── app-locks
│ │ └── demo-service.kbld.lock.yml
│ ├── umbrella-chart
│ │ ├── charts
│ │ │ ├── demo-service
│ │ ├── Chart.lock
│ │ ├── Chart.yaml
│ │ └── values.yaml
├── demo-service-repository (example application)
│ ├── build.sh
│ ├── Dockerfile
│ └── kbld.yaml

Prerequisites

  • helm — Package manager
  • kbld — Image building and image pushing
  • kapp — Deployment tool
  • kpt — Fetch, update, and sync configuration files using git
  • kubeval (optional) — Validate your Kubernetes configuration files
  • kube-score (optional) — Static code analysis
  • sops (optional) — Secret encryption

Helm introduction

Helm is the package manager for Kubernetes. It provides an interface to manage chart dependencies. Helm guaranteed reproducible builds if you are working with the same helm values. Because all files are checked into git we can reproduce the helm templates at any commit.

Dependency Management

  • helm dependency build - Rebuild the charts/ directory based on the Chart.lock file
  • helm dependency list - List the dependencies for the given chart
  • helm dependency update - Update charts/ based on the contents of Chart.yaml

Chart distribution

Chart Repository

In big teams sharing charts can be exhausting tasks. In that situation, you should think about a solution to host your own Chart Repository. You can use chartmuseum.

S3

The simpler approach is to host your charts on S3 and use the helm plugin S3 to make them manageable with the helm cli.

kpt

There is another very interesting approach to share charts or configurations in general. Google has developed a tool called kpt. One of the features is to sync arbitrary files/subdirectories from a git repository. You can even merge upstream updates. This makes it very easy to share files across teams without working in multiple repositories at the same time. The solution would be to fetch a list of chart repositories and store them to umbrella/charts/ and call helm build. Your Chart.yaml dependencies must be prefixed with file://.

# fetch team B order-service subdirectory
kpt pkg get https://github.com/myorg/charts/order-service@VERSION \
umbrella-chart/charts/order-service
# lock dependencies
helm build
# make changes, merge changes and tag that version in the remote repository
kpt pkg update umbrella-chart/charts/order-service@gNEW_VERSION --strategy=resource-merge

Distribute configurations with containers

With kpt fn you can generate, transform, and validate configuration files from images, starlark scripts, or binary executables. The command below will provide DIR/ as an input to a container instance of gcr.io/example.com/my-fn executing the function in it and store the output in charts/order-service. This has great potential to align your tooling with containers.

# run a function using explicit sources and sinks
kpt fn source DIR/ |
kpt fn run --image gcr.io/example.com/my-fn |
kpt fn sink charts/order-service/

Advanced templating

Sometimes helm is not enough. This can have several reasons:

  • You want to keep base charts simple.
  • You want to abstract environments.
# this approach allows you to patch specific files because file stucture is preserved
helm template my-app ./umbrella-chart --output-dir .release
# this requires a local kustomize.yaml
kustomize build .release
# or with ytt, this will template all files and update the original files
helm template my-app ./umbrella-chart --output-dir .release
ytt -f .release --ignore-unknown-comments --output-files .release

✔️ Helm solves:

  • Build an application composed of multiple components.
  • Manage dependencies.
  • Distribute configurations.

The application repository

If you practice CI you will test, build and deploy new images continuously in your CI. Every build produces an immutable image tag that must be replaced in your helm manifests. In order to automate and standardize this process, we use kbld. kbld handles the workflow for building and pushing images. In your pipeline you need to run:

./demo-service-repository/build.sh

Define your application images

Before we can build images, we must create some sources and image destinations so that kbld is able to know which images belong to your application. They are managed in the application repository demo-service-repository/kbld.yaml. They look like CRD's but they aren't applied to your cluster.

The config repository

The directory config-repository/.release refers to the temporary desired state of your cluster. It's generated on your CI pipeline. The folder contains all kubernetes manifest files.

Release snapshot

This command will prerender your umbrella chart to config-repository/.release/state.yaml, builds and push all necessary images and replace all image references in your manifests. It's important to note that no image is built in this step. We reuse all prerendered images references from app-locks. The result is a snapshot of your desired cluster state at a particular commit. The CD pipeline will deploy it straight to your cluster.

$ ./config-repository/render.sh

✔️ kbld solves:

  • One way to build, tag and push images.
  • Agnostic to how manifests are generated.
  • Desired system state versioned in Git.
  • Every commit points to a specific image configuration of all maintained applications.

Deployment

We use kapp to deploy our resources to kubernetes. Kapp ensures that all resources are properly installed in the right order. It provides an enhanced interface to understand what has really changed in your cluster. If you want to learn more you should check the homepage.

$ ./config-repository/deploy.sh

Clean up resources

If you need to delete your app. You only need to call:

$ ./config-repository/delete.sh

✔️ kapp solves:

  • One way to diffing, labeling, deployment and deletion
  • Agnostic to how manifests are generated.

Environment Management

In order to manage multiple environments like development and staging, you can create different branches. Every branch has a different set of image references (stored in config-repository/app-locks) and values for your helm charts.

Secret Management

sops

You can use sops to encrypt yaml files. The files must be encrypted before they are distributed in helm charts. In the deployment process, you can decrypt them with a single command. Sops support several KMS services (Hashicorp Vault, AWS Secrets Manager, etc).

# As a chart maintainer I can encrypt my secrets with:
find ./.release -name "*secret*" -exec sops -e -i {} \;
# Before deployment I will decrypt my secrets so kubernetes can read them.
kapp deploy -n default -a my-app -f <(sops -d ./.umbrella-state/state.yaml)

Rollback / Releasing

The big strength of GitOps is that any commit represent a releasable version of your infrastructure setup.

Controller

Where kubernetes controller really show its strength is locality. They are deployed in your cluster and are protected by their environment. We can use that fact and deploy a controller like secretgen-controller which is responsible to generate secrets on the cluster. secretgen-controller works with CRD's. In that way, the procedure to generate the secret is stored in git but you can't run into the situation where you accidentally commit your password. You will never touch the secret.

Closing words

Demo

Check out the demo to see how it looks like.

More

References

Fullstack Engineer specialized in Web and Distributed Systems. Cloud-Native Applications | DevOps | CI-CD | Test Automation