PlatformEngr® Progressive delivery course

Module 00 · Foundations · 20 min · You are: Platform and product

Orientation: deploy is not release

In this module

  • You learn: Why "the pods are running" is not the same as "users should get this version", and the two ways to go slower: blue-green (a full copy, then a switch) and canary (a share of users first). Who owns which part.
  • You do: Tell the course who you are, check your tools, clone the reference repo, and check that your Argo CD platform from the Argo CD course is still healthy. Rebuild it if the cluster is gone. Then look at how a release goes out today.
  • You end with: Nothing new is installed. Your three repos and the cluster are as the Argo CD course left them, every Application is Synced and Healthy, and the reference repo is cloned next to your repos.

Argo CD made deploying safe to repeat: Git says what runs, and the cluster follows. It did not make a release safe. When checkout 6.8.0 has a bug, a Deployment replaces every pod with it as soon as the pods pass their readiness probe, and every user meets the bug at once. Progressive delivery separates the two moments: the new version runs first, and users get it a share at a time, while metrics decide whether it goes on.

Open this module in the interactive course to track progress and switch tracks.

Release path practised: Committed → Deployed → Exposed → Measured → Decided → Promoted → Rolled back

Concepts

Deploy vs release
Deploy: the new version runs in the cluster. Release: users get it. A Deployment does both at once; a Rollout puts steps between them.
Blue-green
Run the new version as a full second copy behind a preview address, test it, then switch all traffic at once. Switching back is instant while the old copy still runs.
Canary
Give the new version a small share of users (5%, 25%, 50%) and look at its errors and latency before each next step. A bad version hurts few users, briefly.
Analysis
A query against metrics (here Prometheus), asked again and again during a release, with a condition for success. A failed analysis aborts the release automatically.
Argo Rollouts
A Kubernetes controller and a Rollout resource: a Deployment with a release strategy. Argo CD deploys it from Git like any other resource and understands its health.

Who owns what in this course

PieceOwnerRepo
Argo Rollouts, Prometheus, Istio, the Gatewayplatform teamargocd-platform
Shared analysis templates (what "healthy" means)platform teamargocd-platform (rollouts/)
Who may promote or abort a releaseplatform teamargocd-platform (Argo CD RBAC)
The Rollout and the strategies it supportsteam-checkoutcheckout-app (the chart)
Which strategy, steps and version per environmentteam-checkoutcheckout-config (helm/)
Promote, abort, revert, the incident noteteam-checkoutcheckout-config (docs/incidents/)

Files to know

Open them in your editor or on GitHub; the bullets say what to look at.

Steps

1. Tell the course who you are

The same details as in the Argo CD course (this browser may still have them). Commands then use your GitHub user and your workspace folder.

Set your GitHub user and workspace folder in the interactive course or the form at the top of the notes index; these notes then show your values.

2. Where your cluster runs

The same cluster as in the Argo CD course. The kind track uses its cluster kind-argocd and the *.localtest.me hosts.

These notes show kind commands.

3. Check your tools

The Argo CD course tools, plus the Argo Rollouts plugin for kubectl and jq to read JSON. Paste the whole block; it changes nothing.

check() {  # tool, works-check, version, problem, where-to-get-it, optional
  if ! command -v "$1" >/dev/null 2>&1; then
    if [ -n "$6" ]; then printf '  optional  %-22s not installed (%s)\n' "$1" "$5"
    else printf '  MISSING   %-22s install it: %s\n' "$1" "$5"; bad=$((bad+1)); fi
  elif ! eval "$2" >/dev/null 2>&1; then printf '  PROBLEM   %-22s %s\n' "$1" "$4"; [ -n "$6" ] || bad=$((bad+1))
  else printf '  ok        %-22s %s\n' "$1" "$(eval "$3" 2>&1 | head -n 1)"; fi
}
bad=0
check 'git' 'git --version' 'git --version' 'installed, but it does not run' 'https://git-scm.com/downloads'
check 'docker' 'docker info' "docker version --format '{{.Server.Version}}'" 'installed, but the Docker daemon is not running: start Docker Desktop (or the docker service)' 'https://docs.docker.com/get-docker/'
check 'kind' 'kind version' 'kind version' 'installed, but it does not run' 'https://kind.sigs.k8s.io/docs/user/quick-start/#installation'
check 'kubectl' 'kubectl version --client' 'kubectl version --client' 'installed, but it does not run' 'https://kubernetes.io/docs/tasks/tools/'
check 'helm' "helm version --short | grep -q '^v4'" 'helm version --short' 'Helm 3 found: this course uses Helm 4' 'https://helm.sh/docs/intro/install/'
check 'argocd' 'argocd version --client' 'argocd version --client --short' 'installed, but it does not run' 'https://argo-cd.readthedocs.io/en/stable/cli_installation/'
check 'kubectl-argo-rollouts' 'kubectl argo rollouts version' 'kubectl argo rollouts version --short' 'installed, but kubectl cannot run it: put it on your PATH as kubectl-argo-rollouts' 'https://argo-rollouts.readthedocs.io/en/stable/installation/#kubectl-plugin-installation'
check 'jq' 'jq --version' 'jq --version' 'installed, but it does not run' 'https://jqlang.org/download/'
check 'curl' 'curl --version' 'curl --version' 'installed, but it does not run' 'your package manager (it ships with macOS and most Linux distributions)'
check 'perl' 'perl -v' "perl -e 'print qq(perl $^V)'" 'installed, but it does not run' 'your package manager (it ships with macOS and most Linux distributions)'
check 'python3' "python3 -c 'import yaml'" 'python3 --version' 'installed, but PyYAML is missing: python3 -m pip install --user pyyaml' 'https://www.python.org/downloads/'
if [ "$bad" -eq 0 ]; then echo "All set: every tool is ready."; else echo "$bad tool(s) need attention. Install or fix them, then run this check again."; fi

Every line should say ok. Install anything marked MISSING and run it again. kubectl-argo-rollouts is a kubectl plugin: once it is on your PATH, kubectl argo rollouts works.

4. Clone the reference repo

You copy files from it, as you did with argocd-gitops. It goes next to your three repos.

git clone -q https://github.com/platformengr/progressive-delivery ~/work/progressive-delivery

Expect your three repos and the two reference clones side by side:

ls ~/work

5. Is the Argo CD platform still there?

This course continues on the cluster and repos from the Argo CD course. Check both.

Is the cluster up? Expect one node, Ready:

kubectl get nodes

Log in as alice (the platform account from the Argo CD course). Expect 'alice:login' logged in successfully:

argocd login argocd.localtest.me:8443 --grpc-web --insecure --username alice --password 'choose-a-long-one'

Every Application should be Synced and Healthy: root, argocd, traefik, external-secrets, the folder apps, checkout-dev and checkout-prod (and the catalog apps if you did the capstone):

argocd app list

6. Only if your cluster is gone: rebuild it from your repos

Your repos hold everything except the values in the secret stores and the account passwords. Skip this step if the previous one showed a healthy platform.

1. A new cluster with the course ports:

kind create cluster --config ~/work/progressive-delivery/cluster/kind.yaml

2. Argo CD from your own values file, with the admin account switched on for a moment, so you can set the other passwords:

helm repo add argo https://argoproj.github.io/argo-helm && helm repo update argo
helm install argocd argo/argo-cd --version 10.11.6 --namespace argocd --create-namespace \
  -f ~/work/argocd-platform/argocd/values.yaml --set 'configs.cm.admin\.enabled=true' --wait

3. In a second terminal, a port-forward that stays running until step 5:

kubectl -n argocd port-forward svc/argocd-server 8080:80

4. Log in as admin and set the account passwords you used in the Argo CD course:

ADMIN="$(kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath='{.data.password}' | base64 -d)"
argocd login localhost:8080 --plaintext --username admin --password "$ADMIN"
argocd account update-password --account alice --current-password "$ADMIN" --new-password 'choose-a-long-one'
argocd account update-password --account checkout-dev --current-password "$ADMIN" --new-password 'another-long-one'

5. The root Application. From here Argo CD installs everything else from Git and switches admin off again. Stop the port-forward afterwards.

kubectl apply -n argocd -f ~/work/argocd-platform/bootstrap/root.yaml

6. The store values. Wait two minutes for the store namespaces, then put back the deploy key and one password per environment:

kubectl -n store-platform create secret generic repo-checkout-config --from-file=sshPrivateKey=~/work/keys/checkout-config
for env in dev prod; do
  kubectl -n store-checkout create secret generic db-$env --from-literal=password="$(openssl rand -hex 16)"
done

Log in as alice through the ingress (as in the previous step) and wait for argocd app list to show every app Synced and Healthy. It takes a few minutes.

7. How a release goes out today

Look at the strategy of the checkout Deployment. This is what you replace in this course.

Expect RollingUpdate with maxSurge and maxUnavailable at 25%: new pods replace old ones as soon as they are ready. Nothing looks at errors or latency, and nothing waits for a person.

kubectl -n checkout-dev get deployment checkout -o jsonpath='{.spec.strategy}'; echo

And the versions running now. Expect 6.7.1 in dev and 6.7.0 in prod, as the Argo CD course left them (yours may differ if you promoted more):

kubectl get deploy -A -l app.kubernetes.io/name=checkout -L app.kubernetes.io/version

Simplified for the course

The course pushes straight to main in all three repos, as the Argo CD course did from module 07. If your main is protected, open a pull request for each change instead: the commands before git push stay the same.

Quiz

  1. What does a Deployment's rolling update check before it sends users to a new pod?
    • The error rate of the new version, compared with the old one
    • Only that the pod passes its readiness probe
    • That a person has approved the new version in Argo CD
  2. What is the main difference between blue-green and canary?
    • Blue-green switches all traffic at once after testing a full copy; canary moves a share of users at a time
    • Blue-green needs a service mesh; canary works without one
    • Canary keeps two full copies running; blue-green runs only one
  3. Who decides which release strategy checkout uses in prod?
    • The platform team, in the Argo Rollouts values
    • Argo CD, from the health of the last release
    • team-checkout, in checkout-config
Answers
  1. Only that the pod passes its readiness probe Readiness says the pod can answer, not that its answers are right.
  2. Blue-green switches all traffic at once after testing a full copy; canary moves a share of users at a time Both run the new version first; they differ in how users move over.
  3. team-checkout, in checkout-config The platform provides the machinery; the product team decides how its own releases go out.

Field note

The question I ask every team that says "we deploy many times a day": how many users saw your last bad release before you knew? The honest answer is usually "all of them, for about ten minutes". This course is about making that number small without slowing the deploys down.

Report a problem with this module