Implementation guide

Prerequisites

Install plan

Each stage depends on the previous one. The phase folders in the repository follow the same order.

1. vault

   Deploy Vault with KMS auto-unseal and a public hostname.

2. identity provider

   Create the Vault login client, team groups and test identities.

3. vault access

   Initialize Vault; map identity groups to per-team policies; prove group-gated reads.

4. controller

   Give the controller its own vault and IdP identities; build and deploy it; run the acceptance test.

5. pull requests

   Register the ArgoCD project and ApplicationSet; protect the branch; create the approval label.

6. workload access

   Install External Secrets; create per-team vault roles; let ArgoCD place a store in every team namespace.

7. image

   Build the controller in CI, push to the registry, and pin the deployment to the image digest.

8. backstage

   Add the request template, team groups and client entities to the catalog; give Backstage a GitHub token and read access to OAuthClients.

1. Vault with auto-unseal

Create the unseal key and a narrowly scoped IAM user

The IAM user may only encrypt, decrypt and describe with this one key. Its access key goes straight into a Kubernetes Secret and is never printed.

aws kms create-key --description "Courier: Vault auto-unseal" --tags TagKey=project,TagValue=courier
aws kms create-alias --alias-name alias/courier-vault-unseal --target-key-id <key-id>
aws iam create-user --user-name courier-vault-unseal
aws iam put-user-policy --user-name courier-vault-unseal --policy-name vault-awskms-unseal \
  --policy-document '{"Version":"2012-10-17","Statement":[{"Effect":"Allow",
    "Action":["kms:Encrypt","kms:Decrypt","kms:DescribeKey"],"Resource":"<key-arn>"}]}'

# Secret keys: AWS_REGION, VAULT_AWSKMS_SEAL_KEY_ID, AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY
kubectl create namespace vault
kubectl -n vault create secret generic vault-awskms --from-env-file=/dev/stdin

Deploy the chart

Values live in deploy/phase0/vault-values.yaml: a single Raft replica on local disk, seal "awskms" {} reading its settings from the Secret, the UI behind the ingress, and an audit volume. Add it to your app-of-apps as an ArgoCD Application for chart vault version 0.34.1 from https://helm.releases.hashicorp.com.

Create DNS for the Vault hostname, pointing at the ingress. Use the current domain, not a legacy internal zone.

2. Identity provider

deploy/phase0/authentik-setup.sh

Runs inside the Authentik server pod, is safe to re-run, and creates:

• OAuth provider and application vault (confidential), with redirect URIs for the Vault UI and CLI and with grant types authorization_code, refresh_token and client_credentials
• Groups vault-admins, team-alpha and team-bravo, each bound to the application
• Service accounts courier-svc-alpha and courier-svc-bravo for automated tests
• Secrets vault/vault-oidc and vault/courier-phase0-testers

3. Vault access model

deploy/phase0/vault-bootstrap.sh   # init, auth methods, policies, identity groups, sample secrets
deploy/phase0/verify.sh            # must print ALL CHECKS PASSED

On first run Vault is initialized with recovery keys, and the keys and root token are written once to ~/courier-vault-init.json (mode 600). The script then configures:

• KV v2 at kv/ and a file audit device
• oidc/ for people (role human) and jwt/ for machines (role machine)
• Policies vault-admin, one per team, and courier (write-only)
• For each IdP group, two external vault groups: team-alpha aliased on oidc/ and team-alpha-jwt aliased on jwt/

Prove automatic unsealing: kubectl -n vault delete pod vault-0. It must return with Sealed false and no one entering keys.

4. Controller

deploy/phase1/authentik-controller-account.sh  # scoped IdP identity → Secret courier-system/courier-authentik
deploy/phase1/vault-kubernetes-auth.sh         # auth/kubernetes/role/courier → policy courier
deploy/phase1/deploy-controller.sh             # build image, import into k3s, apply CRD + RBAC + manager
deploy/phase1/acceptance.sh                    # must print PHASE 1 ACCEPTANCE PASSED

What the identities can do

Controller configuration is set by deploy/phase1/controller/manager-wiring.yaml; every flag is listed in the reference.

Optional developer check against the real systems: deploy/phase1/integration-test.sh.

5. Pull request workflow

Register with ArgoCD

Add one Application to your app-of-apps with source path deploy/argocd and destination namespace argocd. It creates:

Protect the repository

In GitHub, under Settings → Branches → rule for main:

• Require a pull request before merging, with at least one approval
• Require review from Code Owners
• Require status checks to pass: Validate requests
• Do not allow bypassing the above settings

Create the label security-approved, and restrict who can apply it to the identity or security team. Replace the users in .github/CODEOWNERS with your GitHub teams.

Onboard a team

1. identity provider

   Create the group (for example team-payments).

2. vault

   Add the team to the group list in vault-bootstrap.sh and re-run it to create the team policy and both identity groups.

3. repository

   Add requests/team-payments/ with its first request, and a CODEOWNERS line for the team's approvers. ArgoCD creates both the request application and the team-access application for the new directory.

4. vault

   Re-run deploy/phase2/vault-team-roles.sh so the team's External Secrets store can sign in.

6. Workload access

Install External Secrets

Add chart external-secrets version 2.10.0 from https://charts.external-secrets.io to your app-of-apps, namespace external-secrets, with installCRDs: true and sync options CreateNamespace=true and ServerSideApply=true.

Per-team vault roles

deploy/phase2/vault-team-roles.sh
# role auth/kubernetes/role/team-alpha -> team-alpha/courier-secrets (policy team-alpha)
# role auth/kubernetes/role/team-bravo -> team-bravo/courier-secrets (policy team-bravo)

One role per directory under requests/, bound to that namespace's courier-secrets service account, carrying only that team's policy. Teams without a vault policy are skipped.

Per-team stores

The courier ArgoCD application also syncs:

Verify

deploy/phase2/verify-external-secrets.sh    # EXTERNAL SECRETS VERIFIED
samples/python/verify.sh team-bravo report-exporter   # PYTHON SAMPLE VERIFIED

7. Controller image

Workflow .github/workflows/controller-image.yml runs on every push to main that touches Go code, go.mod or the Dockerfile. It pushes ghcr.io/paimonsoror/courier-controller with two tags: sha-<short commit> and main.

The deployment never follows a moving tag. deploy/phase1/controller/kustomization.yaml pins a digest:

images:
  - name: controller
    newName: ghcr.io/paimonsoror/courier-controller
    digest: sha256:ada7be47e996b26c10f7d358396ef2d267a0ee7cfc8e0534220eec9c5400e35e   # sha-2ed3a9d

Upgrade procedure

1. developer

   Merge the change; wait for Controller image to succeed.

2. developer

   Resolve the digest of sha-<short commit> (registry UI, or crane digest ghcr.io/paimonsoror/courier-controller:sha-<short>).

3. developer

   Update the digest in the overlay in a pull request. After merge, ArgoCD rolls out the new controller.

8. Backstage self-service

Everything that changes as teams and clients are added lives in the Courier repository, so Backstage never needs a rebuild for a new team or request.

Backstage configuration

# app-config.production.yaml: catalog locations
- type: url
  target: https://github.com/paimonsoror/courier/blob/main/backstage/catalog/teams.yaml
  rules: [{ allow: [Group] }]
- type: url
  target: https://github.com/paimonsoror/courier/blob/main/backstage/templates/oauth-client/template.yaml
  rules: [{ allow: [Template] }]
- type: url
  target: https://github.com/paimonsoror/courier/blob/main/requests/*/*.catalog.yml
  rules: [{ allow: [Resource] }]

# app-config.yaml: show OAuthClients on the Kubernetes tab
kubernetes:
  customResources:
    - group: 'courier.sororlab.dev'
      apiVersion: 'v1alpha1'
      plural: 'oauthclients'

Frontend: add the scaffolder and org plugins (@backstage/plugin-scaffolder/alpha, @backstage/plugin-org/alpha) to the app's features. Backend: the scaffolder backend and its GitHub module.

Access Backstage needs

# create the token secret without it touching shell history or the process list
read -rs GH_TOKEN
printf %s "$GH_TOKEN" | kubectl -n backstage create secret generic backstage-github-token --from-file=token=/dev/stdin
unset GH_TOKEN

Rules the validator adds for catalog entries

• A .catalog.yml must have a matching request in the same team directory.
• It must be a backstage.io/v1alpha1 Resource named <team>-<name>, type oauth-client, owned by group:default/<team>.
• Its namespace and label-selector annotations must point at its own request, and the request must carry label courier.sororlab.dev/client: <name>.

ArgoCD only syncs *.yaml, so .catalog.yml files are never applied to the cluster.

Verification

Day-2 operations

Troubleshooting

Each of these was hit while building the reference environment.

Hardening checklist

The proof of value takes shortcuts that production should not.