One request, end to end

At a glance

Merge to usable credentials: about 66 seconds, including a manual ArgoCD refresh (see step 5). No person saw, copied or transmitted the secret.

1. The request

The requester added a single file on branch request/team-bravo-report-exporter. The team directory makes team-bravo the owner; nothing in the file can point the credentials anywhere else.

# requests/team-bravo/report-exporter.yaml
apiVersion: courier.sororlab.dev/v1alpha1
kind: OAuthClient
metadata:
  name: report-exporter
  namespace: team-bravo
spec:
  displayName: "team-bravo: nightly report exporter"
  clientType: confidential
  grantTypes:
    - client_credentials
  scopes:
    - openid
    - profile
    - groups

Check it yourself

go run ./cmd/courier validate requests

2. The policy gate stops it

Opening the pull request triggered the Client requests / Validate requests check. The file was valid, but client_credentials lets a workload act as itself, so the repository policy requires an explicit approval label. The check failed in 18 seconds and put this annotation on the file:

requests/team-bravo/report-exporter.yaml
needs the "security-approved" label on the pull request (grant requires approval)

The run summary showed reviewers exactly what merging would do, before anything existed:

3. Approval re-runs the gate

At 20:53:08 the security-approved label was added. The workflow listens for labeled events, so the check re-ran without a new commit and passed at 20:53:29.

Check it yourself

curl -s https://api.github.com/repos/paimonsoror/courier/commits/976f8ce/check-runs \
  | jq '.check_runs[] | select(.name=="Validate requests") | {conclusion, started_at, completed_at}'

4. Merge

The pull request merged at 21:40:39 as commit de172bd, and the branch was deleted.

5. ArgoCD picks up the merge

The courier-requests ApplicationSet saw a new directory, requests/team-bravo, and generated an application for it. The locked project allowed it to create exactly one kind of object: the OAuthClient in namespace team-bravo.

$ kubectl -n argocd get application -l app.kubernetes.io/part-of=courier
NAME                          SYNC     HEALTH    REV
courier-controller            Synced   Healthy   de172bd07d1f67f87205325ff5fbde04ee12ea2d
courier-requests-team-alpha   Synced   Healthy   de172bd07d1f67f87205325ff5fbde04ee12ea2d
courier-requests-team-bravo   Synced   Healthy   de172bd07d1f67f87205325ff5fbde04ee12ea2d

kubectl -n argocd annotate applicationset courier-requests \
  argocd.argoproj.io/application-set-refresh=true --overwrite

6. The controller delivers

The controller validated the object again with the same rules as the pull request check, wrote the new secret to the vault as pending, created the client in Authentik with that secret, bound it to group team-bravo, and promoted the vault entry to active.

$ kubectl get oauthclients -A
NAMESPACE    NAME              READY   TYPE           CLIENT ID                                  SECRET PATH
team-alpha   billing-sync      True    confidential   BmNrqqyjaqWKLHgXPC7CtpBmKNo2Pye6DQjNMj6F   kv/teams/team-alpha/oauth-clients/team-alpha-billing-sync
team-bravo   report-exporter   True    confidential   8BhnTaEg83o1lfhlwLSbBqHL6N5rMGOdsUi9rfA9   kv/teams/team-bravo/oauth-clients/team-bravo-report-exporter

The controller log records the event with the client ID and path only:

2026-09-13T21:41:45Z  INFO  issued credentials
  OAuthClient: team-bravo/report-exporter
  client:      team-bravo-report-exporter
  clientId:    8BhnTaEg83o1lfhlwLSbBqHL6N5rMGOdsUi9rfA9
  path:        kv/teams/team-bravo/oauth-clients/team-bravo-report-exporter

Check it yourself

kubectl -n team-bravo describe oauthclient report-exporter
kubectl -n courier-system logs deploy/courier-controller-manager | grep report-exporter

7. Access verified

Delivery is only half the claim. The other half is that the right team, and only that team, can use it. deploy/phase2/check-access.sh signs in to the vault as each team's test service account, exactly as a workload would, and tries to read the path.

$ deploy/phase2/check-access.sh team-bravo report-exporter
request team-bravo/report-exporter is Ready; credentials at kv/teams/team-bravo/oauth-clients/team-bravo-report-exporter
PASS  team-alpha is denied
PASS  team-bravo (owner) reads the credentials
PASS  team-bravo credentials get a token from Authentik

ACCESS CHECK PASSED

$ deploy/phase2/check-access.sh team-bravo report-exporter
request team-bravo/report-exporter is Ready; credentials at kv/teams/team-bravo/oauth-clients/team-bravo-report-exporter
PASS  team-alpha is denied
PASS  team-bravo (owner) reads the credentials
PASS  Authentik accepts the delivered secret
PASS  Authentik rejects a wrong secret (so the check above can fail)

ACCESS CHECK PASSED

What never happened

The strongest evidence is what is missing. For this request, the secret did not appear in:

A second request, from Backstage (pull request #2)

Pull request #1 was written by hand. The second request came from the Backstage form, to prove the self-service path end to end: team-alpha asks for orders-portal, a web application that signs users in. It needs no approval label because it does not use client_credentials.

Form to usable credentials: about four minutes, almost all of it the review and merge.

Delivered, and visible to the team

$ deploy/phase2/check-access.sh team-alpha orders-portal
request team-alpha/orders-portal is Ready; credentials at kv/teams/team-alpha/oauth-clients/team-alpha-orders-portal
PASS  team-alpha (owner) reads the credentials
SKIP  token test: authorization_code refresh_token clients need a user sign-in, not client_credentials
PASS  team-bravo is denied

ACCESS CHECK PASSED

Re-run on 14 September with the corrected secret check (see the correction in step 7), which now applies to sign-in clients too:

$ deploy/phase2/check-access.sh team-alpha orders-portal
request team-alpha/orders-portal is Ready; credentials at kv/teams/team-alpha/oauth-clients/team-alpha-orders-portal
PASS  team-alpha (owner) reads the credentials
PASS  Authentik accepts the delivered secret
PASS  Authentik rejects a wrong secret (so the check above can fail)
PASS  team-bravo is denied

ACCESS CHECK PASSED

Rotation and adoption (pull requests #3 and #4)

Phase 4 added two lifecycle operations, both requested the same way as everything else: a small change in a pull request. The Phase 4 controller was rolled out at 01:31:38 UTC on 14 September; both pull requests were merged after it.

Pull request #3: rotate team-bravo's secret

The only change is one annotation on requests/team-bravo/report-exporter.yaml. Any new value asks for exactly one rotation.

metadata:
  name: report-exporter
  annotations:
    courier.sororlab.dev/rotate: "2026-09-14-phase4-demo"

$ deploy/phase4/verify-rotation.sh
==> request status
lastSecretIssued: 2026-09-14T01:38:54Z
rotationHandled:  2026-09-14-phase4-demo
==> before External Secrets refresh
      vault version 8, state active, secret fingerprint 9b96329eec4b
      kubernetes Secret fingerprint 2f2a72d01e19 (client_id unchanged)
PASS  vault holds a new secret; the synced copy still holds the previous one
PASS  client ID is unchanged by rotation
PASS  Authentik rejects the previous secret
PASS  Authentik accepts the new secret
==> force External Secrets to refresh
      vault version 8, state active, secret fingerprint 9b96329eec4b
      kubernetes Secret fingerprint 9b96329eec4b (client_id unchanged)
PASS  External Secrets now holds the new secret
PASS  Authentik accepts the synced copy

ROTATION VERIFIED

The window between "vault has the new secret" and "every copy has it" is the refresh interval of whatever caches it. Workloads that read the vault directly see the new secret on their next read; External Secrets copies wait for their refresh (one hour here) unless forced.

Pull request #4: adopt a hand-built client

Before the pull request, deploy/phase4/create-legacy-client.sh built legacy-crm by hand in Authentik, the way clients were made before Courier, and kept its "emailed" secret aside so we could prove it stops working. Team-alpha then requested crm with one extra annotation:

metadata:
  name: crm
  annotations:
    courier.sororlab.dev/adopt: legacy-crm

$ deploy/phase4/verify-adoption.sh
NAME   READY   TYPE           CLIENT ID                                  SECRET PATH
crm    True    confidential   Fx2FjQztQf1w2bxpAk9MrDawX2XlGlMVaal9JaEU   kv/teams/team-alpha/oauth-clients/legacy-crm
==> Authentik application
      name: team-alpha: CRM integration (adopted) | meta_description: managed-by: courier; owner: team-alpha
PASS  team-alpha reads the adopted client's credentials
PASS  client ID is unchanged by adoption
PASS  Authentik rejects the emailed secret
PASS  Authentik accepts the secret Courier stored in vault
PASS  team-bravo is denied

ADOPTION VERIFIED

Integrations that use the client keep their client ID and redirect settings; they only need to read the new secret from the vault. The copy that was emailed is now worthless.

Drift repair that stays quiet

Before Phase 4, every ten-minute resync rewrote every client in Authentik: 12 model_updated events an hour for report-exporter alone. Phase 4 compares first and writes only on a difference. In the first 16 minutes after rollout, billing-sync had no events and report-exporter had one, the rotation. orders-portal still had one every resync, because Authentik adds redirect_uri_type to redirect URIs and the controller did not send it; that was fixed in 8e454ad.

What we learned

• The label gate works as designed. It blocked a high-risk grant, told the requester exactly what was needed, and cleared itself on approval without a new commit.
• Branch protection is a required setup step, not optional. Without it, the check still reports but merges do not wait for code-owner review.
• Sync latency is ArgoCD's polling interval. Delivery itself took seconds; add an ArgoCD webhook if teams expect credentials immediately after merge.
• Verification must include the negative case. "The owner can read it" alone would also pass if every team could read everything.
• A check that cannot fail proves nothing. "The credentials get a token" also passed with a rotated-out secret and with a made-up one. Every secret check now also sends a wrong secret and requires a refusal.

Reproduce it

1. requester

   Add requests/<team>/<name>.yaml on a branch and open a pull request. See Request a client.

2. approver

   For client_credentials, add security-approved; wait for Validate requests to pass; review and merge.

3. operator

   Optionally refresh the ApplicationSet, then wait for Ready:
   kubectl -n <team> wait oauthclient/<name> --for=condition=Ready --timeout=180s

4. operator

   Prove access: deploy/phase2/check-access.sh <team> <name> must print ACCESS CHECK PASSED.