At a glance

The walkthrough is seven steps from "fresh app repo" to "running in prod":

1. Conventional justfile + Containerfile — standard recipes the workflow YAML calls by name.
2. Author deploy/<unit>.container.tmpl — the quadlet template lives next to the code, frozen at each release.
3. Open a registration PR in shyft-infra — claims the slug, provisions ECR repo + OIDC role + scrape target.
4. Wire up build-image.yml — push image to ECR on every main commit and on tags.
5. Wire up release.yml — on tag, render the template and open the bump-PR.
6. Cut your first release — git tag v0.1.0 and watch it propagate.
7. Register with the portal — admin UI; existing Portal registration flow.

Before the walkthrough, three orientation sections — read once, refer back as needed:

• Architecture — diagrams of the three flows (image, config, telemetry) that make up the pipeline
• What lives where — file-by-file inventory for each repo, registry, and host
• Deploy lifecycle — what happens between git tag v0.4.0 and a running container

And two reference sections to bookmark:

• Image–config coupling — why the template lives in the app repo, what changes go where
• Rollback — image+shape rollback, instance-value rollback, the 2am emergency hand-edit

Architecture

Two repos, one registry, the hosts, and the monitoring VM — that's the whole system. Every piece exists either to build an artifact (a container image), to describe how that artifact runs (the rendered quadlet), or to reconcile the description against what's actually running.

Three independent flows run between the same set of nodes — easier to read separately than as one tangled graph.

The deliberate property worth internalising: image content and runtime shape travel together. When internal-portal cuts a v0.4.0 release, the same CI run that pushes :v0.4.0 to ECR also renders the quadlet template that shipped at that tag and opens a PR carrying the rendered output to shyft-infra. The image and its expected env vars / ports / volumes arrive in lockstep, are reviewed together, and roll back together — covered in detail under image–config coupling below.

What lives where

Each repo, registry, and host has a defined set of files. Use these as the checklist when onboarding: anything missing from the per-app repo or shyft-infra shape on the day you're shipping is a CI failure waiting to happen.

Per-app repo (e.g. internal-portal)

Owned by the app team. Everything that defines what the app is and how it gets built.

internal-portal/
├── Containerfile                       # Multi-stage image build (non-root, standalone)
├── justfile                            # Standard recipes — see /docs/just-standard
├── DEPLOY.md                           # Deploy contract: env vars, ports, dependencies
├── deploy/
│   ├── portal.container.tmpl           # Quadlet template — runtime shape, with placeholders
│   └── portal-db.container.tmpl        # …one .tmpl per container the app *owns*
└── .github/workflows/
  ├── build-image.yml                 # On push to main + on v* tag: lint/test/build/publish
  └── release.yml                     # On v* tag: render .tmpl + open bump-PR in shyft-infra

Pre-existing files unrelated to deploy (src/, package.json, tsconfig.json, etc.) sit alongside — the deploy contract is additive.

Central infra repo (shyft-infra)

Owned by the platform team with app-team contributions via PR. Single source of truth for "what runs where, on what shape of box."

shyft-infra/
├── quadlets/
│   └── portal/                         # One subdir per app, name = slug
│       ├── values.yaml                 # Host-instance values: network, secrets, ports — hand-edited
│       ├── portal.container            # Rendered — auto-managed by bump-PRs
│       ├── portal.network              # Networks — hand-edited (shape rarely changes)
│       └── deps/
│           └── portal-db.container     # Dependency containers (upstream images) — hand-edited
├── terraform/
│   ├── bootstrap/                      # One-shot: S3 state bucket + DynamoDB lock table
│   ├── modules/
│   │   ├── ec2-app-host/               # EC2 + EIP + SG + IAM profile + cloud-init
│   │   ├── ecr-repo/                   # ECR repo with lifecycle policy (one per app)
│   │   ├── gha-oidc-role/              # IAM role for app-repo CI to assume via OIDC
│   │   └── monitoring-host/            # Dedicated Prom+Grafana VM
│   └── live/
│       └── prod/                       # Concrete env: backend.tf + per-host .tf files
├── monitoring/
│   ├── prometheus/
│   │   ├── prometheus.yml              # Base config
│   │   └── targets/                    # File-based SD, one .yml per host
│   ├── grafana/
│   │   └── dashboards/                 # JSON dashboards-as-code
│   └── alertmanager/                   # Routing + templates
├── tools/
│   └── render/                         # CLI: .tmpl + values.yaml + image → rendered .container
├── inventory.yml                       # Host ↔ app mapping; drives scrape-target generation
└── .github/
  ├── workflows/
  │   ├── tf-plan.yml                 # Comment terraform plan on PRs
  │   └── tf-apply.yml                # Manual workflow_dispatch to apply
  └── actions/
      └── render/                     # Reusable action wrapping tools/render/ for app repos

The PR that adds a new app touches the values file, the inventory, a scrape-target file, and (one-time) a terraform module call for the ECR repo + OIDC role. The rendered .container shows up later via the first bump-PR from CI — you don't hand-write it (except for cut-over, once per app).

AWS ECR

Owned by CI only — no human pushes to prod-bound tags. One ECR repository per app, with lifecycle policy to expire untagged images after 14 days.

internal/portal                          # one repo per app under the "internal/" namespace
├── :v0.3.0                            # immutable, semver — what prod pins to
├── :v0.3.1
├── :sha-abc1234                       # immutable, git-sha — ad-hoc dev deploys
└── :stable                            # mutable, opt-in per app — only for low-stakes services
internal/future-app-N                    # …one per app

Tags pushed by build-image.yml; tag strategy in D2 (recommendation: always-push the immutable pair, push :stable only when the app opts in).

Host (Rocky EC2)

Owned by nobody, in the sense that any human edit gets reconciled away on the pull-agent's next tick. Drift detection is implicit.

~/.config/containers/systemd/
├── portal.container                    # Reconciled from shyft-infra/quadlets/portal/
├── portal.network
├── portal-db.container                 # Dependency, from deps/
├── node_exporter.container             # Telemetry, from monitoring/host-agents/
└── cadvisor.container
~/shyft-infra-checkout/                  # Pull-agent's local clone (read-only mirror)
~/.local/share/containers/storage/secrets/  # Podman secrets (database-url, jwt-keys, etc.)
/etc/systemd/system/
├── shyft-pull-agent.service            # The reconciler — installed once at host bring-up
└── shyft-pull-agent.timer              # Fires every 2 min

The pull-agent service + timer are installed once during host bring-up (terraform's cloud-init does it); after that, nothing on the host is hand-edited in steady state.

Monitoring VM

Owned by the platform team. Same shape as an app host (the same pull-agent runs there, reconciling against shyft-infra/monitoring/).

~/.config/containers/systemd/
├── prometheus.container                # Scrapes every host listed in inventory.yml
├── grafana.container
├── alertmanager.container
└── *.volume                            # Persistent storage (~30d retention to start)
+ the same shyft-pull-agent.{service,timer} as any other host

The monitoring VM is provisioned by the monitoring-host terraform module. Adding a metric source is a PR against shyft-infra/monitoring/; the agent picks it up.

The deploy lifecycle

What happens between git tag v0.4.0 and the new container running in prod.

A few properties worth calling out:

• No push from CI to host. CI never SSHes anywhere. CI pushes an image to ECR and opens a PR; the host pulls. Hosts can be unreachable for hours and catch up cleanly on the next tick.
• Bump-PR is the audit trail. git log on shyft-infra answers "what was running on $date?" without registry archaeology.
• One human gate per release. The bump-PR merge is the only manual step in the happy path. Everything before is automated; everything after is reconciliation.

Image–config coupling (why the template lives in the app repo)

A v0.4.0 image may need a new env var, a new port, or a new volume that v0.3.0's quadlet doesn't know about. If image version and quadlet shape drift apart, a tag bump silently breaks the deploy. The pipeline avoids this by keeping the shape next to the code:

• internal-portal/deploy/portal.container.tmpl — the shape of the quadlet (env var names, ports, volumes, network attachments, healthcheck path). Image= and a handful of host-instance fields are template placeholders. Changes to this file are reviewed in the same PR as the code change that needs them.
• shyft-infra/quadlets/portal/values.yaml — the host-instance values plugged into the template (which network, which podman secrets, host-specific port maps).
• shyft-infra/quadlets/portal/portal.container — the rendered output. Auto-managed by the bump-PR; hand-edited only for emergencies.

Integration walkthrough — adding a new app

Concrete order for taking a brand-new app from zero to a running container in prod. Roughly half a day if you've done it once; longer the first time. Pre-reqs: app already builds with npm/uv/etc.; the portal admin team has agreed on the slug.

1. Conventional justfile + Containerfile

Lift the recipes from just-standard — install, dev, build, test, lint, typecheck, image, publish. The workflow YAML calls these by name, so anything non-conventional becomes workflow-YAML drift later.

Author a multi-stage Containerfile that produces a runtime image. The internal-portal Containerfile is the canonical worked example — multi-stage, non-root, standalone bundle.

2. Author deploy/<unit>.container.tmpl

The template is a plain quadlet .container file with Image=, network, and any host-instance fields replaced by Go-template placeholders. The shape — env var names, ports, volume mounts, healthcheck — is frozen by this file; values come from the infra repo at render time.

[Unit]
Description=Internal portal
After=network-online.target

[Container]
Image={{ .image }}
Network={{ .network }}
PublishPort={{ .hostPort }}:3000
Environment=NODE_ENV=production
Environment=PORT=3000
Environment=AUTHENTIK_ISSUER={{ .authentikIssuer }}
Secret=portal-database-url,type=env,target=DATABASE_URL
Secret=portal-jwt-keys,type=mount,target=/run/secrets/jwt-keys.json
HealthCmd=wget -q -O - http://localhost:3000/api/health
HealthInterval=30s

[Service]
Restart=always

[Install]
WantedBy=default.target

Anything that doesn't change across hosts is hardcoded. Anything that does is a placeholder fed by values.yaml.

3. Open a PR in shyft-infra to register the app

You're claiming a slot. This PR alone doesn't deploy anything — it sets up the infra and instance values the bump-PR workflow will later write into.

The PR adds:

network: portal.network
hostPort: 3000
authentikIssuer: https://auth.shyftsolutions.io/application/o/portal/
# secret names referenced by the .tmpl — declared here so a missing
# secret on the host is a loud render-time failure, not a silent runtime one
secrets:
- portal-database-url
- portal-jwt-keys

…and (in the same PR, gated by a platform-team review):

• A call to the ecr-repo terraform module for this app's image repo
• A call to the gha-oidc-role terraform module — grants the app's GitHub repo permission to push to its (and only its) ECR repo via OIDC
• An entry in inventory.yml mapping the app to a host
• A scrape-target file in monitoring/prometheus/targets/

Once this PR merges, terraform apply runs from CI and the app has its ECR repo and its OIDC role.

4. Wire up build-image.yml

Drop a standardized build-image.yml into the app repo's .github/workflows/. It's the same shape for every app — calls just, assumes ECR via OIDC.

name: build-image
on:
push:
  branches: [main]
  tags: ['v*']

permissions:
id-token: write   # OIDC → AWS
contents: read

jobs:
build:
  runs-on: ubuntu-latest
  steps:
    - uses: actions/checkout@v4
    - uses: extractions/setup-just@v2
    - uses: aws-actions/configure-aws-credentials@v4
      with:
        role-to-assume: arn:aws:iam::973309142234:role/gha-${{ github.event.repository.name }}
        aws-region: us-east-2
    - run: just install
    - run: just lint
    - run: just typecheck
    - run: just test
    - run: just build
    - run: just publish

On a v* tag, publish pushes :vX.Y.Z + :sha-<sha>; on main, just :sha-<sha> (and :main if the app opts in via D2).

5. Wire up release.yml (the bump-PR)

On v* tag, after the image is in ECR, render the template and open the bump-PR against shyft-infra. The render CLI lives in shyft-infra/tools/render/ and is shipped as a reusable GitHub Action so every app repo calls it the same way.

name: release
on:
push:
  tags: ['v*']

jobs:
bump:
  runs-on: ubuntu-latest
  needs: build         # waits for build-image.yml on the same tag
  steps:
    - uses: actions/checkout@v4
      with: { path: app }
    - uses: actions/checkout@v4
      with:
        repository: ShyftSolutions/shyft-infra
        token: ${{ secrets.SHYFT_INFRA_PAT }}
        path: infra
    - uses: ShyftSolutions/shyft-infra/actions/render@v1
      with:
        template: app/deploy/portal.container.tmpl
        values:   infra/quadlets/portal/values.yaml
        set:      image=973309142234.dkr.ecr.us-east-2.amazonaws.com/internal/portal:${{ github.ref_name }}
        out:      infra/quadlets/portal/portal.container
    - uses: peter-evans/create-pull-request@v6
      with:
        path: infra
        token: ${{ secrets.SHYFT_INFRA_PAT }}
        branch: bump/portal-${{ github.ref_name }}
        title:  'portal: bump to ${{ github.ref_name }}'
        body:   'Auto-generated bump-PR. Diff carries image-tag change + any shape change shipped with this release.'

The PR description is curt on purpose — the diff is the description. Reviewers focus on the rendered quadlet change.

6. Cut your first release

git tag v0.1.0
git push origin v0.1.0

Within a few minutes:

1. build-image.yml pushes :v0.1.0 + :sha-… to ECR
2. release.yml opens bump/portal-v0.1.0 in shyft-infra
3. You (or a platform reviewer) review the rendered portal.container diff and merge
4. Within ~2min, the pull-agent on the target host syncs the change, pulls the new image, and restarts the unit

7. Register with the portal

Last step is telling the portal where to find the app. This is the existing Portal registration flow — picks a Base URL given your topology, sets up role discovery, gates by Authentik group.

For a co-located app on the portal's podman network the Base URL is just http://<container-name>:<port> — the rest of the page covers the other topologies.

Rollback

Image+shape rollback

Revert the bump-PR in shyft-infra. Image tag and quadlet shape revert atomically — by design, you can't accidentally roll back one and not the other.

# in the shyft-infra checkout
gh pr close <bump-pr-num>
# or, post-merge:
git revert <bump-pr-merge-sha>
git push

Pull-agent picks up the revert on its next tick.

Instance-value rollback

Same shape — git revert the central-repo commit that changed values.yaml.

Emergency: 2am, bump-PR can't merge

You can always SSH to the host and hand-edit ~/.config/containers/systemd/<unit>.container, then systemctl --user daemon-reload && systemctl --user restart <unit>. The next pull-agent tick will reconcile your hand-edit away — so this is for buying time while a proper revert is being pushed, not for permanent fixes.

What this page is not

This page focuses on how to ship an app. Each layer has its own page with the gritty details. All six are currently "coming soon" stubs — the parent page is the source of truth until each underlying piece ships:

• CI image build — what build-image.yml actually contains, tag-strategy options, multi-arch
• Central infra repo — shyft-infra layout in detail, how the modules compose
• Image–config coupling — the template language, the render CLI, edge cases when the template breaks
• Terraform bring-up — adding a brand-new host vs. adding an app to an existing host
• GitOps pull-agent — what the pull-agent does on each tick, how to debug a stuck reconcile, lockfile semantics
• Monitoring — scrape targets, dashboards-as-code, alert routing

Live status of which milestone is in flight is in .claude/features/deploy-pipeline/{plan,tasks}.md.