How the stack fits together

Apps on this platform run as rootless podman containers managed by systemd quadlet files.

Logs

Write to stdout. That's it — Promtail scrapes the systemd journal centrally and ships everything to Loki. No log library, no sidecar, no per-app config.

Viewing your logs in Grafana

Open the Logs dashboard in Grafana — your app's logs will appear there automatically once they're flowing. Use the user_unit filter at the top of the dashboard to select your service and see only its output.

The unit name comes from your quadlet .container file — whatever the file is named is what systemd calls the unit (e.g., my-app.container → my-app.service). If you're unsure of the exact name, run this as the service user:

systemctl --user list-units --type=service

For further querying — searching for a specific string, filtering to errors only, tailing live output — go to Explore in the left sidebar, set the data source to Loki, and query by your service directly:

{user_unit="my-app.service"}

Hit the Live toggle in the top-right to tail in real time while your app is running.

Nothing showing up? Work through this in order:

1. Confirm the container is actually running: systemctl --user status my-app.service
2. Check the journal directly — if logs appear here, Promtail will pick them up within a few seconds: journalctl --user -u my-app.service -f
3. Make sure your app is writing to stdout and not to a file. If it's writing to a file inside the container, the journal never sees it.

Container metrics

Container CPU, memory, network I/O, and restart counts come from prometheus-podman-exporter. Each service user needs their own instance because rootless podman is scoped per user — the exporter for one user cannot see another user's containers.

Setup has four steps: verify prerequisites, configure the exporter override, reload and enable, then add a scrape target to Prometheus.

1. Prerequisites

The binary must be installed. prometheus-podman-exporter should already be on the box. Confirm:

which prometheus-podman-exporter

If the command is not found, install it:

sudo dnf install prometheus-podman-exporter

Linger must be enabled for the service user. Linger lets systemd user units start automatically at boot without a login session. Without it, the exporter only runs while someone is logged in as that user — meaning Prometheus scrapes nothing until then.

loginctl enable-linger <service-user>

Confirm: loginctl show-user <service-user> | grep Linger should print Linger=yes.

2. Configure the exporter

The exporter ships as a systemd user service. Override its default listen address to assign it a dedicated port. Create the drop-in directory and override file as the service user:

mkdir -p ~/.config/systemd/user/prometheus-podman-exporter.service.d

[Service]
ExecStart=
ExecStart=/usr/bin/prometheus-podman-exporter --collector.enable-all --web.listen-address :<port>

The first ExecStart= clears the default command; the second sets yours with the chosen port and --collector.enable-all to expose container-level CPU, memory, network, and restart metrics.

Choosing a port: the convention on this host is to use the 988x range (e.g., 9882, 9883, 9884, …). Check what's already taken before picking one:

ss -tlnp | grep ':98'

You can also look at the existing scrape targets in prometheus.yml to see which ports are already claimed.

3. Reload and enable

systemctl --user daemon-reload
systemctl --user enable --now prometheus-podman-exporter.service

The enable is so the service starts automatically when linger brings the user's session up at boot. Verify it's running:

systemctl --user status prometheus-podman-exporter.service

4. Add a scrape target

The monitoring user's prometheus.yml needs a new job for your app. The target address must use host.containers.internal — Prometheus runs inside a container and cannot reach another user's loopback directly; this alias is the cross-user bridge that rootless podman's network shim provides.

This is in /home/monitoring/prometheus/prometheus.yml

  - job_name: 'podman-<app-name>'
  static_configs:
    - targets: ['host.containers.internal:<port>']
      labels:
        owner: '<service-user>'

Job naming convention: prefix with podman- followed by your app's name (e.g., podman-stipend-tracker). The owner label should be the username the app runs under — useful for filtering in Grafana when multiple apps share the same host.

If you have access to the monitoring user, apply the change:

systemctl --user restart prometheus

Viewing in Grafana

Once the exporter is running and Prometheus is scraping it, open the Podman Exporter Dashboard in Grafana. Your container will appear there automatically alongside the other active containers. Use the container dropdown at the top of the dashboard to select yours and view just its CPU, memory, network I/O, and restart stats.

For more metric exploration, go to Explore → Prometheus and type podman_container_ in the metric field — autocomplete will show everything the exporter is currently exposing for your container.