Merge pull request 'feat(runner): täglicher Aufräumer für das, was abgebrochene Jobs liegen lassen' (#13) from feat/runner-cleanup into main
CI / Shell-Lint (bash -n, source-check, Validierungs-Tests) (push) Successful in 3s
CI / Shell-Lint (bash -n, source-check, Validierungs-Tests) (push) Successful in 3s
This commit was merged in pull request #13.
This commit is contained in:
@@ -36,6 +36,9 @@ jobs:
|
||||
- name: Validierungs-Unit-Tests (lib/build.func)
|
||||
run: bash tests/test_validation.sh
|
||||
|
||||
- name: Runner-Aufräumer (install/runner-cleanup.sh, ohne Docker)
|
||||
run: bash tests/test_runner_cleanup.sh
|
||||
|
||||
- name: shellcheck (falls auf dem Runner installiert)
|
||||
run: |
|
||||
if command -v shellcheck >/dev/null 2>&1; then
|
||||
|
||||
@@ -13,7 +13,7 @@ Inspired by [community-scripts/ProxmoxVE](https://github.com/community-scripts/P
|
||||
| [nexus](ct/nexus.sh) | App LXC for [nexus](https://gitea.luki-net.org/l.kirchner/nexus-hub) (Family Knowledge Hub): host-mode runner (label `nexus`, CI + deploy), service skeleton, `/opt/nexus` layout. Runtime is provisioned/extended via [`install/nexus-runtime.sh`](install/nexus-runtime.sh) (idempotent, re-runnable) | ✅ in production |
|
||||
| [nexus-db](ct/nexus-db.sh) | PostgreSQL 16 + pgvector for nexus — least-privilege role, pg_hba allowlist (only the nexus LXC), DSN handed over via credentials file | ✅ in production |
|
||||
| [authentik](ct/authentik.sh) | Central homelab IdP (official Docker Compose via LXC nesting) — headless bootstrap admin **and** API token for agent-driven blueprint configuration, blueprints mount, permanent auth domain (WebAuthn RP-ID) | ✅ in production |
|
||||
| [runner](ct/runner.sh) | General **instance-wide** Gitea Actions runner (label `homelab`) — Docker for throwaway CI test containers, deliberately **no** sudoers/deploy rights | ✅ in production |
|
||||
| [runner](ct/runner.sh) | General **instance-wide** Gitea Actions runner (label `homelab`) — Docker for throwaway CI test containers, deliberately **no** sudoers/deploy rights. A daily cleaner ([`install/runner-cleanup.sh`](install/runner-cleanup.sh), `ci-runner-cleanup.timer`) removes what cancelled jobs left behind: only what a project claims under `/etc/ci-runner-cleanup.d/` (containers by its label `org.luki-net.project` and name, tmp entries by its prefix) and only past 4 h, above act_runner's 3 h job timeout (containers from their last start, tmp entries by ctime); `runner-install.sh` sets it up on new runners; runners installed before it do not have it until it is set up there | ✅ in production |
|
||||
|
||||
Run any one-liner on a Proxmox VE host as root:
|
||||
|
||||
@@ -74,7 +74,7 @@ Caveat: auditd is a no-op in unprivileged LXCs (the host owns the audit subsyste
|
||||
|
||||
**All changes go through a pull request with cross-review** (Claude Code ↔ Codex, or a human) — no direct pushes to `main`. This rule exists because of two real incidents: a pasted VLAN tag carrying an invisible non-UTF-8 byte broke `pct create` mid-run, and the "missing `source build.func`" bug shipped twice — caught in review on the nexus-db PR, but reaching production via an un-reviewed authentik commit (see wiki → Lessons).
|
||||
|
||||
CI (`.gitea/workflows/ci.yml`, instance-wide `homelab` runner from [ct/runner.sh](ct/runner.sh)) enforces on every PR: `bash -n` over all scripts, the "every `ct/*.sh` sources `build.func`" check ([tests/check_ct_source.sh](tests/check_ct_source.sh)) and the validation unit tests ([tests/test_validation.sh](tests/test_validation.sh)); shellcheck runs when available on the runner.
|
||||
CI (`.gitea/workflows/ci.yml`, instance-wide `homelab` runner from [ct/runner.sh](ct/runner.sh)) enforces on every PR: `bash -n` over all scripts, the "every `ct/*.sh` sources `build.func`" check ([tests/check_ct_source.sh](tests/check_ct_source.sh)), the validation unit tests ([tests/test_validation.sh](tests/test_validation.sh)) and the runner cleaner's tests ([tests/test_runner_cleanup.sh](tests/test_runner_cleanup.sh)); shellcheck runs when available on the runner.
|
||||
|
||||
Build new app prompts on `prompt_validated`/`require_valid` from `lib/build.func` instead of bare `read`. How to add a script: [docs/adding-a-script.md](docs/adding-a-script.md).
|
||||
|
||||
|
||||
Executable
+254
@@ -0,0 +1,254 @@
|
||||
#!/usr/bin/env bash
|
||||
# Daily cleaner for the instance-wide Gitea Actions runner (ct/runner.sh).
|
||||
# Installed by install/runner-install.sh as /usr/local/sbin/ci-runner-cleanup,
|
||||
# run by ci-runner-cleanup.timer once a day.
|
||||
#
|
||||
# A cancelled job never reaches its own teardown step, so what it started
|
||||
# stays behind: a test database container, and a copy of the tree from
|
||||
# counter-check in act-runner's private /tmp (l.kirchner/patchmgr, D64).
|
||||
# This removes exactly that, and only what a project has claimed as its own.
|
||||
# No `docker prune`: the runner works for every repository of the instance,
|
||||
# and their containers and volumes are not ours to judge.
|
||||
#
|
||||
# A project claims its leftovers with one file in /etc/ci-runner-cleanup.d/:
|
||||
#
|
||||
# PROJECT=patchmgr
|
||||
# CONTAINER_NAME_PATTERN='^patchmgr-testdb-'
|
||||
# TMP_PATTERN='patchmgr-counter-check-*'
|
||||
#
|
||||
# The file is read, not run: three keys, a value either bare or in single
|
||||
# quotes, comments and blank lines; anything else and the file is skipped.
|
||||
#
|
||||
# Ownership:
|
||||
# - A project name is lower-case letters and digits only, no hyphen, so
|
||||
# that "<project>-" is a prefix no other project's names can start with.
|
||||
# It is handed out by whoever administers the runner, once per instance:
|
||||
# the files here are root's, and two files naming the same project are
|
||||
# both refused.
|
||||
# - A container must carry the label org.luki-net.project=$PROJECT (set by
|
||||
# the project itself) and its name must match CONTAINER_NAME_PATTERN:
|
||||
# ^$PROJECT- and then only letters, digits and _.- -- a literal prefix,
|
||||
# with no quantifier that could make the hyphen optional.
|
||||
# - A tmp entry must match TMP_PATTERN: $PROJECT- and then only letters,
|
||||
# digits and _.-*? -- a glob that names no directory.
|
||||
# A container goes with its anonymous volumes (`rm -f -v`); named volumes
|
||||
# are left alone.
|
||||
#
|
||||
# Known limit, accepted: label and name prefix say which project a leftover
|
||||
# belongs to, not which repository. A fork of a registered project runs the
|
||||
# same scripts unchanged on this runner and sets the same label and prefix,
|
||||
# so its leftovers older than MAX_AGE_HOURS are removed as well. Accepted
|
||||
# for patchmgr on 11.10.2026: the repository is private, has no forks, and
|
||||
# only its owner and bots contribute. When it opens up for forks, its
|
||||
# deploy/testdb.sh gets a repository label as well, and this script filters
|
||||
# on that one too; tmp entries carry no label, so the prefix counter-check
|
||||
# gives its directory has to name the repository then as well.
|
||||
#
|
||||
# Age: act_runner v0.2.13 ends a job after 3 h (runner.timeout, measured on
|
||||
# runner-01..03 on 08.10.2026: no config file, LoadDefault sets 3 h), and
|
||||
# each runner runs one job at a time. MAX_AGE_HOURS defaults to 4 and cannot
|
||||
# be set to 3 or lower. What is older than that was not created or started
|
||||
# by any job still inside its time limit:
|
||||
# - a container is as old as the later of its creation and its last start,
|
||||
# so one a job started again counts from that start;
|
||||
# - a tmp entry is as old as its ctime. Not its mtime: cp -a, rsync -a and
|
||||
# tar -x carry an old mtime over to a fresh copy, and nothing sets a
|
||||
# ctime back.
|
||||
# A project registers only containers its jobs create themselves with
|
||||
# `docker run` and never start again, and only tmp entries its jobs create
|
||||
# fresh for each run (mktemp) and never use again: a job that starts an
|
||||
# existing container or works in an existing directory could do so between
|
||||
# this script's look at its age and the removal, or without touching the
|
||||
# directory's ctime at all, and nothing here can close that.
|
||||
#
|
||||
# DRY_RUN: empty or 0 removes; any other value only lists.
|
||||
# Exit 1 when the cleanup of any project failed; the others still run.
|
||||
set -euo pipefail
|
||||
export LC_ALL=C
|
||||
|
||||
JOB_TIMEOUT_HOURS=3 # act_runner v0.2.13 runner.timeout, see above
|
||||
MAX_AGE_HOURS=${MAX_AGE_HOURS:-4}
|
||||
DRY_RUN=${DRY_RUN:-}
|
||||
CONF_DIR=${CONF_DIR:-/etc/ci-runner-cleanup.d}
|
||||
TMP_ROOT=${TMP_ROOT:-/tmp}
|
||||
|
||||
# At most three digits and no leading zero. Without an upper bound a large
|
||||
# enough number (4611686018427387904, for one) overflows the seconds below
|
||||
# to anything, 0 included; 999 h is far below that and longer than any age
|
||||
# worth waiting for. A leading zero reads as octal.
|
||||
if ! [[ "$MAX_AGE_HOURS" =~ ^[1-9][0-9]{0,2}$ ]] || [ "$MAX_AGE_HOURS" -le "$JOB_TIMEOUT_HOURS" ]; then
|
||||
echo "MAX_AGE_HOURS must be a whole number from $((JOB_TIMEOUT_HOURS + 1)) to 999, above the job timeout of ${JOB_TIMEOUT_HOURS} h, not '$MAX_AGE_HOURS'" >&2
|
||||
exit 1
|
||||
fi
|
||||
dry=0
|
||||
case "$DRY_RUN" in
|
||||
'' | 0) ;;
|
||||
*) dry=1 ;;
|
||||
esac
|
||||
|
||||
now=$(date +%s)
|
||||
limit=$((MAX_AGE_HOURS * 3600))
|
||||
|
||||
# read_conf reads one project's file into PROJECT, CONTAINER_NAME_PATTERN and
|
||||
# TMP_PATTERN, or says what is wrong with it and fails.
|
||||
read_conf() {
|
||||
local conf=$1 line key val
|
||||
PROJECT='' CONTAINER_NAME_PATTERN='' TMP_PATTERN=''
|
||||
while IFS= read -r line || [ -n "$line" ]; do
|
||||
[[ "$line" =~ ^[[:space:]]*(#.*)?$ ]] && continue
|
||||
if ! [[ "$line" =~ ^([A-Z_]+)=(.*)$ ]]; then
|
||||
echo "$conf: not KEY=value: $line" >&2
|
||||
return 1
|
||||
fi
|
||||
key=${BASH_REMATCH[1]} val=${BASH_REMATCH[2]}
|
||||
if [[ "$val" =~ ^\'([^\']*)\'$ ]]; then
|
||||
val=${BASH_REMATCH[1]}
|
||||
elif ! [[ "$val" =~ ^[A-Za-z0-9_^*.-]*$ ]]; then
|
||||
echo "$conf: $key: a value with anything but letters, digits and _^*.- goes in single quotes" >&2
|
||||
return 1
|
||||
fi
|
||||
case "$key" in
|
||||
PROJECT | CONTAINER_NAME_PATTERN | TMP_PATTERN) printf -v "$key" '%s' "$val" ;;
|
||||
*)
|
||||
echo "$conf: unknown key $key" >&2
|
||||
return 1
|
||||
;;
|
||||
esac
|
||||
done <"$conf"
|
||||
}
|
||||
|
||||
# clean_project removes one project's leftovers and fails if anything it
|
||||
# tried did not work. Run with errexit on, in a subshell of its own.
|
||||
clean_project() {
|
||||
local conf=$1 failed=0 n=0
|
||||
if ! read_conf "$conf"; then
|
||||
echo "$conf: skipped" >&2
|
||||
return 1
|
||||
fi
|
||||
if ! [[ "$PROJECT" =~ ^[a-z0-9]+$ ]]; then
|
||||
echo "$conf: PROJECT '$PROJECT' is not a project name (a-z, 0-9, no hyphen) -- skipped" >&2
|
||||
return 1
|
||||
fi
|
||||
if [ -n "$CONTAINER_NAME_PATTERN" ]; then
|
||||
if ! [[ "$CONTAINER_NAME_PATTERN" =~ ^\^$PROJECT-[A-Za-z0-9_.-]*$ ]]; then
|
||||
echo "$conf: CONTAINER_NAME_PATTERN must be ^$PROJECT- followed only by letters, digits and _.- -- skipped" >&2
|
||||
failed=1
|
||||
else
|
||||
local ids id fields name created started t_created t_started age
|
||||
if ! ids=$(docker ps -aq --filter "label=org.luki-net.project=$PROJECT"); then
|
||||
echo "$conf: docker ps failed" >&2
|
||||
failed=1
|
||||
ids=''
|
||||
fi
|
||||
for id in $ids; do
|
||||
if ! fields=$(docker inspect -f '{{.Name}} {{.Created}} {{.State.StartedAt}}' "$id"); then
|
||||
echo "$conf: docker inspect $id failed" >&2
|
||||
failed=1
|
||||
continue
|
||||
fi
|
||||
read -r name created started <<<"$fields"
|
||||
name=${name#/}
|
||||
[[ "$name" =~ $CONTAINER_NAME_PATTERN ]] || continue
|
||||
if ! t_created=$(date -d "$created" +%s) || ! t_started=$(date -d "$started" +%s); then
|
||||
echo "$conf: $name: cannot read its times '$created' '$started'" >&2
|
||||
failed=1
|
||||
continue
|
||||
fi
|
||||
# A container never started reports 0001-01-01 here.
|
||||
[ "$t_started" -gt "$t_created" ] && t_created=$t_started
|
||||
age=$((now - t_created))
|
||||
[ "$age" -gt "$limit" ] || continue
|
||||
echo "container $name ($((age / 3600)) h)"
|
||||
[ "$dry" -eq 0 ] || continue
|
||||
if docker rm -f -v "$id" >/dev/null; then
|
||||
n=$((n + 1))
|
||||
else
|
||||
echo "$conf: docker rm $name failed" >&2
|
||||
failed=1
|
||||
fi
|
||||
done
|
||||
fi
|
||||
fi
|
||||
if [ -n "$TMP_PATTERN" ]; then
|
||||
if ! [[ "$TMP_PATTERN" =~ ^$PROJECT-[A-Za-z0-9_.*?-]*$ ]]; then
|
||||
echo "$conf: TMP_PATTERN must be $PROJECT- followed only by letters, digits and _.-*? -- skipped" >&2
|
||||
failed=1
|
||||
else
|
||||
local dir list entry ctime age
|
||||
local -a entries
|
||||
list=$(mktemp)
|
||||
# act-runner runs with PrivateTmp: its /tmp is under the host's.
|
||||
for dir in "$TMP_ROOT"/systemd-private-*-act-runner.service-*/tmp; do
|
||||
[ -d "$dir" ] || continue
|
||||
if ! find "$dir" -mindepth 1 -maxdepth 1 -name "$TMP_PATTERN" -print0 >"$list"; then
|
||||
echo "$conf: find in $dir failed" >&2
|
||||
failed=1
|
||||
continue
|
||||
fi
|
||||
mapfile -d '' entries <"$list"
|
||||
for entry in "${entries[@]}"; do
|
||||
if ! ctime=$(stat -c %Z -- "$entry"); then
|
||||
echo "$conf: stat $entry failed" >&2
|
||||
failed=1
|
||||
continue
|
||||
fi
|
||||
age=$((now - ctime))
|
||||
[ "$age" -gt "$limit" ] || continue
|
||||
echo "tmp $entry ($((age / 3600)) h)"
|
||||
[ "$dry" -eq 0 ] || continue
|
||||
if rm -rf -- "$entry"; then
|
||||
n=$((n + 1))
|
||||
else
|
||||
echo "$conf: rm $entry failed" >&2
|
||||
failed=1
|
||||
fi
|
||||
done
|
||||
done
|
||||
rm -f -- "$list"
|
||||
fi
|
||||
fi
|
||||
if [ "$dry" -eq 0 ]; then
|
||||
echo "removed for $PROJECT: $n"
|
||||
else
|
||||
echo "removed for $PROJECT: none (DRY_RUN, the lines above are what would go)"
|
||||
fi
|
||||
return "$failed"
|
||||
}
|
||||
|
||||
# project_of says which project a file claims, or nothing if it cannot be
|
||||
# read; what is wrong with such a file is said by its own run below.
|
||||
project_of() { (read_conf "$1" >/dev/null 2>&1 && printf '%s' "$PROJECT"); }
|
||||
|
||||
# A project named in two files is claimed twice, and neither file can be
|
||||
# told from a copy that should not be there: both are refused.
|
||||
status=0
|
||||
shopt -s nullglob
|
||||
declare -A claims=()
|
||||
for conf in "$CONF_DIR"/*.conf; do
|
||||
p=$(project_of "$conf") || continue
|
||||
[ -z "$p" ] || claims[$p]=$((${claims[$p]:-0} + 1))
|
||||
done
|
||||
|
||||
# One project at a time, each in a subshell of its own, with errexit on
|
||||
# inside it: a failure one project's cleanup did not foresee still fails that
|
||||
# project, and only that one. The run says so in its exit code.
|
||||
for conf in "$CONF_DIR"/*.conf; do
|
||||
p=$(project_of "$conf") || p=''
|
||||
if [ -n "$p" ] && [ "${claims[$p]}" -gt 1 ]; then
|
||||
echo "$conf: PROJECT $p is claimed by ${claims[$p]} files -- skipped" >&2
|
||||
status=1
|
||||
continue
|
||||
fi
|
||||
set +e
|
||||
(
|
||||
set -e
|
||||
clean_project "$conf"
|
||||
)
|
||||
rc=$?
|
||||
set -e
|
||||
if [ "$rc" -ne 0 ]; then
|
||||
echo "$conf: cleanup failed" >&2
|
||||
status=1
|
||||
fi
|
||||
done
|
||||
exit "$status"
|
||||
@@ -6,6 +6,7 @@
|
||||
# - Docker (für Wegwerf-Test-Container in CI, z. B. pgvector)
|
||||
# - act_runner als unprivilegierter User "runner" in der docker-Gruppe
|
||||
# - systemd-Unit act-runner.service (Requires=docker.service)
|
||||
# - ci-runner-cleanup.timer: täglicher Aufräumer nach Label, Präfix und Alter
|
||||
#
|
||||
# SICHERHEITSMODELL (Review-Finding 88, bewusst akzeptiert & dokumentiert):
|
||||
# docker-Gruppe == de-facto root IN DIESEM LXC. Da der Runner instanzweit
|
||||
@@ -213,6 +214,58 @@ systemctl daemon-reload
|
||||
systemctl enable --now act-runner.service
|
||||
msg_ok "act-runner.service installed + started"
|
||||
|
||||
# ── Täglicher Aufräumer (l.kirchner/patchmgr D64) ─────────────────────────────
|
||||
# Ein abgebrochener Job erreicht seinen eigenen Abbau-Schritt nie: Test-Datenbank
|
||||
# und Baumkopien bleiben liegen. Gemessen auf runner-01 am 07.10.2026: 11
|
||||
# Container, 209 Volumes (19 GB) und 4 Kopien (je 1,2 GB) allein aus patchmgr.
|
||||
# Der Aufräumer entfernt NUR, was ein Projekt unter /etc/ci-runner-cleanup.d/
|
||||
# als seines anmeldet — Container per Label org.luki-net.project=<projekt> UND
|
||||
# Namensmuster, tmp-Einträge per Muster mit Projektpräfix — und nur, was älter
|
||||
# als 4 h ist (Container ab ihrem letzten Start, tmp-Einträge nach ctime):
|
||||
# act_runner v0.2.13 beendet einen Job nach 3 h (runner.timeout, gemessen auf
|
||||
# runner-01..03 am 08.10.2026), plus Rand. Die Datei wird gelesen, nicht
|
||||
# ausgeführt; Projektnamen ohne Bindestrich, damit das Präfix eindeutig ist. Kein `docker prune`,
|
||||
# denn dieser Runner arbeitet für jedes Repo der Instanz. Ein Projekt meldet
|
||||
# sich mit einer eigenen Datei an; das Skript bleibt gleich.
|
||||
msg_info "Installing ci-runner-cleanup (daily, by project label, prefix and age)..."
|
||||
curl -fsSL "${LIB_URL%/lib}/install/runner-cleanup.sh" -o /usr/local/sbin/ci-runner-cleanup
|
||||
chmod 755 /usr/local/sbin/ci-runner-cleanup
|
||||
mkdir -p /etc/ci-runner-cleanup.d
|
||||
cat >/etc/ci-runner-cleanup.d/patchmgr.conf <<'CONF_EOF'
|
||||
# l.kirchner/patchmgr (D64): test databases from deploy/testdb.sh (labelled
|
||||
# org.luki-net.project=patchmgr), tree copies from cmd/counter-check.
|
||||
PROJECT=patchmgr
|
||||
CONTAINER_NAME_PATTERN='^patchmgr-testdb-'
|
||||
TMP_PATTERN='patchmgr-counter-check-*'
|
||||
CONF_EOF
|
||||
cat >/etc/systemd/system/ci-runner-cleanup.service <<'UNIT_EOF'
|
||||
[Unit]
|
||||
Description=Remove what cancelled CI jobs left behind (by project label, prefix and age)
|
||||
Requires=docker.service
|
||||
After=docker.service
|
||||
|
||||
[Service]
|
||||
Type=oneshot
|
||||
EnvironmentFile=-/etc/default/ci-runner-cleanup
|
||||
ExecStart=/usr/local/sbin/ci-runner-cleanup
|
||||
UNIT_EOF
|
||||
cat >/etc/systemd/system/ci-runner-cleanup.timer <<'UNIT_EOF'
|
||||
[Unit]
|
||||
Description=Daily cleanup of the CI runner
|
||||
|
||||
[Timer]
|
||||
OnCalendar=daily
|
||||
RandomizedDelaySec=1h
|
||||
Persistent=true
|
||||
|
||||
[Install]
|
||||
WantedBy=timers.target
|
||||
UNIT_EOF
|
||||
systemctl daemon-reload
|
||||
# Nur der Timer: der erste Lauf kommt mit ihm, nicht mit der Installation.
|
||||
systemctl enable --now ci-runner-cleanup.timer
|
||||
msg_ok "ci-runner-cleanup.timer installed (DRY_RUN=1 ci-runner-cleanup lists only)"
|
||||
|
||||
CRED_FILE="/root/runner.credentials"
|
||||
cat >"$CRED_FILE" <<EOF
|
||||
Allgemeiner Gitea-Actions-Runner (instanzweit)
|
||||
@@ -222,6 +275,8 @@ Name/Label: $RUNNER_NAME [$RUNNER_LABELS]
|
||||
User: $APP_USER (docker-Gruppe, KEIN sudo)
|
||||
Workdir: $RUNNER_DIR
|
||||
Logs: journalctl -u act-runner -f
|
||||
Aufräumer: ci-runner-cleanup.timer täglich; Muster in /etc/ci-runner-cleanup.d/
|
||||
(DRY_RUN=1 /usr/local/sbin/ci-runner-cleanup listet nur)
|
||||
|
||||
Repos nutzen ihn mit: runs-on: ${RUNNER_LABELS%%:*}
|
||||
|
||||
|
||||
@@ -0,0 +1,204 @@
|
||||
#!/usr/bin/env bash
|
||||
# Tests for install/runner-cleanup.sh, the runner's daily cleaner. Runs without
|
||||
# Docker and without root: docker and stat are stand-ins on PATH that answer
|
||||
# from tables, and act-runner's private /tmp is a directory under a temp root.
|
||||
# stat is stood in because a ctime cannot be set back, and the cleaner goes by
|
||||
# ctime; find and rm are real, under the temp root only. The cleaner's own
|
||||
# list file (mktemp) goes to a directory of the test too, not to /tmp.
|
||||
# Aufruf: bash tests/test_runner_cleanup.sh
|
||||
set -u
|
||||
|
||||
REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
|
||||
CLEANUP="$REPO_ROOT/install/runner-cleanup.sh"
|
||||
|
||||
PASS=0
|
||||
FAIL=0
|
||||
ok() { PASS=$((PASS + 1)); echo "ok - $1"; }
|
||||
nok() { FAIL=$((FAIL + 1)); echo "NOT OK - $1"; }
|
||||
|
||||
# Setup must not half-work: a stand-in that is not executable would let the
|
||||
# real docker on PATH answer.
|
||||
die() { echo "setup failed: $1" >&2; exit 1; }
|
||||
T="$(mktemp -d)" || die mktemp
|
||||
trap 'rm -rf "$T"' EXIT
|
||||
mkdir -p "$T/conf" "$T/bin" "$T/own" || die mkdir
|
||||
|
||||
# patchmgr's file, as runner-install.sh lays it down.
|
||||
cat >"$T/conf/patchmgr.conf" <<'CONF_EOF' || die conf
|
||||
# l.kirchner/patchmgr (D64)
|
||||
PROJECT=patchmgr
|
||||
CONTAINER_NAME_PATTERN='^patchmgr-testdb-'
|
||||
TMP_PATTERN='patchmgr-counter-check-*'
|
||||
CONF_EOF
|
||||
|
||||
# docker: by the label org.luki-net.project=patchmgr it knows four containers
|
||||
# -- "old" (created and started two days ago), "new" (one hour), "stray"
|
||||
# (two days, labelled but named like something else) and "restarted"
|
||||
# (created two days ago, started again an hour ago). An unlabelled container
|
||||
# never comes back from a filter by label. rm of an id listed in
|
||||
# $T/rm-fails fails.
|
||||
OLD="$(date -u -d '2 days ago' +%FT%TZ)"
|
||||
NEW="$(date -u -d '1 hour ago' +%FT%TZ)"
|
||||
cat >"$T/bin/docker" <<DOCKER_EOF || die docker
|
||||
#!/usr/bin/env bash
|
||||
echo "\$*" >> "$T/calls"
|
||||
case "\$1" in
|
||||
ps) [ "\$4" = "label=org.luki-net.project=patchmgr" ] && printf 'old\nnew\nstray\nrestarted\n' ;;
|
||||
inspect)
|
||||
case "\$4" in
|
||||
old) echo "/patchmgr-testdb-1-x $OLD $OLD" ;;
|
||||
new) echo "/patchmgr-testdb-2-x $NEW $NEW" ;;
|
||||
stray) echo "/other-db $OLD $OLD" ;;
|
||||
restarted) echo "/patchmgr-testdb-3-x $OLD $NEW" ;;
|
||||
esac ;;
|
||||
rm) grep -qx "\$4" "$T/rm-fails" 2>/dev/null && exit 1 ;;
|
||||
esac
|
||||
exit 0
|
||||
DOCKER_EOF
|
||||
# stat -c %Z: the ctime from $T/ctimes by base name, "now" for anything else.
|
||||
cat >"$T/bin/stat" <<STAT_EOF || die stat
|
||||
#!/usr/bin/env bash
|
||||
[ "\$1 \$2" = "-c %Z" ] || exit 2
|
||||
b=\$(basename "\$4")
|
||||
t=\$(awk -v b="\$b" '\$1 == b { print \$2 }' "$T/ctimes")
|
||||
echo "\${t:-\$(date +%s)}"
|
||||
STAT_EOF
|
||||
chmod +x "$T/bin/docker" "$T/bin/stat" || die chmod
|
||||
[ -x "$T/bin/docker" ] && [ -x "$T/bin/stat" ] || die "stand-ins not executable"
|
||||
TWO_DAYS_AGO=$(($(date +%s) - 2 * 86400))
|
||||
|
||||
# act-runner's PrivateTmp: patchmgr's old and new copy, a copy with an old
|
||||
# mtime but a fresh ctime (cp -a of an old tree), and old entries that are
|
||||
# not patchmgr's.
|
||||
fresh_tmp() {
|
||||
rm -rf "$T/root"
|
||||
R="$T/root/systemd-private-x-act-runner.service-y/tmp"
|
||||
mkdir -p "$R/patchmgr-counter-check-old" "$R/patchmgr-counter-check-new" "$R/patchmgr-counter-check-copied" \
|
||||
"$R/counter-check-old" "$R/go-build-old" "$R/patchmgrx-old" || die tmp
|
||||
touch -d '2 days ago' "$R/patchmgr-counter-check-copied" || die touch
|
||||
printf '%s %s\n' patchmgr-counter-check-old "$TWO_DAYS_AGO" counter-check-old "$TWO_DAYS_AGO" \
|
||||
go-build-old "$TWO_DAYS_AGO" patchmgrx-old "$TWO_DAYS_AGO" >"$T/ctimes"
|
||||
: >"$T/calls"
|
||||
rm -f "$T/rm-fails"
|
||||
}
|
||||
|
||||
cleanup() { PATH="$T/bin:$PATH" CONF_DIR="$T/conf" TMP_ROOT="$T/root" TMPDIR="$T/own" bash "$CLEANUP"; }
|
||||
removed_any() { grep -q '^rm' "$T/calls"; }
|
||||
untouched() { [ -d "$R/counter-check-old" ] && [ -d "$R/go-build-old" ] && [ -d "$R/patchmgrx-old" ] && [ -d "$R/patchmgr-counter-check-old" ]; }
|
||||
|
||||
# ── DRY_RUN: empty or 0 removes, anything else only lists ─────────────────────
|
||||
for v in 1 true yes; do
|
||||
fresh_tmp
|
||||
out="$(DRY_RUN=$v cleanup 2>&1)"
|
||||
if removed_any || [ ! -d "$R/patchmgr-counter-check-old" ]; then
|
||||
nok "DRY_RUN=$v removed something"
|
||||
else
|
||||
ok "DRY_RUN=$v removes nothing"
|
||||
fi
|
||||
done
|
||||
if grep -q 'container patchmgr-testdb-1-x' <<<"$out"; then ok "a dry run lists what it would remove"; else nok "a dry run did not list: $out"; fi
|
||||
if grep -q 'removed for patchmgr: none (DRY_RUN' <<<"$out"; then ok "a dry run says it removed nothing"; else nok "a dry run's last line: $out"; fi
|
||||
for v in 0 ''; do
|
||||
fresh_tmp
|
||||
DRY_RUN=$v cleanup >/dev/null 2>&1
|
||||
if removed_any; then ok "DRY_RUN='$v' removes"; else nok "DRY_RUN='$v' removed nothing"; fi
|
||||
done
|
||||
|
||||
# ── The real run: only patchmgr's, by label and name, older than 4 h ─────────
|
||||
fresh_tmp
|
||||
if cleanup >/dev/null 2>&1; then ok "a clean run exits 0"; else nok "a clean run exited non-zero"; fi
|
||||
if grep -qx 'rm -f -v old' "$T/calls"; then ok "the old container goes, with its volumes (rm -f -v)"; else nok "the old container was not removed with -v: $(grep '^rm' "$T/calls")"; fi
|
||||
if grep -q '^rm .*new' "$T/calls"; then nok "a container younger than 4 h was removed"; else ok "a container younger than 4 h stays"; fi
|
||||
if grep -q '^rm .*restarted' "$T/calls"; then nok "a container started again an hour ago was removed"; else ok "a container counts from its last start, not its creation"; fi
|
||||
if grep -q '^rm .*stray' "$T/calls"; then nok "a labelled container with another name was removed"; else ok "the label alone is not enough: the name must match too"; fi
|
||||
if grep -q -- '--filter label=org.luki-net.project=patchmgr' "$T/calls"; then ok "containers are found by the project label"; else nok "docker was not asked by label: $(cat "$T/calls")"; fi
|
||||
if grep -q -- '--filter name=' "$T/calls"; then nok "docker was asked by name, not by label"; else ok "no filter by name alone"; fi
|
||||
if [ ! -e "$R/patchmgr-counter-check-old" ]; then ok "patchmgr's old copy goes"; else nok "patchmgr's old copy is still there"; fi
|
||||
if [ -d "$R/patchmgr-counter-check-new" ]; then ok "a copy younger than 4 h stays"; else nok "the new copy was removed"; fi
|
||||
if [ -d "$R/patchmgr-counter-check-copied" ]; then ok "a fresh copy with an old mtime stays: the age is the ctime"; else nok "a copy with an old mtime but a fresh ctime was removed"; fi
|
||||
if [ -d "$R/counter-check-old" ] && [ -d "$R/go-build-old" ] && [ -d "$R/patchmgrx-old" ]; then ok "entries without patchmgr's prefix stay"; else nok "an entry without the prefix was removed"; fi
|
||||
if grep -q -E '^(system|volume|image|container) prune' "$T/calls"; then nok "docker was asked to prune"; else ok "no docker prune"; fi
|
||||
|
||||
# ── A failure says so in the exit code, and the other projects still run ─────
|
||||
fresh_tmp
|
||||
echo old >"$T/rm-fails"
|
||||
printf "PROJECT=other\nTMP_PATTERN='other-*'\n" >"$T/conf/zz-other.conf"
|
||||
mkdir -p "$R/other-old"
|
||||
echo "other-old $TWO_DAYS_AGO" >>"$T/ctimes"
|
||||
cleanup >"$T/out" 2>&1
|
||||
rc=$?
|
||||
rm -f "$T/conf/zz-other.conf"
|
||||
if [ "$rc" -eq 1 ] && grep -q 'docker rm patchmgr-testdb-1-x failed' "$T/out"; then ok "a failed docker rm makes the run exit 1"; else nok "a failed docker rm: exit $rc: $(cat "$T/out")"; fi
|
||||
if [ ! -e "$R/other-old" ] && [ ! -e "$R/patchmgr-counter-check-old" ]; then ok "the rest still runs after one failure"; else nok "a failure stopped the rest"; fi
|
||||
|
||||
# ── A file cannot claim another project's things, or everything ──────────────
|
||||
refused() { # $1: file content, $2: name for the test
|
||||
fresh_tmp
|
||||
mv "$T/conf/patchmgr.conf" "$T/patchmgr.conf.keep"
|
||||
printf '%s\n' "$1" >"$T/conf/bad.conf"
|
||||
out="$(cleanup 2>&1)"
|
||||
rc=$?
|
||||
rm -f "$T/conf/bad.conf"
|
||||
mv "$T/patchmgr.conf.keep" "$T/conf/patchmgr.conf"
|
||||
if removed_any || ! untouched; then
|
||||
nok "$2 removed something"
|
||||
elif [ "$rc" -ne 0 ] && grep -q 'skipped' <<<"$out"; then
|
||||
ok "$2 is refused"
|
||||
else
|
||||
nok "$2 was not refused (exit $rc): $out"
|
||||
fi
|
||||
}
|
||||
refused "PROJECT=patchmgr
|
||||
CONTAINER_NAME_PATTERN='^.*'" "CONTAINER_NAME_PATTERN='^.*'"
|
||||
refused "PROJECT=patchmgr
|
||||
CONTAINER_NAME_PATTERN='^other-'" "CONTAINER_NAME_PATTERN='^other-'"
|
||||
refused "PROJECT=patchmgr
|
||||
CONTAINER_NAME_PATTERN='^patchmgr-testdb-|.*'" "an alternation after the prefix"
|
||||
refused "PROJECT=patchmgr
|
||||
CONTAINER_NAME_PATTERN='^patchmgr-{0}.*'" "a quantifier that makes the hyphen optional"
|
||||
refused "PROJECT=patchmgr
|
||||
CONTAINER_NAME_PATTERN='^patchmgr-testdb-['" "a pattern that is not a regex"
|
||||
refused "PROJECT=patchmgr
|
||||
TMP_PATTERN='patchmgr-\$(id)'" "a quoted \$(...) in TMP_PATTERN"
|
||||
refused "PROJECT=patchmgr
|
||||
TMP_PATTERN='patchmgr-[a-z]*'" "brackets in TMP_PATTERN"
|
||||
refused "PROJECT=patchmgr
|
||||
TMP_PATTERN='?*'" "TMP_PATTERN='?*'"
|
||||
refused "PROJECT=patchmgr
|
||||
TMP_PATTERN='*'" "TMP_PATTERN='*'"
|
||||
refused "PROJECT=patchmgr
|
||||
TMP_PATTERN='other-*'" "TMP_PATTERN='other-*'"
|
||||
refused "PROJECT=patchmgr
|
||||
TMP_PATTERN='patchmgr-x/y'" "TMP_PATTERN with a directory"
|
||||
refused "PROJECT='Patch Mgr'
|
||||
TMP_PATTERN='*'" "a PROJECT that is not a name"
|
||||
refused "PROJECT=patchmgr-old
|
||||
TMP_PATTERN='patchmgr-old-*'" "a PROJECT with a hyphen (its prefix would overlap patchmgr's)"
|
||||
refused "PROJECT=patchmgr
|
||||
MAX_AGE_HOURS=1
|
||||
TMP_PATTERN='patchmgr-*'" "a file that sets MAX_AGE_HOURS"
|
||||
refused "PROJECT=patchmgr\$(touch $T/ran)
|
||||
TMP_PATTERN='patchmgr-*'" "a file with a command in it"
|
||||
if [ -e "$T/ran" ]; then nok "a command in a file was run"; else ok "a file is read, not run"; fi
|
||||
|
||||
# ── A project claimed by two files is claimed by neither ─────────────────────
|
||||
fresh_tmp
|
||||
printf "PROJECT=patchmgr\nTMP_PATTERN='patchmgr-*'\n" >"$T/conf/copy.conf"
|
||||
out="$(cleanup 2>&1)"
|
||||
rc=$?
|
||||
rm -f "$T/conf/copy.conf"
|
||||
if [ "$rc" -eq 1 ] && ! removed_any && untouched && [ "$(grep -c 'claimed by 2 files' <<<"$out")" -eq 2 ]; then ok "a project in two files is refused in both"; else nok "a project in two files (exit $rc): $out"; fi
|
||||
|
||||
# ── The age cannot go below the job timeout of 3 h ───────────────────────────
|
||||
for age in 0 3 abc 04 1000 4611686018427387904; do
|
||||
if MAX_AGE_HOURS="$age" cleanup >/dev/null 2>&1; then nok "MAX_AGE_HOURS=$age was accepted"; else ok "MAX_AGE_HOURS=$age is refused"; fi
|
||||
done
|
||||
fresh_tmp
|
||||
if MAX_AGE_HOURS=4 cleanup >/dev/null 2>&1; then ok "MAX_AGE_HOURS=4 is accepted"; else nok "MAX_AGE_HOURS=4 was refused"; fi
|
||||
|
||||
# ── The cleaner's list files stay in the test and are removed ────────────────
|
||||
if [ -z "$(ls -A "$T/own")" ]; then ok "no list file is left behind"; else nok "list files left behind: $(ls -A "$T/own")"; fi
|
||||
|
||||
echo
|
||||
echo "pass: $PASS, fail: $FAIL"
|
||||
[ "$FAIL" -eq 0 ]
|
||||
Reference in New Issue
Block a user