Author SHA1 Message Date
claude-bot dd758e6806 feat(runner): install zstd so the actions cache is not gzipped single-threaded
CI / Shell-Lint (bash -n, source-check, Validierungs-Tests) (pull_request) Successful in 2s
actions/cache -- and actions/setup-go, which builds on it -- packs its archive
with zstdmt when zstd is present and falls back to single-threaded gzip when
it is not. The archive name says which happened: cache.tzst against cache.tgz.

Measured in l.kirchner/patchmgr, CI run 1957. The cache is GOMODCACHE plus
GOCACHE and runs to 2-5 GB. runner-gpu has zstd and writes cache.tzst;
runner-01, -02 and -03 write cache.tgz, and go test -race (stable) spent 605
seconds packing it for a job with 41 seconds of work. Across all eight jobs of
that run, 2312 of 3146 seconds went into this step.

This does not settle whether the cache is wanted at all -- in host mode with a
persistent home both directories survive between jobs anyway, and patchmgr is
switching it off for that reason. But as long as any repository on the
instance uses it, it should not be compressed on one core.

zstd is a package for the post step, not for jobs, so it sits with the others
rather than in a workflow: this runner installs nothing at job time, by
design.
2026-09-03 01:04:02 +02:00
l.kirchner 0ec6738217 Merge pull request 'security: version the CIS Tier-A hardening pass' (#11) from security/cis-tierA-hardening into main
CI / Shell-Lint (bash -n, source-check, Validierungs-Tests) (push) Successful in 2s
Reviewed-on: #11
2026-08-19 11:08:14 +02:00
claude-bot 5130b82639 security: version the CIS Tier-A hardening pass
CI / Shell-Lint (bash -n, source-check, Validierungs-Tests) (pull_request) Successful in 2s
The script that sets the SSH, PAM, pwquality and auditd baseline on twelve
containers existed only as a root-owned copy on the machines it hardens.
Bring it into the repo so a change reaches one place instead of twelve.

Two substantive changes over the copy that shipped on 2026-07-24:

- Deny forwarding per option instead of via DisableForwarding. Same effect,
  but DisableForwarding overrides every other forwarding option and is
  invisible in sshd -T, which makes a rejected port-forward read as a
  configuration that should work.
- Quote the command substitution in MODDIR (SC2046).

The header and README now record the rollout command, the containers left
unhardened as break-glass foundation, and why host-specific exceptions must
live in a drop-in that sorts after 99-cis-hardening.conf.
2026-08-19 11:02:15 +02:00
l.kirchner 0ab4f98f4e feat(runner): install the build toolchain compiled languages need (#10)
CI / Shell-Lint (bash -n, source-check, Validierungs-Tests) (push) Successful in 2s
2026-08-18 23:02:16 +02:00
claude-bot 3c77d34b7e docs(runner): name the protoc coupling and fix a mechanism claim
CI / Shell-Lint (bash -n, source-check, Validierungs-Tests) (pull_request) Successful in 2s
Both from the cross-review, both fair.

"Go setzt CGO_ENABLED=0" is right about the effect and wrong about the
mechanism: Go does not set the variable, cgo simply stays off when no C
compiler is found, and go env then reports 0. Reworded.

And protoc on an instance-wide runner ties every repository to the
distribution's version -- 3.21.x on Debian 12. Unlike make and gcc that is a
code generator, so a distro upgrade changes generated code for all users at
once. The comment says so now, and says where a project that needs its own
version should pin it instead of raising it here for everybody.
2026-08-18 23:01:34 +02:00
claude-bot 8c83fb372b feat(runner): install the build toolchain compiled languages need
CI / Shell-Lint (bash -n, source-check, Validierungs-Tests) (pull_request) Successful in 2s
The runner is host-mode, so there is no image bringing tools along: what is
not on this LXC, no job has. Measured on l.kirchner/patchmgr, a Go project,
where all six CI jobs were assigned and every one of them died in the first
seconds:

    make all         make: command not found
    go test -race    go: -race requires cgo; enable cgo by setting CGO_ENABLED=1
    make proto       sudo: command not found

Four packages, each for a reason:

    make               the gate commands are make targets
    gcc                Go turns CGO_ENABLED off when it finds no C compiler,
                       and the race detector cannot be built without cgo
    protobuf-compiler  protoc itself
    libprotobuf-dev    the well-known .proto includes under
                       /usr/include/google/protobuf; without them protoc fails
                       even though the binary is there

sudo stays absent on purpose. A workflow must not be able to install anything
on this runner -- what is needed is declared here, in the script, and not in
somebody's pipeline. That also keeps the security note at the top of this file
honest: the LXC owns nothing, and it gains nothing at a workflow's request.

Go is not in the list. Projects fetch it through actions/setup-go, because CI
matrices run more than one version.

Applied to the running LXC (301 on pve-gamer) while the runner was idle, then
verified: make 4.3, gcc 12.2.0, libprotoc 3.21.12, 11 .proto includes present.
The service PATH already contains /usr/bin, so no restart was needed.
2026-08-18 22:54:54 +02:00
l.kirchner 6fadb27080 Merge pull request 'fix(nexus-db): UTF8-Encoding zur Installationszeit erzwingen (#8)' (#9) from fix/nexus-db-utf8-encoding into main
CI / Shell-Lint (bash -n, source-check, Validierungs-Tests) (push) Successful in 1s
fix(nexus-db): UTF-8-Encoding erzwingen + Post-Install-Check (PR #9, Closes #8) — Locale aktiv zum initdb-Zeitpunkt, Re-Run-Guard, su-statt-sudo-Doku
2026-06-13 14:38:58 +02:00
claude-bot 6543fd77d8 fix(nexus-db): address codex review — early encoding guard, robust helpers
CI / Shell-Lint (bash -n, source-check, Validierungs-Tests) (pull_request) Successful in 2s
- assert encoding BEFORE role/password mutation on re-run, so an old
  SQL_ASCII DB aborts with no side effects (codex finding 1)
- ensure_utf8_locale_active honours its locale argument consistently in
  match, locale.gen line and export (codex finding 2)
- assert_db_encoding_utf8 uses argv-clean runuser psql with :'db' literal
  binding instead of nested su -c shell; docs keep su - postgres -c
  (codex finding 3)
2026-06-13 14:34:35 +02:00
claude-bot bd4293f53e fix(nexus-db): enforce UTF8 encoding at install time (issue #8)
CI / Shell-Lint (bash -n, source-check, Validierungs-Tests) (pull_request) Successful in 1s
A PostgreSQL cluster/database freezes its encoding at initdb / CREATE
DATABASE time; a C (non-UTF-8) locale yields a SQL_ASCII cluster, which
makes psycopg3 return bytes and crashes SQLAlchemy. Harden the installer
and add a reusable pattern for future DB installers:

- ensure_utf8_locale_active: generate AND activate en_US.UTF-8 for the
  install process before the server package runs initdb; abort if the
  locale is not actually available
- create the database explicitly with TEMPLATE template0 ENCODING 'UTF8'
  LC_COLLATE/LC_CTYPE 'en_US.UTF-8' instead of inheriting the cluster
  default
- assert_db_encoding_utf8: post-install guard, abort with an actionable
  message if pg_encoding_to_char is not UTF8 (catches old SQL_ASCII DBs
  on re-run too)
- credentials/README docs use su - postgres -c (minimal LXCs have no sudo)
2026-06-13 14:30:40 +02:00
l.kirchner 553b923445 Merge pull request 'docs: README-Dedup — Contributing konsolidiert, Stand aktualisiert' (#7) from chore/readme-dedup into main
CI / Shell-Lint (bash -n, source-check, Validierungs-Tests) (push) Successful in 2s
docs: README-Dedup nach K-114-Merge (PR #7) — ein Contributing-Abschnitt, Status/CI aktuell
2026-06-12 19:42:48 +02:00
5 changed files with 312 additions and 8 deletions
+22 -1
View File
@@ -41,6 +41,8 @@ Two files per app, both sourcing the shared libs via `curl`:
- **`ct/<app>.sh`** runs on the PVE host: prompts → unprivileged LXC → pushes a config env file into the container → bootstraps the installer. Apps that need Docker (authentik, runner) enable `nesting+keyctl` automatically. The standard prompts include an **SSH root login choice** (`SSH_ROOT_LOGIN`, default yes for homelab convenience; `no` keeps the Debian key-only default) — applied inside the container as an sshd drop-in by `configure_ssh_root_login`. - **`ct/<app>.sh`** runs on the PVE host: prompts → unprivileged LXC → pushes a config env file into the container → bootstraps the installer. Apps that need Docker (authentik, runner) enable `nesting+keyctl` automatically. The standard prompts include an **SSH root login choice** (`SSH_ROOT_LOGIN`, default yes for homelab convenience; `no` keeps the Debian key-only default) — applied inside the container as an sshd drop-in by `configure_ssh_root_login`.
- **`install/<app>-install.sh`** runs inside the LXC: packages, unprivileged app user, secrets generated on-host (never printed), systemd units, a `/root/<app>.credentials` notes file — then shreds the bootstrap env. Idempotent where it matters: re-runs skip what exists. `setup_base_apt` also fixes the bare-template **locale situation**: `C.UTF-8` is exported up front (glibc built-in, covers the first apt run without perl warnings), then `en_US.UTF-8` is generated and set as the system default. - **`install/<app>-install.sh`** runs inside the LXC: packages, unprivileged app user, secrets generated on-host (never printed), systemd units, a `/root/<app>.credentials` notes file — then shreds the bootstrap env. Idempotent where it matters: re-runs skip what exists. `setup_base_apt` also fixes the bare-template **locale situation**: `C.UTF-8` is exported up front (glibc built-in, covers the first apt run without perl warnings), then `en_US.UTF-8` is generated and set as the system default.
**DB installers** carry one extra rule: a PostgreSQL cluster/database freezes its encoding at initdb / `CREATE DATABASE` time and it can never be changed afterwards. A C (non-UTF-8) locale yields a `SQL_ASCII` cluster — psycopg3 then hands text back as bytes and SQLAlchemy crashes. So the pattern (helpers `ensure_utf8_locale_active` + `assert_db_encoding_utf8` in `lib/install.func`) is: make a UTF-8 locale **active** before the server package runs initdb, create the database **explicitly** with `TEMPLATE template0 ENCODING 'UTF8' LC_COLLATE/LC_CTYPE 'en_US.UTF-8'` (never inherit the cluster default), and **verify** `pg_encoding_to_char` returns `UTF8` before finishing — a wrong encoding aborts the install (it's DB damage, see wiki → Lessons). Maintenance examples use `su - postgres -c …`, not `sudo` — these minimal LXCs have no sudo.
Shared libs: [`lib/build.func`](lib/build.func) (host-side: prompts, LXC create, bootstrap) and [`lib/install.func`](lib/install.func) (in-container: apt, users, systemd, http-wait). Shared libs: [`lib/build.func`](lib/build.func) (host-side: prompts, LXC create, bootstrap) and [`lib/install.func`](lib/install.func) (in-container: apt, users, systemd, http-wait).
## Security conventions ## Security conventions
@@ -50,6 +52,24 @@ Shared libs: [`lib/build.func`](lib/build.func) (host-side: prompts, LXC create,
- Secrets are generated on the target host and live in `0600` files — never in the repo, the wiki or chat logs. - Secrets are generated on the target host and live in `0600` files — never in the repo, the wiki or chat logs.
- **Runner separation:** the nexus runner is repo-scoped and lives on the production LXC *because* it holds deploy rights; the general `runner` LXC is instance-wide *because* it holds none. Don't mix these scopes. - **Runner separation:** the nexus runner is repo-scoped and lives on the production LXC *because* it holds deploy rights; the general `runner` LXC is instance-wide *because* it holds none. Don't mix these scopes.
### CIS Tier-A hardening
[`install/cis-tierA.sh`](install/cis-tierA.sh) applies the Tier-A baseline to an existing Debian 12 LXC: SSH drop-in, `login.defs` aging + YESCRYPT, pwquality/faillock, PAM (pwquality, pwhistory, faillock, `nullok` removed) and auditd rules. It is idempotent, backs everything up to `/root/cis-hardening-backup-<ts>/`, gates the SSH restart on `sshd -t` and rolls PAM back if a referenced module is missing.
```bash
pct push <id> install/cis-tierA.sh /root/cis-tierA.sh
pct exec <id> -- bash /root/cis-tierA.sh
```
Applied to authentik first (2026-07-24), then to 11 further containers. The PVE hosts and the Wazuh manager VM stay unhardened on purpose — they are the break-glass foundation.
Two things worth knowing before touching it:
- **Section 1 rewrites `99-cis-hardening.conf` wholesale on every run.** Host-specific exceptions belong in a separate drop-in that sorts *after* it (e.g. `99-zz-local-forward.conf`), never in that file. A `Match` block extends until the next `Match` — across `Include` file boundaries — so such a drop-in has to stay alphabetically last.
- **Forwarding is denied per option, not via `DisableForwarding`.** Both are equivalent in effect, but `DisableForwarding` overrides every other forwarding option *and* does not appear in `sshd -T` output. A denied port-forward then reports `administratively prohibited` while `sshd -T` cheerfully claims `allowtcpforwarding yes`, which costs hours to diagnose.
Caveat: auditd is a no-op in unprivileged LXCs (the host owns the audit subsystem); the script probes for it and removes the package again if it cannot load rules. Together with the Section-1 partition/kernel checks that are equally N/A in a container, the achievable SCA score stays well below 100 %.
## Contributing ## Contributing
**All changes go through a pull request with cross-review** (Claude Code ↔ Codex, or a human) — no direct pushes to `main`. This rule exists because of two real incidents: a pasted VLAN tag carrying an invisible non-UTF-8 byte broke `pct create` mid-run, and the "missing `source build.func`" bug shipped twice — caught in review on the nexus-db PR, but reaching production via an un-reviewed authentik commit (see wiki → Lessons). **All changes go through a pull request with cross-review** (Claude Code ↔ Codex, or a human) — no direct pushes to `main`. This rule exists because of two real incidents: a pasted VLAN tag carrying an invisible non-UTF-8 byte broke `pct create` mid-run, and the "missing `source build.func`" bug shipped twice — caught in review on the nexus-db PR, but reaching production via an un-reviewed authentik commit (see wiki → Lessons).
@@ -63,7 +83,8 @@ Build new app prompts on `prompt_validated`/`require_valid` from `lib/build.func
``` ```
. .
├── ct/ # Host-side scripts, one per app ├── ct/ # Host-side scripts, one per app
├── install/ # In-container installers (+ nexus-runtime.sh re-provisioner) ├── install/ # In-container installers (+ nexus-runtime.sh re-provisioner,
│ # cis-tierA.sh hardening pass)
├── lib/ ├── lib/
│ ├── build.func # Shared host-side helpers (prompts, LXC create, bootstrap) │ ├── build.func # Shared host-side helpers (prompts, LXC create, bootstrap)
│ └── install.func # Shared in-container helpers (apt, systemd, users, http-wait) │ └── install.func # Shared in-container helpers (apt, systemd, users, http-wait)
+163
View File
@@ -0,0 +1,163 @@
#!/bin/bash
# CIS Tier-A hardening for a Debian 12 LXC. Idempotent, backs up everything,
# gates the SSH restart on `sshd -t`, auto-rolls-back PAM on sanity failure.
# Recovery path if anything breaks: `pct exec <id> -- bash` from the PVE host.
#
# Rollout: copy into the target LXC and run as root, e.g.
# pct push <id> install/cis-tierA.sh /root/cis-tierA.sh
# pct exec <id> -- bash /root/cis-tierA.sh
# First applied to authentik on 2026-07-24, then to 11 further containers.
#
# NOT hardened on purpose (break-glass foundation): the PVE hosts pve-gamer /
# pve-i5 and the Wazuh manager VM. See the wiki for the break-glass chain.
#
# SSH: section 1 rewrites /etc/ssh/sshd_config.d/99-cis-hardening.conf WHOLESALE
# on every run. Host-specific exceptions therefore do NOT belong in that file —
# put them in a separate drop-in that sorts AFTER it, e.g.
# 99-zz-local-forward.conf. Mind that a Match block extends until the next
# Match, across Include file boundaries, so such a drop-in must stay last.
#
# Forwarding is denied per option (AllowTcpForwarding / AllowAgentForwarding /
# AllowStreamLocalForwarding / X11Forwarding), not via DisableForwarding. Both
# are equivalent in effect, but DisableForwarding overrides every other
# forwarding option AND is invisible in `sshd -T` output — which makes a denied
# port-forward practically undiagnosable. See the wiki entry on that.
set -u
TS=$(date +%Y%m%d-%H%M%S)
BK=/root/cis-hardening-backup-$TS
mkdir -p "$BK"
export DEBIAN_FRONTEND=noninteractive
say(){ echo "[cis] $*"; }
########## 1. SSH hardening (drop-in, validated) ##########
SSHD=/etc/ssh/sshd_config.d/99-cis-hardening.conf
grep -q "Include /etc/ssh/sshd_config.d" /etc/ssh/sshd_config || echo "Include /etc/ssh/sshd_config.d/*.conf" > /tmp/_noinc
[ -f "$SSHD" ] && cp "$SSHD" "$BK/" 2>/dev/null
printf '%s\n' "Warning: Authorized access only. All activity is monitored." > /etc/issue.net
cat > "$SSHD" <<'EOF'
PermitRootLogin prohibit-password
MaxAuthTries 4
LoginGraceTime 60
ClientAliveInterval 15
ClientAliveCountMax 3
Banner /etc/issue.net
MaxStartups 10:30:60
AllowTcpForwarding no
AllowAgentForwarding no
AllowStreamLocalForwarding no
X11Forwarding no
Ciphers chacha20-poly1305@openssh.com,aes256-gcm@openssh.com,aes128-gcm@openssh.com,aes256-ctr,aes192-ctr,aes128-ctr
MACs hmac-sha2-512-etm@openssh.com,hmac-sha2-256-etm@openssh.com,umac-128-etm@openssh.com
KexAlgorithms curve25519-sha256,curve25519-sha256@libssh.org,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512
EOF
if sshd -t 2>/tmp/sshderr; then
systemctl restart ssh 2>/dev/null || systemctl restart sshd 2>/dev/null
say "SSH: drop-in applied + restarted (sshd -t OK, active=$(systemctl is-active ssh 2>/dev/null || systemctl is-active sshd 2>/dev/null))"
else
rm -f "$SSHD"; say "SSH: sshd -t FAILED -> reverted ($(cat /tmp/sshderr))"
fi
########## 2. login.defs (password aging + hashing) ##########
cp /etc/login.defs "$BK/"
setdef(){ if grep -qE "^\s*$1\b" /etc/login.defs; then sed -i -E "s|^\s*$1\b.*|$1 $2|" /etc/login.defs; else echo "$1 $2" >> /etc/login.defs; fi; }
setdef PASS_MAX_DAYS 365
setdef PASS_MIN_DAYS 1
setdef PASS_WARN_AGE 7
setdef ENCRYPT_METHOD YESCRYPT
useradd -D -f 30 2>/dev/null
say "login.defs: aging + YESCRYPT set"
########## 3. pwquality ##########
apt-get install -y libpam-pwquality >/dev/null 2>&1
[ -f /etc/security/pwquality.conf ] && cp /etc/security/pwquality.conf "$BK/"
cat > /etc/security/pwquality.conf <<'EOF'
minlen = 14
minclass = 4
maxrepeat = 3
dictcheck = 1
enforcing = 1
EOF
[ -f /etc/security/faillock.conf ] && cp /etc/security/faillock.conf "$BK/"
cat > /etc/security/faillock.conf <<'EOF'
deny = 5
unlock_time = 900
fail_interval = 900
EOF
say "pwquality.conf + faillock.conf written"
########## 4. PAM module enablement (backup + sanity + rollback) ##########
CA=/etc/pam.d/common-auth
CP=/etc/pam.d/common-password
cp "$CA" "$BK/"; cp "$CP" "$BK/"
# 5.3.3.4.1 remove nullok
sed -i 's/[[:space:]]*nullok//g' "$CA" "$CP"
# common-password: full, correctly-formed pwquality + pwhistory lines before pam_unix
grep -q pam_pwquality.so "$CP" || sed -i '0,/^password[[:space:]].*pam_unix.so/s//password requisite pam_pwquality.so retry=3\n&/' "$CP"
grep -q pam_pwhistory.so "$CP" || sed -i '0,/^password[[:space:]].*pam_unix.so/s//password required pam_pwhistory.so remember=5 use_authtok\n&/' "$CP"
grep -qE 'pam_unix.so.*use_authtok' "$CP" || sed -i -E 's/(^password[[:space:]].*pam_unix.so.*)/\1 use_authtok/' "$CP"
# common-auth: faillock preauth/authfail/authsucc (full lines, idempotent)
if ! grep -q pam_faillock.so "$CA"; then
sed -i '1i auth required pam_faillock.so preauth' "$CA"
awk 'BEGIN{d=0}{print} (/pam_unix.so/ && d==0){print "auth [default=die] pam_faillock.so authfail"; print "auth sufficient pam_faillock.so authsucc"; d=1}' "$CA" > "$CA.tmp" && mv "$CA.tmp" "$CA"
fi
# sanity: every referenced module must exist; pam_unix + pam_deny must remain
MODDIR=$(dirname "$(find /lib /usr/lib -name pam_unix.so 2>/dev/null | head -1)")
ok=1
for m in $(grep -hoE 'pam_[a-z_]+\.so' "$CA" "$CP" | sort -u); do
[ -f "$MODDIR/$m" ] || { say "PAM sanity: missing $m"; ok=0; }
done
grep -q pam_unix.so "$CA" && grep -q pam_unix.so "$CP" || ok=0
if [ "$ok" != "1" ]; then
cp "$BK/common-auth" "$CA"; cp "$BK/common-password" "$CP"
say "PAM: sanity FAILED -> rolled back common-auth/common-password"
else
say "PAM: pwquality/pwhistory/faillock enabled, nullok removed (sanity OK)"
fi
########## 5. auditd (probe LXC support) ##########
apt-get install -y auditd audispd-plugins >/dev/null 2>&1
AUOK=0
if auditctl -l >/dev/null 2>&1 && auditctl -a always,exit -F arch=b64 -S adjtimex -k _probe 2>/dev/null; then
auditctl -d always,exit -F arch=b64 -S adjtimex -k _probe 2>/dev/null; AUOK=1
fi
if [ "$AUOK" = "1" ]; then
cp -n /etc/audit/rules.d/audit.rules "$BK/" 2>/dev/null
cat > /etc/audit/rules.d/cis.rules <<'EOF'
-w /etc/group -p wa -k identity
-w /etc/passwd -p wa -k identity
-w /etc/shadow -p wa -k identity
-w /etc/gshadow -p wa -k identity
-w /etc/security/opasswd -p wa -k identity
-w /etc/sudoers -p wa -k scope
-w /etc/sudoers.d/ -p wa -k scope
-w /var/log/sudo.log -p wa -k sudo_log
-a always,exit -F arch=b64 -S adjtimex,settimeofday,clock_settime -k time-change
-a always,exit -F arch=b64 -S sethostname,setdomainname -k system-locale
-w /etc/hosts -p wa -k system-locale
-w /etc/network/ -p wa -k system-locale
-w /var/log/wtmp -p wa -k session
-w /var/log/btmp -p wa -k session
-w /var/run/utmp -p wa -k session
-w /var/log/lastlog -p wa -k logins
-w /var/run/faillock/ -p wa -k logins
-a always,exit -F arch=b64 -S mount -F auid>=1000 -F auid!=4294967295 -k mounts
-a always,exit -F arch=b64 -S unlink,unlinkat,rename,renameat -F auid>=1000 -F auid!=4294967295 -k delete
-a always,exit -F arch=b64 -S chmod,fchmod,fchmodat,chown,fchown,fchownat,lchown -F auid>=1000 -F auid!=4294967295 -k perm_mod
-a always,exit -F arch=b64 -S init_module,delete_module,finit_module -k modules
-w /usr/sbin/usermod -p x -k usermod
EOF
systemctl enable auditd >/dev/null 2>&1
augenrules --load >/dev/null 2>&1
systemctl restart auditd 2>/dev/null || service auditd restart 2>/dev/null
say "auditd: FUNCTIONAL in this LXC -> CIS rules loaded ($(auditctl -l 2>/dev/null | wc -l) rules)"
else
systemctl disable --now auditd >/dev/null 2>&1
apt-get purge -y auditd audispd-plugins >/dev/null 2>&1
say "auditd: NOT supported in this LXC (host owns audit subsystem) -> N/A, removed"
fi
say "DONE. Backup: $BK"
+27 -5
View File
@@ -47,6 +47,12 @@ else
msg_warn "PGDG repo already present, skipping" msg_warn "PGDG repo already present, skipping"
fi fi
# The server package runs initdb for the `main` cluster on install — its
# encoding is frozen there. Make a UTF-8 locale active for THIS process first
# so the cluster is never created as SQL_ASCII (issue #8: a pre-fix LXC was
# provisioned under LANG=C and ended up SQL_ASCII).
ensure_utf8_locale_active en_US.UTF-8
msg_info "Installing PostgreSQL $PG_MAJOR + pgvector..." msg_info "Installing PostgreSQL $PG_MAJOR + pgvector..."
apt-get install -y -qq "postgresql-$PG_MAJOR" "postgresql-$PG_MAJOR-pgvector" >/dev/null apt-get install -y -qq "postgresql-$PG_MAJOR" "postgresql-$PG_MAJOR-pgvector" >/dev/null
msg_ok "PostgreSQL $(psql --version | awk '{print $3}') installed" msg_ok "PostgreSQL $(psql --version | awk '{print $3}') installed"
@@ -95,6 +101,13 @@ systemctl restart postgresql
# ── role + database + extension (idempotent, password kept on re-run) ───────── # ── role + database + extension (idempotent, password kept on re-run) ─────────
run_psql() { runuser -u postgres -- psql -v ON_ERROR_STOP=1 -qAt "$@"; } run_psql() { runuser -u postgres -- psql -v ON_ERROR_STOP=1 -qAt "$@"; }
# Re-run safety (issue #8): if the database already exists, verify its
# encoding BEFORE touching roles/passwords. An old SQL_ASCII database must
# abort the run with NO side effects — not after rotating credentials.
if [[ "$(run_psql -c "SELECT 1 FROM pg_database WHERE datname='$DB_NAME'")" == "1" ]]; then
assert_db_encoding_utf8 "$DB_NAME"
fi
if [[ "$(run_psql -c "SELECT 1 FROM pg_roles WHERE rolname='$DB_USER'")" != "1" ]]; then if [[ "$(run_psql -c "SELECT 1 FROM pg_roles WHERE rolname='$DB_USER'")" != "1" ]]; then
msg_info "Creating role $DB_USER + database $DB_NAME..." msg_info "Creating role $DB_USER + database $DB_NAME..."
DB_PASS="$(openssl rand -base64 32 | tr -d '/+=' | head -c 32)" DB_PASS="$(openssl rand -base64 32 | tr -d '/+=' | head -c 32)"
@@ -114,12 +127,18 @@ else
fi fi
if [[ "$(run_psql -c "SELECT 1 FROM pg_database WHERE datname='$DB_NAME'")" != "1" ]]; then if [[ "$(run_psql -c "SELECT 1 FROM pg_database WHERE datname='$DB_NAME'")" != "1" ]]; then
run_psql -c "CREATE DATABASE $DB_NAME OWNER $DB_USER" # Explicit encoding/collation from template0 — never inherit the cluster
# default, which may be SQL_ASCII if initdb ran under a broken locale
# (issue #8). template0 is required to override LC_COLLATE/LC_CTYPE.
run_psql -c "CREATE DATABASE $DB_NAME OWNER $DB_USER ENCODING 'UTF8' LC_COLLATE 'en_US.UTF-8' LC_CTYPE 'en_US.UTF-8' TEMPLATE template0"
# Only the owner may connect — no PUBLIC access. # Only the owner may connect — no PUBLIC access.
run_psql -c "REVOKE CONNECT ON DATABASE $DB_NAME FROM PUBLIC" run_psql -c "REVOKE CONNECT ON DATABASE $DB_NAME FROM PUBLIC"
msg_ok "Database $DB_NAME created (owner $DB_USER, PUBLIC revoked)" # Verify what we just created (the pre-existing case was already checked
# before the role block, issue #8).
assert_db_encoding_utf8 "$DB_NAME"
msg_ok "Database $DB_NAME created (UTF8, owner $DB_USER, PUBLIC revoked)"
else else
msg_warn "Database $DB_NAME already exists, skipping" msg_warn "Database $DB_NAME already exists, skipping creation"
fi fi
# pgvector: CREATE EXTENSION needs superuser; installed now (per ADR-0002: # pgvector: CREATE EXTENSION needs superuser; installed now (per ADR-0002:
@@ -141,9 +160,12 @@ Password: $DB_PASS
DSN for /etc/nexus/env on the nexus LXC (NEXUS_DATABASE_URL): DSN for /etc/nexus/env on the nexus LXC (NEXUS_DATABASE_URL):
postgresql+psycopg://$DB_USER:$DB_PASS@$IP_SELF:$DB_PORT/$DB_NAME postgresql+psycopg://$DB_USER:$DB_PASS@$IP_SELF:$DB_PORT/$DB_NAME
Encoding: UTF8 (LC_COLLATE/LC_CTYPE en_US.UTF-8) — verified at install time.
Access policy (pg_hba): only $NEXUS_APP_IP/32 may connect; all other Access policy (pg_hba): only $NEXUS_APP_IP/32 may connect; all other
hosts are rejected. Local socket stays peer-auth for maintenance: hosts are rejected. Local socket stays peer-auth for maintenance (these
pct exec <CTID> -- runuser -u postgres -- psql -d $DB_NAME minimal LXCs have no sudo — use su, not sudo):
pct exec <CTID> -- su - postgres -c "psql -d $DB_NAME"
EOF EOF
chmod 600 "$CRED_FILE" chmod 600 "$CRED_FILE"
msg_ok "Credentials written to $CRED_FILE (chmod 600)" msg_ok "Credentials written to $CRED_FILE (chmod 600)"
+46 -2
View File
@@ -63,8 +63,52 @@ valid_token_word "$RUNNER_LABELS" || { msg_err "RUNNER_LABELS enthält unzuläss
# der unprivilegierte User darf dort nicht schreiben. # der unprivilegierte User darf dort nicht schreiben.
run_user() { runuser -u "$APP_USER" -- env -C "$RUNNER_DIR" HOME="$APP_HOME" "$@"; } run_user() { runuser -u "$APP_USER" -- env -C "$RUNNER_DIR" HOME="$APP_HOME" "$@"; }
# ── Pakete: git/rsync, Node (checkout-Action), Docker (Test-Container) ──────── # ── Pakete: git/rsync, Node (checkout-Action), Docker (Test-Container),
setup_base_apt git rsync ca-certificates curl # Build-Werkzeuge (Compiler-Sprachen in CI) ─────────────────────────────────
#
# make/gcc/protobuf: Der Runner faehrt Host-Mode, also gibt es kein Image, das
# Werkzeuge mitbringt — was hier nicht liegt, hat kein Job. Konkret gemessen an
# l.kirchner/patchmgr (Go):
# make "make: command not found" in jedem Gate-Job
# gcc ohne gefundenen C-Compiler bleibt cgo aus (go env
# meldet dann CGO_ENABLED=0), und "go test -race" ist
# nicht baubar
# protobuf-compiler protoc fuer die Codegenerierung
# libprotobuf-dev liefert /usr/include/google/protobuf/*.proto; ohne die
# Includes scheitert protoc trotz vorhandenem Binary
#
# ACHTUNG protoc: Das bindet jedes Repo der Instanz an die protoc-Fassung der
# Distribution (Debian 12: 3.21.x). Anders als make/gcc ist protoc ein
# Codegenerator — ein Distro-Upgrade aendert erzeugten Code fuer alle Nutzer
# gleichzeitig. Wer eine eigene Fassung braucht, pinnt sie im Projekt
# (Release-Tarball, buf, oder Docker — der Runner hat Docker) statt sie hier
# zu heben.
#
# zstd: kein Werkzeug fuer Jobs, sondern fuer den Nachlauf. `actions/cache`
# — und damit auch `actions/setup-go`, das darauf aufsetzt — packt seinen
# Cache mit `zstdmt`, wenn zstd vorhanden ist, und faellt sonst auf
# einkerniges gzip zurueck. Der Unterschied ist am Archivnamen zu sehen:
# `cache.tzst` gegen `cache.tgz`.
#
# Gemessen am 02./03.09.2026 in l.kirchner/patchmgr, CI-Lauf 1957: Der Cache
# aus GOMODCACHE und GOCACHE ist dort 2 bis 5 GB gross. Auf runner-gpu (hat
# zstd) heisst er `cache.tzst`; auf runner-01/02/03 `cache.tgz`, und
# `go test -race (stable)` verbrachte damit **605 Sekunden** im Nachlauf bei
# 41 Sekunden Arbeit. Ueber alle acht Jobs waren es 2312 von 3146 Sekunden.
#
# Das ersetzt nicht die Frage, ob dieser Cache ueberhaupt gebraucht wird — im
# Host-Mode mit dauerhaftem Zuhause ueberleben beide Verzeichnisse ohnehin
# zwischen den Jobs, und patchmgr schaltet ihn deshalb ab. Aber solange
# irgendein Repo der Instanz ihn nutzt, soll er nicht einkernig komprimiert
# werden. Belege: .specs/reports/ci-laufzeit-und-cache-2026-09-02.md dort.
#
# Bewusst NICHT installiert: sudo. Ein Workflow soll auf diesem Runner nichts
# nachinstallieren koennen — was gebraucht wird, steht hier.
#
# Go selbst gehoert nicht hierher: Projekte holen es ueber actions/setup-go,
# weil CI-Matrizen mehrere Fassungen fahren.
setup_base_apt git rsync ca-certificates curl \
make gcc protobuf-compiler libprotobuf-dev zstd
NODE_HAVE="$(command -v node >/dev/null 2>&1 && node -v | sed -E 's/^v([0-9]+).*/\1/' || echo 0)" NODE_HAVE="$(command -v node >/dev/null 2>&1 && node -v | sed -E 's/^v([0-9]+).*/\1/' || echo 0)"
if [[ "$NODE_HAVE" != "$NODE_MAJOR" ]]; then if [[ "$NODE_HAVE" != "$NODE_MAJOR" ]]; then
+54
View File
@@ -54,6 +54,60 @@ apt_cleanup() {
apt-get autoclean -qq >/dev/null || true apt-get autoclean -qq >/dev/null || true
} }
# ── database installers: UTF-8 locale before initdb ──────────────────────────
# Pattern for every DB installer. A PostgreSQL cluster/database freezes its
# encoding at initdb / CREATE DATABASE time and it cannot be changed later —
# a C (non-UTF-8) locale yields a SQL_ASCII cluster. psycopg3 then returns
# text as bytes and SQLAlchemy crashes on server-version detection; the app
# reports "db: unreachable". So: GENERATE the UTF-8 locale AND make it active
# for THIS process before the server package runs its automatic initdb, then
# fail loudly if it is not actually available (generating alone is not enough
# — the locale must be active when initdb runs).
# Generates+activates a UTF-8 locale (default en_US.UTF-8); the argument
# honours other UTF-8 locales consistently (match, locale.gen line and the
# exported value all derive from it). Matching normalises case and dashes so
# the canonical `en_US.UTF-8` matches `locale -a`'s `en_US.utf8`.
ensure_utf8_locale_active() {
local loc="${1:-en_US.UTF-8}"
local norm; norm="$(printf '%s' "$loc" | tr 'A-Z' 'a-z' | tr -d '-')"
_locale_present() { locale -a 2>/dev/null | tr 'A-Z' 'a-z' | tr -d '-' | grep -qx "$norm"; }
msg_info "Ensuring $loc is generated and active (DB encoding is frozen at initdb)..."
if ! _locale_present; then
# Uncomment the matching `# <loc> UTF-8` line, then generate.
sed -i "s/^# *${loc} UTF-8/${loc} UTF-8/" /etc/locale.gen
locale-gen >/dev/null
fi
if ! _locale_present; then
msg_err "Locale $loc not available after locale-gen — refusing to continue (initdb would create a SQL_ASCII cluster)"
return 1
fi
# Activate for the current process so any automatic initdb during the
# server package install inherits a UTF-8 locale, not the bare-template C.
export LANG="$loc" LC_ALL="$loc"
msg_ok "Locale active for initdb: LANG=$LANG"
}
# Post-install guard: a database MUST be UTF8. Encoding is irreversible, so a
# wrong value is database damage — abort with a clear, actionable message
# instead of shipping a broken cluster. Uses argv-clean `runuser ... psql`
# with a quoted :'db' literal binding (robust regardless of caller); the
# credentials/README docs use `su - postgres -c` for hand maintenance (these
# minimal LXCs have no sudo).
assert_db_encoding_utf8() {
local db="$1" enc
enc="$(runuser -u postgres -- psql -X -qAt -v db="$db" \
-c "SELECT pg_encoding_to_char(encoding) FROM pg_database WHERE datname = :'db'")"
if [[ "$enc" != "UTF8" ]]; then
msg_err "Database '$db' has encoding '${enc:-<not found>}', expected UTF8."
msg_err "Encoding is frozen at creation time — this is DB damage, not cosmetic."
msg_err "Fix: regenerate the locale (locale-gen en_US.UTF-8) and recreate the DB"
msg_err " with: CREATE DATABASE $db ... TEMPLATE template0 ENCODING 'UTF8'"
msg_err " LC_COLLATE 'en_US.UTF-8' LC_CTYPE 'en_US.UTF-8';"
return 1
fi
msg_ok "Encoding check: database '$db' is UTF8"
}
# ── ssh ────────────────────────────────────────────────────────────────────── # ── ssh ──────────────────────────────────────────────────────────────────────
# SSH-Root-Login gemäß Host-Prompt (prompt_lxc_config setzt SSH_ROOT_LOGIN, # SSH-Root-Login gemäß Host-Prompt (prompt_lxc_config setzt SSH_ROOT_LOGIN,
# bootstrap_install_script reicht es als Env durch; Default: yes). # bootstrap_install_script reicht es als Env durch; Default: yes).