Merge pull request 'fix(nexus-db): UTF8-Encoding zur Installationszeit erzwingen (#8)' (#9) from fix/nexus-db-utf8-encoding into main

fix(nexus-db): UTF-8-Encoding erzwingen + Post-Install-Check (PR #9, Closes #8) — Locale aktiv zum initdb-Zeitpunkt, Re-Run-Guard, su-statt-sudo-Doku

README.md

@@ -13,7 +13,7 @@ Inspired by [community-scripts/ProxmoxVE](https://github.com/community-scripts/P
 | [nexus](ct/nexus.sh) | App LXC for [nexus](https://gitea.luki-net.org/l.kirchner/nexus-hub) (Family Knowledge Hub): host-mode runner (label `nexus`, CI + deploy), service skeleton, `/opt/nexus` layout. Runtime is provisioned/extended via [`install/nexus-runtime.sh`](install/nexus-runtime.sh) (idempotent, re-runnable) | ✅ in production |
 | [nexus-db](ct/nexus-db.sh) | PostgreSQL 16 + pgvector for nexus — least-privilege role, pg_hba allowlist (only the nexus LXC), DSN handed over via credentials file | ✅ in production |
 | [authentik](ct/authentik.sh) | Central homelab IdP (official Docker Compose via LXC nesting) — headless bootstrap admin **and** API token for agent-driven blueprint configuration, blueprints mount, permanent auth domain (WebAuthn RP-ID) | ✅ in production |
-| [runner](ct/runner.sh) | General **instance-wide** Gitea Actions runner (label `homelab`) — Docker for throwaway CI test containers, deliberately **no** sudoers/deploy rights | 🔄 in review ([PR #6](https://gitea.luki-net.org/luki-net/proxmox-scripts/pulls/6)) |
+| [runner](ct/runner.sh) | General **instance-wide** Gitea Actions runner (label `homelab`) — Docker for throwaway CI test containers, deliberately **no** sudoers/deploy rights | ✅ in production |
 
 Run any one-liner on a Proxmox VE host as root:
 
@@ -32,12 +32,16 @@ CTID=200 HOSTNAME=devpi DISK_SIZE=30 RAM=4096 CORES=4 IPCFG=dhcp \
 
 All defaults (`DEFAULT_HOSTNAME`, `DEFAULT_DISK`, …) and app config values are settable per call via env vars.
 
+Every input is validated (digits-only for CTID/disk/cores/RAM/VLAN, IP/CIDR/gateway format, hostname/storage charsets). Invalid interactive input re-prompts — including pasted values with invisible control/non-ASCII bytes, which are rejected rather than silently stripped. Env-provided values are validated too and abort the run when malformed (no re-prompt loop in non-interactive use).
+
 ## The pattern
 
 Two files per app, both sourcing the shared libs via `curl`:
 
-- **`ct/<app>.sh`** runs on the PVE host: prompts → unprivileged LXC → pushes a config env file into the container → bootstraps the installer. Apps that need Docker (authentik, runner) enable `nesting+keyctl` automatically.
-- **`install/<app>-install.sh`** runs inside the LXC: packages, unprivileged app user, secrets generated on-host (never printed), systemd units, a `/root/<app>.credentials` notes file — then shreds the bootstrap env. Idempotent where it matters: re-runs skip what exists.
+- **`ct/<app>.sh`** runs on the PVE host: prompts → unprivileged LXC → pushes a config env file into the container → bootstraps the installer. Apps that need Docker (authentik, runner) enable `nesting+keyctl` automatically. The standard prompts include an **SSH root login choice** (`SSH_ROOT_LOGIN`, default yes for homelab convenience; `no` keeps the Debian key-only default) — applied inside the container as an sshd drop-in by `configure_ssh_root_login`.
+- **`install/<app>-install.sh`** runs inside the LXC: packages, unprivileged app user, secrets generated on-host (never printed), systemd units, a `/root/<app>.credentials` notes file — then shreds the bootstrap env. Idempotent where it matters: re-runs skip what exists. `setup_base_apt` also fixes the bare-template **locale situation**: `C.UTF-8` is exported up front (glibc built-in, covers the first apt run without perl warnings), then `en_US.UTF-8` is generated and set as the system default.
+
+**DB installers** carry one extra rule: a PostgreSQL cluster/database freezes its encoding at initdb / `CREATE DATABASE` time and it can never be changed afterwards. A C (non-UTF-8) locale yields a `SQL_ASCII` cluster — psycopg3 then hands text back as bytes and SQLAlchemy crashes. So the pattern (helpers `ensure_utf8_locale_active` + `assert_db_encoding_utf8` in `lib/install.func`) is: make a UTF-8 locale **active** before the server package runs initdb, create the database **explicitly** with `TEMPLATE template0 ENCODING 'UTF8' LC_COLLATE/LC_CTYPE 'en_US.UTF-8'` (never inherit the cluster default), and **verify** `pg_encoding_to_char` returns `UTF8` before finishing — a wrong encoding aborts the install (it's DB damage, see wiki → Lessons). Maintenance examples use `su - postgres -c …`, not `sudo` — these minimal LXCs have no sudo.
 
 Shared libs: [`lib/build.func`](lib/build.func) (host-side: prompts, LXC create, bootstrap) and [`lib/install.func`](lib/install.func) (in-container: apt, users, systemd, http-wait).
 
@@ -50,15 +54,11 @@ Shared libs: [`lib/build.func`](lib/build.func) (host-side: prompts, LXC create,
 
 ## Contributing
 
-**All changes go through a pull request with cross-review** (Claude Code ↔ Codex, or a human) — no direct pushes to `main`. This rule exists because of two real incidents where an un-reviewed script shipped a missing `source build.func` (see wiki → Lessons). CI (`bash -n`, source-check, validation suite) is being introduced with [PR #5](https://gitea.luki-net.org/luki-net/proxmox-scripts/pulls/5) and runs on the `homelab` runner.
+**All changes go through a pull request with cross-review** (Claude Code ↔ Codex, or a human) — no direct pushes to `main`. This rule exists because of two real incidents: a pasted VLAN tag carrying an invisible non-UTF-8 byte broke `pct create` mid-run, and the "missing `source build.func`" bug shipped twice — caught in review on the nexus-db PR, but reaching production via an un-reviewed authentik commit (see wiki → Lessons).
 
-How to add a script: [docs/adding-a-script.md](docs/adding-a-script.md).
+CI (`.gitea/workflows/ci.yml`, instance-wide `homelab` runner from [ct/runner.sh](ct/runner.sh)) enforces on every PR: `bash -n` over all scripts, the "every `ct/*.sh` sources `build.func`" check ([tests/check_ct_source.sh](tests/check_ct_source.sh)) and the validation unit tests ([tests/test_validation.sh](tests/test_validation.sh)); shellcheck runs when available on the runner.
 
-## Contributing (verbindlich seit K-114)
-
-**Alle Änderungen laufen als PR mit Cross-Review** — keine Direkt-Commits auf `main`. Hintergrund: Der „`build.func` nicht gesourct"-Bug hat es einmal bis in die Produktion geschafft (authentik-Anlage), während dieselbe Fehlerklasse im nexus-db-PR vom Review gefangen wurde. Die CI (`.gitea/workflows/ci.yml`, Runner-Label `homelab` — instanzweiter Runner aus `ct/runner.sh`) erzwingt zusätzlich: `bash -n` über alle Scripts, „jedes `ct/*.sh` sourct `build.func`" (`tests/check_ct_source.sh`) und die Validierungs-Unit-Tests (`tests/test_validation.sh`).
-
-Eingaben in `prompt_lxc_config` sind validiert (Ziffern-Checks, IP/CIDR/Gateway-Format, Re-Prompt bei unsichtbaren Steuer-/Non-ASCII-Zeichen — Lesson vom 2026-06-11). Neue App-Prompts bitte über `prompt_validated`/`require_valid` aus `lib/build.func` bauen statt nacktem `read`.
+Build new app prompts on `prompt_validated`/`require_valid` from `lib/build.func` instead of bare `read`. How to add a script: [docs/adding-a-script.md](docs/adding-a-script.md).
 
 ## Repo layout
 
@@ -69,6 +69,9 @@ Eingaben in `prompt_lxc_config` sind validiert (Ziffern-Checks, IP/CIDR/Gateway-
 ├── lib/
 │   ├── build.func      # Shared host-side helpers (prompts, LXC create, bootstrap)
 │   └── install.func    # Shared in-container helpers (apt, systemd, users, http-wait)
+├── tests/
+│   ├── test_validation.sh   # Unit tests for the input validation helpers
+│   └── check_ct_source.sh   # Every ct/*.sh must source build.func
 ├── docs/
 │   └── adding-a-script.md
 ├── README.md

install/nexus-db-install.sh

@@ -47,6 +47,12 @@ else
   msg_warn "PGDG repo already present, skipping"
 fi
 
+# The server package runs initdb for the `main` cluster on install — its
+# encoding is frozen there. Make a UTF-8 locale active for THIS process first
+# so the cluster is never created as SQL_ASCII (issue #8: a pre-fix LXC was
+# provisioned under LANG=C and ended up SQL_ASCII).
+ensure_utf8_locale_active en_US.UTF-8
+
 msg_info "Installing PostgreSQL $PG_MAJOR + pgvector..."
 apt-get install -y -qq "postgresql-$PG_MAJOR" "postgresql-$PG_MAJOR-pgvector" >/dev/null
 msg_ok "PostgreSQL $(psql --version | awk '{print $3}') installed"
@@ -95,6 +101,13 @@ systemctl restart postgresql
 # ── role + database + extension (idempotent, password kept on re-run) ─────────
 run_psql() { runuser -u postgres -- psql -v ON_ERROR_STOP=1 -qAt "$@"; }
 
+# Re-run safety (issue #8): if the database already exists, verify its
+# encoding BEFORE touching roles/passwords. An old SQL_ASCII database must
+# abort the run with NO side effects — not after rotating credentials.
+if [[ "$(run_psql -c "SELECT 1 FROM pg_database WHERE datname='$DB_NAME'")" == "1" ]]; then
+  assert_db_encoding_utf8 "$DB_NAME"
+fi
+
 if [[ "$(run_psql -c "SELECT 1 FROM pg_roles WHERE rolname='$DB_USER'")" != "1" ]]; then
   msg_info "Creating role $DB_USER + database $DB_NAME..."
   DB_PASS="$(openssl rand -base64 32 | tr -d '/+=' | head -c 32)"
@@ -114,12 +127,18 @@ else
 fi
 
 if [[ "$(run_psql -c "SELECT 1 FROM pg_database WHERE datname='$DB_NAME'")" != "1" ]]; then
-  run_psql -c "CREATE DATABASE $DB_NAME OWNER $DB_USER"
+  # Explicit encoding/collation from template0 — never inherit the cluster
+  # default, which may be SQL_ASCII if initdb ran under a broken locale
+  # (issue #8). template0 is required to override LC_COLLATE/LC_CTYPE.
+  run_psql -c "CREATE DATABASE $DB_NAME OWNER $DB_USER ENCODING 'UTF8' LC_COLLATE 'en_US.UTF-8' LC_CTYPE 'en_US.UTF-8' TEMPLATE template0"
   # Only the owner may connect — no PUBLIC access.
   run_psql -c "REVOKE CONNECT ON DATABASE $DB_NAME FROM PUBLIC"
-  msg_ok "Database $DB_NAME created (owner $DB_USER, PUBLIC revoked)"
+  # Verify what we just created (the pre-existing case was already checked
+  # before the role block, issue #8).
+  assert_db_encoding_utf8 "$DB_NAME"
+  msg_ok "Database $DB_NAME created (UTF8, owner $DB_USER, PUBLIC revoked)"
 else
-  msg_warn "Database $DB_NAME already exists, skipping"
+  msg_warn "Database $DB_NAME already exists, skipping creation"
 fi
 
 # pgvector: CREATE EXTENSION needs superuser; installed now (per ADR-0002:
@@ -141,9 +160,12 @@ Password: $DB_PASS
 DSN for /etc/nexus/env on the nexus LXC (NEXUS_DATABASE_URL):
   postgresql+psycopg://$DB_USER:$DB_PASS@$IP_SELF:$DB_PORT/$DB_NAME
 
+Encoding: UTF8 (LC_COLLATE/LC_CTYPE en_US.UTF-8) — verified at install time.
+
 Access policy (pg_hba): only $NEXUS_APP_IP/32 may connect; all other
-hosts are rejected. Local socket stays peer-auth for maintenance:
-  pct exec <CTID> -- runuser -u postgres -- psql -d $DB_NAME
+hosts are rejected. Local socket stays peer-auth for maintenance (these
+minimal LXCs have no sudo — use su, not sudo):
+  pct exec <CTID> -- su - postgres -c "psql -d $DB_NAME"
 EOF
   chmod 600 "$CRED_FILE"
   msg_ok "Credentials written to $CRED_FILE (chmod 600)"

lib/install.func

@@ -54,6 +54,60 @@ apt_cleanup() {
   apt-get autoclean -qq >/dev/null || true
 }
 
+# ── database installers: UTF-8 locale before initdb ──────────────────────────
+# Pattern for every DB installer. A PostgreSQL cluster/database freezes its
+# encoding at initdb / CREATE DATABASE time and it cannot be changed later —
+# a C (non-UTF-8) locale yields a SQL_ASCII cluster. psycopg3 then returns
+# text as bytes and SQLAlchemy crashes on server-version detection; the app
+# reports "db: unreachable". So: GENERATE the UTF-8 locale AND make it active
+# for THIS process before the server package runs its automatic initdb, then
+# fail loudly if it is not actually available (generating alone is not enough
+# — the locale must be active when initdb runs).
+# Generates+activates a UTF-8 locale (default en_US.UTF-8); the argument
+# honours other UTF-8 locales consistently (match, locale.gen line and the
+# exported value all derive from it). Matching normalises case and dashes so
+# the canonical `en_US.UTF-8` matches `locale -a`'s `en_US.utf8`.
+ensure_utf8_locale_active() {
+  local loc="${1:-en_US.UTF-8}"
+  local norm; norm="$(printf '%s' "$loc" | tr 'A-Z' 'a-z' | tr -d '-')"
+  _locale_present() { locale -a 2>/dev/null | tr 'A-Z' 'a-z' | tr -d '-' | grep -qx "$norm"; }
+  msg_info "Ensuring $loc is generated and active (DB encoding is frozen at initdb)..."
+  if ! _locale_present; then
+    # Uncomment the matching `# <loc> UTF-8` line, then generate.
+    sed -i "s/^# *${loc} UTF-8/${loc} UTF-8/" /etc/locale.gen
+    locale-gen >/dev/null
+  fi
+  if ! _locale_present; then
+    msg_err "Locale $loc not available after locale-gen — refusing to continue (initdb would create a SQL_ASCII cluster)"
+    return 1
+  fi
+  # Activate for the current process so any automatic initdb during the
+  # server package install inherits a UTF-8 locale, not the bare-template C.
+  export LANG="$loc" LC_ALL="$loc"
+  msg_ok "Locale active for initdb: LANG=$LANG"
+}
+
+# Post-install guard: a database MUST be UTF8. Encoding is irreversible, so a
+# wrong value is database damage — abort with a clear, actionable message
+# instead of shipping a broken cluster. Uses argv-clean `runuser ... psql`
+# with a quoted :'db' literal binding (robust regardless of caller); the
+# credentials/README docs use `su - postgres -c` for hand maintenance (these
+# minimal LXCs have no sudo).
+assert_db_encoding_utf8() {
+  local db="$1" enc
+  enc="$(runuser -u postgres -- psql -X -qAt -v db="$db" \
+    -c "SELECT pg_encoding_to_char(encoding) FROM pg_database WHERE datname = :'db'")"
+  if [[ "$enc" != "UTF8" ]]; then
+    msg_err "Database '$db' has encoding '${enc:-<not found>}', expected UTF8."
+    msg_err "Encoding is frozen at creation time — this is DB damage, not cosmetic."
+    msg_err "Fix: regenerate the locale (locale-gen en_US.UTF-8) and recreate the DB"
+    msg_err "     with: CREATE DATABASE $db ... TEMPLATE template0 ENCODING 'UTF8'"
+    msg_err "          LC_COLLATE 'en_US.UTF-8' LC_CTYPE 'en_US.UTF-8';"
+    return 1
+  fi
+  msg_ok "Encoding check: database '$db' is UTF8"
+}
+
 # ── ssh ──────────────────────────────────────────────────────────────────────
 # SSH-Root-Login gemäß Host-Prompt (prompt_lxc_config setzt SSH_ROOT_LOGIN,
 # bootstrap_install_script reicht es als Env durch; Default: yes).