Cologne-Geomorphological-Software-Lab/CGDB: CGDB v2.0.0 "Arica"

Abstract

This is a major release: three changes break a straight git pull upgrade from v1.1.0 and require the manual steps in the Upgrade Tutorial below: migrating dependency management from pip to uv, adding Node/npm as a required build-time dependency on the server, and moving scheduled maintenance jobs onto a persistent Dagster daemon process. Changelog Breaking Changes Dependency management: pip → uv. requirements.txt is gone; dependencies are now declared in pyproject.toml and installed via uv sync. Existing venvs built with pip need to be rebuilt. Node.js + npm now required on the server. The map dashboard's frontend is a Vite app that must be built during deploy (npm ci && npm run build). v1.1.0 had no frontend build step at all. Maintenance jobs now require a running dagster-daemon process. The old subprocess.Popen-based trigger from the admin is gone; jobs are submitted via dagster job launch and need the daemon (plus a DAGSTER_HOME) to actually run. Existing deployments need a new supervised process (systemd unit or equivalent) — see Upgrade Tutorial. Python 3.13+ now required. Permission system rewritten. Group/permission definitions changed; a one-time --reset of the predefined groups is recommended after upgrading (applied automatically otherwise via post_migrate, but --reset guarantees a clean rebuild — see Upgrade Tutorial). Security Fixes Fixed an IDOR in project re-parenting in the admin layer (58bb604). Fixed a bbox filter bypass and hardened import_landforms against malformed input (ec3eeee). FieldPhoto.file is no longer served as a raw media URL — downloads now go through a project-scoped, permission-checked view (5c7cb4d). Fixed a fallback in ReferenceAdmin.has_delete_permission that could grant unintended delete access (007758a). Fixed a 500 error path in RasterSceneAdmin for non-superusers that could leak state (fb74c61). Broader admin-layer security review: fixed several access-control issues found during an internal architecture/security audit (7a044a1, 84906f0). CI hardened with a full bandit, vulture, xenon, basedpyright and mypy sweep, plus pylint, to catch this class of issue earlier (d0959f6). New Features Raster data app: import, admin, and metadata recomputation for raster scenes; corpus_path/file precedence fixed during recompute (5f2b101). Geodata app: extended API and location import; GPS accuracy tracking added to location capture. Map: Vite-based rebuild; Google Satellite basemap. Analysis: cosmogenic nuclide dating model and admin support. Admin/UX: sample-admin now summarizes all sample-related measurements in one place; luminescence, grainsize, and location admin interfaces revised. Internal data_quality flag added to LuminescenceDating and RadiocarbonDating, with test coverage. Permission system rewrite, covering project-, group- and object-level access. deploy management command for scripted, repeatable production deploys. Dagster orchestration for scheduled database maintenance, including per-table DuckDB export failure tracking (MaintenanceRun.log now populated for daemon-triggered runs). Bug Fixes & Reliability Fixed inconsistent on_delete behavior across Sample-related foreign keys, preventing orphaned or unexpectedly cascaded records (fa3fb7c). Fixed Meta.ordering inheritance and duplicate ordering on M2M relations; fixed eager validator binding (47b4517). Fixed process/notes property casing mismatch in landform import (82aa5a2). Added error handling around three previously unguarded map data loaders (23a0d66). Smaller fixes across UI, dashboard, datetime handling, and admin resources (campaign, sample, researcher, manufacturer). Infrastructure & Developer Experience CI pipeline added for tests and linting, with a throwaway local_settings.py so Django settings actually load in CI (9a0db28). Pre-commit hardened: full basedpyright coverage across all 8 apps, plus bandit, vulture, xenon, pylint duplicate-code detection. Dead code and leftover Dagster boilerplate removed. Test coverage extended (analysis, field_data, geodata, laboratory, raster_data, orchestration), including direct reachability tests for permission fallbacks and coverage for CosmogenicNuclideDating. Routine dependency updates via Dependabot (Pillow, GitPython, and others). Documentation Sphinx-based documentation site added (docs/), with a GitHub Actions workflow that builds it on every push (publishing to GitHub Pages is not yet enabled). CONTRIBUTING.md, CODE_OF_CONDUCT.md, and a revised security policy/reporting process added. Pull request template added. README overhauled: deploy workflow, Vite frontend, Dagster daemon migration, and full test structure documented. Upgrade Tutorial Audience: operators upgrading an existing v1.1.0 (or earlier) CGDB deployment to v2.0.0 "Arica". Steps 1–3 are one-time migration steps specific to this release; step 4 onward is the normal deploy flow. Before you start: Take a full database backup yourself, independent of the automatic pre-migrate backup manage.py deploy takes — this upgrade touches more than a single migration. Schedule a maintenance window. Steps 1–3 involve service restarts and are not zero-downtime. Confirm shell access with the same privileges your normal deploy uses (the app-owning user for most steps, sudo only for manage.py deploy itself). 1. Install prerequisites on the server Python 3.13+ — confirm with python3 --version; upgrade the interpreter first if it's older. uv — install if not already present. Node.js + npm — required from this release on, as a one-shot build tool only (nothing Node-based runs persistently). Any reasonably current LTS Node works; there's no pinned minimum version. Confirm with node --version / npm --version. 2. Migrate dependency management from pip to uv Do this once, before the first uv sync on the server: # from the project root, with the OLD pip-based venv still in place deactivate 2>/dev/null || true rm -rf .venv # the old pip-managed venv — uv creates its own requirements.txt no longer exists in the new codebase; uv sync (run automatically by manage.py deploy, see step 4) reads pyproject.toml/uv.lock instead and creates a fresh .venv. If anything outside this repo activates the old venv by hard-coded path (a systemd unit, a cron job, a supervisor config), update those paths — uv sync still creates .venv in the same location, so most setups need no changes here, but double-check anything referencing .venv/bin/pip directly. 3. Migrate scheduled maintenance jobs to the Dagster daemon Skip this step if you never enabled the optional Dagster orchestration on this deployment. Older deployments triggered maintenance jobs via a detached subprocess.Popen from the admin, with no daemon and no dedicated Dagster run storage. This release requires a persistent dagster-daemon process instead: Deploy the new code first (step 4 below covers this) — uv sync pulls in dagster>=1.13.16, dagster-webserver>=1.13.11, dagster-postgres>=0.29.11. No Django migration is involved in this particular change; it's a config/behavior change, not a schema change. Make orchestration/dagster_home writable by the same user your web server process runs as (e.g. www-data on Debian/Ubuntu, apache on RHEL/CentOS — check whichever user your existing WSGIDaemonProcess, or equivalent, is configured with). Triggering a maintenance job from the admin runs dagster job launch synchronously in the web request (orchestration/admin.py, via subprocess.run) — not just the daemon. Both that in-request launch and the daemon itself need write access to DAGSTER_HOME (SQLite run storage creates a history/ subdirectory there on first use). Since the directory typically arrives owned by whoever ran git pull (your personal account) or root (from sudo manage.py deploy), fix ownership explicitly: sudo chown -R : /orchestration/dagster_home (Optional — only if you want PostgreSQL run storage instead of the SQLite default) create a dedicated Postgres database for Dagster's own run storage — reuse the same Postgres instance/host/credentials your app already runs on, just a separate database name (e.g. dagster): CREATE DATABASE dagster; GRANT ALL PRIVILEGES ON DATABASE dagster TO ; Then edit orchestration/dagster_home/dagster.yaml: comment out the sqlite storage block, uncomment the postgres block, and set: export DAGSTER_PG_USER=dagster export DAGSTER_PG_PASSWORD=... export DAGSTER_PG_HOST=localhost export DAGSTER_PG_DB=dagster The SQLite default needs none of this — just DAGSTER_HOME (next step). Add the daemon as its own supervised process. If served via Apache/mod_wsgi, add a new, separate systemd unit — Apache doesn't need to know about it. First check which user your existing WSGIDaemonProcess runs as (/etc/apache2/sites-available/*.conf, the user=/group= on that directive) so the daemon runs as the same user rather than a new one. Example /etc/systemd/system/cgdb-dagster-daemon.service: [Unit] Description=CGDB Dagster daemon After=network.target [Service] Type=simple User= Group= WorkingDirectory= Environment=DAGSTER_HOME= /orchestration/dagster_home # The four lines below are only needed if you switched dagster.yaml to # PostgreSQL storage (step 3 above) — omit them for the SQLite default. # Environment=DAGSTER_PG_USER=dagster # Environment=DAGSTER_PG_PASSWORD=... # Environment=DAGSTER_PG_HOST=localhost # Environment=DAGSTER_PG_DB=dagster ExecStart= /.venv/bin/dagster-daemon run Restart=on-failure RestartSec=5 [Install] WantedBy=multi-user.target sudo systemctl daemon-reload sudo systemctl enable --now cgdb-dagster-daemon 4. Deploy git pull --ff-only sudo python manage.py deploy This runs, in order: a clean-working-tree check, a pre-migrate database backup, uv sync (now against pyproject.toml, per step 2), np