docs: Landscape Program + dataset groups design spec

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-05 12:57:53 +05:00
parent 1e0844ca82
commit c930f5928b
1 changed files with 317 additions and 0 deletions
--- a/docs/superpowers/specs/2026-09-05-landscape-program-design.md
+++ b/docs/superpowers/specs/2026-09-05-landscape-program-design.md
@@ -0,0 +1,317 @@
+# Landscape Program (ЛП) + dataset groups and dates — design
+
+Date: 2026-09-05. Source: `LP_20260902.excalidraw`, `TZ_functional_zoning_go.md`,
+`zoning_core.py`, `Landscape.ipynb` (reference numpy implementation).
+Decisions taken with the product owner are recorded inline as **Decision**.
+
+## 1. Goals
+
+1. **Dataset versioning.** Datasets can belong to a *dataset group* ("одноимённые"
+   datasets), carry a `start_date` / `end_date` period, be filtered by those, and the
+   "freshest" dataset of a group can be fetched.
+2. **Landscape Program pipeline.** A DAG of raster computations (Z, U, E, C, P_max,
+   C_max, functional zoning V1–V3) executed by the worker, with results stored as
+   ordinary raster datasets (COG + SLD) served by the existing dataset endpoints,
+   incremental recompute, and a dedicated upload for input rasters.
+
+Non-goals: spatial operations (resampling, reprojection), GUI, changing the
+vector_with_kato flow beyond the column rename below.
+
+## 2. Part 1 — Dataset groups and dates
+
+### 2.1 Schema (migration `00010_dataset_groups_and_dates.sql`)
+
+```sql
+CREATE TABLE dataset_groups (
+    id               UUID PRIMARY KEY DEFAULT gen_random_uuid(),
+    code             VARCHAR(255) NOT NULL UNIQUE,   -- slug, same rules as categories.code
+    name             VARCHAR(255) NOT NULL,
+    description      TEXT NOT NULL DEFAULT '',
+    sld_storage_key  TEXT,
+    sld_filename     VARCHAR(255),
+    sld_content_type VARCHAR(255),
+    created_at       TIMESTAMPTZ NOT NULL DEFAULT now(),
+    updated_at       TIMESTAMPTZ NOT NULL DEFAULT now()
+);
+
+ALTER TABLE datasets ADD COLUMN dg_id      UUID REFERENCES dataset_groups(id) ON DELETE RESTRICT;
+ALTER TABLE datasets ADD COLUMN start_date DATE;
+ALTER TABLE datasets ADD COLUMN end_date   DATE;
+CREATE INDEX idx_datasets_dg_id_end_date ON datasets (dg_id, end_date DESC NULLS LAST, created_at DESC);
+CREATE INDEX idx_datasets_start_date ON datasets (start_date);
+CREATE INDEX idx_datasets_end_date   ON datasets (end_date);
+
+-- Decision: rename the observation/mapping period columns (breaking API change).
+ALTER TABLE dataset_observations RENAME COLUMN date    TO start_date;
+ALTER TABLE dataset_observations RENAME COLUMN date_to TO end_date;
+-- unique index and idx_dataset_observations_kato_date recreated on the new names.
+-- datasets.year_columns JSONB: every element's keys "date" -> "start_date",
+-- "date_to" -> "end_date" (UPDATE with jsonb_agg over jsonb_array_elements; NULL-safe).
+```
+
+Constraint: `end_date >= start_date` when both set (CHECK constraint + service validation).
+
+### 2.2 Domain changes
+
+- `domain.DatasetGroup{ID, Code, Name, Description, SldStorageKey, SldFilename,
+  SldContentType, CreatedAt, UpdatedAt}`.
+- `domain.Dataset` and `domain.DatasetSummary` gain `DatasetGroupID *uuid.UUID`
+  (`json:"dg_id"`), `StartDate *string` (`json:"start_date"`, `YYYY-MM-DD`),
+  `EndDate *string` (`json:"end_date"`).
+- `domain.Observation`: `Date`→`StartDate` (`json:"start_date"`), `DateTo`→`EndDate`
+  (`json:"end_date"`). `domain.YearColumn`: same rename. Mapping request body uses
+  `start_date` (required) / `end_date` (optional). Everything that previously said
+  `date`/`date_to` in code, OpenAPI and README is renamed. GeoJSON output keeps its
+  `data` (keyed by start date) and `periods` (start→end) shape.
+- `domain.DatasetFilter` gains `DatasetGroupID *uuid.UUID`, `StartDateFrom`,
+  `StartDateTo`, `EndDateFrom`, `EndDateTo *string` (inclusive bounds).
+
+### 2.3 API
+
+| Method & path | Behaviour |
+|---|---|
+| `GET /dataset-groups` | paginated list, `?code=` exact filter optional |
+| `POST /dataset-groups` | `{code, name, description}`; 201 |
+| `GET /dataset-groups/{id}` | group |
+| `PUT /dataset-groups/{id}` | replace `{code, name, description}` |
+| `DELETE /dataset-groups/{id}` | 204; 409 when datasets reference it; also removes group SLD object |
+| `GET /dataset-groups/{id}/latest` | freshest dataset of the group: order by `end_date DESC NULLS LAST, created_at DESC`; optional `?status=`, `?file_type=`; 404 when none |
+| `POST /dataset-groups/{id}/sld` | multipart `file` (`.sld`/`.xml`), replaces; mirrors dataset SLD upload |
+| `GET /dataset-groups/{id}/sld` | download; 404 when none |
+| `GET /datasets` | new filters `dg_id`, `start_date_from`, `start_date_to`, `end_date_from`, `end_date_to` (all `YYYY-MM-DD`, inclusive, any combination) |
+| `POST /datasets` | new optional form fields `dg_id`, `start_date`, `end_date` |
+| `PUT /datasets/{id}` | body gains `dg_id`, `start_date`, `end_date` (nullable; omitted/null clears) |
+| `PATCH /datasets/{id}` | sparse; same three fields, `null` clears |
+| `GET /datasets/{id}/sld` | **fallback**: when the dataset has no SLD but its group has one, the group SLD is served |
+
+Validation: `dg_id` must reference an existing group (404/422 as for `category_id`);
+dates parse as `2006-01-02`; `end_date` not before `start_date`.
+
+`DatasetSummary` (list responses) includes `dg_id`, `start_date`, `end_date`.
+
+### 2.4 Freshness rule
+
+"Самый свежий" = greatest `end_date`; datasets with `NULL end_date` sort after all
+dated ones; ties broken by `created_at DESC`. The same rule is used by the pipeline
+to resolve inputs (with `status = ready`, `file_type = raster`).
+
+## 3. Part 2 — Landscape Program
+
+### 3.1 Computation graph
+
+Node code `XY`: `X` = stage, `Y` = function in the stage. A stage starts only when
+every node of all lower stages finished. Every input and output of every node is a
+**dataset group**; leaf groups are the *slots* users upload to.
+
+| Node | Name | Inputs (group codes) | Outputs (group codes) |
+|---|---|---|---|
+| 00 | z_integrated | `lp-z-biodiversity`, `lp-z-lcr`, `lp-z-soil-protection`, `lp-z-wr` | `lp-z-integrated` |
+| 01 | u_integrated | `lp-u-surf-water`, `lp-u-soil`, `lp-u-crop`, `lp-u-veg` | `lp-u-integrated` |
+| 02 | z_max | Z1..Z4 as above | `lp-z-max` |
+| 03 | u_max | U as above | `lp-u-max` |
+| 10 | e_ecological_constraint | `lp-z-integrated`, `lp-u-integrated` | `lp-e` |
+| 20 | c_conflicts | `lp-p-irrigation`, `lp-p-snow`, `lp-p-crop`, `lp-p-pasture`, `lp-p-tourism`, `lp-e` | `lp-c-irrigation`, `lp-c-snow`, `lp-c-crop`, `lp-c-pasture`, `lp-c-tourism` |
+| 21 | p_max | five P groups | `lp-p-max` |
+| 30 | func_zoning | `lp-e`, five C groups | `lp-v1-conservative`, `lp-v2-balanced`, `lp-v3-development` |
+| 31 | c_max | five C groups | `lp-c-max` |
+
+Sector order everywhere: 1 irrigation, 2 snow, 3 crop, 4 pasture, 5 tourism.
+**Decision:** func_zoning takes S_k = C_k (folder `300-…-Equals_102_202`), not P_k.
+
+Groups and the category `landscape` ("Ландшафтная программа") are seeded by
+migration `00011_landscape.sql` (`INSERT … ON CONFLICT (code) DO NOTHING`). The
+graph itself lives in Go (`internal/landscape/graph.go`), not in the DB.
+
+### 3.2 Pixel rules (from the reference notebook; must match it bitwise)
+
+Notation: a value is *valid* iff `1 <= v <= 5`; anything else (including nodata) is
+invalid. Output nodata is `-9999`. `classify(x)` maps a float32 score to a class:
+`x <= 1.8 → 1`, `<= 2.6 → 2`, `<= 3.4 → 3`, `<= 4.2 → 4`, `<= 5.0 → 5` (and requires
+`x >= 1.0`); comparisons are done in float32 against float32 constants, exactly as
+numpy does with a float32 array.
+
+| Node | Valid domain (pixel) | Value |
+|---|---|---|
+| z_integrated | all four Z valid | `classify(0.25*Z1 + 0.25*Z2 + 0.25*Z3 + 0.25*Z4)` (float32) |
+| u_integrated | water ∧ soil ∧ (crop ∨ veg) | `classify(num/den)`, `num = 0.25*water + 0.25*soil + [0.25*crop if crop valid] + [0.25*veg if veg valid]`, `den = 0.5 + [0.25] + [0.25]` |
+| z_max | all four valid | `max(Z1..Z4)` |
+| u_max | water ∧ soil ∧ (crop ∨ veg) | `max` over the valid ones of water, soil, crop, veg |
+| e | Z ∧ U valid | `max(Z, U)` |
+| c_conflicts (5 outputs) | P_j ∧ E valid | `min(P_j, E)` |
+| p_max | at least one P valid | `max` over valid P |
+| c_max | at least one C valid | `max` over valid C |
+| func_zoning (3 outputs) | see TZ §4 | see below |
+
+Functional zoning (TZ `TZ_functional_zoning_go.md`, reference `zoning_core.py`):
+
+```
+eValid = valid(E); base_k = eValid && valid(S_k); analysisValid = eValid && any(base_k)
+if !analysisValid → -9999 in all scenarios
+candidate_k = base_k && rule_v(E, S_k)
+  V1: (E <= 2 && S >= v1_min_s_e12) || (E == 3 && S >= v1_min_s_e3)       (defaults 3, 4)
+  V2: (E <= 2) || (E == 3 && S >= v2_min_s_e3) || (E == 4 && S >= v2_min_s_e4)  (defaults 3, 5)
+  V3: (E <= 4)
+no candidates → class 1 (regulated); else winners = candidates with S == max(S over candidates);
+sector = first of tie_priority (default [1,2,3,4,5]) among winners; class = sector + 1
+```
+
+Output classes: 1 regulated, 2 irrigation, 3 snow, 4 crop, 5 pasture, 6 tourism.
+
+### 3.3 Raster I/O abstraction (**Decision:** godal behind an interface)
+
+Package `internal/raster` (existing) gains an engine-agnostic interface so the
+implementation can be swapped (godal today; `gdal_calc`, ENVI+CLI or another binding
+later) without touching the pipeline:
+
+```go
+type Grid struct { Width, Height int; GeoTransform [6]float64; ProjectionWKT string }
+type Window struct { X, Y, Width, Height int }
+type Reader interface {
+    Grid() Grid
+    NoData() (value float64, ok bool)
+    ReadFloat64(w Window, dst []float64) error   // band 1, converted by the driver
+    Close() error
+}
+type WriterOptions struct { NoData int16; TileSize int; Compression string /* DEFLATE|ZSTD */ }
+type Writer interface {
+    WriteInt16(w Window, src []int16) error       // band 1
+    Close() error
+}
+type IO interface {
+    Open(ctx context.Context, path string) (Reader, error)
+    Create(ctx context.Context, path string, g Grid, o WriterOptions) (Writer, error)
+    SameGrid(a, b Grid) bool   // size, transform (abs tol 1e-9), CRS (GDAL IsSame)
+}
+```
+
+`internal/raster/godal.go` implements `IO` with `github.com/airbusgeo/godal` (cgo).
+Output GeoTIFF: `int16`, nodata `-9999`, `TILED=YES`, `BLOCKXSIZE=BLOCKYSIZE=512`,
+`COMPRESS=DEFLATE`, `PREDICTOR=2`, `BIGTIFF=IF_SAFER`; georeference copied from the
+node's first input (for func_zoning: E). Inputs are read as float64 regardless of
+their storage type; multi-band inputs use band 1.
+
+The existing `RasterConverter` (gdal_translate / gdalinfo / ogr2ogr CLI) stays as is
+and keeps producing the COG for every result dataset.
+
+Build: `CGO_ENABLED=1`; `go.mod` gets `toolchain go1.26.2` (the local SDK is
+incomplete) and the godal dependency. Dockerfile: builder `golang:1.26-alpine` +
+`gdal-dev gcc musl-dev pkgconfig`; runtime `alpine:3.22` + `gdal gdal-tools`.
+
+### 3.4 Compute engine (`internal/landscape`)
+
+- `graph.go`: slots, nodes, stages, group codes, node-level param subsets.
+- `kernels.go`: pure per-window functions over `[]float64` inputs → `[]int16`
+  outputs, one per node. No I/O, fully unit-tested (TZ §7 edge cases included).
+- `compute.go`: `Compute(ctx, io raster.IO, node Node, inputs map[string]string
+  /*group→local path*/, outDir string, p Params) (map[string]string /*group→path*/, error)`.
+  Opens inputs, **fails if grids differ** (**Decision:** no resampling), creates
+  outputs on the first input's grid, iterates windows of `p.WindowSize`
+  (default 2048) with a worker pool: each worker owns its own `Reader` handles
+  (GDAL datasets are not goroutine-safe); writes are serialised per output through a
+  mutex. Window order does not affect results.
+- Nodes of one stage run concurrently (`errgroup`); stages run sequentially.
+
+### 3.5 Runs, incremental recompute
+
+Migration `00011_landscape.sql`:
+
+```sql
+CREATE TABLE lp_runs (
+    id          UUID PRIMARY KEY DEFAULT gen_random_uuid(),
+    status      VARCHAR(32) NOT NULL,          -- queued | running | succeeded | failed
+    from_stage  SMALLINT,                      -- NULL = incremental
+    params      JSONB NOT NULL,
+    overrides   JSONB NOT NULL DEFAULT '{}',   -- {group_code: dataset_id}
+    error       TEXT,
+    created_at  TIMESTAMPTZ NOT NULL DEFAULT now(),
+    started_at  TIMESTAMPTZ,
+    finished_at TIMESTAMPTZ
+);
+CREATE TABLE lp_run_nodes (
+    id              UUID PRIMARY KEY DEFAULT gen_random_uuid(),
+    run_id          UUID NOT NULL REFERENCES lp_runs(id) ON DELETE CASCADE,
+    node_code       VARCHAR(8) NOT NULL,
+    stage           SMALLINT NOT NULL,
+    status          VARCHAR(32) NOT NULL,      -- pending | running | skipped | succeeded | failed
+    fingerprint     TEXT NOT NULL,
+    input_datasets  JSONB NOT NULL,            -- [{group, dataset_id}]
+    output_datasets JSONB,                     -- [{group, dataset_id}]
+    error           TEXT,
+    started_at      TIMESTAMPTZ,
+    finished_at     TIMESTAMPTZ,
+    UNIQUE (run_id, node_code)
+);
+CREATE INDEX idx_lp_run_nodes_lookup ON lp_run_nodes (node_code, fingerprint, status);
+```
+
+Run execution (worker, queue `gis.landscape.run`, routing key `landscape.run`):
+
+1. Mark run `running`. For each stage in order, for each node concurrently:
+2. **Resolve inputs** per input group: `overrides[group]` if present; else the dataset
+   produced for that group earlier in *this* run; else the freshest `ready` raster of
+   the group (§2.4). Missing → node fails with a clear message.
+3. **Fingerprint** = sha256 of node code, sorted `(group, dataset_id)` pairs, and the
+   node-relevant params. If `from_stage` is NULL or `stage < from_stage`, and a
+   previous `succeeded` node row with the same fingerprint exists whose output
+   datasets all still exist → mark `skipped`, reuse those outputs. Otherwise compute.
+4. Download inputs to a temp dir, `Compute`, then for each output create a raster
+   dataset (§3.6), record `output_datasets`, mark `succeeded`.
+5. Any node failure fails the stage; remaining stages are not started; run `failed`
+   with the error. Otherwise run `succeeded`.
+
+`from_stage = 0` recomputes everything ("триггер на перерасчёт всего").
+
+### 3.6 Result datasets
+
+Every output is stored through a new service method
+`DatasetService.CreateRasterFromFile(ctx, CreateRasterInput{CategoryID, GroupID,
+Name, Filename, Path, StartDate, EndDate, Meta})`: uploads the file, inserts a
+`raster` dataset (`automated = true`, `status = processing`), enqueues the existing
+COG conversion. `Name` = `<Output name> <run started_at date>`, `Filename` =
+`<group>.tif`, `start_date`/`end_date` = min/max over the node's inputs (NULL if all
+NULL), `meta = {"lp_run_id", "lp_node", "inputs": [{group, dataset_id}]}`.
+Downstream nodes read the original object (`storage_key`), not the COG, so they never
+wait for conversion. SLD for a result comes from its group (dataset SLD fallback, §2.3).
+
+### 3.7 Landscape API (`/landscape`)
+
+| Method & path | Behaviour |
+|---|---|
+| `GET /landscape/slots` | every leaf slot: `code`, `title`, group id, latest dataset summary (any status) or null |
+| `POST /landscape/slots/{code}` | **dedicated input upload**: multipart `file` (`.tif/.tiff`, TIFF magic check), optional `name`, `start_date`, `end_date`. Creates a `raster` dataset in the slot's group under the `landscape` category via the regular upload path (COG conversion enqueued). 201 Dataset. Unknown slot → 404 |
+| `POST /landscape/runs` | body `{from_stage?: 0..3, overrides?: {group_code: dataset_id}, params?: {window_size?, tie_priority?, v1_min_s_e12?, v1_min_s_e3?, v2_min_s_e3?, v2_min_s_e4?}}`; validates overrides (existing ready raster), inserts run `queued` + pending node rows, publishes job; 202 with run |
+| `GET /landscape/runs` | paginated, newest first |
+| `GET /landscape/runs/{id}` | run with nodes (status, inputs, outputs, error, timings) |
+| `GET /landscape/results` | per output group: `group`, `title`, `node`, latest `ready` dataset (or null) with `cog_url = /datasets/{id}.cog`, `sld_url` when the group has an SLD |
+
+Params defaults: `window_size 2048`, `tie_priority [1,2,3,4,5]`, `v1 3/4`, `v2 3/5`.
+Validation: tie_priority is a permutation of 1..5; thresholds in 1..5; window_size
+256..8192.
+
+### 3.8 Error handling
+
+- Grid mismatch, invalid CRS, missing input, GDAL error → node `failed` with message;
+  run `failed`. No retry (consistent with existing consumers: nack without requeue).
+- Non-nodata values outside 1..5 are treated as invalid pixels (TZ §2), not fatal.
+- Result dataset creation failure after compute → node failed, temp files removed.
+- Temp files always cleaned with `defer`.
+
+### 3.9 Testing
+
+- Kernels: table tests incl. TZ §7 cases (invalid E, all S invalid, E=5, E=4 with
+  several S=5, all S equal, applicability of crop/veg for U, "any valid" for max nodes).
+- `Compute`: tests with an in-memory fake `raster.IO` (10×10 grids), including grid
+  mismatch failure and window tiling with sizes that do not divide the raster.
+- godal implementation: round-trip test on a temp GeoTIFF (runs when GDAL present).
+- Services: stub repositories as in `dataset_test.go`; run planning (fingerprint,
+  skip/reuse, from_stage), input resolution order (override → this run → latest).
+- HTTP: handler tests for validation paths.
+- Part 1: filter clause building, latest ordering (nulls last), rename in mapping and
+  observations, SLD fallback.
+
+### 3.10 Out of scope / follow-ups
+
+- Resampling/alignment of inputs (fails by design).
+- Continuous (float) intermediate rasters and diagnostic layers from the notebook
+  (tie counts, component counts, critical flags).
+- Real-data acceptance run against the Drive test set (inputs there are not aligned).