From c930f5928b8c12a23980b679a72a5aaba2df103a Mon Sep 17 00:00:00 2001 From: Bakhtiyar Issakhmetov Date: Sat, 5 Sep 2026 12:57:53 +0500 Subject: [PATCH] docs: Landscape Program + dataset groups design spec Co-Authored-By: Claude Fable 5.1 --- .../2026-09-05-landscape-program-design.md | 317 ++++++++++++++++++ 1 file changed, 317 insertions(+) create mode 100644 docs/superpowers/specs/2026-09-05-landscape-program-design.md diff --git a/docs/superpowers/specs/2026-09-05-landscape-program-design.md b/docs/superpowers/specs/2026-09-05-landscape-program-design.md new file mode 100644 index 0000000..77268b1 --- /dev/null +++ b/docs/superpowers/specs/2026-09-05-landscape-program-design.md @@ -0,0 +1,317 @@ +# Landscape Program (ЛП) + dataset groups and dates — design + +Date: 2026-09-05. Source: `LP_20260902.excalidraw`, `TZ_functional_zoning_go.md`, +`zoning_core.py`, `Landscape.ipynb` (reference numpy implementation). +Decisions taken with the product owner are recorded inline as **Decision**. + +## 1. Goals + +1. **Dataset versioning.** Datasets can belong to a *dataset group* ("одноимённые" + datasets), carry a `start_date` / `end_date` period, be filtered by those, and the + "freshest" dataset of a group can be fetched. +2. **Landscape Program pipeline.** A DAG of raster computations (Z, U, E, C, P_max, + C_max, functional zoning V1–V3) executed by the worker, with results stored as + ordinary raster datasets (COG + SLD) served by the existing dataset endpoints, + incremental recompute, and a dedicated upload for input rasters. + +Non-goals: spatial operations (resampling, reprojection), GUI, changing the +vector_with_kato flow beyond the column rename below. + +## 2. Part 1 — Dataset groups and dates + +### 2.1 Schema (migration `00010_dataset_groups_and_dates.sql`) + +```sql +CREATE TABLE dataset_groups ( + id UUID PRIMARY KEY DEFAULT gen_random_uuid(), + code VARCHAR(255) NOT NULL UNIQUE, -- slug, same rules as categories.code + name VARCHAR(255) NOT NULL, + description TEXT NOT NULL DEFAULT '', + sld_storage_key TEXT, + sld_filename VARCHAR(255), + sld_content_type VARCHAR(255), + created_at TIMESTAMPTZ NOT NULL DEFAULT now(), + updated_at TIMESTAMPTZ NOT NULL DEFAULT now() +); + +ALTER TABLE datasets ADD COLUMN dg_id UUID REFERENCES dataset_groups(id) ON DELETE RESTRICT; +ALTER TABLE datasets ADD COLUMN start_date DATE; +ALTER TABLE datasets ADD COLUMN end_date DATE; +CREATE INDEX idx_datasets_dg_id_end_date ON datasets (dg_id, end_date DESC NULLS LAST, created_at DESC); +CREATE INDEX idx_datasets_start_date ON datasets (start_date); +CREATE INDEX idx_datasets_end_date ON datasets (end_date); + +-- Decision: rename the observation/mapping period columns (breaking API change). +ALTER TABLE dataset_observations RENAME COLUMN date TO start_date; +ALTER TABLE dataset_observations RENAME COLUMN date_to TO end_date; +-- unique index and idx_dataset_observations_kato_date recreated on the new names. +-- datasets.year_columns JSONB: every element's keys "date" -> "start_date", +-- "date_to" -> "end_date" (UPDATE with jsonb_agg over jsonb_array_elements; NULL-safe). +``` + +Constraint: `end_date >= start_date` when both set (CHECK constraint + service validation). + +### 2.2 Domain changes + +- `domain.DatasetGroup{ID, Code, Name, Description, SldStorageKey, SldFilename, + SldContentType, CreatedAt, UpdatedAt}`. +- `domain.Dataset` and `domain.DatasetSummary` gain `DatasetGroupID *uuid.UUID` + (`json:"dg_id"`), `StartDate *string` (`json:"start_date"`, `YYYY-MM-DD`), + `EndDate *string` (`json:"end_date"`). +- `domain.Observation`: `Date`→`StartDate` (`json:"start_date"`), `DateTo`→`EndDate` + (`json:"end_date"`). `domain.YearColumn`: same rename. Mapping request body uses + `start_date` (required) / `end_date` (optional). Everything that previously said + `date`/`date_to` in code, OpenAPI and README is renamed. GeoJSON output keeps its + `data` (keyed by start date) and `periods` (start→end) shape. +- `domain.DatasetFilter` gains `DatasetGroupID *uuid.UUID`, `StartDateFrom`, + `StartDateTo`, `EndDateFrom`, `EndDateTo *string` (inclusive bounds). + +### 2.3 API + +| Method & path | Behaviour | +|---|---| +| `GET /dataset-groups` | paginated list, `?code=` exact filter optional | +| `POST /dataset-groups` | `{code, name, description}`; 201 | +| `GET /dataset-groups/{id}` | group | +| `PUT /dataset-groups/{id}` | replace `{code, name, description}` | +| `DELETE /dataset-groups/{id}` | 204; 409 when datasets reference it; also removes group SLD object | +| `GET /dataset-groups/{id}/latest` | freshest dataset of the group: order by `end_date DESC NULLS LAST, created_at DESC`; optional `?status=`, `?file_type=`; 404 when none | +| `POST /dataset-groups/{id}/sld` | multipart `file` (`.sld`/`.xml`), replaces; mirrors dataset SLD upload | +| `GET /dataset-groups/{id}/sld` | download; 404 when none | +| `GET /datasets` | new filters `dg_id`, `start_date_from`, `start_date_to`, `end_date_from`, `end_date_to` (all `YYYY-MM-DD`, inclusive, any combination) | +| `POST /datasets` | new optional form fields `dg_id`, `start_date`, `end_date` | +| `PUT /datasets/{id}` | body gains `dg_id`, `start_date`, `end_date` (nullable; omitted/null clears) | +| `PATCH /datasets/{id}` | sparse; same three fields, `null` clears | +| `GET /datasets/{id}/sld` | **fallback**: when the dataset has no SLD but its group has one, the group SLD is served | + +Validation: `dg_id` must reference an existing group (404/422 as for `category_id`); +dates parse as `2006-01-02`; `end_date` not before `start_date`. + +`DatasetSummary` (list responses) includes `dg_id`, `start_date`, `end_date`. + +### 2.4 Freshness rule + +"Самый свежий" = greatest `end_date`; datasets with `NULL end_date` sort after all +dated ones; ties broken by `created_at DESC`. The same rule is used by the pipeline +to resolve inputs (with `status = ready`, `file_type = raster`). + +## 3. Part 2 — Landscape Program + +### 3.1 Computation graph + +Node code `XY`: `X` = stage, `Y` = function in the stage. A stage starts only when +every node of all lower stages finished. Every input and output of every node is a +**dataset group**; leaf groups are the *slots* users upload to. + +| Node | Name | Inputs (group codes) | Outputs (group codes) | +|---|---|---|---| +| 00 | z_integrated | `lp-z-biodiversity`, `lp-z-lcr`, `lp-z-soil-protection`, `lp-z-wr` | `lp-z-integrated` | +| 01 | u_integrated | `lp-u-surf-water`, `lp-u-soil`, `lp-u-crop`, `lp-u-veg` | `lp-u-integrated` | +| 02 | z_max | Z1..Z4 as above | `lp-z-max` | +| 03 | u_max | U as above | `lp-u-max` | +| 10 | e_ecological_constraint | `lp-z-integrated`, `lp-u-integrated` | `lp-e` | +| 20 | c_conflicts | `lp-p-irrigation`, `lp-p-snow`, `lp-p-crop`, `lp-p-pasture`, `lp-p-tourism`, `lp-e` | `lp-c-irrigation`, `lp-c-snow`, `lp-c-crop`, `lp-c-pasture`, `lp-c-tourism` | +| 21 | p_max | five P groups | `lp-p-max` | +| 30 | func_zoning | `lp-e`, five C groups | `lp-v1-conservative`, `lp-v2-balanced`, `lp-v3-development` | +| 31 | c_max | five C groups | `lp-c-max` | + +Sector order everywhere: 1 irrigation, 2 snow, 3 crop, 4 pasture, 5 tourism. +**Decision:** func_zoning takes S_k = C_k (folder `300-…-Equals_102_202`), not P_k. + +Groups and the category `landscape` ("Ландшафтная программа") are seeded by +migration `00011_landscape.sql` (`INSERT … ON CONFLICT (code) DO NOTHING`). The +graph itself lives in Go (`internal/landscape/graph.go`), not in the DB. + +### 3.2 Pixel rules (from the reference notebook; must match it bitwise) + +Notation: a value is *valid* iff `1 <= v <= 5`; anything else (including nodata) is +invalid. Output nodata is `-9999`. `classify(x)` maps a float32 score to a class: +`x <= 1.8 → 1`, `<= 2.6 → 2`, `<= 3.4 → 3`, `<= 4.2 → 4`, `<= 5.0 → 5` (and requires +`x >= 1.0`); comparisons are done in float32 against float32 constants, exactly as +numpy does with a float32 array. + +| Node | Valid domain (pixel) | Value | +|---|---|---| +| z_integrated | all four Z valid | `classify(0.25*Z1 + 0.25*Z2 + 0.25*Z3 + 0.25*Z4)` (float32) | +| u_integrated | water ∧ soil ∧ (crop ∨ veg) | `classify(num/den)`, `num = 0.25*water + 0.25*soil + [0.25*crop if crop valid] + [0.25*veg if veg valid]`, `den = 0.5 + [0.25] + [0.25]` | +| z_max | all four valid | `max(Z1..Z4)` | +| u_max | water ∧ soil ∧ (crop ∨ veg) | `max` over the valid ones of water, soil, crop, veg | +| e | Z ∧ U valid | `max(Z, U)` | +| c_conflicts (5 outputs) | P_j ∧ E valid | `min(P_j, E)` | +| p_max | at least one P valid | `max` over valid P | +| c_max | at least one C valid | `max` over valid C | +| func_zoning (3 outputs) | see TZ §4 | see below | + +Functional zoning (TZ `TZ_functional_zoning_go.md`, reference `zoning_core.py`): + +``` +eValid = valid(E); base_k = eValid && valid(S_k); analysisValid = eValid && any(base_k) +if !analysisValid → -9999 in all scenarios +candidate_k = base_k && rule_v(E, S_k) + V1: (E <= 2 && S >= v1_min_s_e12) || (E == 3 && S >= v1_min_s_e3) (defaults 3, 4) + V2: (E <= 2) || (E == 3 && S >= v2_min_s_e3) || (E == 4 && S >= v2_min_s_e4) (defaults 3, 5) + V3: (E <= 4) +no candidates → class 1 (regulated); else winners = candidates with S == max(S over candidates); +sector = first of tie_priority (default [1,2,3,4,5]) among winners; class = sector + 1 +``` + +Output classes: 1 regulated, 2 irrigation, 3 snow, 4 crop, 5 pasture, 6 tourism. + +### 3.3 Raster I/O abstraction (**Decision:** godal behind an interface) + +Package `internal/raster` (existing) gains an engine-agnostic interface so the +implementation can be swapped (godal today; `gdal_calc`, ENVI+CLI or another binding +later) without touching the pipeline: + +```go +type Grid struct { Width, Height int; GeoTransform [6]float64; ProjectionWKT string } +type Window struct { X, Y, Width, Height int } +type Reader interface { + Grid() Grid + NoData() (value float64, ok bool) + ReadFloat64(w Window, dst []float64) error // band 1, converted by the driver + Close() error +} +type WriterOptions struct { NoData int16; TileSize int; Compression string /* DEFLATE|ZSTD */ } +type Writer interface { + WriteInt16(w Window, src []int16) error // band 1 + Close() error +} +type IO interface { + Open(ctx context.Context, path string) (Reader, error) + Create(ctx context.Context, path string, g Grid, o WriterOptions) (Writer, error) + SameGrid(a, b Grid) bool // size, transform (abs tol 1e-9), CRS (GDAL IsSame) +} +``` + +`internal/raster/godal.go` implements `IO` with `github.com/airbusgeo/godal` (cgo). +Output GeoTIFF: `int16`, nodata `-9999`, `TILED=YES`, `BLOCKXSIZE=BLOCKYSIZE=512`, +`COMPRESS=DEFLATE`, `PREDICTOR=2`, `BIGTIFF=IF_SAFER`; georeference copied from the +node's first input (for func_zoning: E). Inputs are read as float64 regardless of +their storage type; multi-band inputs use band 1. + +The existing `RasterConverter` (gdal_translate / gdalinfo / ogr2ogr CLI) stays as is +and keeps producing the COG for every result dataset. + +Build: `CGO_ENABLED=1`; `go.mod` gets `toolchain go1.26.2` (the local SDK is +incomplete) and the godal dependency. Dockerfile: builder `golang:1.26-alpine` + +`gdal-dev gcc musl-dev pkgconfig`; runtime `alpine:3.22` + `gdal gdal-tools`. + +### 3.4 Compute engine (`internal/landscape`) + +- `graph.go`: slots, nodes, stages, group codes, node-level param subsets. +- `kernels.go`: pure per-window functions over `[]float64` inputs → `[]int16` + outputs, one per node. No I/O, fully unit-tested (TZ §7 edge cases included). +- `compute.go`: `Compute(ctx, io raster.IO, node Node, inputs map[string]string + /*group→local path*/, outDir string, p Params) (map[string]string /*group→path*/, error)`. + Opens inputs, **fails if grids differ** (**Decision:** no resampling), creates + outputs on the first input's grid, iterates windows of `p.WindowSize` + (default 2048) with a worker pool: each worker owns its own `Reader` handles + (GDAL datasets are not goroutine-safe); writes are serialised per output through a + mutex. Window order does not affect results. +- Nodes of one stage run concurrently (`errgroup`); stages run sequentially. + +### 3.5 Runs, incremental recompute + +Migration `00011_landscape.sql`: + +```sql +CREATE TABLE lp_runs ( + id UUID PRIMARY KEY DEFAULT gen_random_uuid(), + status VARCHAR(32) NOT NULL, -- queued | running | succeeded | failed + from_stage SMALLINT, -- NULL = incremental + params JSONB NOT NULL, + overrides JSONB NOT NULL DEFAULT '{}', -- {group_code: dataset_id} + error TEXT, + created_at TIMESTAMPTZ NOT NULL DEFAULT now(), + started_at TIMESTAMPTZ, + finished_at TIMESTAMPTZ +); +CREATE TABLE lp_run_nodes ( + id UUID PRIMARY KEY DEFAULT gen_random_uuid(), + run_id UUID NOT NULL REFERENCES lp_runs(id) ON DELETE CASCADE, + node_code VARCHAR(8) NOT NULL, + stage SMALLINT NOT NULL, + status VARCHAR(32) NOT NULL, -- pending | running | skipped | succeeded | failed + fingerprint TEXT NOT NULL, + input_datasets JSONB NOT NULL, -- [{group, dataset_id}] + output_datasets JSONB, -- [{group, dataset_id}] + error TEXT, + started_at TIMESTAMPTZ, + finished_at TIMESTAMPTZ, + UNIQUE (run_id, node_code) +); +CREATE INDEX idx_lp_run_nodes_lookup ON lp_run_nodes (node_code, fingerprint, status); +``` + +Run execution (worker, queue `gis.landscape.run`, routing key `landscape.run`): + +1. Mark run `running`. For each stage in order, for each node concurrently: +2. **Resolve inputs** per input group: `overrides[group]` if present; else the dataset + produced for that group earlier in *this* run; else the freshest `ready` raster of + the group (§2.4). Missing → node fails with a clear message. +3. **Fingerprint** = sha256 of node code, sorted `(group, dataset_id)` pairs, and the + node-relevant params. If `from_stage` is NULL or `stage < from_stage`, and a + previous `succeeded` node row with the same fingerprint exists whose output + datasets all still exist → mark `skipped`, reuse those outputs. Otherwise compute. +4. Download inputs to a temp dir, `Compute`, then for each output create a raster + dataset (§3.6), record `output_datasets`, mark `succeeded`. +5. Any node failure fails the stage; remaining stages are not started; run `failed` + with the error. Otherwise run `succeeded`. + +`from_stage = 0` recomputes everything ("триггер на перерасчёт всего"). + +### 3.6 Result datasets + +Every output is stored through a new service method +`DatasetService.CreateRasterFromFile(ctx, CreateRasterInput{CategoryID, GroupID, +Name, Filename, Path, StartDate, EndDate, Meta})`: uploads the file, inserts a +`raster` dataset (`automated = true`, `status = processing`), enqueues the existing +COG conversion. `Name` = ` `, `Filename` = +`.tif`, `start_date`/`end_date` = min/max over the node's inputs (NULL if all +NULL), `meta = {"lp_run_id", "lp_node", "inputs": [{group, dataset_id}]}`. +Downstream nodes read the original object (`storage_key`), not the COG, so they never +wait for conversion. SLD for a result comes from its group (dataset SLD fallback, §2.3). + +### 3.7 Landscape API (`/landscape`) + +| Method & path | Behaviour | +|---|---| +| `GET /landscape/slots` | every leaf slot: `code`, `title`, group id, latest dataset summary (any status) or null | +| `POST /landscape/slots/{code}` | **dedicated input upload**: multipart `file` (`.tif/.tiff`, TIFF magic check), optional `name`, `start_date`, `end_date`. Creates a `raster` dataset in the slot's group under the `landscape` category via the regular upload path (COG conversion enqueued). 201 Dataset. Unknown slot → 404 | +| `POST /landscape/runs` | body `{from_stage?: 0..3, overrides?: {group_code: dataset_id}, params?: {window_size?, tie_priority?, v1_min_s_e12?, v1_min_s_e3?, v2_min_s_e3?, v2_min_s_e4?}}`; validates overrides (existing ready raster), inserts run `queued` + pending node rows, publishes job; 202 with run | +| `GET /landscape/runs` | paginated, newest first | +| `GET /landscape/runs/{id}` | run with nodes (status, inputs, outputs, error, timings) | +| `GET /landscape/results` | per output group: `group`, `title`, `node`, latest `ready` dataset (or null) with `cog_url = /datasets/{id}.cog`, `sld_url` when the group has an SLD | + +Params defaults: `window_size 2048`, `tie_priority [1,2,3,4,5]`, `v1 3/4`, `v2 3/5`. +Validation: tie_priority is a permutation of 1..5; thresholds in 1..5; window_size +256..8192. + +### 3.8 Error handling + +- Grid mismatch, invalid CRS, missing input, GDAL error → node `failed` with message; + run `failed`. No retry (consistent with existing consumers: nack without requeue). +- Non-nodata values outside 1..5 are treated as invalid pixels (TZ §2), not fatal. +- Result dataset creation failure after compute → node failed, temp files removed. +- Temp files always cleaned with `defer`. + +### 3.9 Testing + +- Kernels: table tests incl. TZ §7 cases (invalid E, all S invalid, E=5, E=4 with + several S=5, all S equal, applicability of crop/veg for U, "any valid" for max nodes). +- `Compute`: tests with an in-memory fake `raster.IO` (10×10 grids), including grid + mismatch failure and window tiling with sizes that do not divide the raster. +- godal implementation: round-trip test on a temp GeoTIFF (runs when GDAL present). +- Services: stub repositories as in `dataset_test.go`; run planning (fingerprint, + skip/reuse, from_stage), input resolution order (override → this run → latest). +- HTTP: handler tests for validation paths. +- Part 1: filter clause building, latest ordering (nulls last), rename in mapping and + observations, SLD fallback. + +### 3.10 Out of scope / follow-ups + +- Resampling/alignment of inputs (fails by design). +- Continuous (float) intermediate rasters and diagnostic layers from the notebook + (tie counts, component counts, critical flags). +- Real-data acceptance run against the Drive test set (inputs there are not aligned).