gis/docs/superpowers/specs/2026-09-05-landscape-program-design.md

# Landscape Program (ЛП) + dataset groups and dates — design

Date: 2026-09-05. Source: `LP_20260902.excalidraw`, `TZ_functional_zoning_go.md`,
`zoning_core.py`, `Landscape.ipynb` (reference numpy implementation).
Decisions taken with the product owner are recorded inline as **Decision**.

## 1. Goals

1. **Dataset versioning.** Datasets can belong to a *dataset group* ("одноимённые"
   datasets), carry a `start_date` / `end_date` period, be filtered by those, and the
   "freshest" dataset of a group can be fetched.
2. **Landscape Program pipeline.** A DAG of raster computations (Z, U, E, C, P_max,
   C_max, functional zoning V1–V3) executed by the worker, with results stored as
   ordinary raster datasets (COG + SLD) served by the existing dataset endpoints,
   incremental recompute, and a dedicated upload for input rasters.

Non-goals: spatial operations (resampling, reprojection), GUI, changing the
vector_with_kato flow beyond the column rename below.

## 2. Part 1 — Dataset groups and dates

### 2.1 Schema (migration `00010_dataset_groups_and_dates.sql`)

```sql
CREATE TABLE dataset_groups (
    id               UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    code             VARCHAR(255) NOT NULL UNIQUE,   -- slug, same rules as categories.code
    name             VARCHAR(255) NOT NULL,
    description      TEXT NOT NULL DEFAULT '',
    sld_storage_key  TEXT,
    sld_filename     VARCHAR(255),
    sld_content_type VARCHAR(255),
    created_at       TIMESTAMPTZ NOT NULL DEFAULT now(),
    updated_at       TIMESTAMPTZ NOT NULL DEFAULT now()
);

ALTER TABLE datasets ADD COLUMN dg_id      UUID REFERENCES dataset_groups(id) ON DELETE RESTRICT;
ALTER TABLE datasets ADD COLUMN start_date DATE;
ALTER TABLE datasets ADD COLUMN end_date   DATE;
CREATE INDEX idx_datasets_dg_id_end_date ON datasets (dg_id, end_date DESC NULLS LAST, created_at DESC);
CREATE INDEX idx_datasets_start_date ON datasets (start_date);
CREATE INDEX idx_datasets_end_date   ON datasets (end_date);

-- Decision: rename the observation/mapping period columns (breaking API change).
ALTER TABLE dataset_observations RENAME COLUMN date    TO start_date;
ALTER TABLE dataset_observations RENAME COLUMN date_to TO end_date;
-- unique index and idx_dataset_observations_kato_date recreated on the new names.
-- datasets.year_columns JSONB: every element's keys "date" -> "start_date",
-- "date_to" -> "end_date" (UPDATE with jsonb_agg over jsonb_array_elements; NULL-safe).
```

Constraint: `end_date >= start_date` when both set (CHECK constraint + service validation).

### 2.2 Domain changes

- `domain.DatasetGroup{ID, Code, Name, Description, SldStorageKey, SldFilename,
  SldContentType, CreatedAt, UpdatedAt}`.
- `domain.Dataset` and `domain.DatasetSummary` gain `DatasetGroupID *uuid.UUID`
  (`json:"dg_id"`), `StartDate *string` (`json:"start_date"`, `YYYY-MM-DD`),
  `EndDate *string` (`json:"end_date"`).
- `domain.Observation`: `Date`→`StartDate` (`json:"start_date"`), `DateTo`→`EndDate`
  (`json:"end_date"`). `domain.YearColumn`: same rename. Mapping request body uses
  `start_date` (required) / `end_date` (optional). Everything that previously said
  `date`/`date_to` in code, OpenAPI and README is renamed. GeoJSON output keeps its
  `data` (keyed by start date) and `periods` (start→end) shape.
- `domain.DatasetFilter` gains `DatasetGroupID *uuid.UUID`, `StartDateFrom`,
  `StartDateTo`, `EndDateFrom`, `EndDateTo *string` (inclusive bounds).

### 2.3 API

| Method & path | Behaviour |
|---|---|
| `GET /dataset-groups` | paginated list, `?code=` exact filter optional |
| `POST /dataset-groups` | `{code, name, description}`; 201 |
| `GET /dataset-groups/{id}` | group |
| `PUT /dataset-groups/{id}` | replace `{code, name, description}` |
| `DELETE /dataset-groups/{id}` | 204; 409 when datasets reference it; also removes group SLD object |
| `GET /dataset-groups/{id}/latest` | freshest dataset of the group: order by `end_date DESC NULLS LAST, created_at DESC`; optional `?status=`, `?file_type=`; 404 when none |
| `POST /dataset-groups/{id}/sld` | multipart `file` (`.sld`/`.xml`), replaces; mirrors dataset SLD upload |
| `GET /dataset-groups/{id}/sld` | download; 404 when none |
| `GET /datasets` | new filters `dg_id`, `start_date_from`, `start_date_to`, `end_date_from`, `end_date_to` (all `YYYY-MM-DD`, inclusive, any combination) |
| `POST /datasets` | new optional form fields `dg_id`, `start_date`, `end_date` |
| `PUT /datasets/{id}` | body gains `dg_id`, `start_date`, `end_date` (nullable; omitted/null clears) |
| `PATCH /datasets/{id}` | sparse; same three fields, `null` clears |
| `GET /datasets/{id}/sld` | **fallback**: when the dataset has no SLD but its group has one, the group SLD is served |

Validation: `dg_id` must reference an existing group (404/422 as for `category_id`);
dates parse as `2006-01-02`; `end_date` not before `start_date`.

`DatasetSummary` (list responses) includes `dg_id`, `start_date`, `end_date`.

### 2.4 Freshness rule

"Самый свежий" = greatest `end_date`; datasets with `NULL end_date` sort after all
dated ones; ties broken by `created_at DESC`. The same rule is used by the pipeline
to resolve inputs (with `status = ready`, `file_type = raster`).

## 3. Part 2 — Landscape Program

### 3.1 Computation graph

Node code `XY`: `X` = stage, `Y` = function in the stage. A stage starts only when
every node of all lower stages finished. Every input and output of every node is a
**dataset group**; leaf groups are the *slots* users upload to.

| Node | Name | Inputs (group codes) | Outputs (group codes) |
|---|---|---|---|
| 00 | z_integrated | `lp-z-biodiversity`, `lp-z-lcr`, `lp-z-soil-protection`, `lp-z-wr` | `lp-z-integrated` |
| 01 | u_integrated | `lp-u-surf-water`, `lp-u-soil`, `lp-u-crop`, `lp-u-veg` | `lp-u-integrated` |
| 02 | z_max | Z1..Z4 as above | `lp-z-max` |
| 03 | u_max | U as above | `lp-u-max` |
| 10 | e_ecological_constraint | `lp-z-integrated`, `lp-u-integrated` | `lp-e` |
| 20 | c_conflicts | `lp-p-irrigation`, `lp-p-snow`, `lp-p-crop`, `lp-p-pasture`, `lp-p-tourism`, `lp-e` | `lp-c-irrigation`, `lp-c-snow`, `lp-c-crop`, `lp-c-pasture`, `lp-c-tourism` |
| 21 | p_max | five P groups | `lp-p-max` |
| 30 | func_zoning | `lp-e`, five C groups | `lp-v1-conservative`, `lp-v2-balanced`, `lp-v3-development` |
| 31 | c_max | five C groups | `lp-c-max` |

Sector order everywhere: 1 irrigation, 2 snow, 3 crop, 4 pasture, 5 tourism.
**Decision:** func_zoning takes S_k = C_k (folder `300-…-Equals_102_202`), not P_k.

Groups and the category `landscape` ("Ландшафтная программа") are seeded by
migration `00011_landscape.sql` (`INSERT … ON CONFLICT (code) DO NOTHING`). The
graph itself lives in Go (`internal/landscape/graph.go`), not in the DB.

### 3.2 Pixel rules (from the reference notebook; must match it bitwise)

Notation: a value is *valid* iff `1 <= v <= 5`; anything else (including nodata) is
invalid. Output nodata is `-9999`. `classify(x)` maps a float32 score to a class:
`x <= 1.8 → 1`, `<= 2.6 → 2`, `<= 3.4 → 3`, `<= 4.2 → 4`, `<= 5.0 → 5` (and requires
`x >= 1.0`); comparisons are done in float32 against float32 constants, exactly as
numpy does with a float32 array.

| Node | Valid domain (pixel) | Value |
|---|---|---|
| z_integrated | all four Z valid | `classify(0.25*Z1 + 0.25*Z2 + 0.25*Z3 + 0.25*Z4)` (float32) |
| u_integrated | water ∧ soil ∧ (crop ∨ veg) | `classify(num/den)`, `num = 0.25*water + 0.25*soil + [0.25*crop if crop valid] + [0.25*veg if veg valid]`, `den = 0.5 + [0.25] + [0.25]` |
| z_max | all four valid | `max(Z1..Z4)` |
| u_max | water ∧ soil ∧ (crop ∨ veg) | `max` over the valid ones of water, soil, crop, veg |
| e | Z ∧ U valid | `max(Z, U)` |
| c_conflicts (5 outputs) | P_j ∧ E valid | `min(P_j, E)` |
| p_max | at least one P valid | `max` over valid P |
| c_max | at least one C valid | `max` over valid C |
| func_zoning (3 outputs) | see TZ §4 | see below |

Functional zoning (TZ `TZ_functional_zoning_go.md`, reference `zoning_core.py`):

```
eValid = valid(E); base_k = eValid && valid(S_k); analysisValid = eValid && any(base_k)
if !analysisValid → -9999 in all scenarios
candidate_k = base_k && rule_v(E, S_k)
  V1: (E <= 2 && S >= v1_min_s_e12) || (E == 3 && S >= v1_min_s_e3)       (defaults 3, 4)
  V2: (E <= 2) || (E == 3 && S >= v2_min_s_e3) || (E == 4 && S >= v2_min_s_e4)  (defaults 3, 5)
  V3: (E <= 4)
no candidates → class 1 (regulated); else winners = candidates with S == max(S over candidates);
sector = first of tie_priority (default [1,2,3,4,5]) among winners; class = sector + 1
```

Output classes: 1 regulated, 2 irrigation, 3 snow, 4 crop, 5 pasture, 6 tourism.

### 3.3 Raster I/O abstraction (**Decision:** godal behind an interface)

Package `internal/raster` (existing) gains an engine-agnostic interface so the
implementation can be swapped (godal today; `gdal_calc`, ENVI+CLI or another binding
later) without touching the pipeline:

```go
type Grid struct { Width, Height int; GeoTransform [6]float64; ProjectionWKT string }
type Window struct { X, Y, Width, Height int }
type Reader interface {
    Grid() Grid
    NoData() (value float64, ok bool)
    ReadFloat64(w Window, dst []float64) error   // band 1, converted by the driver
    Close() error
}
type WriterOptions struct { NoData int16; TileSize int; Compression string /* DEFLATE|ZSTD */ }
type Writer interface {
    WriteInt16(w Window, src []int16) error       // band 1
    Close() error
}
type IO interface {
    Open(ctx context.Context, path string) (Reader, error)
    Create(ctx context.Context, path string, g Grid, o WriterOptions) (Writer, error)
    SameGrid(a, b Grid) bool   // size, transform (abs tol 1e-9), CRS (GDAL IsSame)
}
```

`internal/raster/godal.go` implements `IO` with `github.com/airbusgeo/godal` (cgo).
Output GeoTIFF: `int16`, nodata `-9999`, `TILED=YES`, `BLOCKXSIZE=BLOCKYSIZE=512`,
`COMPRESS=DEFLATE`, `PREDICTOR=2`, `BIGTIFF=IF_SAFER`; georeference copied from the
node's first input (for func_zoning: E). Inputs are read as float64 regardless of
their storage type; multi-band inputs use band 1.

The existing `RasterConverter` (gdal_translate / gdalinfo / ogr2ogr CLI) stays as is
and keeps producing the COG for every result dataset.

Build: `CGO_ENABLED=1`; `go.mod` gets `toolchain go1.26.2` (the local SDK is
incomplete) and the godal dependency. Dockerfile: builder `golang:1.26-alpine` +
`gdal-dev gcc musl-dev pkgconfig`; runtime `alpine:3.22` + `gdal gdal-tools`.

### 3.4 Compute engine (`internal/landscape`)

- `graph.go`: slots, nodes, stages, group codes, node-level param subsets.
- `kernels.go`: pure per-window functions over `[]float64` inputs → `[]int16`
  outputs, one per node. No I/O, fully unit-tested (TZ §7 edge cases included).
- `compute.go`: `Compute(ctx, io raster.IO, node Node, inputs map[string]string
  /*group→local path*/, outDir string, p Params) (map[string]string /*group→path*/, error)`.
  Opens inputs, **fails if grids differ** (**Decision:** no resampling), creates
  outputs on the first input's grid, iterates windows of `p.WindowSize`
  (default 2048) with a worker pool: each worker owns its own `Reader` handles
  (GDAL datasets are not goroutine-safe); writes are serialised per output through a
  mutex. Window order does not affect results.
- Nodes of one stage run concurrently (`errgroup`); stages run sequentially.

### 3.5 Runs, incremental recompute

Migration `00011_landscape.sql`:

```sql
CREATE TABLE lp_runs (
    id          UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    status      VARCHAR(32) NOT NULL,          -- queued | running | succeeded | failed
    from_stage  SMALLINT,                      -- NULL = incremental
    params      JSONB NOT NULL,
    overrides   JSONB NOT NULL DEFAULT '{}',   -- {group_code: dataset_id}
    error       TEXT,
    created_at  TIMESTAMPTZ NOT NULL DEFAULT now(),
    started_at  TIMESTAMPTZ,
    finished_at TIMESTAMPTZ
);
CREATE TABLE lp_run_nodes (
    id              UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    run_id          UUID NOT NULL REFERENCES lp_runs(id) ON DELETE CASCADE,
    node_code       VARCHAR(8) NOT NULL,
    stage           SMALLINT NOT NULL,
    status          VARCHAR(32) NOT NULL,      -- pending | running | skipped | succeeded | failed
    fingerprint     TEXT NOT NULL,
    input_datasets  JSONB NOT NULL,            -- [{group, dataset_id}]
    output_datasets JSONB,                     -- [{group, dataset_id}]
    error           TEXT,
    started_at      TIMESTAMPTZ,
    finished_at     TIMESTAMPTZ,
    UNIQUE (run_id, node_code)
);
CREATE INDEX idx_lp_run_nodes_lookup ON lp_run_nodes (node_code, fingerprint, status);
```

Run execution (worker, queue `gis.landscape.run`, routing key `landscape.run`):

1. Mark run `running`. For each stage in order, for each node concurrently:
2. **Resolve inputs** per input group: `overrides[group]` if present; else the dataset
   produced for that group earlier in *this* run; else the freshest `ready` raster of
   the group (§2.4). Missing → node fails with a clear message.
3. **Fingerprint** = sha256 of node code, sorted `(group, dataset_id)` pairs, and the
   node-relevant params. If `from_stage` is NULL or `stage < from_stage`, and a
   previous `succeeded` node row with the same fingerprint exists whose output
   datasets all still exist → mark `skipped`, reuse those outputs. Otherwise compute.
4. Download inputs to a temp dir, `Compute`, then for each output create a raster
   dataset (§3.6), record `output_datasets`, mark `succeeded`.
5. Any node failure fails the stage; remaining stages are not started; run `failed`
   with the error. Otherwise run `succeeded`.

`from_stage = 0` recomputes everything ("триггер на перерасчёт всего").

### 3.6 Result datasets

Every output is stored through a new service method
`DatasetService.CreateRasterFromFile(ctx, CreateRasterInput{CategoryID, GroupID,
Name, Filename, Path, StartDate, EndDate, Meta})`: uploads the file, inserts a
`raster` dataset (`automated = true`, `status = processing`), enqueues the existing
COG conversion. `Name` = `<Output name> <run started_at date>`, `Filename` =
`<group>.tif`, `start_date`/`end_date` = min/max over the node's inputs (NULL if all
NULL), `meta = {"lp_run_id", "lp_node", "inputs": [{group, dataset_id}]}`.
Downstream nodes read the original object (`storage_key`), not the COG, so they never
wait for conversion. SLD for a result comes from its group (dataset SLD fallback, §2.3).

### 3.7 Landscape API (`/landscape`)

| Method & path | Behaviour |
|---|---|
| `GET /landscape/slots` | every leaf slot: `code`, `title`, group id, latest dataset summary (any status) or null |
| `POST /landscape/slots/{code}` | **dedicated input upload**: multipart `file` (`.tif/.tiff`, TIFF magic check), optional `name`, `start_date`, `end_date`. Creates a `raster` dataset in the slot's group under the `landscape` category via the regular upload path (COG conversion enqueued). 201 Dataset. Unknown slot → 404 |
| `POST /landscape/runs` | body `{from_stage?: 0..3, overrides?: {group_code: dataset_id}, params?: {window_size?, tie_priority?, v1_min_s_e12?, v1_min_s_e3?, v2_min_s_e3?, v2_min_s_e4?}}`; validates overrides (existing ready raster), inserts run `queued` + pending node rows, publishes job; 202 with run |
| `GET /landscape/runs` | paginated, newest first |
| `GET /landscape/runs/{id}` | run with nodes (status, inputs, outputs, error, timings) |
| `GET /landscape/results` | per output group: `group`, `title`, `node`, latest `ready` dataset (or null) with `cog_url = /datasets/{id}.cog`, `sld_url` when the group has an SLD |

Params defaults: `window_size 2048`, `tie_priority [1,2,3,4,5]`, `v1 3/4`, `v2 3/5`.
Validation: tie_priority is a permutation of 1..5; thresholds in 1..5; window_size
256..8192.

### 3.8 Error handling

- Grid mismatch, invalid CRS, missing input, GDAL error → node `failed` with message;
  run `failed`. No retry (consistent with existing consumers: nack without requeue).
- Non-nodata values outside 1..5 are treated as invalid pixels (TZ §2), not fatal.
- Result dataset creation failure after compute → node failed, temp files removed.
- Temp files always cleaned with `defer`.

### 3.9 Testing

- Kernels: table tests incl. TZ §7 cases (invalid E, all S invalid, E=5, E=4 with
  several S=5, all S equal, applicability of crop/veg for U, "any valid" for max nodes).
- `Compute`: tests with an in-memory fake `raster.IO` (10×10 grids), including grid
  mismatch failure and window tiling with sizes that do not divide the raster.
- godal implementation: round-trip test on a temp GeoTIFF (runs when GDAL present).
- Services: stub repositories as in `dataset_test.go`; run planning (fingerprint,
  skip/reuse, from_stage), input resolution order (override → this run → latest).
- HTTP: handler tests for validation paths.
- Part 1: filter clause building, latest ordering (nulls last), rename in mapping and
  observations, SLD fallback.

### 3.10 Out of scope / follow-ups

- Resampling/alignment of inputs (fails by design).
- Continuous (float) intermediate rasters and diagnostic layers from the notebook
  (tie counts, component counts, critical flags).
- Real-data acceptance run against the Drive test set (inputs there are not aligned).