gis/docs/superpowers/specs/2026-09-05-landscape-program-design.md

Landscape Program (ЛП) + dataset groups and dates — design

Date: 2026-09-05. Source: LP_20260902.excalidraw, TZ_functional_zoning_go.md, zoning_core.py, Landscape.ipynb (reference numpy implementation). Decisions taken with the product owner are recorded inline as Decision.

1. Goals

1. Dataset versioning. Datasets can belong to a dataset group ("одноимённые" datasets), carry a start_date / end_date period, be filtered by those, and the "freshest" dataset of a group can be fetched.
2. Landscape Program pipeline. A DAG of raster computations (Z, U, E, C, P_max, C_max, functional zoning V1–V3) executed by the worker, with results stored as ordinary raster datasets (COG + SLD) served by the existing dataset endpoints, incremental recompute, and a dedicated upload for input rasters.

Non-goals: spatial operations (resampling, reprojection), GUI, changing the vector_with_kato flow beyond the column rename below.

2. Part 1 — Dataset groups and dates

2.1 Schema (migration 00010_dataset_groups_and_dates.sql)

CREATE TABLE dataset_groups (
    id               UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    code             VARCHAR(255) NOT NULL UNIQUE,   -- slug, same rules as categories.code
    name             VARCHAR(255) NOT NULL,
    description      TEXT NOT NULL DEFAULT '',
    sld_storage_key  TEXT,
    sld_filename     VARCHAR(255),
    sld_content_type VARCHAR(255),
    created_at       TIMESTAMPTZ NOT NULL DEFAULT now(),
    updated_at       TIMESTAMPTZ NOT NULL DEFAULT now()
);

ALTER TABLE datasets ADD COLUMN dg_id      UUID REFERENCES dataset_groups(id) ON DELETE RESTRICT;
ALTER TABLE datasets ADD COLUMN start_date DATE;
ALTER TABLE datasets ADD COLUMN end_date   DATE;
CREATE INDEX idx_datasets_dg_id_end_date ON datasets (dg_id, end_date DESC NULLS LAST, created_at DESC);
CREATE INDEX idx_datasets_start_date ON datasets (start_date);
CREATE INDEX idx_datasets_end_date   ON datasets (end_date);

-- Decision: rename the observation/mapping period columns (breaking API change).
ALTER TABLE dataset_observations RENAME COLUMN date    TO start_date;
ALTER TABLE dataset_observations RENAME COLUMN date_to TO end_date;
-- unique index and idx_dataset_observations_kato_date recreated on the new names.
-- datasets.year_columns JSONB: every element's keys "date" -> "start_date",
-- "date_to" -> "end_date" (UPDATE with jsonb_agg over jsonb_array_elements; NULL-safe).

Constraint: end_date >= start_date when both set (CHECK constraint + service validation).

2.2 Domain changes

• domain.DatasetGroup{ID, Code, Name, Description, SldStorageKey, SldFilename, SldContentType, CreatedAt, UpdatedAt}.
• domain.Dataset and domain.DatasetSummary gain DatasetGroupID *uuid.UUID (json:"dg_id"), StartDate *string (json:"start_date", YYYY-MM-DD), EndDate *string (json:"end_date").
• domain.Observation: Date→StartDate (json:"start_date"), DateTo→EndDate (json:"end_date"). domain.YearColumn: same rename. Mapping request body uses start_date (required) / end_date (optional). Everything that previously said date/date_to in code, OpenAPI and README is renamed. GeoJSON output keeps its data (keyed by start date) and periods (start→end) shape.
• domain.DatasetFilter gains DatasetGroupID *uuid.UUID, StartDateFrom, StartDateTo, EndDateFrom, EndDateTo *string (inclusive bounds).

2.3 API

Method & path	Behaviour
GET /dataset-groups	paginated list, ?code= exact filter optional
POST /dataset-groups	{code, name, description}; 201
GET /dataset-groups/{id}	group
PUT /dataset-groups/{id}	replace {code, name, description}
DELETE /dataset-groups/{id}	204; 409 when datasets reference it; also removes group SLD object
GET /dataset-groups/{id}/latest	freshest dataset of the group: order by end_date DESC NULLS LAST, created_at DESC; optional ?status=, ?file_type=; 404 when none
POST /dataset-groups/{id}/sld	multipart file (.sld/.xml), replaces; mirrors dataset SLD upload
GET /dataset-groups/{id}/sld	download; 404 when none
GET /datasets	new filters dg_id, start_date_from, start_date_to, end_date_from, end_date_to (all YYYY-MM-DD, inclusive, any combination)
POST /datasets	new optional form fields dg_id, start_date, end_date
PUT /datasets/{id}	body gains dg_id, start_date, end_date (nullable; omitted/null clears)
PATCH /datasets/{id}	sparse; same three fields, null clears
GET /datasets/{id}/sld	fallback: when the dataset has no SLD but its group has one, the group SLD is served

Validation: dg_id must reference an existing group (404/422 as for category_id); dates parse as 2006-01-02; end_date not before start_date.

DatasetSummary (list responses) includes dg_id, start_date, end_date.

2.4 Freshness rule

"Самый свежий" = greatest end_date; datasets with NULL end_date sort after all dated ones; ties broken by created_at DESC. The same rule is used by the pipeline to resolve inputs (with status = ready, file_type = raster).

3. Part 2 — Landscape Program

3.1 Computation graph

Node code XY: X = stage, Y = function in the stage. A stage starts only when every node of all lower stages finished. Every input and output of every node is a dataset group; leaf groups are the slots users upload to.

Node	Name	Inputs (group codes)	Outputs (group codes)
00	z_integrated	lp-z-biodiversity, lp-z-lcr, lp-z-soil-protection, lp-z-wr	lp-z-integrated
01	u_integrated	lp-u-surf-water, lp-u-soil, lp-u-crop, lp-u-veg	lp-u-integrated
02	z_max	Z1..Z4 as above	lp-z-max
03	u_max	U as above	lp-u-max
10	e_ecological_constraint	lp-z-integrated, lp-u-integrated	lp-e
20	c_conflicts	lp-p-irrigation, lp-p-snow, lp-p-crop, lp-p-pasture, lp-p-tourism, lp-e	lp-c-irrigation, lp-c-snow, lp-c-crop, lp-c-pasture, lp-c-tourism
21	p_max	five P groups	lp-p-max
30	func_zoning	lp-e, five C groups	lp-v1-conservative, lp-v2-balanced, lp-v3-development
31	c_max	five C groups	lp-c-max

Sector order everywhere: 1 irrigation, 2 snow, 3 crop, 4 pasture, 5 tourism. Decision: func_zoning takes S_k = C_k (folder 300-…-Equals_102_202), not P_k.

Groups and the category landscape ("Ландшафтная программа") are seeded by migration 00011_landscape.sql (INSERT … ON CONFLICT (code) DO NOTHING). The graph itself lives in Go (internal/landscape/graph.go), not in the DB.

3.2 Pixel rules (from the reference notebook; must match it bitwise)

Notation: a value is valid iff 1 <= v <= 5; anything else (including nodata) is invalid. Output nodata is -9999. classify(x) maps a float32 score to a class: x <= 1.8 → 1, <= 2.6 → 2, <= 3.4 → 3, <= 4.2 → 4, <= 5.0 → 5 (and requires x >= 1.0); comparisons are done in float32 against float32 constants, exactly as numpy does with a float32 array.

Node	Valid domain (pixel)	Value
z_integrated	all four Z valid	classify(0.25*Z1 + 0.25*Z2 + 0.25*Z3 + 0.25*Z4) (float32)
u_integrated	water ∧ soil ∧ (crop ∨ veg)	classify(num/den), num = 0.25*water + 0.25*soil + [0.25*crop if crop valid] + [0.25*veg if veg valid], den = 0.5 + [0.25] + [0.25]
z_max	all four valid	max(Z1..Z4)
u_max	water ∧ soil ∧ (crop ∨ veg)	max over the valid ones of water, soil, crop, veg
e	Z ∧ U valid	max(Z, U)
c_conflicts (5 outputs)	P_j ∧ E valid	min(P_j, E)
p_max	at least one P valid	max over valid P
c_max	at least one C valid	max over valid C
func_zoning (3 outputs)	see TZ §4	see below

Functional zoning (TZ TZ_functional_zoning_go.md, reference zoning_core.py):

eValid = valid(E); base_k = eValid && valid(S_k); analysisValid = eValid && any(base_k)
if !analysisValid → -9999 in all scenarios
candidate_k = base_k && rule_v(E, S_k)
  V1: (E <= 2 && S >= v1_min_s_e12) || (E == 3 && S >= v1_min_s_e3)       (defaults 3, 4)
  V2: (E <= 2) || (E == 3 && S >= v2_min_s_e3) || (E == 4 && S >= v2_min_s_e4)  (defaults 3, 5)
  V3: (E <= 4)
no candidates → class 1 (regulated); else winners = candidates with S == max(S over candidates);
sector = first of tie_priority (default [1,2,3,4,5]) among winners; class = sector + 1

Output classes: 1 regulated, 2 irrigation, 3 snow, 4 crop, 5 pasture, 6 tourism.

3.3 Raster I/O abstraction (Decision: godal behind an interface)

Package internal/raster (existing) gains an engine-agnostic interface so the implementation can be swapped (godal today; gdal_calc, ENVI+CLI or another binding later) without touching the pipeline:

type Grid struct { Width, Height int; GeoTransform [6]float64; ProjectionWKT string }
type Window struct { X, Y, Width, Height int }
type Reader interface {
    Grid() Grid
    NoData() (value float64, ok bool)
    ReadFloat64(w Window, dst []float64) error   // band 1, converted by the driver
    Close() error
}
type WriterOptions struct { NoData int16; TileSize int; Compression string /* DEFLATE|ZSTD */ }
type Writer interface {
    WriteInt16(w Window, src []int16) error       // band 1
    Close() error
}
type IO interface {
    Open(ctx context.Context, path string) (Reader, error)
    Create(ctx context.Context, path string, g Grid, o WriterOptions) (Writer, error)
    SameGrid(a, b Grid) bool   // size, transform (abs tol 1e-9), CRS (GDAL IsSame)
}

internal/raster/godal.go implements IO with github.com/airbusgeo/godal (cgo). Output GeoTIFF: int16, nodata -9999, TILED=YES, BLOCKXSIZE=BLOCKYSIZE=512, COMPRESS=DEFLATE, PREDICTOR=2, BIGTIFF=IF_SAFER; georeference copied from the node's first input (for func_zoning: E). Inputs are read as float64 regardless of their storage type; multi-band inputs use band 1.

The existing RasterConverter (gdal_translate / gdalinfo / ogr2ogr CLI) stays as is and keeps producing the COG for every result dataset.

Build: CGO_ENABLED=1; go.mod gets toolchain go1.26.2 (the local SDK is incomplete) and the godal dependency. Dockerfile: builder golang:1.26-alpine + gdal-dev gcc musl-dev pkgconfig; runtime alpine:3.22 + gdal gdal-tools.

3.4 Compute engine (internal/landscape)

• graph.go: slots, nodes, stages, group codes, node-level param subsets.
• kernels.go: pure per-window functions over []float64 inputs → []int16 outputs, one per node. No I/O, fully unit-tested (TZ §7 edge cases included).
• compute.go: Compute(ctx, io raster.IO, node Node, inputs map[string]string /*group→local path*/, outDir string, p Params) (map[string]string /*group→path*/, error). Opens inputs, fails if grids differ (Decision: no resampling), creates outputs on the first input's grid, iterates windows of p.WindowSize (default 2048) with a worker pool: each worker owns its own Reader handles (GDAL datasets are not goroutine-safe); writes are serialised per output through a mutex. Window order does not affect results.
• Nodes of one stage run concurrently (errgroup); stages run sequentially.

3.5 Runs, incremental recompute

Migration 00011_landscape.sql:

CREATE TABLE lp_runs (
    id          UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    status      VARCHAR(32) NOT NULL,          -- queued | running | succeeded | failed
    from_stage  SMALLINT,                      -- NULL = incremental
    params      JSONB NOT NULL,
    overrides   JSONB NOT NULL DEFAULT '{}',   -- {group_code: dataset_id}
    error       TEXT,
    created_at  TIMESTAMPTZ NOT NULL DEFAULT now(),
    started_at  TIMESTAMPTZ,
    finished_at TIMESTAMPTZ
);
CREATE TABLE lp_run_nodes (
    id              UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    run_id          UUID NOT NULL REFERENCES lp_runs(id) ON DELETE CASCADE,
    node_code       VARCHAR(8) NOT NULL,
    stage           SMALLINT NOT NULL,
    status          VARCHAR(32) NOT NULL,      -- pending | running | skipped | succeeded | failed
    fingerprint     TEXT NOT NULL,
    input_datasets  JSONB NOT NULL,            -- [{group, dataset_id}]
    output_datasets JSONB,                     -- [{group, dataset_id}]
    error           TEXT,
    started_at      TIMESTAMPTZ,
    finished_at     TIMESTAMPTZ,
    UNIQUE (run_id, node_code)
);
CREATE INDEX idx_lp_run_nodes_lookup ON lp_run_nodes (node_code, fingerprint, status);

Run execution (worker, queue gis.landscape.run, routing key landscape.run):

1. Mark run running. For each stage in order, for each node concurrently:
2. Resolve inputs per input group: overrides[group] if present; else the dataset produced for that group earlier in this run; else the freshest ready raster of the group (§2.4). Missing → node fails with a clear message.
3. Fingerprint = sha256 of node code, sorted (group, dataset_id) pairs, and the node-relevant params. If from_stage is NULL or stage < from_stage, and a previous succeeded node row with the same fingerprint exists whose output datasets all still exist → mark skipped, reuse those outputs. Otherwise compute.
4. Download inputs to a temp dir, Compute, then for each output create a raster dataset (§3.6), record output_datasets, mark succeeded.
5. Any node failure fails the stage; remaining stages are not started; run failed with the error. Otherwise run succeeded.

from_stage = 0 recomputes everything ("триггер на перерасчёт всего").

3.6 Result datasets

Every output is stored through a new service method DatasetService.CreateRasterFromFile(ctx, CreateRasterInput{CategoryID, GroupID, Name, Filename, Path, StartDate, EndDate, Meta}): uploads the file, inserts a raster dataset (automated = true, status = processing), enqueues the existing COG conversion. Name = <Output name> <run started_at date>, Filename = <group>.tif, start_date/end_date = min/max over the node's inputs (NULL if all NULL), meta = {"lp_run_id", "lp_node", "inputs": [{group, dataset_id}]}. Downstream nodes read the original object (storage_key), not the COG, so they never wait for conversion. SLD for a result comes from its group (dataset SLD fallback, §2.3).

3.7 Landscape API (/landscape)

Method & path	Behaviour
GET /landscape/slots	every leaf slot: code, title, group id, latest dataset summary (any status) or null
POST /landscape/slots/{code}	dedicated input upload: multipart file (.tif/.tiff, TIFF magic check), optional name, start_date, end_date. Creates a raster dataset in the slot's group under the landscape category via the regular upload path (COG conversion enqueued). 201 Dataset. Unknown slot → 404
POST /landscape/runs	body {from_stage?: 0..3, overrides?: {group_code: dataset_id}, params?: {window_size?, tie_priority?, v1_min_s_e12?, v1_min_s_e3?, v2_min_s_e3?, v2_min_s_e4?}}; validates overrides (existing ready raster), inserts run queued + pending node rows, publishes job; 202 with run
GET /landscape/runs	paginated, newest first
GET /landscape/runs/{id}	run with nodes (status, inputs, outputs, error, timings)
GET /landscape/results	per output group: group, title, node, latest ready dataset (or null) with cog_url = /datasets/{id}.cog, sld_url when the group has an SLD

Params defaults: window_size 2048, tie_priority [1,2,3,4,5], v1 3/4, v2 3/5. Validation: tie_priority is a permutation of 1..5; thresholds in 1..5; window_size 256..8192.

3.8 Error handling

• Grid mismatch, invalid CRS, missing input, GDAL error → node failed with message; run failed. No retry (consistent with existing consumers: nack without requeue).
• Non-nodata values outside 1..5 are treated as invalid pixels (TZ §2), not fatal.
• Result dataset creation failure after compute → node failed, temp files removed.
• Temp files always cleaned with defer.

3.9 Testing

• Kernels: table tests incl. TZ §7 cases (invalid E, all S invalid, E=5, E=4 with several S=5, all S equal, applicability of crop/veg for U, "any valid" for max nodes).
• Compute: tests with an in-memory fake raster.IO (10×10 grids), including grid mismatch failure and window tiling with sizes that do not divide the raster.
• godal implementation: round-trip test on a temp GeoTIFF (runs when GDAL present).
• Services: stub repositories as in dataset_test.go; run planning (fingerprint, skip/reuse, from_stage), input resolution order (override → this run → latest).
• HTTP: handler tests for validation paths.
• Part 1: filter clause building, latest ordering (nulls last), rename in mapping and observations, SLD fallback.

3.10 Out of scope / follow-ups

• Resampling/alignment of inputs (fails by design).
• Continuous (float) intermediate rasters and diagnostic layers from the notebook (tie counts, component counts, critical flags).
• Real-data acceptance run against the Drive test set (inputs there are not aligned).