feat(landscape): HTTP API, worker consumer, wiring, docs

Adds the /landscape routes (slots, slot upload, runs, run detail, results),
mounts them in the router, wires the landscape repository, job publisher,
godal raster IO and Compute into the composition root, and registers the
gis-landscape-runner consumer in the worker.

Also fixes the Dockerfile so the image actually runs: godal's cgo needs g++,
and the runtime base must match the builder's Alpine release or libgdal's
soname does not resolve.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
2026-09-05 14:11:57 +05:00
parent d74277552f
commit ad3c9fbdcd
8 changed files with 827 additions and 7 deletions

128
README.md
View File

@@ -19,6 +19,8 @@ internal/
transport/http/ chi router, middleware, handlers
storage/s3/ MinIO/S3 object storage
messaging/rabbitmq/ connection, publisher, consumer
landscape/ Landscape Program graph, kernels, window executor
raster/ GDAL raster I/O (COG conversion + godal engine)
platform/logger/ slog setup
pkg/httputil/ generic JSON/validation HTTP helpers
migrations/ embedded goose SQL migrations
@@ -126,6 +128,126 @@ which acts as the fallback style for every dataset in the group:
otherwise falls back to its group's — 404 only when neither exists. Deleting a
group also removes its stored SLD object.
## Landscape Program (ЛП)
The **Landscape Program** is a DAG of per-pixel raster computations that turns
ecological value (Z), vulnerability (U) and sector suitability (P) class rasters
into an ecological constraint (E), per-sector conflicts (C) and three functional
zoning scenarios (V1–V3). Every input and output of every node is a *dataset
group*; the leaf inputs are the **slots** users upload to, and every result is an
ordinary `raster` dataset (COG + SLD) served by the existing dataset endpoints.
The graph lives in Go (`internal/landscape/graph.go`); the groups and the
`landscape` category are seeded by migration `00011_landscape.sql`. A node code
`XY` reads as `X` = stage, `Y` = function within the stage: a stage starts only
after every node of all lower stages finished, and nodes of one stage run
concurrently.
| Node | Name | Inputs (group codes) | Outputs (group codes) |
|---|---|---|---|
| 00 | z_integrated | `lp-z-biodiversity`, `lp-z-lcr`, `lp-z-soil-protection`, `lp-z-wr` | `lp-z-integrated` |
| 01 | u_integrated | `lp-u-surf-water`, `lp-u-soil`, `lp-u-crop`, `lp-u-veg` | `lp-u-integrated` |
| 02 | z_max | Z1..Z4 as above | `lp-z-max` |
| 03 | u_max | U as above | `lp-u-max` |
| 10 | e_ecological_constraint | `lp-z-integrated`, `lp-u-integrated` | `lp-e` |
| 20 | c_conflicts | `lp-p-irrigation`, `lp-p-snow`, `lp-p-crop`, `lp-p-pasture`, `lp-p-tourism`, `lp-e` | `lp-c-irrigation`, `lp-c-snow`, `lp-c-crop`, `lp-c-pasture`, `lp-c-tourism` |
| 21 | p_max | five P groups | `lp-p-max` |
| 30 | func_zoning | `lp-e`, five C groups | `lp-v1-conservative`, `lp-v2-balanced`, `lp-v3-development` |
| 31 | c_max | five C groups | `lp-c-max` |
Sector order everywhere: 1 irrigation, 2 snow, 3 crop, 4 pasture, 5 tourism.
### Pixel rules
A pixel value is **valid** iff `1 <= v <= 5`; anything else (nodata included) is
invalid. Output rasters are `int16` with nodata `-9999`. `classify(x)` maps a
float32 score to a class: `<= 1.8 → 1`, `<= 2.6 → 2`, `<= 3.4 → 3`, `<= 4.2 → 4`,
`<= 5.0 → 5` (and requires `x >= 1.0`).
| Node | Valid domain (pixel) | Value |
|---|---|---|
| z_integrated | all four Z valid | `classify(0.25*Z1 + 0.25*Z2 + 0.25*Z3 + 0.25*Z4)` |
| u_integrated | water ∧ soil ∧ (crop ∨ veg) | `classify(num/den)` with crop/veg contributing `0.25` each only when valid |
| z_max / u_max | as above | `max` over the valid inputs |
| e_ecological_constraint | Z ∧ U valid | `max(Z, U)` |
| c_conflicts | `P_j` ∧ E valid | `min(P_j, E)` per sector |
| p_max / c_max | at least one input valid | `max` over the valid ones |
| func_zoning | E valid ∧ at least one `C_k` valid | winning sector + 1, else 1 |
Functional zoning takes `S_k = C_k`. A sector is a candidate when `S_k` and `E`
are valid and the scenario rule holds — V1: `E <= 2 && S >= v1_min_s_e12` or
`E == 3 && S >= v1_min_s_e3`; V2: `E <= 2`, or `E == 3 && S >= v2_min_s_e3`, or
`E == 4 && S >= v2_min_s_e4`; V3: `E <= 4`. With no candidate the pixel is class
1 (regulated); otherwise the candidate with the greatest `S` wins, ties broken by
`tie_priority`, and the class is `sector + 1` (2 irrigation, 3 snow, 4 crop,
5 pasture, 6 tourism).
Inputs are **not** resampled or reprojected: a node whose inputs do not share one
grid (size, geotransform, CRS) fails with a clear message, and so does the run.
### Uploading inputs, running, reading results
```sh
# 1. fill a slot (repeat per input group; see GET /landscape/slots for the codes)
curl -X POST https://dssgis.dwh.kz/landscape/slots/lp-z-biodiversity \
-F file=@z_biodiversity.tif -F name="Z Biodiversity 2025" \
-F start_date=2025-01-01 -F end_date=2025-12-31
# 2. queue a run (202); the worker executes the graph asynchronously
curl -X POST https://dssgis.dwh.kz/landscape/runs \
-H 'Content-Type: application/json' -d '{"from_stage": 0}'
# 3. follow it, then read the layers
curl https://dssgis.dwh.kz/landscape/runs/<run-id>
curl https://dssgis.dwh.kz/landscape/results
```
`GET /landscape/slots` lists every leaf slot with the dataset currently filling
it (any status), so the UI can show what is still missing. The upload is the
regular raster path: `.tif`/`.tiff` with a TIFF magic-byte check, stored in the
slot's group under the `landscape` category, COG conversion enqueued.
**Input resolution.** For each input group a node takes, in order: the run's
`overrides[group]`, else whatever this run already produced for that group, else
the freshest `ready` raster of the group (greatest `end_date`, NULLs last, newest
`created_at` first). A missing input fails the node and the run.
**Incremental recompute.** Each node is fingerprinted over its code, its resolved
inputs and the params that actually affect it. Without `from_stage`, a node whose
fingerprint matches an earlier `succeeded` node — with all of its output datasets
still present and `ready` — is marked `skipped` and its outputs reused. With
`from_stage: N`, every node of stage `N` and above is recomputed regardless;
`from_stage: 0` recomputes everything. `overrides` pins a specific dataset
(existing, `raster`, `ready`) for a group instead of resolving the freshest one.
Run params: `window_size` (256..8192, default 2048), `tie_priority` (a
permutation of 1..5, default `[1,2,3,4,5]`), `v1_min_s_e12`/`v1_min_s_e3`
(defaults 3/4) and `v2_min_s_e3`/`v2_min_s_e4` (defaults 3/5).
**Results.** Every output is a normal dataset in its group, named
`<Output name> <run date>`, `automated = true`, with
`meta = {lp_run_id, lp_node, inputs}` and the min/max period of its inputs.
`GET /landscape/results` returns one entry per output group with its freshest
`ready` dataset, a `cog_url` (`/datasets/{id}.cog`) and, when the group carries
one, an `sld_url` (`/dataset-groups/{id}/sld`). A node failure fails its stage;
later stages are not started and the run ends `failed` with the message. Jobs are
nacked without requeue, so there is no automatic retry — start a new run.
### Build requirements
The raster math uses [godal](https://github.com/airbusgeo/godal), a **cgo**
binding to GDAL, so the binary must be built with `CGO_ENABLED=1` against GDAL
headers and libraries:
```sh
brew install gdal # macOS; Linux: libgdal-dev / gdal-dev
go build ./... # cgo is on by default for a native build
```
`build/package/Dockerfile` already handles the image: the builder stage adds
`gdal-dev gcc musl-dev pkgconfig` and builds with `CGO_ENABLED=1`, and the
runtime stage installs `gdal` (the shared library godal needs) plus `gdal-tools`
(`gdal_translate`/`gdalinfo` for the COG conversion).
## Getting started
```sh
@@ -172,6 +294,12 @@ server runs it is served at `/openapi.yaml`, with an interactive **Redoc** UI at
| GET | `/dataset-groups/{id}/latest` | freshest dataset of the group (`?status=`, `?file_type=`) |
| POST | `/dataset-groups/{id}/sld` | upload/replace the group's shared SLD |
| GET | `/dataset-groups/{id}/sld` | download the group's shared SLD |
| GET | `/landscape/slots` | ЛП input slots + the dataset filling each |
| POST | `/landscape/slots/{code}` | upload a GeoTIFF into a slot (multipart: `file`, `name?`, `start_date?`, `end_date?`) |
| GET | `/landscape/runs` | paginated runs, newest first (`?page=`, `?page_size=`) |
| POST | `/landscape/runs` | queue a run (`from_stage?`, `overrides?`, `params?`); 202 |
| GET | `/landscape/runs/{id}` | run with per-node status, inputs, outputs, timings |
| GET | `/landscape/results` | per output group: latest layer, `cog_url`, `sld_url` |
Example upload: