feat(landscape): HTTP API, worker consumer, wiring, docs
Adds the /landscape routes (slots, slot upload, runs, run detail, results), mounts them in the router, wires the landscape repository, job publisher, godal raster IO and Compute into the composition root, and registers the gis-landscape-runner consumer in the worker. Also fixes the Dockerfile so the image actually runs: godal's cgo needs g++, and the runtime base must match the builder's Alpine release or libgdal's soname does not resolve. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
128
README.md
128
README.md
@@ -19,6 +19,8 @@ internal/
|
||||
transport/http/ chi router, middleware, handlers
|
||||
storage/s3/ MinIO/S3 object storage
|
||||
messaging/rabbitmq/ connection, publisher, consumer
|
||||
landscape/ Landscape Program graph, kernels, window executor
|
||||
raster/ GDAL raster I/O (COG conversion + godal engine)
|
||||
platform/logger/ slog setup
|
||||
pkg/httputil/ generic JSON/validation HTTP helpers
|
||||
migrations/ embedded goose SQL migrations
|
||||
@@ -126,6 +128,126 @@ which acts as the fallback style for every dataset in the group:
|
||||
otherwise falls back to its group's — 404 only when neither exists. Deleting a
|
||||
group also removes its stored SLD object.
|
||||
|
||||
## Landscape Program (ЛП)
|
||||
|
||||
The **Landscape Program** is a DAG of per-pixel raster computations that turns
|
||||
ecological value (Z), vulnerability (U) and sector suitability (P) class rasters
|
||||
into an ecological constraint (E), per-sector conflicts (C) and three functional
|
||||
zoning scenarios (V1–V3). Every input and output of every node is a *dataset
|
||||
group*; the leaf inputs are the **slots** users upload to, and every result is an
|
||||
ordinary `raster` dataset (COG + SLD) served by the existing dataset endpoints.
|
||||
|
||||
The graph lives in Go (`internal/landscape/graph.go`); the groups and the
|
||||
`landscape` category are seeded by migration `00011_landscape.sql`. A node code
|
||||
`XY` reads as `X` = stage, `Y` = function within the stage: a stage starts only
|
||||
after every node of all lower stages finished, and nodes of one stage run
|
||||
concurrently.
|
||||
|
||||
| Node | Name | Inputs (group codes) | Outputs (group codes) |
|
||||
|---|---|---|---|
|
||||
| 00 | z_integrated | `lp-z-biodiversity`, `lp-z-lcr`, `lp-z-soil-protection`, `lp-z-wr` | `lp-z-integrated` |
|
||||
| 01 | u_integrated | `lp-u-surf-water`, `lp-u-soil`, `lp-u-crop`, `lp-u-veg` | `lp-u-integrated` |
|
||||
| 02 | z_max | Z1..Z4 as above | `lp-z-max` |
|
||||
| 03 | u_max | U as above | `lp-u-max` |
|
||||
| 10 | e_ecological_constraint | `lp-z-integrated`, `lp-u-integrated` | `lp-e` |
|
||||
| 20 | c_conflicts | `lp-p-irrigation`, `lp-p-snow`, `lp-p-crop`, `lp-p-pasture`, `lp-p-tourism`, `lp-e` | `lp-c-irrigation`, `lp-c-snow`, `lp-c-crop`, `lp-c-pasture`, `lp-c-tourism` |
|
||||
| 21 | p_max | five P groups | `lp-p-max` |
|
||||
| 30 | func_zoning | `lp-e`, five C groups | `lp-v1-conservative`, `lp-v2-balanced`, `lp-v3-development` |
|
||||
| 31 | c_max | five C groups | `lp-c-max` |
|
||||
|
||||
Sector order everywhere: 1 irrigation, 2 snow, 3 crop, 4 pasture, 5 tourism.
|
||||
|
||||
### Pixel rules
|
||||
|
||||
A pixel value is **valid** iff `1 <= v <= 5`; anything else (nodata included) is
|
||||
invalid. Output rasters are `int16` with nodata `-9999`. `classify(x)` maps a
|
||||
float32 score to a class: `<= 1.8 → 1`, `<= 2.6 → 2`, `<= 3.4 → 3`, `<= 4.2 → 4`,
|
||||
`<= 5.0 → 5` (and requires `x >= 1.0`).
|
||||
|
||||
| Node | Valid domain (pixel) | Value |
|
||||
|---|---|---|
|
||||
| z_integrated | all four Z valid | `classify(0.25*Z1 + 0.25*Z2 + 0.25*Z3 + 0.25*Z4)` |
|
||||
| u_integrated | water ∧ soil ∧ (crop ∨ veg) | `classify(num/den)` with crop/veg contributing `0.25` each only when valid |
|
||||
| z_max / u_max | as above | `max` over the valid inputs |
|
||||
| e_ecological_constraint | Z ∧ U valid | `max(Z, U)` |
|
||||
| c_conflicts | `P_j` ∧ E valid | `min(P_j, E)` per sector |
|
||||
| p_max / c_max | at least one input valid | `max` over the valid ones |
|
||||
| func_zoning | E valid ∧ at least one `C_k` valid | winning sector + 1, else 1 |
|
||||
|
||||
Functional zoning takes `S_k = C_k`. A sector is a candidate when `S_k` and `E`
|
||||
are valid and the scenario rule holds — V1: `E <= 2 && S >= v1_min_s_e12` or
|
||||
`E == 3 && S >= v1_min_s_e3`; V2: `E <= 2`, or `E == 3 && S >= v2_min_s_e3`, or
|
||||
`E == 4 && S >= v2_min_s_e4`; V3: `E <= 4`. With no candidate the pixel is class
|
||||
1 (regulated); otherwise the candidate with the greatest `S` wins, ties broken by
|
||||
`tie_priority`, and the class is `sector + 1` (2 irrigation, 3 snow, 4 crop,
|
||||
5 pasture, 6 tourism).
|
||||
|
||||
Inputs are **not** resampled or reprojected: a node whose inputs do not share one
|
||||
grid (size, geotransform, CRS) fails with a clear message, and so does the run.
|
||||
|
||||
### Uploading inputs, running, reading results
|
||||
|
||||
```sh
|
||||
# 1. fill a slot (repeat per input group; see GET /landscape/slots for the codes)
|
||||
curl -X POST https://dssgis.dwh.kz/landscape/slots/lp-z-biodiversity \
|
||||
-F file=@z_biodiversity.tif -F name="Z Biodiversity 2025" \
|
||||
-F start_date=2025-01-01 -F end_date=2025-12-31
|
||||
|
||||
# 2. queue a run (202); the worker executes the graph asynchronously
|
||||
curl -X POST https://dssgis.dwh.kz/landscape/runs \
|
||||
-H 'Content-Type: application/json' -d '{"from_stage": 0}'
|
||||
|
||||
# 3. follow it, then read the layers
|
||||
curl https://dssgis.dwh.kz/landscape/runs/<run-id>
|
||||
curl https://dssgis.dwh.kz/landscape/results
|
||||
```
|
||||
|
||||
`GET /landscape/slots` lists every leaf slot with the dataset currently filling
|
||||
it (any status), so the UI can show what is still missing. The upload is the
|
||||
regular raster path: `.tif`/`.tiff` with a TIFF magic-byte check, stored in the
|
||||
slot's group under the `landscape` category, COG conversion enqueued.
|
||||
|
||||
**Input resolution.** For each input group a node takes, in order: the run's
|
||||
`overrides[group]`, else whatever this run already produced for that group, else
|
||||
the freshest `ready` raster of the group (greatest `end_date`, NULLs last, newest
|
||||
`created_at` first). A missing input fails the node and the run.
|
||||
|
||||
**Incremental recompute.** Each node is fingerprinted over its code, its resolved
|
||||
inputs and the params that actually affect it. Without `from_stage`, a node whose
|
||||
fingerprint matches an earlier `succeeded` node — with all of its output datasets
|
||||
still present and `ready` — is marked `skipped` and its outputs reused. With
|
||||
`from_stage: N`, every node of stage `N` and above is recomputed regardless;
|
||||
`from_stage: 0` recomputes everything. `overrides` pins a specific dataset
|
||||
(existing, `raster`, `ready`) for a group instead of resolving the freshest one.
|
||||
Run params: `window_size` (256..8192, default 2048), `tie_priority` (a
|
||||
permutation of 1..5, default `[1,2,3,4,5]`), `v1_min_s_e12`/`v1_min_s_e3`
|
||||
(defaults 3/4) and `v2_min_s_e3`/`v2_min_s_e4` (defaults 3/5).
|
||||
|
||||
**Results.** Every output is a normal dataset in its group, named
|
||||
`<Output name> <run date>`, `automated = true`, with
|
||||
`meta = {lp_run_id, lp_node, inputs}` and the min/max period of its inputs.
|
||||
`GET /landscape/results` returns one entry per output group with its freshest
|
||||
`ready` dataset, a `cog_url` (`/datasets/{id}.cog`) and, when the group carries
|
||||
one, an `sld_url` (`/dataset-groups/{id}/sld`). A node failure fails its stage;
|
||||
later stages are not started and the run ends `failed` with the message. Jobs are
|
||||
nacked without requeue, so there is no automatic retry — start a new run.
|
||||
|
||||
### Build requirements
|
||||
|
||||
The raster math uses [godal](https://github.com/airbusgeo/godal), a **cgo**
|
||||
binding to GDAL, so the binary must be built with `CGO_ENABLED=1` against GDAL
|
||||
headers and libraries:
|
||||
|
||||
```sh
|
||||
brew install gdal # macOS; Linux: libgdal-dev / gdal-dev
|
||||
go build ./... # cgo is on by default for a native build
|
||||
```
|
||||
|
||||
`build/package/Dockerfile` already handles the image: the builder stage adds
|
||||
`gdal-dev gcc musl-dev pkgconfig` and builds with `CGO_ENABLED=1`, and the
|
||||
runtime stage installs `gdal` (the shared library godal needs) plus `gdal-tools`
|
||||
(`gdal_translate`/`gdalinfo` for the COG conversion).
|
||||
|
||||
## Getting started
|
||||
|
||||
```sh
|
||||
@@ -172,6 +294,12 @@ server runs it is served at `/openapi.yaml`, with an interactive **Redoc** UI at
|
||||
| GET | `/dataset-groups/{id}/latest` | freshest dataset of the group (`?status=`, `?file_type=`) |
|
||||
| POST | `/dataset-groups/{id}/sld` | upload/replace the group's shared SLD |
|
||||
| GET | `/dataset-groups/{id}/sld` | download the group's shared SLD |
|
||||
| GET | `/landscape/slots` | ЛП input slots + the dataset filling each |
|
||||
| POST | `/landscape/slots/{code}` | upload a GeoTIFF into a slot (multipart: `file`, `name?`, `start_date?`, `end_date?`) |
|
||||
| GET | `/landscape/runs` | paginated runs, newest first (`?page=`, `?page_size=`) |
|
||||
| POST | `/landscape/runs` | queue a run (`from_stage?`, `overrides?`, `params?`); 202 |
|
||||
| GET | `/landscape/runs/{id}` | run with per-node status, inputs, outputs, timings |
|
||||
| GET | `/landscape/results` | per output group: latest layer, `cog_url`, `sld_url` |
|
||||
|
||||
Example upload:
|
||||
|
||||
|
||||
Reference in New Issue
Block a user