Compare commits

...

26 Commits

Author SHA1 Message Date
ca87d6961b test(acceptance): reproducible Stage 8 acceptance package
Add test/acceptance/ so the customer's reviewers can re-run what the
Stage 8 documents claim on an isolated stand built from this branch:

- compose.acceptance.yml + stand.sh: compose project gis-acceptance
  (app 18080, postgres 15432, minio 19000/19001, rabbitmq mgmt 15672),
  POSTGIS_PLATFORM / MINIO_IMAGE knobs for arm64 and amd64 hosts.
- functional.sh: the 89 test-stand checks of Приложение 7 section 5
  (F01-F38, N01-N38, L02-L05, L07-L15), own prefixed test data,
  PostGIS/MinIO confirmation via the stand containers, full cleanup.
- lp_gen.py / lp_run.sh / lp_verify.py: synthetic ЛП inputs, a run from
  stage 0, and an independent numpy re-implementation of every node that
  compares all 15 outputs pixel by pixel; read-only (GET) mode works
  against any stand, in row/column windows for 22219x14977 rasters.
- load.sh + seed.sh: scenarios S1-S6 of Приложение 3 with hey 0.1.5
  running inside the stand network; CSV/JSON summaries, criteria K1-K4.
- scan.sh: govulncheck (source and binary), gosec, trivy fs/image,
  golangci-lint; fails on any High/Critical.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-15 22:37:28 +05:00
c6229c46f7 build(deps): go1.26.9 and golang.org/x/net v0.60.0
Close stdlib (net/http, crypto/tls, mime/multipart, net/http/httputil)
and x/net advisories published after the go1.26.8 release
(GO-2026-6603..6617, CVE-2026-78667/78669/97031).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-15 21:53:51 +05:00
5b44cbda2f fix(security): close High/Critical findings from the Stage 8 scan
- Bump github.com/rabbitmq/amqp091-go 1.12.0 -> 1.13.0 (3 Critical, 7 High,
  reachable via rabbitmq.Connect) and golang.org/x/{text,net,crypto},
  klauspost/compress; toolchain go1.26.8, builder image pinned to
  golang:1.26.8-alpine3.24.
- Build Content-Disposition per RFC 6266/8187 (ASCII fallback +
  filename*=UTF-8''), stripping CR/LF, quotes and backslashes (CWE-113).
- Cap request bodies with MaxBytesReader (HTTP_MAX_UPLOAD_BYTES, default
  2 GiB); multipart uploads over the limit return 413 (CWE-400).
- /readyz reports "unavailable" instead of raw backend errors (CWE-209).
- Drop deprecated middleware.RealIP; add X-Content-Type-Options: nosniff.
- docker-compose credentials come from ${VAR:-default}.
- Check the RegisterValidation error; annotate verified gosec false
  positives with #nosec and a reason.

Re-scan: govulncheck 0 reachable, trivy fs/image 0 High/Critical,
gosec High 0. go test -race passes against PostGIS.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-15 21:44:10 +05:00
2544d4e146 fix(landscape): func_zoning ranks the original P suitability rasters, not C
The reference notebook ("Three scenarios" cells) and zoning_core.py feed E plus
the five P_* WEIGHTED_CLASS rasters into functional zoning; the diagram folder
300-…-Equals_102_202 pointing at the C layers was a drawing error. C_k =
min(P_k, E) stay diagnostic conflict layers. Node 30 keeps its code/stage;
its changed inputs change every fingerprint, so V1–V3 recompute on the next run.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-11 21:45:39 +05:00
dae9294b4d fix(landscape): cast run status parameter so SetRunStatus type-checks in Postgres
The status parameter was used both as the assigned value (varchar column)
and in CASE comparisons against text literals, so Postgres deduced two
types for $2 and every run failed at SetRunStatus(running) with SQLSTATE
42P08; the failure also prevented marking the run failed, leaving it
queued. Cast every use to text.

Add TEST_DB_URL-gated repository tests (skipped without a database) that
exercise the landscape run/node SQL and the new dataset period/group
queries against a real Postgres, and document them in the README.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-11 18:02:24 +05:00
d06d67436b test(landscape): make node-failure test deterministic under sibling cancellation
Scope the stub publisher failure to node 00's output; when every stage-0
publish failed, the first failing sibling cancelled node 00 and its
recorded error was "context canceled" instead of the publish error.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-05 16:01:33 +05:00
4051506b95 fix(landscape): guard redelivery, detach terminal writes, bound workers
F2: Execute had no status guard, so a redelivered or stale message reran a
finished run. Runs that are neither queued nor running are logged and acked
without work; "running" still executes so a worker that crashed mid-run
recovers.

F4: failRun, finishNodeFailed and MarkRunFailed wrote the terminal status on
the caller's context, which is exactly the context that is already cancelled
when the run failed because of a shutdown or a sibling node — the write was
refused and the run stayed stuck in "running". They now write on a
context.WithoutCancel copy with a 10s timeout; the consumer's best-effort
MarkRunFailed does the same.

F5: peak memory scaled with the host's core count. LP_MAX_WORKERS (default 8)
is now the total window-worker budget of a run and LP_MAX_NODES (default 2) the
node fan-out; each node gets max(1, workers/nodes) through the new
landscape.ComputeWithWorkers. Compute and service.ComputeFunc keep their
signatures: Compute delegates with MaxWorkersPerNode, and a legacy ComputeFunc
handed to NewLandscapeService is wrapped.

F6: the stage loop used a plain errgroup, so a failing node let its siblings
run to completion. It now uses errgroup.WithContext and passes the group
context into runNode, cancelling siblings mid-compute.

F7: wrap every float32 product in an explicit conversion so the compiler
cannot fuse it into an FMA, keeping the kernels bit-identical to numpy's
per-op rounding.

F8: README's builder package list now matches the Dockerfile (adds g++), and
the Landscape section documents the memory model and the two new knobs.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-05 14:44:34 +05:00
bcda76003c fix(datasets): accept BigTIFF headers and sanitise object keys
F1: ValidateFileContent rejected the pipeline's own output. GDAL writes a
BigTIFF header ("II+\0" / "MM\0+") whenever BIGTIFF=IF_SAFER kicks in for a
large raster, so accept both alongside the classic TIFF magic and reword the
error to "file is not a valid TIFF/BigTIFF GeoTIFF". GodalIO.Create now
delegates to an unexported createGTiff so a test can force BIGTIFF=YES and
assert the produced header passes the check.

F3: a user-supplied filename went straight into the S3 object key for dataset
uploads and for both dataset and dataset-group SLD uploads, so "../../x.sld"
escaped the dataset's own prefix. Reduce every such name to its base with a
shared safeObjectName helper, rejecting names with no usable base.

F9: rename the shadowing inner "raw" in the PATCH dg_id case to "s".

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-05 14:44:21 +05:00
6aec00ee29 fix(landscape): accept an empty run body; pin the builder's Alpine release
POST /landscape/runs documents an optional body, but DecodeJSON returned
io.EOF on an empty one and the handler turned that into 400 — breaking the
most common call, an all-defaults run. decodeRunRequest now treats io.EOF as
the zero request; a malformed body is still 400.

The Dockerfile's Alpine ABI coupling was enforced only by a comment: a
floating golang:1.26-alpine builder against a pinned alpine:3.24 runtime
would build green and die at startup on a libgdal soname mismatch as soon as
the Go image rolls forward. Both stages are now pinned to 3.24.

Also: the route-mount test walks the router so all six routes are asserted
(GET /slots and GET /results included), and the spec documents the 404 those
two return when migration 00011's seeded groups are absent.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-05 14:22:26 +05:00
ad3c9fbdcd feat(landscape): HTTP API, worker consumer, wiring, docs
Adds the /landscape routes (slots, slot upload, runs, run detail, results),
mounts them in the router, wires the landscape repository, job publisher,
godal raster IO and Compute into the composition root, and registers the
gis-landscape-runner consumer in the worker.

Also fixes the Dockerfile so the image actually runs: godal's cgo needs g++,
and the runtime base must match the builder's Alpine release or libgdal's
soname does not resolve.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-05 14:11:57 +05:00
d74277552f fix(landscape): bound node fan-out, cover override resolution, test the run consumer
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-05 13:49:27 +05:00
5fba77b59c feat(landscape): run planning/execution service with fingerprint reuse; RabbitMQ job
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-05 13:40:25 +05:00
ff6d5fc412 fix(landscape): bound Compute's per-worker memory and clean up on failure
Review round 1: size per-worker buffers off the largest actual window
instead of WindowSize^2 (avoids allocating ~200+ MiB per worker for
small/partial rasters), cap workers at 8 regardless of GOMAXPROCS, remove
partially-written outputs when a run fails, wrap input-open errors with
node/group context, guard checkGrids against a node with no inputs, and
clamp a non-positive WindowSize so raster.Windows can't loop forever.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-05 13:30:23 +05:00
bc524f3d98 feat(dataset-groups): CRUD, latest-in-group, group SLD with dataset fallback, CreateRasterFromFile
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-05 13:29:01 +05:00
32eba016b3 feat(landscape): window executor Compute with worker pool and grid checks
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-05 13:21:12 +05:00
831ddd1154 feat(landscape): computation graph, params and per-pixel kernels
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-05 13:16:17 +05:00
922812b765 feat(datasets): periods (start_date/end_date), dg_id column, date filters; rename observation dates
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-05 13:16:07 +05:00
37d4f3519b feat(raster): IO abstraction with godal implementation; cgo Dockerfile
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-05 13:14:48 +05:00
5ca6d9e662 feat(landscape): run/node domain, seed migration, repository
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-05 13:13:32 +05:00
91c2cab66e docs: Landscape Program implementation plan
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-05 13:08:25 +05:00
bea7e116a0 build: pin toolchain go1.26.2 (local SDK is incomplete)
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-05 13:00:01 +05:00
c930f5928b docs: Landscape Program + dataset groups design spec
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-05 12:57:53 +05:00
1e0844ca82 feat: Add SLD endpoints, add date_to to vector_with_kato 2026-08-01 22:09:59 +05:00
00a0cd1b8e feat: Add PUT and PATCH datasets endpoints 2026-07-21 00:19:52 +05:00
47b2e8653d fix: Fix processing for vector attributres table 2026-07-10 23:34:19 +05:00
017f869380 feat: Remove geometry and properties from GET /datasets/{id} endpoint, new endpoint for download cog file 2026-07-02 16:27:36 +05:00
83 changed files with 16200 additions and 325 deletions

View File

@@ -9,4 +9,8 @@ S3_SECRET_KEY: minioadmin
S3_BUCKET: geofiles
S3_USE_SSL: "false"
DB_PORT=5423
DB_PORT=5423
# Landscape Program parallelism (bounds worker peak memory)
LP_MAX_WORKERS=8
LP_MAX_NODES=2

227
README.md
View File

@@ -19,6 +19,8 @@ internal/
transport/http/ chi router, middleware, handlers
storage/s3/ MinIO/S3 object storage
messaging/rabbitmq/ connection, publisher, consumer
landscape/ Landscape Program graph, kernels, window executor
raster/ GDAL raster I/O (COG conversion + godal engine)
platform/logger/ slog setup
pkg/httputil/ generic JSON/validation HTTP helpers
migrations/ embedded goose SQL migrations
@@ -32,6 +34,9 @@ api/openapi.yaml OpenAPI 3.1.1 spec (embedded + served at /openapi.yaml)
- **Category** — hierarchical (self-referencing `parent_id`). Full CRUD; cycle-safe
on update.
- **DatasetGroup** — a named group of "одноимённые" datasets (see *Dataset
groups* below). Full CRUD; deleting a group still referenced by datasets is
rejected with 409.
- **Dataset** — a geo file uploaded to S3/MinIO (`file_type`: `vector_with_kato |
vector | raster`), belonging to one Category. Carries `code`/`name`/`description`/
`unit` metadata, a user-defined `meta` (JSONB) blob, an `automated` flag, a
@@ -72,10 +77,10 @@ the KATO column and the year columns:
`parse_error`).
3. The client polls `GET /datasets/{id}` until `awaiting_mapping`, then submits
`POST /datasets/{id}/mapping` with the chosen `kato_column` and a
`year_columns` map (each `{column, date}`). Validated against the detected
columns; `status` → `extracting`.
`year_columns` map (each `{column, start_date}`, optionally `end_date` for
a period). Validated against the detected columns; `status` → `extracting`.
4. A second worker job **unpivots** the attribute table into long-format
`dataset_observations` — one row per `(kato_code, date)` with a numeric
`dataset_observations` — one row per `(kato_code, start_date)` with a numeric
`value` (or `value_text` for non-numeric cells); `status` → `ready`. Read
them via `GET /datasets/{id}/observations` (paginated, optional
`?kato_code=`).
@@ -84,12 +89,191 @@ the KATO column and the year columns:
curl -X POST https://dssgis.dwh.kz/datasets/<id>/mapping -H 'Content-Type: application/json' -d '{
"kato_column": "като",
"year_columns": [
{"column": "F_2023", "date": "2023-01-01"},
{"column": "D_2025", "date": "2025-01-01"}
{"column": "F_2023", "start_date": "2023-01-01"},
{"column": "D_2025", "start_date": "2025-01-01"}
]
}'
```
### Dataset periods
A dataset carries an optional period — `start_date` and `end_date`
(`YYYY-MM-DD`, both nullable, `end_date` inclusive and never before
`start_date`) — plus an optional `dg_id` linking it to a *dataset group*: the
"одноимённые" datasets that describe the same thing for different periods. All
three are accepted on `POST /datasets` (form fields), `PUT /datasets/{id}` and
`PATCH /datasets/{id}` (an explicit `null`, or omission on PUT, clears them).
`GET /datasets` filters on them with `?dg_id=`, and with the inclusive bounds
`?start_date_from=`, `?start_date_to=`, `?end_date_from=` and `?end_date_to=`
(each `YYYY-MM-DD`).
### Dataset groups
A **dataset group** (`/dataset-groups`) collects the datasets that describe the
same indicator for different periods — the "одноимённые" datasets. A group has a
slug `code` (unique), a `name` and a `description`, and datasets join it through
their `dg_id`.
**Freshness.** `GET /dataset-groups/{id}/latest` returns the group's freshest
dataset: the greatest `end_date` wins, datasets without an `end_date` come last,
and ties are broken by the newest `created_at`. Narrow the candidates with
`?status=` and `?file_type=` (for example `?status=ready&file_type=raster` to
get the latest published raster). A group with no matching dataset yields 404.
**Shared SLD.** A group can carry one SLD (Styled Layer Descriptor) style file
(`POST`/`GET /dataset-groups/{id}/sld`, multipart `file`, `.sld` or `.xml`),
which acts as the fallback style for every dataset in the group:
`GET /datasets/{id}/sld` returns the dataset's own SLD when it has one, and
otherwise falls back to its group's — 404 only when neither exists. Deleting a
group also removes its stored SLD object.
## Landscape Program (ЛП)
The **Landscape Program** is a DAG of per-pixel raster computations that turns
ecological value (Z), vulnerability (U) and sector suitability (P) class rasters
into an ecological constraint (E), per-sector conflicts (C) and three functional
zoning scenarios (V1–V3). Every input and output of every node is a *dataset
group*; the leaf inputs are the **slots** users upload to, and every result is an
ordinary `raster` dataset (COG + SLD) served by the existing dataset endpoints.
The graph lives in Go (`internal/landscape/graph.go`); the groups and the
`landscape` category are seeded by migration `00011_landscape.sql`. A node code
`XY` reads as `X` = stage, `Y` = function within the stage: a stage starts only
after every node of all lower stages finished, and nodes of one stage run
concurrently.
| Node | Name | Inputs (group codes) | Outputs (group codes) |
|---|---|---|---|
| 00 | z_integrated | `lp-z-biodiversity`, `lp-z-lcr`, `lp-z-soil-protection`, `lp-z-wr` | `lp-z-integrated` |
| 01 | u_integrated | `lp-u-surf-water`, `lp-u-soil`, `lp-u-crop`, `lp-u-veg` | `lp-u-integrated` |
| 02 | z_max | Z1..Z4 as above | `lp-z-max` |
| 03 | u_max | U as above | `lp-u-max` |
| 10 | e_ecological_constraint | `lp-z-integrated`, `lp-u-integrated` | `lp-e` |
| 20 | c_conflicts | `lp-p-irrigation`, `lp-p-snow`, `lp-p-crop`, `lp-p-pasture`, `lp-p-tourism`, `lp-e` | `lp-c-irrigation`, `lp-c-snow`, `lp-c-crop`, `lp-c-pasture`, `lp-c-tourism` |
| 21 | p_max | five P groups | `lp-p-max` |
| 30 | func_zoning | `lp-e`, five P groups (original suitability) | `lp-v1-conservative`, `lp-v2-balanced`, `lp-v3-development` |
| 31 | c_max | five C groups | `lp-c-max` |
Sector order everywhere: 1 irrigation, 2 snow, 3 crop, 4 pasture, 5 tourism.
### Pixel rules
A pixel value is **valid** iff `1 <= v <= 5`; anything else (nodata included) is
invalid. Output rasters are `int16` with nodata `-9999`. `classify(x)` maps a
float32 score to a class: `<= 1.8 → 1`, `<= 2.6 → 2`, `<= 3.4 → 3`, `<= 4.2 → 4`,
`<= 5.0 → 5` (and requires `x >= 1.0`).
| Node | Valid domain (pixel) | Value |
|---|---|---|
| z_integrated | all four Z valid | `classify(0.25*Z1 + 0.25*Z2 + 0.25*Z3 + 0.25*Z4)` |
| u_integrated | water ∧ soil ∧ (crop ∨ veg) | `classify(num/den)` with crop/veg contributing `0.25` each only when valid |
| z_max / u_max | as above | `max` over the valid inputs |
| e_ecological_constraint | Z ∧ U valid | `max(Z, U)` |
| c_conflicts | `P_j` ∧ E valid | `min(P_j, E)` per sector |
| p_max / c_max | at least one input valid | `max` over the valid ones |
| func_zoning | E valid ∧ at least one `P_k` valid | winning sector + 1, else 1 |
Functional zoning takes `S_k = P_k`, the original suitability rasters (the conflict
layers `C_k` are diagnostics and are not zoning inputs). A sector is a candidate when `S_k` and `E`
are valid and the scenario rule holds — V1: `E <= 2 && S >= v1_min_s_e12` or
`E == 3 && S >= v1_min_s_e3`; V2: `E <= 2`, or `E == 3 && S >= v2_min_s_e3`, or
`E == 4 && S >= v2_min_s_e4`; V3: `E <= 4`. With no candidate the pixel is class
1 (regulated); otherwise the candidate with the greatest `S` wins, ties broken by
`tie_priority`, and the class is `sector + 1` (2 irrigation, 3 snow, 4 crop,
5 pasture, 6 tourism).
Inputs are **not** resampled or reprojected: a node whose inputs do not share one
grid (size, geotransform, CRS) fails with a clear message, and so does the run.
### Uploading inputs, running, reading results
```sh
# 1. fill a slot (repeat per input group; see GET /landscape/slots for the codes)
curl -X POST https://dssgis.dwh.kz/landscape/slots/lp-z-biodiversity \
-F file=@z_biodiversity.tif -F name="Z Biodiversity 2025" \
-F start_date=2025-01-01 -F end_date=2025-12-31
# 2. queue a run (202); the worker executes the graph asynchronously
curl -X POST https://dssgis.dwh.kz/landscape/runs \
-H 'Content-Type: application/json' -d '{"from_stage": 0}'
# 3. follow it, then read the layers
curl https://dssgis.dwh.kz/landscape/runs/<run-id>
curl https://dssgis.dwh.kz/landscape/results
```
`GET /landscape/slots` lists every leaf slot with the dataset currently filling
it (any status), so the UI can show what is still missing. The upload is the
regular raster path: `.tif`/`.tiff` with a TIFF magic-byte check, stored in the
slot's group under the `landscape` category, COG conversion enqueued.
**Input resolution.** For each input group a node takes, in order: the run's
`overrides[group]`, else whatever this run already produced for that group, else
the freshest `ready` raster of the group (greatest `end_date`, NULLs last, newest
`created_at` first). A missing input fails the node and the run.
**Incremental recompute.** Each node is fingerprinted over its code, its resolved
inputs and the params that actually affect it. Without `from_stage`, a node whose
fingerprint matches an earlier `succeeded` node — with all of its output datasets
still present and `ready` — is marked `skipped` and its outputs reused. With
`from_stage: N`, every node of stage `N` and above is recomputed regardless;
`from_stage: 0` recomputes everything. `overrides` pins a specific dataset
(existing, `raster`, `ready`) for a group instead of resolving the freshest one.
Run params: `window_size` (256..8192, default 2048), `tie_priority` (a
permutation of 1..5, default `[1,2,3,4,5]`), `v1_min_s_e12`/`v1_min_s_e3`
(defaults 3/4) and `v2_min_s_e3`/`v2_min_s_e4` (defaults 3/5).
**Results.** Every output is a normal dataset in its group, named
`<Output name> <run date>`, `automated = true`, with
`meta = {lp_run_id, lp_node, inputs}` and the min/max period of its inputs.
`GET /landscape/results` returns one entry per output group with its freshest
`ready` dataset, a `cog_url` (`/datasets/{id}.cog`) and, when the group carries
one, an `sld_url` (`/dataset-groups/{id}/sld`). A node failure fails its stage;
later stages are not started and the run ends `failed` with the message. Jobs are
nacked without requeue, so there is no automatic retry — start a new run.
### Memory and parallelism
A run has two nested levels of parallelism: nodes of a stage run concurrently,
and each node processes its windows with its own worker pool. Every worker holds
one window buffer per input **and** per output, so the worker's peak resident set
is roughly
```
nodes × workers-per-node × window_size² × (inputs + outputs) × 8 bytes
```
(the widest node, `20 c_conflicts`, has 6 inputs and 5 outputs). Two environment
variables bound that product, independently of how many cores the host has:
| Variable | Default | Meaning |
| --- | --- | --- |
| `LP_MAX_WORKERS` | `8` | Total window-worker budget shared by all nodes of a run. |
| `LP_MAX_NODES` | `2` | How many graph nodes of one stage run at the same time. |
Each node gets `max(1, LP_MAX_WORKERS / LP_MAX_NODES)` workers, further capped by
`GOMAXPROCS` and by the number of windows, so the total stays at
`LP_MAX_WORKERS`. At the defaults and `window_size: 2048` that is about
8 × 2048² × 11 × 8 B ≈ 2.9 GiB worst case — lower `LP_MAX_WORKERS` (or
`window_size`) on a small host, raise it on a large one.
### Build requirements
The raster math uses [godal](https://github.com/airbusgeo/godal), a **cgo**
binding to GDAL, so the binary must be built with `CGO_ENABLED=1` against GDAL
headers and libraries:
```sh
brew install gdal # macOS; Linux: libgdal-dev / gdal-dev
go build ./... # cgo is on by default for a native build
```
`build/package/Dockerfile` already handles the image: the builder stage adds
`gdal-dev gcc g++ musl-dev pkgconfig` and builds with `CGO_ENABLED=1`, and the
runtime stage installs `gdal` (the shared library godal needs) plus `gdal-tools`
(`gdal_translate`/`gdalinfo` for the COG conversion).
## Getting started
```sh
@@ -117,15 +301,31 @@ server runs it is served at `/openapi.yaml`, with an interactive **Redoc** UI at
| GET | `/categories/{id}` | get |
| PUT | `/categories/{id}` | update |
| DELETE | `/categories/{id}` | delete |
| GET | `/datasets` | paginated list of summaries (`?page=`, `?page_size=`, `?category_id=`) |
| POST | `/datasets` | upload (multipart: `file`, `file_type`, `category_id`, `code`, `name`, `description?`, `unit?`, `meta?` (JSON), `automated?` (bool)) |
| GET | `/datasets` | paginated list of summaries (`?page=`, `?page_size=`, `?category_id=`, `?dg_id=`, `?start_date_from=`, `?start_date_to=`, `?end_date_from=`, `?end_date_to=`) |
| POST | `/datasets` | upload (multipart: `file`, `file_type`, `category_id`, `code`, `name`, `description?`, `unit?`, `meta?` (JSON), `automated?` (bool), `dg_id?`, `start_date?`, `end_date?`) |
| GET | `/datasets/{id}` | full dataset (geometry as GeoJSON, `bbox` for rasters) |
| GET | `/datasets/{id}.geojson` | GeoJSON `FeatureCollection`; plain `vector` returns its geometry as a single feature with the extracted attribute table as top-level properties; `vector_with_kato` always ignores its own geometry and joins the `districts` table by KATO, mapping observations onto each polygon |
| GET | `/datasets/{id}/status` | processing status; long-polls with `?current=<status>` (holds up to `?wait=` secs, default 25, max 60) |
| GET | `/datasets/{id}/download` | download the stored file |
| POST | `/datasets/{id}/mapping` | set KATO column + year→date map (vector_with_kato) |
| POST | `/datasets/{id}/mapping` | set KATO column + year→period map (vector_with_kato) |
| GET | `/datasets/{id}/observations` | paginated unpivoted values (`?kato_code=`, `?page=`, `?page_size=`) |
| POST | `/datasets/{id}/sld` | upload/replace the dataset's SLD (multipart `file`, `.sld`/`.xml`) |
| GET | `/datasets/{id}/sld` | download the dataset's SLD, falling back to its group's |
| DELETE | `/datasets/{id}` | delete (row + object) |
| GET | `/dataset-groups` | paginated list (`?code=`, `?page=`, `?page_size=`) |
| POST | `/dataset-groups` | create (`code`, `name`, `description?`) |
| GET | `/dataset-groups/{id}` | get |
| PUT | `/dataset-groups/{id}` | update |
| DELETE | `/dataset-groups/{id}` | delete (409 while datasets reference it) |
| GET | `/dataset-groups/{id}/latest` | freshest dataset of the group (`?status=`, `?file_type=`) |
| POST | `/dataset-groups/{id}/sld` | upload/replace the group's shared SLD |
| GET | `/dataset-groups/{id}/sld` | download the group's shared SLD |
| GET | `/landscape/slots` | ЛП input slots + the dataset filling each |
| POST | `/landscape/slots/{code}` | upload a GeoTIFF into a slot (multipart: `file`, `name?`, `start_date?`, `end_date?`) |
| GET | `/landscape/runs` | paginated runs, newest first (`?page=`, `?page_size=`) |
| POST | `/landscape/runs` | queue a run (`from_stage?`, `overrides?`, `params?`); 202 |
| GET | `/landscape/runs/{id}` | run with per-node status, inputs, outputs, timings |
| GET | `/landscape/results` | per output group: latest layer, `cog_url`, `sld_url` |
Example upload:
@@ -165,6 +365,17 @@ make lint # golangci-lint (if installed)
CI (`.github/workflows/ci.yml`) runs build, vet, `go test -race`, and golangci-lint
on every push and pull request.
Repository tests in `internal/repository/postgres` run against a real database
when `TEST_DB_URL` points at one with migrations applied, and are skipped
otherwise. They catch Postgres-only failures (parameter type deduction, casts,
constraints) that in-memory stubs cannot:
```sh
docker run -d --name gis-test-pg -e POSTGRES_USER=gis -e POSTGRES_PASSWORD=gis -e POSTGRES_DB=gis -p 5434:5432 postgis/postgis:17-3.5
DB_URL='postgres://gis:gis@127.0.0.1:5434/gis?sslmode=disable' S3_ENDPOINT=x S3_ACCESS_KEY=x S3_SECRET_KEY=x RABBITMQ_URL=amqp://x go run ./cmd/gis migrate up
TEST_DB_URL='postgres://gis:gis@127.0.0.1:5434/gis?sslmode=disable' go test ./internal/repository/postgres/ -v
```
## Adding a feature
Each new domain is one vertical slice mirroring Category/Dataset:

File diff suppressed because it is too large Load Diff

View File

@@ -1,5 +1,12 @@
# Build context is the repo root: docker build -f build/package/Dockerfile .
FROM golang:1.26.1-alpine AS builder
# Go patch release pinned (not the floating 1.26 tag) so the shipped standard
# library is reproducible and includes the security fixes up to go1.26.9;
# keep in step with the `toolchain` line in go.mod.
FROM golang:1.26.9-alpine3.24 AS builder
# godal (GDAL bindings) needs cgo, a C/C++ toolchain and the GDAL headers.
# g++ is required: parts of the binding compile as C++ against GDAL's headers.
RUN apk add --no-cache gdal-dev gcc g++ musl-dev pkgconfig
WORKDIR /src
@@ -7,13 +14,18 @@ COPY go.mod go.sum ./
RUN go mod download
COPY . .
RUN CGO_ENABLED=0 GOOS=linux go build -trimpath -o /out/gis ./cmd/gis
RUN CGO_ENABLED=1 GOOS=linux go build -trimpath -o /out/gis ./cmd/gis
FROM alpine:3.20
# Must stay on the same Alpine release as the builder — hence the pinned
# `golang:1.26.9-alpine3.24` above rather than the floating `-alpine` tag. A
# mismatch ships a different libgdal soname, and the image still builds green
# while the binary dies at startup with "libgdal.so.NN: No such file".
# Bump both lines together.
FROM alpine:3.24
# gdal-tools provides gdal_translate / gdalinfo for raster COG conversion and
# ogr2ogr for extracting vector feature geometry (worker).
RUN apk add --no-cache ca-certificates tzdata gdal-tools \
# gdal: shared library for godal (raster math in the worker).
# gdal-tools: gdal_translate / gdalinfo (COG conversion) and ogr2ogr.
RUN apk add --no-cache ca-certificates tzdata gdal gdal-tools \
&& adduser -D -u 10001 app
WORKDIR /app

View File

@@ -2,6 +2,9 @@
# HTTP server
PORT=8080
# Upper bound for any request body, uploads included (bytes; default 2 GiB).
# Larger requests are rejected with 413. 0 disables the cap.
HTTP_MAX_UPLOAD_BYTES=2147483648
# Postgres
DB_URL=postgres://gis:gis@127.0.0.1:5432/gis?sslmode=disable
@@ -28,3 +31,7 @@ RABBITMQ_QUEUE=gis.events.example
# Host port mapping for the postgres container (docker-compose)
DB_PORT=5432
# Landscape Program parallelism (bounds worker peak memory)
LP_MAX_WORKERS=8
LP_MAX_NODES=2

View File

@@ -1,16 +1,21 @@
# Credentials below default to well-known development values so that
# `docker compose up` works out of the box. They MUST be overridden on any
# shared or reachable host: put strong values for POSTGRES_PASSWORD,
# MINIO_ROOT_USER, MINIO_ROOT_PASSWORD, RABBITMQ_DEFAULT_USER and
# RABBITMQ_DEFAULT_PASS in deployments/.env (not committed).
x-app-common: &app-common
build:
context: ..
dockerfile: build/package/Dockerfile
environment: &app-environment
PORT: 8080
DB_URL: postgres://gis:gis@postgres:5432/gis?sslmode=disable
DB_URL: postgres://gis:${POSTGRES_PASSWORD:-gis}@postgres:5432/gis?sslmode=disable
S3_ENDPOINT: minio:9000
S3_ACCESS_KEY: minioadmin
S3_SECRET_KEY: minioadmin
S3_ACCESS_KEY: ${MINIO_ROOT_USER:-minioadmin}
S3_SECRET_KEY: ${MINIO_ROOT_PASSWORD:-minioadmin}
S3_BUCKET: geofiles
S3_USE_SSL: "false"
RABBITMQ_URL: amqp://guest:guest@rabbitmq:5672/
RABBITMQ_URL: amqp://${RABBITMQ_DEFAULT_USER:-guest}:${RABBITMQ_DEFAULT_PASS:-guest}@rabbitmq:5672/
RABBITMQ_EXCHANGE: gis.events
RABBITMQ_QUEUE: gis.events.example
depends_on:
@@ -47,7 +52,7 @@ services:
image: postgis/postgis:17-3.5
environment:
POSTGRES_USER: gis
POSTGRES_PASSWORD: gis
POSTGRES_PASSWORD: ${POSTGRES_PASSWORD:-gis}
POSTGRES_DB: gis
ports:
- "${DB_PORT:-5432}:5432"
@@ -67,8 +72,8 @@ services:
- "9000:9000"
- "9001:9001"
environment:
MINIO_ROOT_USER: minioadmin
MINIO_ROOT_PASSWORD: minioadmin
MINIO_ROOT_USER: ${MINIO_ROOT_USER:-minioadmin}
MINIO_ROOT_PASSWORD: ${MINIO_ROOT_PASSWORD:-minioadmin}
volumes:
- minio_data:/data
healthcheck:
@@ -84,8 +89,8 @@ services:
- "5672:5672"
- "15672:15672"
environment:
RABBITMQ_DEFAULT_USER: guest
RABBITMQ_DEFAULT_PASS: guest
RABBITMQ_DEFAULT_USER: ${RABBITMQ_DEFAULT_USER:-guest}
RABBITMQ_DEFAULT_PASS: ${RABBITMQ_DEFAULT_PASS:-guest}
volumes:
- rabbitmq_data:/var/lib/rabbitmq
healthcheck:

File diff suppressed because it is too large Load Diff

View File

@@ -0,0 +1,320 @@
# Landscape Program (ЛП) + dataset groups and dates — design
Date: 2026-09-05. Source: `LP_20260902.excalidraw`, `TZ_functional_zoning_go.md`,
`zoning_core.py`, `Landscape.ipynb` (reference numpy implementation).
Decisions taken with the product owner are recorded inline as **Decision**.
## 1. Goals
1. **Dataset versioning.** Datasets can belong to a *dataset group* ("одноимённые"
datasets), carry a `start_date` / `end_date` period, be filtered by those, and the
"freshest" dataset of a group can be fetched.
2. **Landscape Program pipeline.** A DAG of raster computations (Z, U, E, C, P_max,
C_max, functional zoning V1–V3) executed by the worker, with results stored as
ordinary raster datasets (COG + SLD) served by the existing dataset endpoints,
incremental recompute, and a dedicated upload for input rasters.
Non-goals: spatial operations (resampling, reprojection), GUI, changing the
vector_with_kato flow beyond the column rename below.
## 2. Part 1 — Dataset groups and dates
### 2.1 Schema (migration `00010_dataset_groups_and_dates.sql`)
```sql
CREATE TABLE dataset_groups (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
code VARCHAR(255) NOT NULL UNIQUE, -- slug, same rules as categories.code
name VARCHAR(255) NOT NULL,
description TEXT NOT NULL DEFAULT '',
sld_storage_key TEXT,
sld_filename VARCHAR(255),
sld_content_type VARCHAR(255),
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
updated_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
ALTER TABLE datasets ADD COLUMN dg_id UUID REFERENCES dataset_groups(id) ON DELETE RESTRICT;
ALTER TABLE datasets ADD COLUMN start_date DATE;
ALTER TABLE datasets ADD COLUMN end_date DATE;
CREATE INDEX idx_datasets_dg_id_end_date ON datasets (dg_id, end_date DESC NULLS LAST, created_at DESC);
CREATE INDEX idx_datasets_start_date ON datasets (start_date);
CREATE INDEX idx_datasets_end_date ON datasets (end_date);
-- Decision: rename the observation/mapping period columns (breaking API change).
ALTER TABLE dataset_observations RENAME COLUMN date TO start_date;
ALTER TABLE dataset_observations RENAME COLUMN date_to TO end_date;
-- unique index and idx_dataset_observations_kato_date recreated on the new names.
-- datasets.year_columns JSONB: every element's keys "date" -> "start_date",
-- "date_to" -> "end_date" (UPDATE with jsonb_agg over jsonb_array_elements; NULL-safe).
```
Constraint: `end_date >= start_date` when both set (CHECK constraint + service validation).
### 2.2 Domain changes
- `domain.DatasetGroup{ID, Code, Name, Description, SldStorageKey, SldFilename,
SldContentType, CreatedAt, UpdatedAt}`.
- `domain.Dataset` and `domain.DatasetSummary` gain `DatasetGroupID *uuid.UUID`
(`json:"dg_id"`), `StartDate *string` (`json:"start_date"`, `YYYY-MM-DD`),
`EndDate *string` (`json:"end_date"`).
- `domain.Observation`: `Date`→`StartDate` (`json:"start_date"`), `DateTo`→`EndDate`
(`json:"end_date"`). `domain.YearColumn`: same rename. Mapping request body uses
`start_date` (required) / `end_date` (optional). Everything that previously said
`date`/`date_to` in code, OpenAPI and README is renamed. GeoJSON output keeps its
`data` (keyed by start date) and `periods` (start→end) shape.
- `domain.DatasetFilter` gains `DatasetGroupID *uuid.UUID`, `StartDateFrom`,
`StartDateTo`, `EndDateFrom`, `EndDateTo *string` (inclusive bounds).
### 2.3 API
| Method & path | Behaviour |
|---|---|
| `GET /dataset-groups` | paginated list, `?code=` exact filter optional |
| `POST /dataset-groups` | `{code, name, description}`; 201 |
| `GET /dataset-groups/{id}` | group |
| `PUT /dataset-groups/{id}` | replace `{code, name, description}` |
| `DELETE /dataset-groups/{id}` | 204; 409 when datasets reference it; also removes group SLD object |
| `GET /dataset-groups/{id}/latest` | freshest dataset of the group: order by `end_date DESC NULLS LAST, created_at DESC`; optional `?status=`, `?file_type=`; 404 when none |
| `POST /dataset-groups/{id}/sld` | multipart `file` (`.sld`/`.xml`), replaces; mirrors dataset SLD upload |
| `GET /dataset-groups/{id}/sld` | download; 404 when none |
| `GET /datasets` | new filters `dg_id`, `start_date_from`, `start_date_to`, `end_date_from`, `end_date_to` (all `YYYY-MM-DD`, inclusive, any combination) |
| `POST /datasets` | new optional form fields `dg_id`, `start_date`, `end_date` |
| `PUT /datasets/{id}` | body gains `dg_id`, `start_date`, `end_date` (nullable; omitted/null clears) |
| `PATCH /datasets/{id}` | sparse; same three fields, `null` clears |
| `GET /datasets/{id}/sld` | **fallback**: when the dataset has no SLD but its group has one, the group SLD is served |
Validation: `dg_id` must reference an existing group (404/422 as for `category_id`);
dates parse as `2006-01-02`; `end_date` not before `start_date`.
`DatasetSummary` (list responses) includes `dg_id`, `start_date`, `end_date`.
### 2.4 Freshness rule
"Самый свежий" = greatest `end_date`; datasets with `NULL end_date` sort after all
dated ones; ties broken by `created_at DESC`. The same rule is used by the pipeline
to resolve inputs (with `status = ready`, `file_type = raster`).
## 3. Part 2 — Landscape Program
### 3.1 Computation graph
Node code `XY`: `X` = stage, `Y` = function in the stage. A stage starts only when
every node of all lower stages finished. Every input and output of every node is a
**dataset group**; leaf groups are the *slots* users upload to.
| Node | Name | Inputs (group codes) | Outputs (group codes) |
|---|---|---|---|
| 00 | z_integrated | `lp-z-biodiversity`, `lp-z-lcr`, `lp-z-soil-protection`, `lp-z-wr` | `lp-z-integrated` |
| 01 | u_integrated | `lp-u-surf-water`, `lp-u-soil`, `lp-u-crop`, `lp-u-veg` | `lp-u-integrated` |
| 02 | z_max | Z1..Z4 as above | `lp-z-max` |
| 03 | u_max | U as above | `lp-u-max` |
| 10 | e_ecological_constraint | `lp-z-integrated`, `lp-u-integrated` | `lp-e` |
| 20 | c_conflicts | `lp-p-irrigation`, `lp-p-snow`, `lp-p-crop`, `lp-p-pasture`, `lp-p-tourism`, `lp-e` | `lp-c-irrigation`, `lp-c-snow`, `lp-c-crop`, `lp-c-pasture`, `lp-c-tourism` |
| 21 | p_max | five P groups | `lp-p-max` |
| 30 | func_zoning | `lp-e`, five P groups (original suitability) | `lp-v1-conservative`, `lp-v2-balanced`, `lp-v3-development` |
| 31 | c_max | five C groups | `lp-c-max` |
Sector order everywhere: 1 irrigation, 2 snow, 3 crop, 4 pasture, 5 tourism.
**Decision (revised 2026-09-11):** func_zoning takes S_k = P_k, the original suitability
rasters, as in the reference notebook ("Three scenarios" cells) and `zoning_core.py`.
The diagram folder `300-…-Equals_102_202` pointing at the C layers was a drawing error;
C_k = min(P_k, E) are diagnostic conflict layers and never replace P_k in zoning.
Groups and the category `landscape` ("Ландшафтная программа") are seeded by
migration `00011_landscape.sql` (`INSERT … ON CONFLICT (code) DO NOTHING`). The
graph itself lives in Go (`internal/landscape/graph.go`), not in the DB.
### 3.2 Pixel rules (from the reference notebook; must match it bitwise)
Notation: a value is *valid* iff `1 <= v <= 5`; anything else (including nodata) is
invalid. Output nodata is `-9999`. `classify(x)` maps a float32 score to a class:
`x <= 1.8 → 1`, `<= 2.6 → 2`, `<= 3.4 → 3`, `<= 4.2 → 4`, `<= 5.0 → 5` (and requires
`x >= 1.0`); comparisons are done in float32 against float32 constants, exactly as
numpy does with a float32 array.
| Node | Valid domain (pixel) | Value |
|---|---|---|
| z_integrated | all four Z valid | `classify(0.25*Z1 + 0.25*Z2 + 0.25*Z3 + 0.25*Z4)` (float32) |
| u_integrated | water ∧ soil ∧ (crop ∨ veg) | `classify(num/den)`, `num = 0.25*water + 0.25*soil + [0.25*crop if crop valid] + [0.25*veg if veg valid]`, `den = 0.5 + [0.25] + [0.25]` |
| z_max | all four valid | `max(Z1..Z4)` |
| u_max | water ∧ soil ∧ (crop ∨ veg) | `max` over the valid ones of water, soil, crop, veg |
| e | Z ∧ U valid | `max(Z, U)` |
| c_conflicts (5 outputs) | P_j ∧ E valid | `min(P_j, E)` |
| p_max | at least one P valid | `max` over valid P |
| c_max | at least one C valid | `max` over valid C |
| func_zoning (3 outputs) | see TZ §4; S_k = P_k | see below |
Functional zoning (TZ `TZ_functional_zoning_go.md`, reference `zoning_core.py`):
```
eValid = valid(E); base_k = eValid && valid(S_k); analysisValid = eValid && any(base_k)
if !analysisValid → -9999 in all scenarios
candidate_k = base_k && rule_v(E, S_k)
V1: (E <= 2 && S >= v1_min_s_e12) || (E == 3 && S >= v1_min_s_e3) (defaults 3, 4)
V2: (E <= 2) || (E == 3 && S >= v2_min_s_e3) || (E == 4 && S >= v2_min_s_e4) (defaults 3, 5)
V3: (E <= 4)
no candidates → class 1 (regulated); else winners = candidates with S == max(S over candidates);
sector = first of tie_priority (default [1,2,3,4,5]) among winners; class = sector + 1
```
Output classes: 1 regulated, 2 irrigation, 3 snow, 4 crop, 5 pasture, 6 tourism.
### 3.3 Raster I/O abstraction (**Decision:** godal behind an interface)
Package `internal/raster` (existing) gains an engine-agnostic interface so the
implementation can be swapped (godal today; `gdal_calc`, ENVI+CLI or another binding
later) without touching the pipeline:
```go
type Grid struct { Width, Height int; GeoTransform [6]float64; ProjectionWKT string }
type Window struct { X, Y, Width, Height int }
type Reader interface {
Grid() Grid
NoData() (value float64, ok bool)
ReadFloat64(w Window, dst []float64) error // band 1, converted by the driver
Close() error
}
type WriterOptions struct { NoData int16; TileSize int; Compression string /* DEFLATE|ZSTD */ }
type Writer interface {
WriteInt16(w Window, src []int16) error // band 1
Close() error
}
type IO interface {
Open(ctx context.Context, path string) (Reader, error)
Create(ctx context.Context, path string, g Grid, o WriterOptions) (Writer, error)
SameGrid(a, b Grid) bool // size, transform (abs tol 1e-9), CRS (GDAL IsSame)
}
```
`internal/raster/godal.go` implements `IO` with `github.com/airbusgeo/godal` (cgo).
Output GeoTIFF: `int16`, nodata `-9999`, `TILED=YES`, `BLOCKXSIZE=BLOCKYSIZE=512`,
`COMPRESS=DEFLATE`, `PREDICTOR=2`, `BIGTIFF=IF_SAFER`; georeference copied from the
node's first input (for func_zoning: E). Inputs are read as float64 regardless of
their storage type; multi-band inputs use band 1.
The existing `RasterConverter` (gdal_translate / gdalinfo / ogr2ogr CLI) stays as is
and keeps producing the COG for every result dataset.
Build: `CGO_ENABLED=1`; `go.mod` gets `toolchain go1.26.2` (the local SDK is
incomplete) and the godal dependency. Dockerfile: builder `golang:1.26-alpine` +
`gdal-dev gcc musl-dev pkgconfig`; runtime `alpine:3.22` + `gdal gdal-tools`.
### 3.4 Compute engine (`internal/landscape`)
- `graph.go`: slots, nodes, stages, group codes, node-level param subsets.
- `kernels.go`: pure per-window functions over `[]float64` inputs → `[]int16`
outputs, one per node. No I/O, fully unit-tested (TZ §7 edge cases included).
- `compute.go`: `Compute(ctx, io raster.IO, node Node, inputs map[string]string
/*group→local path*/, outDir string, p Params) (map[string]string /*group→path*/, error)`.
Opens inputs, **fails if grids differ** (**Decision:** no resampling), creates
outputs on the first input's grid, iterates windows of `p.WindowSize`
(default 2048) with a worker pool: each worker owns its own `Reader` handles
(GDAL datasets are not goroutine-safe); writes are serialised per output through a
mutex. Window order does not affect results.
- Nodes of one stage run concurrently (`errgroup`); stages run sequentially.
### 3.5 Runs, incremental recompute
Migration `00011_landscape.sql`:
```sql
CREATE TABLE lp_runs (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
status VARCHAR(32) NOT NULL, -- queued | running | succeeded | failed
from_stage SMALLINT, -- NULL = incremental
params JSONB NOT NULL,
overrides JSONB NOT NULL DEFAULT '{}', -- {group_code: dataset_id}
error TEXT,
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
started_at TIMESTAMPTZ,
finished_at TIMESTAMPTZ
);
CREATE TABLE lp_run_nodes (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
run_id UUID NOT NULL REFERENCES lp_runs(id) ON DELETE CASCADE,
node_code VARCHAR(8) NOT NULL,
stage SMALLINT NOT NULL,
status VARCHAR(32) NOT NULL, -- pending | running | skipped | succeeded | failed
fingerprint TEXT NOT NULL,
input_datasets JSONB NOT NULL, -- [{group, dataset_id}]
output_datasets JSONB, -- [{group, dataset_id}]
error TEXT,
started_at TIMESTAMPTZ,
finished_at TIMESTAMPTZ,
UNIQUE (run_id, node_code)
);
CREATE INDEX idx_lp_run_nodes_lookup ON lp_run_nodes (node_code, fingerprint, status);
```
Run execution (worker, queue `gis.landscape.run`, routing key `landscape.run`):
1. Mark run `running`. For each stage in order, for each node concurrently:
2. **Resolve inputs** per input group: `overrides[group]` if present; else the dataset
produced for that group earlier in *this* run; else the freshest `ready` raster of
the group (§2.4). Missing → node fails with a clear message.
3. **Fingerprint** = sha256 of node code, sorted `(group, dataset_id)` pairs, and the
node-relevant params. If `from_stage` is NULL or `stage < from_stage`, and a
previous `succeeded` node row with the same fingerprint exists whose output
datasets all still exist → mark `skipped`, reuse those outputs. Otherwise compute.
4. Download inputs to a temp dir, `Compute`, then for each output create a raster
dataset (§3.6), record `output_datasets`, mark `succeeded`.
5. Any node failure fails the stage; remaining stages are not started; run `failed`
with the error. Otherwise run `succeeded`.
`from_stage = 0` recomputes everything ("триггер на перерасчёт всего").
### 3.6 Result datasets
Every output is stored through a new service method
`DatasetService.CreateRasterFromFile(ctx, CreateRasterInput{CategoryID, GroupID,
Name, Filename, Path, StartDate, EndDate, Meta})`: uploads the file, inserts a
`raster` dataset (`automated = true`, `status = processing`), enqueues the existing
COG conversion. `Name` = `<Output name> <run started_at date>`, `Filename` =
`<group>.tif`, `start_date`/`end_date` = min/max over the node's inputs (NULL if all
NULL), `meta = {"lp_run_id", "lp_node", "inputs": [{group, dataset_id}]}`.
Downstream nodes read the original object (`storage_key`), not the COG, so they never
wait for conversion. SLD for a result comes from its group (dataset SLD fallback, §2.3).
### 3.7 Landscape API (`/landscape`)
| Method & path | Behaviour |
|---|---|
| `GET /landscape/slots` | every leaf slot: `code`, `title`, group id, latest dataset summary (any status) or null |
| `POST /landscape/slots/{code}` | **dedicated input upload**: multipart `file` (`.tif/.tiff`, TIFF magic check), optional `name`, `start_date`, `end_date`. Creates a `raster` dataset in the slot's group under the `landscape` category via the regular upload path (COG conversion enqueued). 201 Dataset. Unknown slot → 404 |
| `POST /landscape/runs` | body `{from_stage?: 0..3, overrides?: {group_code: dataset_id}, params?: {window_size?, tie_priority?, v1_min_s_e12?, v1_min_s_e3?, v2_min_s_e3?, v2_min_s_e4?}}`; validates overrides (existing ready raster), inserts run `queued` + pending node rows, publishes job; 202 with run |
| `GET /landscape/runs` | paginated, newest first |
| `GET /landscape/runs/{id}` | run with nodes (status, inputs, outputs, error, timings) |
| `GET /landscape/results` | per output group: `group`, `title`, `node`, latest `ready` dataset (or null) with `cog_url = /datasets/{id}.cog`, `sld_url` when the group has an SLD |
Params defaults: `window_size 2048`, `tie_priority [1,2,3,4,5]`, `v1 3/4`, `v2 3/5`.
Validation: tie_priority is a permutation of 1..5; thresholds in 1..5; window_size
256..8192.
### 3.8 Error handling
- Grid mismatch, invalid CRS, missing input, GDAL error → node `failed` with message;
run `failed`. No retry (consistent with existing consumers: nack without requeue).
- Non-nodata values outside 1..5 are treated as invalid pixels (TZ §2), not fatal.
- Result dataset creation failure after compute → node failed, temp files removed.
- Temp files always cleaned with `defer`.
### 3.9 Testing
- Kernels: table tests incl. TZ §7 cases (invalid E, all S invalid, E=5, E=4 with
several S=5, all S equal, applicability of crop/veg for U, "any valid" for max nodes).
- `Compute`: tests with an in-memory fake `raster.IO` (10×10 grids), including grid
mismatch failure and window tiling with sizes that do not divide the raster.
- godal implementation: round-trip test on a temp GeoTIFF (runs when GDAL present).
- Services: stub repositories as in `dataset_test.go`; run planning (fingerprint,
skip/reuse, from_stage), input resolution order (override → this run → latest).
- HTTP: handler tests for validation paths.
- Part 1: filter clause building, latest ordering (nulls last), rename in mapping and
observations, SLD fallback.
### 3.10 Out of scope / follow-ups
- Resampling/alignment of inputs (fails by design).
- Continuous (float) intermediate rasters and diagnostic layers from the notebook
(tie counts, component counts, critical flags).
- Real-data acceptance run against the Drive test set (inputs there are not aligned).

23
go.mod
View File

@@ -2,7 +2,10 @@ module gis
go 1.26.1
toolchain go1.26.9
require (
github.com/airbusgeo/godal v0.0.18
github.com/caarlos0/env/v11 v11.4.1
github.com/go-chi/chi/v5 v5.3.0
github.com/go-playground/validator/v10 v10.30.3
@@ -11,10 +14,10 @@ require (
github.com/joho/godotenv v1.5.1
github.com/minio/minio-go/v7 v7.2.0
github.com/pressly/goose/v3 v3.27.1
github.com/rabbitmq/amqp091-go v1.12.0
github.com/rabbitmq/amqp091-go v1.13.0
github.com/spf13/cobra v1.10.2
golang.org/x/sync v0.20.0
golang.org/x/text v0.37.0
golang.org/x/sync v0.23.0
golang.org/x/text v0.42.0
modernc.org/sqlite v1.53.0
)
@@ -28,10 +31,9 @@ require (
github.com/jackc/pgpassfile v1.0.0 // indirect
github.com/jackc/pgservicefile v0.0.0-20240606120523-5a60cdf6a761 // indirect
github.com/jackc/puddle/v2 v2.2.2 // indirect
github.com/klauspost/compress v1.18.6 // indirect
github.com/klauspost/compress v1.18.7 // indirect
github.com/klauspost/cpuid/v2 v2.2.11 // indirect
github.com/klauspost/crc32 v1.3.0 // indirect
github.com/kr/text v0.2.0 // indirect
github.com/leodido/go-urn v1.4.0 // indirect
github.com/mattn/go-isatty v0.0.21 // indirect
github.com/mfridman/interpolate v0.0.2 // indirect
@@ -43,13 +45,16 @@ require (
github.com/rs/xid v1.6.0 // indirect
github.com/sethvargo/go-retry v0.3.0 // indirect
github.com/spf13/pflag v1.0.10 // indirect
github.com/spiffe/go-spiffe/v2 v2.9.0 // indirect
github.com/stretchr/testify v1.12.1 // indirect
github.com/tinylib/msgp v1.6.1 // indirect
github.com/zeebo/xxh3 v1.1.0 // indirect
go.uber.org/multierr v1.11.0 // indirect
go.yaml.in/yaml/v3 v3.0.4 // indirect
golang.org/x/crypto v0.52.0 // indirect
golang.org/x/net v0.54.0 // indirect
golang.org/x/sys v0.45.0 // indirect
go.yaml.in/yaml/v3 v3.0.5 // indirect
golang.org/x/crypto v0.57.0 // indirect
golang.org/x/net v0.60.0 // indirect
golang.org/x/sys v0.48.0 // indirect
google.golang.org/protobuf v1.36.12 // indirect
gopkg.in/ini.v1 v1.67.2 // indirect
modernc.org/libc v1.73.4 // indirect
modernc.org/mathutil v1.7.1 // indirect

142
go.sum
View File

@@ -1,18 +1,58 @@
cel.dev/expr v0.19.2 h1:V354PbqIXr9IQdwy4SYA4xa0HXaWq1BUPAGzugBY5V4=
cel.dev/expr v0.19.2/go.mod h1:MrpN08Q+lEBs+bGYdLxxHkZoUSsCp0nSKTs0nTymJgw=
cloud.google.com/go v0.118.3 h1:jsypSnrE/w4mJysioGdMBg4MiW/hHx/sArFpaBWHdME=
cloud.google.com/go v0.118.3/go.mod h1:Lhs3YLnBlwJ4KA6nuObNMZ/fCbOQBPuWKPoE0Wa/9Vc=
cloud.google.com/go/auth v0.15.0 h1:Ly0u4aA5vG/fsSsxu98qCQBemXtAtJf+95z9HK+cxps=
cloud.google.com/go/auth v0.15.0/go.mod h1:WJDGqZ1o9E9wKIL+IwStfyn/+s59zl4Bi+1KQNVXLZ8=
cloud.google.com/go/auth/oauth2adapt v0.2.7 h1:/Lc7xODdqcEw8IrZ9SvwnlLX6j9FHQM74z6cBk9Rw6M=
cloud.google.com/go/auth/oauth2adapt v0.2.7/go.mod h1:NTbTTzfvPl1Y3V1nPpOgl2w6d/FjO7NNUQaWSox6ZMc=
cloud.google.com/go/compute/metadata v0.6.0 h1:A6hENjEsCDtC1k8byVsgwvVcioamEHvZ4j01OwKxG9I=
cloud.google.com/go/compute/metadata v0.6.0/go.mod h1:FjyFAW1MW0C203CEOMDTu3Dk1FlqW3Rga40jzHL4hfg=
cloud.google.com/go/iam v1.4.1 h1:cFC25Nv+u5BkTR/BT1tXdoF2daiVbZ1RLx2eqfQ9RMM=
cloud.google.com/go/iam v1.4.1/go.mod h1:2vUEJpUG3Q9p2UdsyksaKpDzlwOrnMzS30isdReIcLM=
cloud.google.com/go/monitoring v1.24.0 h1:csSKiCJ+WVRgNkRzzz3BPoGjFhjPY23ZTcaenToJxMM=
cloud.google.com/go/monitoring v1.24.0/go.mod h1:Bd1PRK5bmQBQNnuGwHBfUamAV1ys9049oEPHnn4pcsc=
cloud.google.com/go/storage v1.51.0 h1:ZVZ11zCiD7b3k+cH5lQs/qcNaoSz3U9I0jgwVzqDlCw=
cloud.google.com/go/storage v1.51.0/go.mod h1:YEJfu/Ki3i5oHC/7jyTgsGZwdQ8P9hqMqvpi5kRKGgc=
github.com/GoogleCloudPlatform/opentelemetry-operations-go/detectors/gcp v1.25.0 h1:3c8yed4lgqTt+oTQ+JNMDo+F4xprBf+O/il4ZC0nRLw=
github.com/GoogleCloudPlatform/opentelemetry-operations-go/detectors/gcp v1.25.0/go.mod h1:obipzmGjfSjam60XLwGfqUkJsfiheAl+TUjG+4yzyPM=
github.com/GoogleCloudPlatform/opentelemetry-operations-go/exporter/metric v0.51.0 h1:fYE9p3esPxA/C0rQ0AHhP0drtPXDRhaWiwg1DPqO7IU=
github.com/GoogleCloudPlatform/opentelemetry-operations-go/exporter/metric v0.51.0/go.mod h1:BnBReJLvVYx2CS/UHOgVz2BXKXD9wsQPxZug20nZhd0=
github.com/GoogleCloudPlatform/opentelemetry-operations-go/internal/resourcemapping v0.51.0 h1:6/0iUd0xrnX7qt+mLNRwg5c0PGv8wpE8K90ryANQwMI=
github.com/GoogleCloudPlatform/opentelemetry-operations-go/internal/resourcemapping v0.51.0/go.mod h1:otE2jQekW/PqXk1Awf5lmfokJx4uwuqcj1ab5SpGeW0=
github.com/airbusgeo/errs v0.0.3 h1:+lUcbfm45tPLz/DMUC38QT+y9N60aKrPv4MXqVb7Noc=
github.com/airbusgeo/errs v0.0.3/go.mod h1:/U1fafLfbdxZXtIqdTGeGGzLqnZcnGJXsJ0S7TaP4P0=
github.com/airbusgeo/godal v0.0.18 h1:NQJ9P7OgSs5zVtlWH/tOta7QHYbbtha7Nttf63YM01Q=
github.com/airbusgeo/godal v0.0.18/go.mod h1:Mpn/6exBzEbIx4DaG2rIjCmAiF9Ww4SJN2ZtwVsSwCo=
github.com/airbusgeo/osio v0.1.3 h1:IxNf1co8lxwqJCziV+5qZyZbkT6JYdmR3L0oomNffrk=
github.com/airbusgeo/osio v0.1.3/go.mod h1:d4qlaDSgsMilFoi+/62vwFZStuYIHIDovYzIRjTK+Jw=
github.com/caarlos0/env/v11 v11.4.1 h1:fYwH0sWEsBSMPG7t4e/PEfTFzrWrpjyygXyUnWiSwEw=
github.com/caarlos0/env/v11 v11.4.1/go.mod h1:qupehSf/Y0TUTsxKywqRt/vJjN5nz6vauiYEUUr8P4U=
github.com/cespare/xxhash/v2 v2.3.0 h1:UL815xU9SqsFlibzuggzjXhog7bL6oX9BbNZnL2UFvs=
github.com/cespare/xxhash/v2 v2.3.0/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs=
github.com/cncf/xds/go v0.0.0-20250121191232-2f005788dc42 h1:Om6kYQYDUk5wWbT0t0q6pvyM49i9XZAv9dDrkDA7gjk=
github.com/cncf/xds/go v0.0.0-20250121191232-2f005788dc42/go.mod h1:W+zGtBO5Y1IgJhy4+A9GOqVhqLpfZi+vwmdNXUehLA8=
github.com/cpuguy83/go-md2man/v2 v2.0.6/go.mod h1:oOW0eioCTA6cOiMLiUPZOpcVxMig6NIQQ7OS05n1F4g=
github.com/creack/pty v1.1.9/go.mod h1:oKZEueFk5CKHvIhNR5MUki03XCEU+Q6VDXinZuGJ33E=
github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/dustin/go-humanize v1.0.1 h1:GzkhY7T5VNhEkwH0PVJgjz+fX1rhBrR7pRT3mDkpeCY=
github.com/dustin/go-humanize v1.0.1/go.mod h1:Mu1zIs6XwVuF/gI1OepvI0qD18qycQx+mFykh5fBlto=
github.com/envoyproxy/go-control-plane/envoy v1.32.4 h1:jb83lalDRZSpPWW2Z7Mck/8kXZ5CQAFYVjQcdVIr83A=
github.com/envoyproxy/go-control-plane/envoy v1.32.4/go.mod h1:Gzjc5k8JcJswLjAx1Zm+wSYE20UrLtt7JZMWiWQXQEw=
github.com/envoyproxy/protoc-gen-validate v1.2.1 h1:DEo3O99U8j4hBFwbJfrz9VtgcDfUKS7KJ7spH3d86P8=
github.com/envoyproxy/protoc-gen-validate v1.2.1/go.mod h1:d/C80l/jxXLdfEIhX1W2TmLfsJ31lvEjwamM4DxlWXU=
github.com/felixge/httpsnoop v1.0.4 h1:NFTV2Zj1bL4mc9sqWACXbQFVBBg2W3GPvqp8/ESS2Wg=
github.com/felixge/httpsnoop v1.0.4/go.mod h1:m8KPJKqk1gH5J9DgRY2ASl2lWCfGKXixSwevea8zH2U=
github.com/gabriel-vasile/mimetype v1.4.13 h1:46nXokslUBsAJE/wMsp5gtO500a4F3Nkz9Ufpk2AcUM=
github.com/gabriel-vasile/mimetype v1.4.13/go.mod h1:d+9Oxyo1wTzWdyVUPMmXFvp4F9tea18J8ufA774AB3s=
github.com/go-chi/chi/v5 v5.3.0 h1:halUjDxhshgXHMrao5bB8eNBXo/rnzwr8m5m36glehM=
github.com/go-chi/chi/v5 v5.3.0/go.mod h1:R+tYY2hNuVUUjxoPtqUdgBqevM9s9njzkTLutVsOCto=
github.com/go-jose/go-jose/v4 v4.1.5 h1:RjgjO2LOtWOJKUC5wpwY9LR3B3vwVAz6JS2YHfYU6eA=
github.com/go-jose/go-jose/v4 v4.1.5/go.mod h1:x4oUasVrzR7071A4TnHLGSPpNOm2a21K9Kf04k1rs08=
github.com/go-logr/logr v1.4.3 h1:CjnDlHq8ikf6E492q6eKboGOC0T8CDaOvkHCIg8idEI=
github.com/go-logr/logr v1.4.3/go.mod h1:9T104GzyrTigFIr8wt5mBrctHMim0Nb2HLGrmQ40KvY=
github.com/go-logr/stdr v1.2.2 h1:hSWxHoqTgW2S2qGc0LTAI563KZ5YKYRhT3MFKZMbjag=
github.com/go-logr/stdr v1.2.2/go.mod h1:mMo/vtBO5dYbehREoey6XUKy/eSumjCCveDpRre4VKE=
github.com/go-playground/assert/v2 v2.2.0 h1:JvknZsQTYeFEAhQwI4qEt9cyV5ONwRHC+lYKSsYSR8s=
github.com/go-playground/assert/v2 v2.2.0/go.mod h1:VDjEfimB/XKnb+ZQfWdccd7VUvScMdVu0Titje2rxJ4=
github.com/go-playground/locales v0.14.1 h1:EWaQ/wswjilfKLTECiXz7Rh+3BjFhfDFKv/oXslEjJA=
@@ -23,8 +63,16 @@ github.com/go-playground/validator/v10 v10.30.3 h1:4MU6YkEwx7GbcPJOZxrtbu+QfF3pJ
github.com/go-playground/validator/v10 v10.30.3/go.mod h1:4Axh7oCNGcoGkqLoE4YWt6n20mcEIsPRlB7vPk3lpyc=
github.com/google/pprof v0.0.0-20250317173921-a4b03ec1a45e h1:ijClszYn+mADRFY17kjQEVQ1XRhq2/JR1M3sGqeJoxs=
github.com/google/pprof v0.0.0-20250317173921-a4b03ec1a45e/go.mod h1:boTsfXsheKC2y+lKOCMpSfarhxDeIzfZG1jqGcPl3cA=
github.com/google/s2a-go v0.1.9 h1:LGD7gtMgezd8a/Xak7mEWL0PjoTQFvpRudN895yqKW0=
github.com/google/s2a-go v0.1.9/go.mod h1:YA0Ei2ZQL3acow2O62kdp9UlnvMmU7kA6Eutn0dXayM=
github.com/google/uuid v1.6.0 h1:NIvaJDMOsjHA8n1jAhLSgzrAzy1Hgr+hNrb57e+94F0=
github.com/google/uuid v1.6.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
github.com/googleapis/enterprise-certificate-proxy v0.3.5 h1:VgzTY2jogw3xt39CusEnFJWm7rlsq5yL5q9XdLOuP5g=
github.com/googleapis/enterprise-certificate-proxy v0.3.5/go.mod h1:MkHOF77EYAE7qfSuSS9PU6g4Nt4e11cnsDUowfwewLA=
github.com/googleapis/gax-go/v2 v2.14.1 h1:hb0FFeiPaQskmvakKu5EbCbpntQn48jyHuvrkurSS/Q=
github.com/googleapis/gax-go/v2 v2.14.1/go.mod h1:Hb/NubMaVM88SrNkvl8X/o8XWwDJEPqouaLeN2IUxoA=
github.com/hashicorp/golang-lru v1.0.2 h1:dV3g9Z/unq5DpblPpw+Oqcv4dU/1omnb4Ok8iPY6p1c=
github.com/hashicorp/golang-lru v1.0.2/go.mod h1:iADmTwqILo4mZ8BN3D2Q6+9jd8WM5uGBxy+E8yxSoD4=
github.com/hashicorp/golang-lru/v2 v2.0.7 h1:a+bsQ5rvGLjzHuww6tVxozPZFVghXaHOwFs4luLUK2k=
github.com/hashicorp/golang-lru/v2 v2.0.7/go.mod h1:QeFd9opnmA6QUJc5vARoKUSoFhyfM2/ZepoAG6RGpeM=
github.com/inconshreveable/mousetrap v1.1.0 h1:wN+x4NVGpMsO7ErUn/mUI3vEoE6Jt13X2s0bqwp9tc8=
@@ -39,17 +87,13 @@ github.com/jackc/puddle/v2 v2.2.2 h1:PR8nw+E/1w0GLuRFSmiioY6UooMp6KJv0/61nB7icHo
github.com/jackc/puddle/v2 v2.2.2/go.mod h1:vriiEXHvEE654aYKXXjOvZM39qJ0q+azkZFrfEOc3H4=
github.com/joho/godotenv v1.5.1 h1:7eLL/+HRGLY0ldzfGMeQkb7vMd0as4CfYvUVzLqw0N0=
github.com/joho/godotenv v1.5.1/go.mod h1:f4LDr5Voq0i2e/R5DDNOoa2zzDfwtkZa6DnEwAbqwq4=
github.com/klauspost/compress v1.18.6 h1:2jupLlAwFm95+YDR+NwD2MEfFO9d4z4Prjl1XXDjuao=
github.com/klauspost/compress v1.18.6/go.mod h1:cwPg85FWrGar70rWktvGQj8/hthj3wpl0PGDogxkrSQ=
github.com/klauspost/compress v1.18.7 h1:aUyZsS4kH3QTKurYhAOwAHxllVPnOthb3vPfnF1Ehjw=
github.com/klauspost/compress v1.18.7/go.mod h1:cwPg85FWrGar70rWktvGQj8/hthj3wpl0PGDogxkrSQ=
github.com/klauspost/cpuid/v2 v2.0.1/go.mod h1:FInQzS24/EEf25PyTYn52gqo7WaD8xa0213Md/qVLRg=
github.com/klauspost/cpuid/v2 v2.2.11 h1:0OwqZRYI2rFrjS4kvkDnqJkKHdHaRnCm68/DY4OxRzU=
github.com/klauspost/cpuid/v2 v2.2.11/go.mod h1:hqwkgyIinND0mEev00jJYCxPNVRVXFQeu1XKlok6oO0=
github.com/klauspost/crc32 v1.3.0 h1:sSmTt3gUt81RP655XGZPElI0PelVTZ6YwCRnPSupoFM=
github.com/klauspost/crc32 v1.3.0/go.mod h1:D7kQaZhnkX/Y0tstFGf8VUzv2UofNGqCjnC3zdHB0Hw=
github.com/kr/pretty v0.3.0 h1:WgNl7dwNpEZ6jJ9k1snq4pZsg7DOEN8hP9Xw0Tsjwk0=
github.com/kr/pretty v0.3.0/go.mod h1:640gp4NfQd8pI5XOwp5fnNeVWj67G7CFk/SaSQn7NBk=
github.com/kr/text v0.2.0 h1:5Nx0Ya0ZqY2ygV366QzturHI13Jq95ApcVaJBhpS+AY=
github.com/kr/text v0.2.0/go.mod h1:eLer722TekiGuMkidMxC/pM04lWEeraHUUmBw8l2grE=
github.com/leodido/go-urn v1.4.0 h1:WT9HwE9SGECu3lg4d/dIA+jxlljEa1/ffXKmRjqdmIQ=
github.com/leodido/go-urn v1.4.0/go.mod h1:bvxc+MVxLKB4z00jd1z+Dvzr47oO32F/QSNjSBOlFxI=
github.com/mattn/go-isatty v0.0.21 h1:xYae+lCNBP7QuW4PUnNG61ffM4hVIfm+zUzDuSzYLGs=
@@ -66,16 +110,15 @@ github.com/ncruces/go-strftime v1.0.0 h1:HMFp8mLCTPp341M/ZnA4qaf7ZlsbTc+miZjCLOF
github.com/ncruces/go-strftime v1.0.0/go.mod h1:Fwc5htZGVVkseilnfgOVb9mKy6w1naJmn9CehxcKcls=
github.com/philhofer/fwd v1.2.0 h1:e6DnBTl7vGY+Gz322/ASL4Gyp1FspeMvx1RNDoToZuM=
github.com/philhofer/fwd v1.2.0/go.mod h1:RqIHx9QI14HlwKwm98g9Re5prTQ6LdeRQn+gXJFxsJM=
github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
github.com/planetscale/vtprotobuf v0.6.1-0.20240319094008-0393e58bdf10 h1:GFCKgmp0tecUJ0sJuv4pzYCqS9+RGSn52M3FUwPs+uo=
github.com/planetscale/vtprotobuf v0.6.1-0.20240319094008-0393e58bdf10/go.mod h1:t/avpk3KcrXxUnYOhZhMXJlSEyie6gQbtLq5NM3loB8=
github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
github.com/pressly/goose/v3 v3.27.1 h1:6uEvcprBybDmW4hcz3gYujhARhye+GoWKhEWyzD5sh4=
github.com/pressly/goose/v3 v3.27.1/go.mod h1:maruOxsPnIG2yHHyo8UqKWXYKFcH7Q76csUV7+7KYoM=
github.com/rabbitmq/amqp091-go v1.12.0 h1:V0v14Iqfs+MwHWihJt/nGS5Ulu0vw572b2Co3mwunkI=
github.com/rabbitmq/amqp091-go v1.12.0/go.mod h1:Hy4jKW5kQART1u+JkDTF9YYOQUHXqMuhrgxOEeS7G4o=
github.com/rabbitmq/amqp091-go v1.13.0 h1:L8NA1WtF76C6KA3LAoufjfLgbist/If1UQYcsOjtxXA=
github.com/rabbitmq/amqp091-go v1.13.0/go.mod h1:Hy4jKW5kQART1u+JkDTF9YYOQUHXqMuhrgxOEeS7G4o=
github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec h1:W09IVJc94icq4NjY3clb7Lk8O1qJ8BdBEF8z0ibU0rE=
github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec/go.mod h1:qqbHyh8v60DhA7CoWK5oRCqLrMHRGoxYCSS9EjAz6Eo=
github.com/rogpeppe/go-internal v1.14.1 h1:UQB4HGPB6osV0SQTLymcB4TgvyWu6ZyliaW0tI/otEQ=
github.com/rogpeppe/go-internal v1.14.1/go.mod h1:MaRKkUm5W0goXpeCfT7UZI6fk/L7L7so1lCWt35ZSgc=
github.com/rs/xid v1.6.0 h1:fV591PaemRlL6JfRxGDEPl69wICngIQ3shQtzfy2gxU=
github.com/rs/xid v1.6.0/go.mod h1:7XoLgs4eV+QndskICGsho+ADou8ySMSjJKDIan90Nz0=
github.com/russross/blackfriday/v2 v2.1.0/go.mod h1:+Rmxgy9KzJVeS9/2gXHxylqXiyQDYRxCVz55jmeOWTM=
@@ -86,6 +129,8 @@ github.com/spf13/cobra v1.10.2/go.mod h1:7C1pvHqHw5A4vrJfjNwvOdzYu0Gml16OCs2GRiT
github.com/spf13/pflag v1.0.9/go.mod h1:McXfInJRrz4CZXVZOBLb0bTZqETkiAhM9Iw0y3An2Bg=
github.com/spf13/pflag v1.0.10 h1:4EBh2KAYBwaONj6b2Ye1GiHfwjqyROoF4RwYO+vPwFk=
github.com/spf13/pflag v1.0.10/go.mod h1:McXfInJRrz4CZXVZOBLb0bTZqETkiAhM9Iw0y3An2Bg=
github.com/spiffe/go-spiffe/v2 v2.9.0 h1:g5qTg3qIP1HXV+GzuyEfSJFVxUCpEP5Ui1OINVJHPB4=
github.com/spiffe/go-spiffe/v2 v2.9.0/go.mod h1:w2CLWKLMTX/PPYUEUPv3ltH0RXsw5S8suwNF46w9/Aw=
github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
github.com/stretchr/objx v0.4.0/go.mod h1:YvHI0jy2hoMjB+UWwv71VJQ9isScKT/TqJzVSSt89Yw=
github.com/stretchr/objx v0.5.0/go.mod h1:Yh+to48EsGEfYuaHDzXPcE3xhTkx73EhmCGUpEOglKo=
@@ -95,41 +140,74 @@ github.com/stretchr/testify v1.7.0/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/
github.com/stretchr/testify v1.7.1/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
github.com/stretchr/testify v1.8.0/go.mod h1:yNjHg4UonilssWZ8iaSj1OCr/vHnekPRkoO+kdMU+MU=
github.com/stretchr/testify v1.8.4/go.mod h1:sz/lmYIOXD/1dqDmKjjqLyZ2RngseejIcXlSw2iwfAo=
github.com/stretchr/testify v1.11.1 h1:7s2iGBzp5EwR7/aIZr8ao5+dra3wiQyKjjFuvgVKu7U=
github.com/stretchr/testify v1.11.1/go.mod h1:wZwfW3scLgRK+23gO65QZefKpKQRnfz6sD981Nm4B6U=
github.com/stretchr/testify v1.12.1 h1:EuwCh5fleGS7H32xRwO3wRGT7DxrDhLAT6FF8MpWDWE=
github.com/stretchr/testify v1.12.1/go.mod h1:MDEgiDPPsNp5cuIrHPPCyornHKgEVbtFUmoNlxoYthg=
github.com/tinylib/msgp v1.6.1 h1:ESRv8eL3u+DNHUoSAAQRE50Hm162zqAnBoGv9PzScPY=
github.com/tinylib/msgp v1.6.1/go.mod h1:RSp0LW9oSxFut3KzESt5Voq4GVWyS+PSulT77roAqEA=
github.com/zeebo/assert v1.3.0 h1:g7C04CbJuIDKNPFHmsk4hwZDO5O+kntRxzaUoNXj+IQ=
github.com/zeebo/assert v1.3.0/go.mod h1:Pq9JiuJQpG8JLJdtkwrJESF0Foym2/D9XMU5ciN/wJ0=
github.com/zeebo/xxh3 v1.1.0 h1:s7DLGDK45Dyfg7++yxI0khrfwq9661w9EN78eP/UZVs=
github.com/zeebo/xxh3 v1.1.0/go.mod h1:IisAie1LELR4xhVinxWS5+zf1lA4p0MW4T+w+W07F5s=
go.opentelemetry.io/auto/sdk v1.2.1 h1:jXsnJ4Lmnqd11kwkBV2LgLoFMZKizbCi5fNZ/ipaZ64=
go.opentelemetry.io/auto/sdk v1.2.1/go.mod h1:KRTj+aOaElaLi+wW1kO/DZRXwkF4C5xPbEe3ZiIhN7Y=
go.opentelemetry.io/contrib/detectors/gcp v1.34.0 h1:JRxssobiPg23otYU5SbWtQC//snGVIM3Tx6QRzlQBao=
go.opentelemetry.io/contrib/detectors/gcp v1.34.0/go.mod h1:cV4BMFcscUR/ckqLkbfQmF0PRsq8w/lMGzdbCSveBHo=
go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc v0.59.0 h1:rgMkmiGfix9vFJDcDi1PK8WEQP4FLQwLDfhp5ZLpFeE=
go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc v0.59.0/go.mod h1:ijPqXp5P6IRRByFVVg9DY8P5HkxkHE5ARIa+86aXPf4=
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.68.0 h1:CqXxU8VOmDefoh0+ztfGaymYbhdB/tT3zs79QaZTNGY=
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.68.0/go.mod h1:BuhAPThV8PBHBvg8ZzZ/Ok3idOdhWIodywz2xEcRbJo=
go.opentelemetry.io/otel v1.43.0 h1:mYIM03dnh5zfN7HautFE4ieIig9amkNANT+xcVxAj9I=
go.opentelemetry.io/otel v1.43.0/go.mod h1:JuG+u74mvjvcm8vj8pI5XiHy1zDeoCS2LB1spIq7Ay0=
go.opentelemetry.io/otel/metric v1.43.0 h1:d7638QeInOnuwOONPp4JAOGfbCEpYb+K6DVWvdxGzgM=
go.opentelemetry.io/otel/metric v1.43.0/go.mod h1:RDnPtIxvqlgO8GRW18W6Z/4P462ldprJtfxHxyKd2PY=
go.opentelemetry.io/otel/sdk v1.34.0 h1:95zS4k/2GOy069d321O8jWgYsW3MzVV+KuSPKp7Wr1A=
go.opentelemetry.io/otel/sdk v1.34.0/go.mod h1:0e/pNiaMAqaykJGKbi+tSjWfNNHMTxoC9qANsCzbyxU=
go.opentelemetry.io/otel/sdk/metric v1.34.0 h1:5CeK9ujjbFVL5c1PhLuStg1wxA7vQv7ce1EK0Gyvahk=
go.opentelemetry.io/otel/sdk/metric v1.34.0/go.mod h1:jQ/r8Ze28zRKoNRdkjCZxfs6YvBTG1+YIqyFVFYec5w=
go.opentelemetry.io/otel/trace v1.43.0 h1:BkNrHpup+4k4w+ZZ86CZoHHEkohws8AY+WTX09nk+3A=
go.opentelemetry.io/otel/trace v1.43.0/go.mod h1:/QJhyVBUUswCphDVxq+8mld+AvhXZLhe+8WVFxiFff0=
go.uber.org/goleak v1.3.0 h1:2K3zAYmnTNqV73imy9J1T3WC+gmCePx2hEGkimedGto=
go.uber.org/goleak v1.3.0/go.mod h1:CoHD4mav9JJNrW/WLlf7HGZPjdw8EucARQHekz1X6bE=
go.uber.org/multierr v1.11.0 h1:blXXJkSxSSfBVBlC76pxqeO+LN3aDfLQo+309xJstO0=
go.uber.org/multierr v1.11.0/go.mod h1:20+QtiLqy0Nd6FdQB9TLXag12DsQkrbs3htMFfDN80Y=
go.yaml.in/yaml/v3 v3.0.4 h1:tfq32ie2Jv2UxXFdLJdh3jXuOzWiL1fo0bu/FbuKpbc=
go.yaml.in/yaml/v3 v3.0.4/go.mod h1:DhzuOOF2ATzADvBadXxruRBLzYTpT36CKvDb3+aBEFg=
golang.org/x/crypto v0.52.0 h1:RMs7fP2rXdep0CftQlK8Uf+kibLm7qkCcradZWYz988=
golang.org/x/crypto v0.52.0/go.mod h1:1QgfPxDqh0T2M/elOJtp9RvuR95kVjir0e6/BvEmGbc=
golang.org/x/mod v0.36.0 h1:JJjpVx6myfUsUdAzZuOSTTmRE0PfZeNWzzvKrP7amb4=
golang.org/x/mod v0.36.0/go.mod h1:moc6ELqsWcOw5Ef3xVprK5ul/MvtVvkIXLziUOICjUQ=
golang.org/x/net v0.54.0 h1:2zJIZAxAHV/OHCDTCOHAYehQzLfSXuf/5SoL/Dv6w/w=
golang.org/x/net v0.54.0/go.mod h1:Sj4oj8jK6XmHpBZU/zWHw3BV3abl4Kvi+Ut7cQcY+cQ=
golang.org/x/sync v0.20.0 h1:e0PTpb7pjO8GAtTs2dQ6jYa5BWYlMuX047Dco/pItO4=
golang.org/x/sync v0.20.0/go.mod h1:9xrNwdLfx4jkKbNva9FpL6vEN7evnE43NNNJQ2LF3+0=
golang.org/x/sys v0.45.0 h1:dO4czNzziLiiXplLQgBCEpCvXQ3dnkn0SdaZSYdQ+FY=
golang.org/x/sys v0.45.0/go.mod h1:4GL1E5IUh+htKOUEOaiffhrAeqysfVGipDYzABqnCmw=
golang.org/x/text v0.37.0 h1:Cqjiwd9eSg8e0QAkyCaQTNHFIIzWtidPahFWR83rTrc=
golang.org/x/text v0.37.0/go.mod h1:a5sjxXGs9hsn/AJVwuElvCAo9v8QYLzvavO5z2PiM38=
golang.org/x/tools v0.45.0 h1:18qN3FAooORvApf5XjCXgsuayZOEtXf6JK18I3+ONa8=
golang.org/x/tools v0.45.0/go.mod h1:LuUGqqaXcXMEFEruIVJVm5mgDD8vww/z/SR1gQ4uE/0=
go.yaml.in/yaml/v3 v3.0.5 h1:N6y/pJk8buWs9NY5ERU2HSMfm+IuD/OtfdAnq6kESPw=
go.yaml.in/yaml/v3 v3.0.5/go.mod h1:HVTZu1O7/Vkt2N+BFy8Zza+lnLsABggaTM2ZpNIGuKg=
golang.org/x/crypto v0.57.0 h1:3ZVCjf8Ggz7zneR/EHRVx68Ctf+2pmIMP2UFhh9cC6M=
golang.org/x/crypto v0.57.0/go.mod h1:Fdz0i5U6CoizGwLda9DttjSk6qlZo25zYNtR+ycvuZA=
golang.org/x/mod v0.41.0 h1:qJmnOUb4YB+FsEuM3HcWucdZASCPGhsX6uljO6pog0c=
golang.org/x/mod v0.41.0/go.mod h1:Ek9pY8RKWXwsWvd3rQiHYtMqkjSUV+s1Rj7j4H5Ur6o=
golang.org/x/net v0.60.0 h1:79p50tfZlm0J9YfoDsSi639qSXNGVwEzOPLCxM2FsYU=
golang.org/x/net v0.60.0/go.mod h1:2DA/G1UfVbCpQPeWTmMPGY7Cs2PkBkwu743bVX5PIVg=
golang.org/x/oauth2 v0.28.0 h1:CrgCKl8PPAVtLnU3c+EDw6x11699EWlsDeWNWKdIOkc=
golang.org/x/oauth2 v0.28.0/go.mod h1:onh5ek6nERTohokkhCD/y2cV4Do3fxFHFuAejCkRWT8=
golang.org/x/sync v0.23.0 h1:KameEIfc1IkluZyXWLn39Wd4tURc6GbCiISGiZm2bQk=
golang.org/x/sync v0.23.0/go.mod h1:sUUOizhqBxiL6pEWpqNLUiaJn1ShEbZ6BBqskPbjZm0=
golang.org/x/sys v0.48.0 h1:bbX/i/6MgT9BVLM9RT1thmxL04yeTAhbEz4SyadbXoo=
golang.org/x/sys v0.48.0/go.mod h1:hNLxWAXmnKAxqDtdwIYC4bM9oQPEecfsnNMuSxOs3og=
golang.org/x/text v0.42.0 h1:JbOZXgfeCPU9gacVtYliJqOhD+zhrEqK4LfdpmlUZqI=
golang.org/x/text v0.42.0/go.mod h1:ojzP1Z+2QtioaF8DTtO8K5q7JWVVYwZKenzujK0Zd0E=
golang.org/x/time v0.10.0 h1:3usCWA8tQn0L8+hFJQNgzpWbd89begxN66o1Ojdn5L4=
golang.org/x/time v0.10.0/go.mod h1:3BpzKBy/shNhVucY/MWOyx10tF3SFh9QdLuxbVysPQM=
golang.org/x/tools v0.49.0 h1:3NI7VXzL9+1WZD52Dx2ttoPwD5DWrFGpl9mFZDlmisI=
golang.org/x/tools v0.49.0/go.mod h1:SJNXV9DBKT0UbdttsQjbfJlAE/q+y36++zo3uL3N0Oo=
google.golang.org/api v0.224.0 h1:Ir4UPtDsNiwIOHdExr3fAj4xZ42QjK7uQte3lORLJwU=
google.golang.org/api v0.224.0/go.mod h1:3V39my2xAGkodXy0vEqcEtkqgw2GtrFL5WuBZlCTCOQ=
google.golang.org/genproto v0.0.0-20250303144028-a0af3efb3deb h1:ITgPrl429bc6+2ZraNSzMDk3I95nmQln2fuPstKwFDE=
google.golang.org/genproto v0.0.0-20250303144028-a0af3efb3deb/go.mod h1:sAo5UzpjUwgFBCzupwhcLcxHVDK7vG5IqI30YnwX2eE=
google.golang.org/genproto/googleapis/api v0.0.0-20250303144028-a0af3efb3deb h1:p31xT4yrYrSM/G4Sn2+TNUkVhFCbG9y8itM2S6Th950=
google.golang.org/genproto/googleapis/api v0.0.0-20250303144028-a0af3efb3deb/go.mod h1:jbe3Bkdp+Dh2IrslsFCklNhweNTBgSYanP1UXhJDhKg=
google.golang.org/genproto/googleapis/rpc v0.0.0-20260420184626-e10c466a9529 h1:XF8+t6QQiS0o9ArVan/HW8Q7cycNPGsJf6GA2nXxYAg=
google.golang.org/genproto/googleapis/rpc v0.0.0-20260420184626-e10c466a9529/go.mod h1:4Hqkh8ycfw05ld/3BWL7rJOSfebL2Q+DVDeRgYgxUU8=
google.golang.org/grpc v1.80.0 h1:Xr6m2WmWZLETvUNvIUmeD5OAagMw3FiKmMlTdViWsHM=
google.golang.org/grpc v1.80.0/go.mod h1:ho/dLnxwi3EDJA4Zghp7k2Ec1+c2jqup0bFkw07bwF4=
google.golang.org/protobuf v1.36.12 h1:pJOKDDOyeXErUroCihFAd5LQuwXBSpVnKGrj5o/fwxc=
google.golang.org/protobuf v1.36.12/go.mod h1:HTf+CrKn2C3g5S8VImy6tdcUvCska2kB7j23XfzDpco=
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c h1:Hei/4ADfdWqJk1ZMxUNpqntNwaWcugrBjAiHlqqRiVk=
gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c/go.mod h1:JHkPIbrfpd72SG/EVd6muEfDQjcINNoR0C8j2r3qZ4Q=
gopkg.in/ini.v1 v1.67.2 h1:JtOSMb9OuaCZKr7h5D/h6iii14sK0hLbplTc6frx4Ss=
gopkg.in/ini.v1 v1.67.2/go.mod h1:x/cyOwCgZqOkJoDIJ3c1KNHMo10+nLGAhh+kn3Zizss=
gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
modernc.org/cc/v4 v4.28.4 h1:Hd/4Es+MBj+/7hSdZaisNyu6bv3V0Dp2MdllyfqaH+c=
modernc.org/cc/v4 v4.28.4/go.mod h1:OnovgIhbbMXMu1aISnJ0wvVD1KnW+cAUJkIrAWh+kVI=

View File

@@ -38,6 +38,8 @@ type App struct {
publisher *rabbitmq.Publisher
categories *service.CategoryService
datasets *service.DatasetService
groups *service.DatasetGroupService
landscape *service.LandscapeService
eventRepo *postgres.EventRepository
}
@@ -68,10 +70,16 @@ func New(ctx context.Context) (*App, error) {
categoryRepo := postgres.NewCategoryRepository(pool)
datasetRepo := postgres.NewDatasetRepository(pool)
groupRepo := postgres.NewDatasetGroupRepository(pool)
eventRepo := postgres.NewEventRepository(pool)
lpRepo := postgres.NewLandscapeRepository(pool)
publisher := rabbitmq.NewPublisher(rabbit)
jobPublisher := rabbitmq.NewDatasetJobPublisher(publisher)
lpJobs := rabbitmq.NewLandscapeJobPublisher(publisher)
datasetsSvc := service.NewDatasetService(datasetRepo, store, categoryRepo, groupRepo, jobPublisher, parser.Columns, parser.Rows, raster.NewGDALConverter())
return &App{
Cfg: cfg,
@@ -81,30 +89,47 @@ func New(ctx context.Context) (*App, error) {
rabbit: rabbit,
publisher: publisher,
categories: service.NewCategoryService(categoryRepo),
datasets: service.NewDatasetService(datasetRepo, store, categoryRepo, jobPublisher, parser.Columns, parser.Rows, raster.NewGDALConverter()),
eventRepo: eventRepo,
datasets: datasetsSvc,
groups: service.NewDatasetGroupService(groupRepo, datasetRepo, store),
// A nil ComputeFunc selects landscape.ComputeWithWorkers, so each node
// gets the per-node share of the configured worker budget.
landscape: service.NewLandscapeService(
lpRepo, datasetRepo, groupRepo, categoryRepo, datasetsSvc, store,
lpJobs, raster.NewGodalIO(), nil, log,
service.LandscapeOptions{
MaxWorkers: cfg.Landscape.MaxWorkers,
MaxConcurrentNodes: cfg.Landscape.MaxConcurrentNodes,
},
),
eventRepo: eventRepo,
}, nil
}
// Handler builds the HTTP handler with all routes and readiness checks wired.
func (a *App) Handler() stdhttp.Handler {
validate := validator.New(validator.WithRequiredStructEnabled())
validate.RegisterValidation("slug", func(fl validator.FieldLevel) bool {
if err := validate.RegisterValidation("slug", func(fl validator.FieldLevel) bool {
return domain.ValidSlug(fl.Field().String())
})
}); err != nil {
// Only fails for an empty tag or nil func: a programming error.
panic(fmt.Sprintf("register slug validation: %v", err))
}
health := transporthttp.NewHealthHandler(map[string]transporthttp.ReadinessCheck{
"postgres": func(ctx context.Context) error { return a.pool.Ping(ctx) },
"s3": func(ctx context.Context) error { return a.store.Ping(ctx) },
"rabbitmq": func(_ context.Context) error { return a.rabbit.Ping() },
})
}, a.Log)
return transporthttp.NewRouter(transporthttp.RouterDeps{
Logger: a.Log,
Health: health,
Categories: transporthttp.NewCategoryHandler(a.categories, validate),
Datasets: transporthttp.NewDatasetHandler(a.datasets, validate),
OpenAPISpec: api.Spec,
Logger: a.Log,
Health: health,
Categories: transporthttp.NewCategoryHandler(a.categories, validate),
Datasets: transporthttp.NewDatasetHandler(a.datasets, validate),
DatasetGroups: transporthttp.NewDatasetGroupHandler(a.groups, validate),
Landscape: transporthttp.NewLandscapeHandler(a.landscape, validate),
OpenAPISpec: api.Spec,
MaxBodyBytes: a.Cfg.HTTP.MaxUploadBytes,
})
}
@@ -120,10 +145,20 @@ func (a *App) Consumers() []*rabbitmq.Consumer {
a.PropertiesConsumer(),
a.ExtractConsumer(),
a.ConvertConsumer(),
a.LandscapeConsumer(),
a.ExampleConsumer(),
}
}
// LandscapeConsumer builds the Landscape Program run consumer.
func (a *App) LandscapeConsumer() *rabbitmq.Consumer {
handler := rabbitmq.NewLandscapeRunHandler(a.landscape, a.Log)
return rabbitmq.NewConsumer(
a.rabbit, rabbitmq.LandscapeRunQueue, rabbitmq.LandscapeRunRoutingKey,
"gis-landscape-runner", handler, a.Log,
)
}
// PropertiesConsumer builds the plain-vector properties-extraction consumer.
func (a *App) PropertiesConsumer() *rabbitmq.Consumer {
handler := rabbitmq.NewPropertiesHandler(a.datasets, a.Log)
@@ -178,11 +213,12 @@ func (a *App) ReprocessDataset(ctx context.Context, id uuid.UUID) (domain.Datase
return a.datasets.Reprocess(ctx, id)
}
// ReprocessAllDatasets re-enqueues the processing job for every dataset,
// restarting each one's asynchronous pipeline. It returns how many jobs were
// enqueued and a per-dataset map of any failures.
func (a *App) ReprocessAllDatasets(ctx context.Context) (int, map[uuid.UUID]error, error) {
return a.datasets.ReprocessAll(ctx)
// ReprocessAllDatasets re-enqueues the processing job for every dataset
// matching fileType (nil matches every file type), restarting each one's
// asynchronous pipeline. It returns how many jobs were enqueued and a
// per-dataset map of any failures.
func (a *App) ReprocessAllDatasets(ctx context.Context, fileType *domain.FileType) (int, map[uuid.UUID]error, error) {
return a.datasets.ReprocessAll(ctx, fileType)
}
// Close releases all resources in reverse order of acquisition.

View File

@@ -2,31 +2,53 @@ package cli
import (
"fmt"
"strings"
"gis/internal/app"
"gis/internal/domain"
"github.com/google/uuid"
"github.com/spf13/cobra"
)
var reprocessAll bool
var (
reprocessAll bool
reprocessType string
)
var reprocessCmd = &cobra.Command{
Use: "reprocess [dataset-id]",
Short: "Re-enqueue the processing job for uploaded datasets",
Long: "Re-publish the RabbitMQ message that drives an uploaded dataset's\n" +
"asynchronous processing, selecting the right step from its file type\n" +
"(vector_with_kato -> parse, vector -> properties, raster -> cog).\n\n" +
"(vector_with_kato -> parse, vector -> properties, raster -> cog).\n" +
"Before re-enqueuing, each dataset's stored geometry and GeoJSON are\n" +
"cleared so a stale (e.g. previously miscomputed) value is never served\n" +
"while the job is pending.\n\n" +
"Pass a dataset id to reprocess a single dataset, or --all to reprocess\n" +
"every dataset.\n\n" +
"every dataset. --type restricts --all to one file type\n" +
"(vector_with_kato, vector, or raster).\n\n" +
"Examples:\n" +
" gis reprocess 06818b2b-1fc5-47d9-a764-db2d4cb3df75\n" +
" gis reprocess --all",
" gis reprocess --all\n" +
" gis reprocess --all --type=vector",
Args: cobra.MaximumNArgs(1),
RunE: func(cmd *cobra.Command, args []string) error {
if reprocessAll == (len(args) == 1) {
return fmt.Errorf("provide exactly one of <dataset-id> or --all")
}
if reprocessType != "" && !reprocessAll {
return fmt.Errorf("--type can only be used together with --all")
}
var fileType *domain.FileType
if reprocessType != "" {
ft := domain.FileType(strings.TrimSpace(reprocessType))
if !ft.Valid() {
return fmt.Errorf("invalid --type %q: must be one of vector_with_kato, vector, raster", reprocessType)
}
fileType = &ft
}
ctx, cancel := signalContext()
defer cancel()
@@ -38,7 +60,7 @@ var reprocessCmd = &cobra.Command{
defer application.Close()
if reprocessAll {
enqueued, failures, err := application.ReprocessAllDatasets(ctx)
enqueued, failures, err := application.ReprocessAllDatasets(ctx, fileType)
if err != nil {
return err
}
@@ -47,7 +69,7 @@ var reprocessCmd = &cobra.Command{
"dataset_id", id, "error", ferr)
}
application.Log.Info("re-enqueued all dataset processing",
"enqueued", enqueued, "failed", len(failures))
"file_type", reprocessType, "enqueued", enqueued, "failed", len(failures))
if len(failures) > 0 {
return fmt.Errorf("%d of %d datasets failed to re-enqueue", len(failures), enqueued+len(failures))
}
@@ -71,4 +93,5 @@ var reprocessCmd = &cobra.Command{
func init() {
reprocessCmd.Flags().BoolVar(&reprocessAll, "all", false, "reprocess every dataset")
reprocessCmd.Flags().StringVar(&reprocessType, "type", "", "restrict --all to one file type (vector_with_kato, vector, raster)")
}

View File

@@ -15,7 +15,7 @@ var publishExample bool
var workerCmd = &cobra.Command{
Use: "worker",
Short: "Run the RabbitMQ consumers (dataset parsing + example)",
Short: "Run the RabbitMQ consumers (dataset processing, landscape runs + example)",
RunE: func(cmd *cobra.Command, args []string) error {
ctx, cancel := signalContext()
defer cancel()

View File

@@ -12,10 +12,11 @@ import (
// Config holds all configuration for the application. Values are read from
// environment variables; a local .env file (if present) is loaded first.
type Config struct {
HTTP HTTPConfig
DB DBConfig
S3 S3Config
RabbitMQ RabbitMQConfig
HTTP HTTPConfig
DB DBConfig
S3 S3Config
RabbitMQ RabbitMQConfig
Landscape LandscapeConfig
}
// HTTPConfig configures the HTTP server.
@@ -26,6 +27,9 @@ type HTTPConfig struct {
WriteTimeout time.Duration `env:"HTTP_WRITE_TIMEOUT" envDefault:"120s"`
IdleTimeout time.Duration `env:"HTTP_IDLE_TIMEOUT" envDefault:"60s"`
ShutdownTimeout time.Duration `env:"HTTP_SHUTDOWN_TIMEOUT" envDefault:"10s"`
// MaxUploadBytes caps the size of any request body, multipart uploads
// included (default 2 GiB); larger requests get 413. 0 disables the cap.
MaxUploadBytes int64 `env:"HTTP_MAX_UPLOAD_BYTES" envDefault:"2147483648"`
}
// Addr returns the listen address for the HTTP server.
@@ -58,6 +62,19 @@ type RabbitMQConfig struct {
Queue string `env:"RABBITMQ_QUEUE" envDefault:"gis.events.example"`
}
// LandscapeConfig bounds the Landscape Program's parallelism, and with it the
// worker's peak memory: a run holds up to MaxConcurrentNodes nodes in flight,
// each running a pool of MaxWorkers/MaxConcurrentNodes goroutines that own one
// window buffer per input and output. The defaults (8 total workers spread
// over 2 nodes) keep that product bounded regardless of how many cores the
// host has.
type LandscapeConfig struct {
// MaxWorkers is the total window-worker budget shared by all nodes of a run.
MaxWorkers int `env:"LP_MAX_WORKERS" envDefault:"8"`
// MaxConcurrentNodes is how many graph nodes of one stage run at a time.
MaxConcurrentNodes int `env:"LP_MAX_NODES" envDefault:"2"`
}
// Load reads configuration from the environment, loading an optional .env file
// from the current working directory first.
func Load() (*Config, error) {

View File

@@ -71,18 +71,30 @@ type DatasetFilter struct {
FileType *FileType
Automated *bool
Status *string
// DatasetGroupID restricts the listing to one dataset group.
DatasetGroupID *uuid.UUID
// StartDateFrom / StartDateTo and EndDateFrom / EndDateTo are inclusive
// YYYY-MM-DD bounds on the dataset's period columns.
StartDateFrom *string
StartDateTo *string
EndDateFrom *string
EndDateTo *string
}
// Observation is a single unpivoted value from a dataset's attribute table,
// keyed by KATO code and date. Exactly one of Value / ValueText is typically
// set (numeric vs non-numeric cell); both may be nil for an empty cell.
// keyed by KATO code and date (optionally a StartDate..EndDate period). Exactly
// one of Value / ValueText is typically set (numeric vs non-numeric cell); both
// may be nil for an empty cell.
type Observation struct {
ID uuid.UUID `json:"id"`
DatasetID uuid.UUID `json:"dataset_id"`
KatoCode string `json:"kato_code"`
Date string `json:"date"`
Value *float64 `json:"value"`
ValueText *string `json:"value_text"`
StartDate string `json:"start_date"`
// EndDate is the period's end date. Nil for a point-in-time observation
// (pre-existing behavior).
EndDate *string `json:"end_date,omitempty"`
Value *float64 `json:"value"`
ValueText *string `json:"value_text"`
}
// allowedExtensions lists the accepted lowercase file extensions (including the
@@ -115,9 +127,23 @@ func ExtensionAllowedFor(ft FileType, ext string) bool {
func ValidateFileContent(ext string, head []byte) error {
switch ext {
case ".tif", ".tiff":
// TIFF: little-endian "II*\0" or big-endian "MM\0*".
if !bytes.HasPrefix(head, []byte("II*\x00")) && !bytes.HasPrefix(head, []byte("MM\x00*")) {
return fmt.Errorf("file is not a valid TIFF/GeoTIFF")
// Classic TIFF: little-endian "II*\0" or big-endian "MM\0*".
// BigTIFF (version 43) uses "II+\0" / "MM\0+"; GDAL writes it for
// large rasters (BIGTIFF=IF_SAFER), so the pipeline's own COG output
// must pass this check too.
tiffMagic := [][]byte{
[]byte("II*\x00"), []byte("MM\x00*"),
[]byte("II+\x00"), []byte("MM\x00+"),
}
ok := false
for _, m := range tiffMagic {
if bytes.HasPrefix(head, m) {
ok = true
break
}
}
if !ok {
return fmt.Errorf("file is not a valid TIFF/BigTIFF GeoTIFF")
}
case ".zip":
// ZIP local-file or empty-archive signature.
@@ -134,10 +160,30 @@ func ValidateFileContent(ext string, head []byte) error {
if b, ok := firstMeaningfulByte(head); !ok || b != '{' {
return fmt.Errorf("file is not valid GeoJSON")
}
case ".sld", ".xml":
// SLD is XML; a lightweight sniff for the opening tag.
if b, ok := firstMeaningfulByte(head); !ok || b != '<' {
return fmt.Errorf("file is not valid XML/SLD")
}
}
return nil
}
// AllowedSLDExtensions lists the accepted lowercase extensions (including the
// dot) for a dataset's SLD (Styled Layer Descriptor) style file.
var AllowedSLDExtensions = []string{".sld", ".xml"}
// ExtensionAllowedForSLD reports whether ext (lowercase, with dot) is a valid
// SLD file extension.
func ExtensionAllowedForSLD(ext string) bool {
for _, e := range AllowedSLDExtensions {
if e == ext {
return true
}
}
return false
}
// firstMeaningfulByte returns the first non-whitespace byte after an optional
// UTF-8 BOM.
func firstMeaningfulByte(head []byte) (byte, bool) {
@@ -160,18 +206,30 @@ type AttributeColumn struct {
Samples []string `json:"samples,omitempty"`
}
// YearColumn maps an attribute column to the date it represents,
// e.g. {"column": "F_2023", "date": "2023-01-01"}.
// YearColumn maps an attribute column to the date (or date period) it
// represents, e.g. {"column": "F_2023", "start_date": "2023-01-01"} for a
// single point in time, or {"column": "F_2020_2023", "start_date":
// "2020-01-01", "end_date": "2023-12-31"} for a period. EndDate is optional;
// when nil the mapping is a single point in time, preserving the original
// (pre-period) behavior.
type YearColumn struct {
Column string `json:"column"`
Date string `json:"date"`
Column string `json:"column"`
StartDate string `json:"start_date"`
EndDate *string `json:"end_date,omitempty"`
}
// DatasetSummary is the lightweight view of a dataset used in list responses.
// It omits the heavy geometry/attribute/JSONB fields.
type DatasetSummary struct {
ID uuid.UUID `json:"id"`
CategoryID uuid.UUID `json:"category_id"`
ID uuid.UUID `json:"id"`
CategoryID uuid.UUID `json:"category_id"`
// DatasetGroupID links the dataset to its dataset group ("одноимённые"
// datasets versioned by period). Nullable.
DatasetGroupID *uuid.UUID `json:"dg_id"`
// StartDate / EndDate is the period the dataset describes (YYYY-MM-DD).
// Both nullable; EndDate drives the "freshest in group" rule.
StartDate *string `json:"start_date"`
EndDate *string `json:"end_date"`
Name string `json:"name"`
Description *string `json:"description"`
Unit *string `json:"unit"`
@@ -184,21 +242,33 @@ type DatasetSummary struct {
// Dataset is a geo file stored in the object store and grouped under a category.
type Dataset struct {
ID uuid.UUID `json:"id"`
CategoryID uuid.UUID `json:"category_id"`
Name string `json:"name"`
Description *string `json:"description"`
Unit *string `json:"unit"`
Filename string `json:"filename"`
StorageKey string `json:"storage_key"`
ID uuid.UUID `json:"id"`
CategoryID uuid.UUID `json:"category_id"`
// DatasetGroupID links the dataset to its dataset group ("одноимённые"
// datasets versioned by period). Nullable.
DatasetGroupID *uuid.UUID `json:"dg_id"`
// StartDate / EndDate is the period the dataset describes (YYYY-MM-DD).
// Both nullable; EndDate drives the "freshest in group" rule.
StartDate *string `json:"start_date"`
EndDate *string `json:"end_date"`
Name string `json:"name"`
Description *string `json:"description"`
Unit *string `json:"unit"`
Filename string `json:"filename"`
StorageKey string `json:"storage_key"`
// CogStorageKey points to the Cloud-Optimized GeoTIFF for rasters. Nullable.
CogStorageKey *string `json:"cog_storage_key"`
FileType FileType `json:"file_type"`
SizeBytes int64 `json:"size_bytes"`
ContentType string `json:"content_type"`
// SldStorageKey points to the dataset's SLD (Styled Layer Descriptor) style
// file, if one has been uploaded. Nullable; applies to every file type.
SldStorageKey *string `json:"sld_storage_key"`
SldFilename *string `json:"sld_filename"`
SldContentType *string `json:"sld_content_type"`
// Properties holds tabular data extracted from the file (e.g. a shapefile's
// attribute table). Nullable.
Properties json.RawMessage `json:"properties"`
// attribute table). Nullable; not exposed in dataset responses.
Properties json.RawMessage `json:"-"`
// Meta holds arbitrary user-defined data. Nullable.
Meta json.RawMessage `json:"meta"`
// Automated is a user-defined flag.
@@ -215,8 +285,9 @@ type Dataset struct {
// ParseError holds the failure reason when Status is failed. Nullable.
ParseError *string `json:"parse_error"`
// Geometry is the dataset's spatial geometry, serialized as GeoJSON.
// Nullable; populated from the file's spatial data.
Geometry json.RawMessage `json:"geometry"`
// Nullable; populated from the file's spatial data. Not exposed in dataset
// responses.
Geometry json.RawMessage `json:"-"`
// GeoJSON is the pre-assembled GeoJSON FeatureCollection served by the
// .geojson endpoint. It is generated and persisted at processing time for
// vector and vector_with_kato datasets. Nullable; not exposed on the dataset
@@ -228,3 +299,37 @@ type Dataset struct {
CreatedAt time.Time `json:"created_at"`
UpdatedAt time.Time `json:"updated_at"`
}
// DatasetFields is the set of editable metadata columns written by
// UpdateFields. Nil pointers clear the corresponding nullable column.
type DatasetFields struct {
CategoryID uuid.UUID
Name string
Description *string
Unit *string
DatasetGroupID *uuid.UUID
StartDate *string
EndDate *string
}
// ValidateDatePeriod checks that start/end (YYYY-MM-DD, either may be nil)
// parse and that end is not before start. Returns an ErrValidation-wrapped
// error naming the offending field.
func ValidateDatePeriod(start, end *string) error {
var from, to time.Time
var err error
if start != nil {
if from, err = time.Parse("2006-01-02", *start); err != nil {
return fmt.Errorf("%w: start_date %q is not YYYY-MM-DD", ErrValidation, *start)
}
}
if end != nil {
if to, err = time.Parse("2006-01-02", *end); err != nil {
return fmt.Errorf("%w: end_date %q is not YYYY-MM-DD", ErrValidation, *end)
}
}
if start != nil && end != nil && to.Before(from) {
return fmt.Errorf("%w: end_date %q is before start_date %q", ErrValidation, *end, *start)
}
return nil
}

View File

@@ -0,0 +1,23 @@
package domain
import (
"time"
"github.com/google/uuid"
)
// DatasetGroup groups "одноимённые" datasets (the same indicator delivered for
// different periods). The freshest dataset of a group is the one with the
// greatest end_date. A group may carry a shared SLD style used as a fallback by
// datasets that have none of their own.
type DatasetGroup struct {
ID uuid.UUID `json:"id"`
Code string `json:"code"`
Name string `json:"name"`
Description string `json:"description"`
SldStorageKey *string `json:"sld_storage_key"`
SldFilename *string `json:"sld_filename"`
SldContentType *string `json:"sld_content_type"`
CreatedAt time.Time `json:"created_at"`
UpdatedAt time.Time `json:"updated_at"`
}

View File

@@ -1,6 +1,9 @@
package domain
import "testing"
import (
"errors"
"testing"
)
func TestFileTypeValid(t *testing.T) {
valid := []FileType{FileTypeVectorWithKato, FileTypeVector, FileTypeRaster}
@@ -27,7 +30,10 @@ func TestValidateFileContent(t *testing.T) {
}{
{"valid tiff LE", ".tif", []byte("II*\x00rest"), true},
{"valid tiff BE", ".tiff", []byte("MM\x00*rest"), true},
{"valid bigtiff LE", ".tif", []byte("II+\x00rest"), true},
{"valid bigtiff BE", ".tiff", []byte("MM\x00+rest"), true},
{"bad tiff", ".tif", []byte("not a tiff"), false},
{"bad tiff near-miss prefix", ".tif", []byte("II-\x00rest"), false},
{"valid zip", ".zip", []byte("PK\x03\x04rest"), true},
{"bad zip", ".zip", []byte("RARrest"), false},
{"valid gpkg", ".gpkg", []byte("SQLite format 3\x00rest"), true},
@@ -72,3 +78,31 @@ func TestExtensionAllowedFor(t *testing.T) {
}
}
}
func TestValidateDatePeriod(t *testing.T) {
s := func(v string) *string { return &v }
cases := []struct {
name string
start, end *string
wantErr bool
}{
{"both nil", nil, nil, false},
{"start only", s("2020-01-01"), nil, false},
{"ordered", s("2020-01-01"), s("2020-12-31"), false},
{"same day", s("2020-01-01"), s("2020-01-01"), false},
{"reversed", s("2020-12-31"), s("2020-01-01"), true},
{"bad start", s("01.01.2020"), nil, true},
{"bad end", nil, s("2020-13-01"), true},
}
for _, tc := range cases {
t.Run(tc.name, func(t *testing.T) {
err := ValidateDatePeriod(tc.start, tc.end)
if (err != nil) != tc.wantErr {
t.Fatalf("err=%v wantErr=%v", err, tc.wantErr)
}
if err != nil && !errors.Is(err, ErrValidation) {
t.Fatalf("expected ErrValidation, got %v", err)
}
})
}
}

View File

@@ -0,0 +1,68 @@
package domain
import (
"encoding/json"
"time"
"github.com/google/uuid"
)
// LPRunStatus is the lifecycle status of a Landscape Program run.
type LPRunStatus string
// Landscape Program run statuses.
const (
LPRunQueued LPRunStatus = "queued"
LPRunRunning LPRunStatus = "running"
LPRunSucceeded LPRunStatus = "succeeded"
LPRunFailed LPRunStatus = "failed"
)
// LPNodeStatus is the lifecycle status of a single node within a Landscape
// Program run.
type LPNodeStatus string
// Landscape Program run node statuses.
const (
LPNodePending LPNodeStatus = "pending"
LPNodeRunning LPNodeStatus = "running"
LPNodeSkipped LPNodeStatus = "skipped"
LPNodeSucceeded LPNodeStatus = "succeeded"
LPNodeFailed LPNodeStatus = "failed"
)
// LPDatasetRef references a dataset produced for, or consumed from, a
// dataset group slot in the Landscape Program graph.
type LPDatasetRef struct {
Group string `json:"group"`
DatasetID uuid.UUID `json:"dataset_id"`
}
// LPRun is a single execution of the Landscape Program graph.
type LPRun struct {
ID uuid.UUID `json:"id"`
Status LPRunStatus `json:"status"`
FromStage *int `json:"from_stage"`
Params json.RawMessage `json:"params"`
Overrides map[string]uuid.UUID `json:"overrides"`
Error *string `json:"error"`
CreatedAt time.Time `json:"created_at"`
StartedAt *time.Time `json:"started_at"`
FinishedAt *time.Time `json:"finished_at"`
Nodes []LPRunNode `json:"nodes"`
}
// LPRunNode is the execution state of a single graph node within an LPRun.
type LPRunNode struct {
ID uuid.UUID `json:"id"`
RunID uuid.UUID `json:"run_id"`
NodeCode string `json:"node_code"`
Stage int `json:"stage"`
Status LPNodeStatus `json:"status"`
Fingerprint string `json:"fingerprint"`
Inputs []LPDatasetRef `json:"inputs"`
Outputs []LPDatasetRef `json:"outputs"`
Error *string `json:"error"`
StartedAt *time.Time `json:"started_at"`
FinishedAt *time.Time `json:"finished_at"`
}

View File

@@ -0,0 +1,225 @@
package landscape
import (
"context"
"fmt"
"os"
"path/filepath"
"runtime"
"sync"
"gis/internal/raster"
"golang.org/x/sync/errgroup"
)
// MaxWorkersPerNode is the per-node worker cap Compute uses. Peak memory is
// roughly nodes x workers x window^2 x (inputs+outputs), so callers that run
// several nodes at once should divide their total worker budget between them
// and call ComputeWithWorkers instead.
const MaxWorkersPerNode = 8
// Compute runs one node over its input rasters with the default per-node
// worker cap (MaxWorkersPerNode). See ComputeWithWorkers.
func Compute(ctx context.Context, rio raster.IO, node Node, inputs map[string]string, outDir string, p Params) (map[string]string, error) {
return ComputeWithWorkers(ctx, rio, node, inputs, outDir, p, MaxWorkersPerNode)
}
// ComputeWithWorkers runs one node over its input rasters with an explicit
// worker cap. See the package doc for the concurrency model: windows are
// processed by a worker pool of at most workers goroutines; each worker owns
// its own Reader handles (GDAL datasets are not goroutine-safe) and writes are
// serialised per output. Window order does not affect the result. On any
// error, no output files are left behind: ComputeWithWorkers best-effort
// removes every output it created before returning. A workers value below 1 is
// treated as 1.
func ComputeWithWorkers(ctx context.Context, rio raster.IO, node Node, inputs map[string]string, outDir string, p Params, workers int) (map[string]string, error) {
p = p.WithDefaults()
if p.WindowSize <= 0 {
// WithDefaults only fills in a zero value; guard against a negative
// one too, since raster.Windows would otherwise loop forever.
p.WindowSize = DefaultParams().WindowSize
}
paths := make([]string, len(node.Inputs))
for i, g := range node.Inputs {
path, ok := inputs[g]
if !ok || path == "" {
return nil, fmt.Errorf("node %s (%s): missing input %q", node.Code, node.Name, g)
}
paths[i] = path
}
// Validate grids with one set of handles before creating anything.
grid, err := checkGrids(ctx, rio, node, paths)
if err != nil {
return nil, err
}
outPaths := make(map[string]string, len(node.Outputs))
writers := make([]raster.Writer, len(node.Outputs))
createdPaths := make([]string, 0, len(node.Outputs))
for j, g := range node.Outputs {
path := filepath.Join(outDir, g+".tif")
w, err := rio.Create(ctx, path, grid, raster.DefaultWriterOptions())
if err != nil {
closeWriters(writers[:j])
removeOutputs(createdPaths)
return nil, fmt.Errorf("node %s: create output %s: %w", node.Code, g, err)
}
writers[j] = w
outPaths[g] = path
createdPaths = append(createdPaths, path)
}
windows := raster.Windows(grid.Width, grid.Height, p.WindowSize)
// The first window raster.Windows produces is always the largest: it
// starts at (0,0) so both its width and height are the un-clipped
// min(WindowSize, dimension). Size per-worker buffers off it instead of
// WindowSize*WindowSize, which would over-allocate whenever the raster is
// smaller than one window (the common case for small/test rasters, and
// for the last row/column of any raster).
var size int
if len(windows) > 0 {
size = windows[0].Width * windows[0].Height
}
jobs := make(chan raster.Window)
var writeMu sync.Mutex
eg, gctx := errgroup.WithContext(ctx)
// Cap workers well below GOMAXPROCS too: each worker holds a full set of
// per-window input/output buffers, and nodes can have many inputs and
// outputs (e.g. node 20: 6 inputs, 5 outputs), so an unbounded worker
// count multiplies memory with core count and with concurrently running
// nodes. The caller's budget is the third bound.
workers = min(runtime.GOMAXPROCS(0), len(windows), max(1, workers))
for w := 0; w < workers; w++ {
eg.Go(func() error {
readers, err := openAll(gctx, rio, node, paths)
if err != nil {
return err
}
defer closeReaders(readers)
in := make([][]float64, len(paths))
for i := range in {
in[i] = make([]float64, size)
}
out := make([][]int16, len(writers))
for j := range out {
out[j] = make([]int16, size)
}
for win := range jobs {
n := win.Width * win.Height
inView := make([][]float64, len(in))
for i, r := range readers {
if err := r.ReadFloat64(win, in[i][:n]); err != nil {
return fmt.Errorf("node %s: read %s window %+v: %w", node.Code, node.Inputs[i], win, err)
}
inView[i] = in[i][:n]
}
outView := make([][]int16, len(out))
for j := range out {
outView[j] = out[j][:n]
}
node.Kernel(inView, outView, p)
writeMu.Lock()
for j, wr := range writers {
if err := wr.WriteInt16(win, outView[j]); err != nil {
writeMu.Unlock()
return fmt.Errorf("node %s: write %s window %+v: %w", node.Code, node.Outputs[j], win, err)
}
}
writeMu.Unlock()
}
return nil
})
}
eg.Go(func() error {
defer close(jobs)
for _, win := range windows {
select {
case jobs <- win:
case <-gctx.Done():
return gctx.Err()
}
}
return nil
})
runErr := eg.Wait()
if err := closeWriters(writers); err != nil && runErr == nil {
runErr = err
}
if runErr != nil {
removeOutputs(createdPaths)
return nil, runErr
}
return outPaths, nil
}
// checkGrids opens every input once, verifies all grids match the first, and
// returns that grid.
func checkGrids(ctx context.Context, rio raster.IO, node Node, paths []string) (raster.Grid, error) {
readers, err := openAll(ctx, rio, node, paths)
if err != nil {
return raster.Grid{}, err
}
defer closeReaders(readers)
if len(readers) == 0 {
return raster.Grid{}, fmt.Errorf("node %s (%s): has no inputs", node.Code, node.Name)
}
ref := readers[0].Grid()
for i := 1; i < len(readers); i++ {
if g := readers[i].Grid(); !rio.SameGrid(ref, g) {
return raster.Grid{}, fmt.Errorf("node %s: input %s grid (%dx%d, %v) differs from %s (%dx%d, %v); inputs must be aligned",
node.Code, node.Inputs[i], g.Width, g.Height, g.GeoTransform, node.Inputs[0], ref.Width, ref.Height, ref.GeoTransform)
}
}
return ref, nil
}
// openAll opens paths in order, wrapping any failure with the node code and
// the input group name (paths is always node.Inputs resolved to files, in
// the same order) for context.
func openAll(ctx context.Context, rio raster.IO, node Node, paths []string) ([]raster.Reader, error) {
readers := make([]raster.Reader, 0, len(paths))
for i, p := range paths {
r, err := rio.Open(ctx, p)
if err != nil {
closeReaders(readers)
name := p
if i < len(node.Inputs) {
name = node.Inputs[i]
}
return nil, fmt.Errorf("node %s: input %s: %w", node.Code, name, err)
}
readers = append(readers, r)
}
return readers, nil
}
func closeReaders(rs []raster.Reader) {
for _, r := range rs {
if r != nil {
_ = r.Close()
}
}
}
func closeWriters(ws []raster.Writer) error {
var first error
for _, w := range ws {
if w == nil {
continue
}
if err := w.Close(); err != nil && first == nil {
first = err
}
}
return first
}
// removeOutputs best-effort deletes every path in paths, ignoring errors. It
// is used to leave no partially-written output behind after a failed run.
func removeOutputs(paths []string) {
for _, p := range paths {
_ = os.Remove(p)
}
}

View File

@@ -0,0 +1,252 @@
package landscape
import (
"context"
"fmt"
"os"
"path/filepath"
"testing"
"gis/internal/raster"
)
func grid(w, h int) raster.Grid {
return raster.Grid{Width: w, Height: h, GeoTransform: [6]float64{0, 30, 0, 0, 0, -30}, ProjectionWKT: "EPSG:32642"}
}
func constRaster(w, h int, v float64) []float64 {
d := make([]float64, w*h)
for i := range d {
d[i] = v
}
return d
}
func TestCompute_E_TilesWindowsAndWritesEveryPixel(t *testing.T) {
fio := newFakeIO()
g := grid(10, 7)
z := constRaster(10, 7, 2)
u := constRaster(10, 7, 4)
z[0] = -9999 // pixel 0 invalid
fio.put("z.tif", g, z)
fio.put("u.tif", g, u)
node, _ := NodeByCode("10")
p := DefaultParams()
p.WindowSize = 4 // forces partial windows (10x7 with 4x4 tiles)
out, err := Compute(context.Background(), fio, node, map[string]string{OutZIntegrated: "z.tif", OutUIntegrated: "u.tif"}, "/out", p)
if err != nil {
t.Fatal(err)
}
path := out[OutE]
if path != filepath.Join("/out", OutE+".tif") {
t.Fatalf("unexpected output path %q", path)
}
res := fio.files[path]
if res.grid != g {
t.Fatal("output grid must copy the first input's grid")
}
if res.opts != raster.DefaultWriterOptions() {
t.Fatalf("output created with %+v, want raster.DefaultWriterOptions() = %+v", res.opts, raster.DefaultWriterOptions())
}
for i, v := range res.data {
want := 4.0
if i == 0 {
want = -9999
}
if v != want {
t.Fatalf("pixel %d = %v want %v", i, v, want)
}
}
}
func TestCompute_FailsOnGridMismatch(t *testing.T) {
fio := newFakeIO()
fio.put("z.tif", grid(10, 7), constRaster(10, 7, 2))
fio.put("u.tif", grid(9, 7), constRaster(9, 7, 4))
node, _ := NodeByCode("10")
_, err := Compute(context.Background(), fio, node, map[string]string{OutZIntegrated: "z.tif", OutUIntegrated: "u.tif"}, "/out", DefaultParams())
if err == nil {
t.Fatal("expected grid mismatch error")
}
if _, created := fio.files[filepath.Join("/out", OutE+".tif")]; created {
t.Fatal("no output must be created on mismatch")
}
}
func TestCompute_FailsOnMissingInput(t *testing.T) {
fio := newFakeIO()
fio.put("z.tif", grid(2, 2), constRaster(2, 2, 2))
node, _ := NodeByCode("10")
if _, err := Compute(context.Background(), fio, node, map[string]string{OutZIntegrated: "z.tif"}, "/out", DefaultParams()); err == nil {
t.Fatal("expected error for missing input group")
}
}
func TestCompute_MultiOutputNode(t *testing.T) {
fio := newFakeIO()
g := grid(3, 1)
paths := map[string]string{}
for i, s := range []string{SlotPIrrigation, SlotPSnow, SlotPCrop, SlotPPasture, SlotPTourism} {
paths[s] = s + ".tif"
fio.put(paths[s], g, constRaster(3, 1, float64(i+1)))
}
paths[OutE] = "e.tif"
fio.put("e.tif", g, constRaster(3, 1, 3))
node, _ := NodeByCode("20")
out, err := Compute(context.Background(), fio, node, paths, "/o", DefaultParams())
if err != nil {
t.Fatal(err)
}
if len(out) != 5 {
t.Fatalf("want 5 outputs, got %d", len(out))
}
if fio.files[out[OutCTourism]].data[0] != 3 || fio.files[out[OutCIrrigation]].data[0] != 1 {
t.Fatal("min(P, E) not applied per output")
}
}
// TestCompute_MultiOutputNode_TiledWindows runs the 6-input/5-output node 20
// with a window size much smaller than the raster, so per-worker output
// buffers are reused across windows of differing pixel counts (partial
// windows at the raster's right/bottom edges).
func TestCompute_MultiOutputNode_TiledWindows(t *testing.T) {
fio := newFakeIO()
w, h := 5, 3
g := grid(w, h)
paths := map[string]string{}
for i, s := range []string{SlotPIrrigation, SlotPSnow, SlotPCrop, SlotPPasture, SlotPTourism} {
paths[s] = s + ".tif"
fio.put(paths[s], g, constRaster(w, h, float64(i+1)))
}
paths[OutE] = "e.tif"
fio.put("e.tif", g, constRaster(w, h, 3))
node, _ := NodeByCode("20")
p := DefaultParams()
p.WindowSize = 2 // 5x3 with 2x2 tiles: several differently-sized windows
out, err := Compute(context.Background(), fio, node, paths, "/o2", p)
if err != nil {
t.Fatal(err)
}
codes := []string{OutCIrrigation, OutCSnow, OutCCrop, OutCPasture, OutCTourism}
for i, code := range codes {
want := min(i+1, 3) // min(P_sector, E=3)
res := fio.files[out[code]]
for px, v := range res.data {
if v != float64(want) {
t.Fatalf("%s pixel %d = %v want %v", code, px, v, want)
}
}
}
}
// TestCompute_ZeroParamsUsesDefaults exercises Compute with a completely
// zero-valued Params, relying on WithDefaults to fill in WindowSize and the
// other node-independent knobs.
func TestCompute_ZeroParamsUsesDefaults(t *testing.T) {
fio := newFakeIO()
g := grid(4, 4)
fio.put("z.tif", g, constRaster(4, 4, 2))
fio.put("u.tif", g, constRaster(4, 4, 3))
node, _ := NodeByCode("10")
out, err := Compute(context.Background(), fio, node, map[string]string{OutZIntegrated: "z.tif", OutUIntegrated: "u.tif"}, "/z", Params{})
if err != nil {
t.Fatal(err)
}
for _, v := range fio.files[out[OutE]].data {
if v != 3 {
t.Fatalf("pixel = %v want 3", v)
}
}
}
// TestCompute_NegativeWindowSizeClampedToDefault guards against
// raster.Windows looping forever on a non-positive window size: WithDefaults
// only fills in a zero value, so Compute must clamp negatives itself.
func TestCompute_NegativeWindowSizeClampedToDefault(t *testing.T) {
fio := newFakeIO()
g := grid(2, 2)
fio.put("z.tif", g, constRaster(2, 2, 2))
fio.put("u.tif", g, constRaster(2, 2, 3))
node, _ := NodeByCode("10")
p := DefaultParams()
p.WindowSize = -5
out, err := Compute(context.Background(), fio, node, map[string]string{OutZIntegrated: "z.tif", OutUIntegrated: "u.tif"}, "/neg", p)
if err != nil {
t.Fatal(err)
}
if fio.files[out[OutE]].data[0] != 3 {
t.Fatal("unexpected result")
}
}
// TestCompute_FailureMidRunReturnsError simulates a read failure partway
// through a run (the fake fails every ReadFloat64 against u.tif from the
// first call on) and checks Compute surfaces the error instead of returning
// a partial result.
func TestCompute_FailureMidRunReturnsError(t *testing.T) {
fio := newFakeIO()
g := grid(10, 7)
fio.put("z.tif", g, constRaster(10, 7, 2))
fio.put("u.tif", g, constRaster(10, 7, 4))
fio.failReadAfterN("u.tif", 0) // fail starting from the very first read
node, _ := NodeByCode("10")
p := DefaultParams()
p.WindowSize = 4
if _, err := Compute(context.Background(), fio, node, map[string]string{OutZIntegrated: "z.tif", OutUIntegrated: "u.tif"}, "/out-fail", p); err == nil {
t.Fatal("expected error from simulated read failure")
}
}
// TestRemoveOutputs_DeletesFiles unit-tests the cleanup helper Compute uses
// after a failed run, against the real filesystem (the fake IO above has no
// on-disk files to assert against).
func TestRemoveOutputs_DeletesFiles(t *testing.T) {
dir := t.TempDir()
var paths []string
for i := 0; i < 3; i++ {
p := filepath.Join(dir, fmt.Sprintf("out%d.tif", i))
if err := os.WriteFile(p, []byte("data"), 0o644); err != nil {
t.Fatal(err)
}
paths = append(paths, p)
}
removeOutputs(paths)
for _, p := range paths {
if _, err := os.Stat(p); !os.IsNotExist(err) {
t.Fatalf("expected %s to be removed, stat err = %v", p, err)
}
}
}
// TestComputeWithWorkers_RespectsCap checks the caller's worker budget bounds
// the pool: each pool worker opens one reader per input and holds a buffer per
// input and output, so the open count bounds peak memory. checkGrids opens one
// extra set of handles before the pool starts, hence the (workers+1) bound.
func TestComputeWithWorkers_RespectsCap(t *testing.T) {
node, _ := NodeByCode("10")
inputs := len(node.Inputs)
for _, workers := range []int{1, 2, 4, 0, -3} {
t.Run(fmt.Sprintf("workers=%d", workers), func(t *testing.T) {
fio := newFakeIO()
g := grid(64, 64)
fio.put("z.tif", g, constRaster(64, 64, 2))
fio.put("u.tif", g, constRaster(64, 64, 4))
p := DefaultParams()
p.WindowSize = 8 // 64 windows, so the pool is never window-bound
if _, err := ComputeWithWorkers(context.Background(), fio, node,
map[string]string{OutZIntegrated: "z.tif", OutUIntegrated: "u.tif"}, "/out", p, workers); err != nil {
t.Fatal(err)
}
effective := max(1, workers)
if got, limit := fio.openCount(), (effective+1)*inputs; got > limit {
t.Fatalf("opened %d readers with workers=%d, want at most %d", got, workers, limit)
}
if fio.openCount() < inputs {
t.Fatalf("opened %d readers, want at least one set of %d", fio.openCount(), inputs)
}
})
}
}

View File

@@ -0,0 +1,122 @@
package landscape
import (
"context"
"fmt"
"sync"
"gis/internal/raster"
)
// fakeIO keeps rasters in memory keyed by path.
type fakeIO struct {
mu sync.Mutex
files map[string]*fakeRaster
// failReadAfter, when set for a path, makes every ReadFloat64 call
// against that path fail once the cumulative read count (across all
// readers/workers) exceeds the configured threshold. Used to simulate a
// mid-run I/O failure.
failReadAfter map[string]int
readCounts map[string]int
// opens counts every successful Open. Each pool worker opens one reader
// per input, so the count bounds the number of concurrent handles.
opens int
}
// openCount returns how many readers have been opened so far.
func (f *fakeIO) openCount() int {
f.mu.Lock()
defer f.mu.Unlock()
return f.opens
}
type fakeRaster struct {
grid raster.Grid
data []float64 // row-major
opts raster.WriterOptions
}
func newFakeIO() *fakeIO { return &fakeIO{files: map[string]*fakeRaster{}} }
func (f *fakeIO) put(path string, g raster.Grid, data []float64) {
f.files[path] = &fakeRaster{grid: g, data: data}
}
// failReadAfterN makes ReadFloat64 against path fail starting from the
// (n+1)-th call to it.
func (f *fakeIO) failReadAfterN(path string, n int) {
f.mu.Lock()
defer f.mu.Unlock()
if f.failReadAfter == nil {
f.failReadAfter = map[string]int{}
f.readCounts = map[string]int{}
}
f.failReadAfter[path] = n
}
type fakeReader struct {
f *fakeIO
path string
r *fakeRaster
}
func (r *fakeReader) Grid() raster.Grid { return r.r.grid }
func (r *fakeReader) NoData() (float64, bool) { return -9999, true }
func (r *fakeReader) Close() error { return nil }
func (r *fakeReader) ReadFloat64(w raster.Window, dst []float64) error {
if r.f != nil {
r.f.mu.Lock()
if n, ok := r.f.failReadAfter[r.path]; ok {
r.f.readCounts[r.path]++
if r.f.readCounts[r.path] > n {
r.f.mu.Unlock()
return fmt.Errorf("fake: simulated read failure for %s", r.path)
}
}
r.f.mu.Unlock()
}
for y := 0; y < w.Height; y++ {
copy(dst[y*w.Width:(y+1)*w.Width], r.r.data[(w.Y+y)*r.r.grid.Width+w.X:(w.Y+y)*r.r.grid.Width+w.X+w.Width])
}
return nil
}
type fakeWriter struct {
f *fakeIO
r *fakeRaster
}
func (w *fakeWriter) WriteInt16(win raster.Window, src []int16) error {
w.f.mu.Lock()
defer w.f.mu.Unlock()
for y := 0; y < win.Height; y++ {
for x := 0; x < win.Width; x++ {
w.r.data[(win.Y+y)*w.r.grid.Width+win.X+x] = float64(src[y*win.Width+x])
}
}
return nil
}
func (w *fakeWriter) Close() error { return nil }
func (f *fakeIO) Open(_ context.Context, path string) (raster.Reader, error) {
f.mu.Lock()
defer f.mu.Unlock()
r, ok := f.files[path]
if !ok {
return nil, fmt.Errorf("fake: %s not found", path)
}
f.opens++
return &fakeReader{f: f, path: path, r: r}, nil
}
func (f *fakeIO) Create(_ context.Context, path string, g raster.Grid, o raster.WriterOptions) (raster.Writer, error) {
f.mu.Lock()
defer f.mu.Unlock()
r := &fakeRaster{grid: g, data: make([]float64, g.Width*g.Height), opts: o}
f.files[path] = r
return &fakeWriter{f: f, r: r}, nil
}
func (f *fakeIO) SameGrid(a, b raster.Grid) bool { return a == b }

157
internal/landscape/graph.go Normal file
View File

@@ -0,0 +1,157 @@
// Package landscape defines the Landscape Program (ЛП) computation graph — the
// per-pixel formulas that turn ecological (Z), vulnerability (U) and
// suitability (P) class rasters into constraints (E), conflicts (C) and
// functional zoning (V1–V3) — together with a window-based executor.
package landscape
import "sort"
const (
// NoData is the output nodata value of every LP raster.
NoData int16 = -9999
// CategoryCode is the dataset category holding every LP dataset.
CategoryCode = "landscape"
)
// Group codes. Slots are leaf inputs users upload; the rest are node outputs.
const (
SlotZBiodiversity = "lp-z-biodiversity"
SlotZLCR = "lp-z-lcr"
SlotZSoilProtection = "lp-z-soil-protection"
SlotZWR = "lp-z-wr"
SlotUSurfWater = "lp-u-surf-water"
SlotUSoil = "lp-u-soil"
SlotUCrop = "lp-u-crop"
SlotUVeg = "lp-u-veg"
SlotPIrrigation = "lp-p-irrigation"
SlotPSnow = "lp-p-snow"
SlotPCrop = "lp-p-crop"
SlotPPasture = "lp-p-pasture"
SlotPTourism = "lp-p-tourism"
OutZIntegrated = "lp-z-integrated"
OutUIntegrated = "lp-u-integrated"
OutZMax = "lp-z-max"
OutUMax = "lp-u-max"
OutE = "lp-e"
OutCIrrigation = "lp-c-irrigation"
OutCSnow = "lp-c-snow"
OutCCrop = "lp-c-crop"
OutCPasture = "lp-c-pasture"
OutCTourism = "lp-c-tourism"
OutPMax = "lp-p-max"
OutV1 = "lp-v1-conservative"
OutV2 = "lp-v2-balanced"
OutV3 = "lp-v3-development"
OutCMax = "lp-c-max"
)
type Slot struct{ Code, Title string }
type Output struct{ Code, Title string }
// Kernel computes one window. in[i] follows Node.Inputs order and out[j]
// follows Node.Outputs order; every slice has exactly window-pixel length.
type Kernel func(in [][]float64, out [][]int16, p Params)
// Node is one function of the graph. Stage = first digit of Code.
type Node struct {
Code, Name string
Stage int
Inputs []string
Outputs []string
Kernel Kernel
}
var Slots = []Slot{
{SlotZBiodiversity, "Z Biodiversity"}, {SlotZLCR, "Z LCR"}, {SlotZSoilProtection, "Z Soil Protection"}, {SlotZWR, "Z Water Resources"},
{SlotUSurfWater, "U Surface Water"}, {SlotUSoil, "U Soil"}, {SlotUCrop, "U Crop"}, {SlotUVeg, "U Vegetation"},
{SlotPIrrigation, "P Irrigation"}, {SlotPSnow, "P Snow"}, {SlotPCrop, "P Crop"}, {SlotPPasture, "P Pasture"}, {SlotPTourism, "P Tourism"},
}
var Outputs = []Output{
{OutZIntegrated, "Z Integrated"}, {OutUIntegrated, "U Integrated"}, {OutZMax, "Z Max"}, {OutUMax, "U Max"},
{OutE, "E Ecological Constraint"},
{OutCIrrigation, "C Irrigation"}, {OutCSnow, "C Snow"}, {OutCCrop, "C Crop"}, {OutCPasture, "C Pasture"}, {OutCTourism, "C Tourism"},
{OutPMax, "P Max"},
{OutV1, "V1 Conservative"}, {OutV2, "V2 Balanced"}, {OutV3, "V3 Development"}, {OutCMax, "C Max"},
}
var zInputs = []string{SlotZBiodiversity, SlotZLCR, SlotZSoilProtection, SlotZWR}
var uInputs = []string{SlotUSurfWater, SlotUSoil, SlotUCrop, SlotUVeg} // order matters: water, soil, crop, veg
var pInputs = []string{SlotPIrrigation, SlotPSnow, SlotPCrop, SlotPPasture, SlotPTourism} // sector order 1..5
var cOutputs = []string{OutCIrrigation, OutCSnow, OutCCrop, OutCPasture, OutCTourism}
var Nodes = []Node{
{Code: "00", Name: "z_integrated", Stage: 0, Inputs: zInputs, Outputs: []string{OutZIntegrated}, Kernel: kernelZIntegrated},
{Code: "01", Name: "u_integrated", Stage: 0, Inputs: uInputs, Outputs: []string{OutUIntegrated}, Kernel: kernelUIntegrated},
{Code: "02", Name: "z_max", Stage: 0, Inputs: zInputs, Outputs: []string{OutZMax}, Kernel: kernelZMax},
{Code: "03", Name: "u_max", Stage: 0, Inputs: uInputs, Outputs: []string{OutUMax}, Kernel: kernelUMax},
{Code: "10", Name: "e_ecological_constraint", Stage: 1, Inputs: []string{OutZIntegrated, OutUIntegrated}, Outputs: []string{OutE}, Kernel: kernelE},
{Code: "20", Name: "c_conflicts", Stage: 2, Inputs: append(append([]string{}, pInputs...), OutE), Outputs: cOutputs, Kernel: kernelCConflicts},
{Code: "21", Name: "p_max", Stage: 2, Inputs: pInputs, Outputs: []string{OutPMax}, Kernel: kernelMaxAnyValid},
// func_zoning ranks the original suitability rasters P1..P5 against E (the
// reference notebook and zoning_core.py); the conflict layers C_k are
// diagnostics and are not zoning inputs. It stays in stage 3 to keep the
// diagram's folder numbering (30x) even though it only depends on stage 1.
{Code: "30", Name: "func_zoning", Stage: 3, Inputs: append([]string{OutE}, pInputs...), Outputs: []string{OutV1, OutV2, OutV3}, Kernel: kernelFuncZoning},
{Code: "31", Name: "c_max", Stage: 3, Inputs: cOutputs, Outputs: []string{OutCMax}, Kernel: kernelMaxAnyValid},
}
func NodeByCode(code string) (Node, bool) {
for _, n := range Nodes {
if n.Code == code {
return n, true
}
}
return Node{}, false
}
func SlotByCode(code string) (Slot, bool) {
for _, s := range Slots {
if s.Code == code {
return s, true
}
}
return Slot{}, false
}
func OutputByCode(code string) (Output, bool) {
for _, o := range Outputs {
if o.Code == code {
return o, true
}
}
return Output{}, false
}
// NodeForOutput returns the node producing the given output group.
func NodeForOutput(group string) (Node, bool) {
for _, n := range Nodes {
for _, o := range n.Outputs {
if o == group {
return n, true
}
}
}
return Node{}, false
}
// Stages groups nodes by stage, ascending; within a stage nodes are sorted by code.
func Stages() [][]Node {
byStage := map[int][]Node{}
for _, n := range Nodes {
byStage[n.Stage] = append(byStage[n.Stage], n)
}
keys := make([]int, 0, len(byStage))
for k := range byStage {
keys = append(keys, k)
}
sort.Ints(keys)
out := make([][]Node, 0, len(keys))
for _, k := range keys {
ns := byStage[k]
sort.Slice(ns, func(i, j int) bool { return ns[i].Code < ns[j].Code })
out = append(out, ns)
}
return out
}

View File

@@ -0,0 +1,153 @@
package landscape
import (
"regexp"
"testing"
)
var slugRe = regexp.MustCompile(`^[a-z0-9]+(?:-[a-z0-9]+)*$`)
// isSlot reports whether code names one of the leaf input slots.
func isSlot(code string) bool {
_, ok := SlotByCode(code)
return ok
}
func TestNodeInputsComeFromSlotsOrLowerStage(t *testing.T) {
for _, n := range Nodes {
for _, in := range n.Inputs {
if isSlot(in) {
continue
}
producer, ok := NodeForOutput(in)
if !ok {
t.Errorf("node %s: input %q is neither a slot nor a known node output", n.Code, in)
continue
}
if producer.Stage >= n.Stage {
t.Errorf("node %s (stage %d): input %q is produced by node %s at stage %d, want stage < %d",
n.Code, n.Stage, in, producer.Code, producer.Stage, n.Stage)
}
}
}
}
func TestEveryOutputHasExactlyOneProducer(t *testing.T) {
count := map[string]int{}
for _, n := range Nodes {
for _, o := range n.Outputs {
count[o]++
}
}
for _, o := range Outputs {
if c := count[o.Code]; c != 1 {
t.Errorf("output %q has %d producers, want exactly 1", o.Code, c)
}
}
// No node should produce something that isn't in the Outputs list.
for code, c := range count {
if _, ok := OutputByCode(code); !ok {
t.Errorf("node output %q (%d producers) is not declared in Outputs", code, c)
}
}
}
func TestStagesGroupingAndOrder(t *testing.T) {
stages := Stages()
if len(stages) != 4 {
t.Fatalf("got %d stages, want 4", len(stages))
}
want := [][]string{
{"00", "01", "02", "03"},
{"10"},
{"20", "21"},
{"30", "31"},
}
for i, group := range stages {
var codes []string
for _, n := range group {
codes = append(codes, n.Code)
}
if len(codes) != len(want[i]) {
t.Fatalf("stage group %d: got codes %v, want %v", i, codes, want[i])
}
for j, c := range codes {
if c != want[i][j] {
t.Errorf("stage group %d: got codes %v, want %v", i, codes, want[i])
break
}
}
}
}
func TestAllGroupCodesAreValidSlugs(t *testing.T) {
for _, s := range Slots {
if !slugRe.MatchString(s.Code) {
t.Errorf("slot code %q is not a valid slug", s.Code)
}
}
for _, o := range Outputs {
if !slugRe.MatchString(o.Code) {
t.Errorf("output code %q is not a valid slug", o.Code)
}
}
}
func TestNodeByCode(t *testing.T) {
n, ok := NodeByCode("00")
if !ok || n.Name != "z_integrated" {
t.Fatalf("NodeByCode(00) = %+v, %v", n, ok)
}
if _, ok := NodeByCode("99"); ok {
t.Fatal("NodeByCode(99) should not be found")
}
}
func TestSlotByCode(t *testing.T) {
s, ok := SlotByCode(SlotZBiodiversity)
if !ok || s.Title != "Z Biodiversity" {
t.Fatalf("SlotByCode(%s) = %+v, %v", SlotZBiodiversity, s, ok)
}
if _, ok := SlotByCode("nope"); ok {
t.Fatal("SlotByCode(nope) should not be found")
}
}
func TestOutputByCode(t *testing.T) {
o, ok := OutputByCode(OutE)
if !ok || o.Title != "E Ecological Constraint" {
t.Fatalf("OutputByCode(%s) = %+v, %v", OutE, o, ok)
}
if _, ok := OutputByCode("nope"); ok {
t.Fatal("OutputByCode(nope) should not be found")
}
}
func TestNodeForOutput(t *testing.T) {
n, ok := NodeForOutput(OutV1)
if !ok || n.Code != "30" {
t.Fatalf("NodeForOutput(%s) = %+v, %v", OutV1, n, ok)
}
if _, ok := NodeForOutput("nope"); ok {
t.Fatal("NodeForOutput(nope) should not be found")
}
}
// Functional zoning ranks the ORIGINAL suitability rasters P1..P5 against E
// (reference notebook cells "Three scenarios" and zoning_core.py); the
// conflict layers C_k are diagnostics and must not replace P_k here.
func TestFuncZoningTakesEAndOriginalP(t *testing.T) {
n, ok := NodeByCode("30")
if !ok {
t.Fatal("node 30 missing")
}
want := []string{OutE, SlotPIrrigation, SlotPSnow, SlotPCrop, SlotPPasture, SlotPTourism}
if len(n.Inputs) != len(want) {
t.Fatalf("node 30 inputs = %v, want %v", n.Inputs, want)
}
for i := range want {
if n.Inputs[i] != want[i] {
t.Fatalf("node 30 input[%d] = %s, want %s (order is E, then sectors 1..5)", i, n.Inputs[i], want[i])
}
}
}

View File

@@ -0,0 +1,218 @@
package landscape
// valid reports whether a class value is in the analysis domain (1..5). Any
// other value, including nodata and NaN, is invalid.
func valid(v float64) bool { return v >= 1 && v <= 5 }
// classify maps a float32 score onto classes 1..5 with the fixed intervals
// used by the reference notebook (comparisons in float32, like numpy on a
// float32 array). Scores outside [1, 5] yield NoData.
func classify(x float32) int16 {
switch {
case x < 1.0:
return NoData
case x <= 1.8:
return 1
case x <= 2.6:
return 2
case x <= 3.4:
return 3
case x <= 4.2:
return 4
case x <= 5.0:
return 5
default:
return NoData
}
}
// kernelZIntegrated: all four Z valid -> classify(0.25*Z1 + 0.25*Z2 + 0.25*Z3 + 0.25*Z4).
func kernelZIntegrated(in [][]float64, out [][]int16, _ Params) {
for i := range out[0] {
z1, z2, z3, z4 := in[0][i], in[1][i], in[2][i], in[3][i]
if !(valid(z1) && valid(z2) && valid(z3) && valid(z4)) {
out[0][i] = NoData
continue
}
// explicit conversions forbid FMA fusion so results match numpy's per-op rounding
score := float32(float32(0.25)*float32(z1)) + float32(float32(0.25)*float32(z2)) +
float32(float32(0.25)*float32(z3)) + float32(float32(0.25)*float32(z4))
out[0][i] = classify(score)
}
}
// kernelUIntegrated: water and soil required, at least one of crop/veg;
// U = (0.25*water + 0.25*soil + [0.25*crop] + [0.25*veg]) / (0.5 + [0.25] + [0.25]).
func kernelUIntegrated(in [][]float64, out [][]int16, _ Params) {
for i := range out[0] {
water, soil, crop, veg := in[0][i], in[1][i], in[2][i], in[3][i]
cropOK, vegOK := valid(crop), valid(veg)
if !(valid(water) && valid(soil) && (cropOK || vegOK)) {
out[0][i] = NoData
continue
}
// explicit conversions forbid FMA fusion so results match numpy's per-op rounding
num := float32(float32(0.25)*float32(water)) + float32(float32(0.25)*float32(soil))
den := float32(0.5)
if cropOK {
num += float32(float32(0.25) * float32(crop))
den += 0.25
}
if vegOK {
num += float32(float32(0.25) * float32(veg))
den += 0.25
}
out[0][i] = classify(num / den)
}
}
// kernelZMax: all four valid -> max.
func kernelZMax(in [][]float64, out [][]int16, _ Params) {
for i := range out[0] {
m, ok := int16(0), true
for _, layer := range in {
v := layer[i]
if !valid(v) {
ok = false
break
}
if int16(v) > m {
m = int16(v)
}
}
if !ok {
m = NoData
}
out[0][i] = m
}
}
// kernelUMax: water and soil required plus one of crop/veg; max over valid layers.
func kernelUMax(in [][]float64, out [][]int16, _ Params) {
for i := range out[0] {
water, soil, crop, veg := in[0][i], in[1][i], in[2][i], in[3][i]
cropOK, vegOK := valid(crop), valid(veg)
if !(valid(water) && valid(soil) && (cropOK || vegOK)) {
out[0][i] = NoData
continue
}
m := max(int16(water), int16(soil))
if cropOK {
m = max(m, int16(crop))
}
if vegOK {
m = max(m, int16(veg))
}
out[0][i] = m
}
}
// kernelE: Z and U valid -> max(Z, U).
func kernelE(in [][]float64, out [][]int16, _ Params) {
for i := range out[0] {
z, u := in[0][i], in[1][i]
if !(valid(z) && valid(u)) {
out[0][i] = NoData
continue
}
out[0][i] = max(int16(z), int16(u))
}
}
// kernelCConflicts: inputs P1..P5, E; output j = min(Pj, E) where both valid.
func kernelCConflicts(in [][]float64, out [][]int16, _ Params) {
e := in[5]
for j := 0; j < 5; j++ {
p := in[j]
for i := range out[j] {
if !(valid(p[i]) && valid(e[i])) {
out[j][i] = NoData
continue
}
out[j][i] = min(int16(p[i]), int16(e[i]))
}
}
}
// kernelMaxAnyValid: max over the valid inputs; NoData when none is valid.
// Used for P_max and C_max.
func kernelMaxAnyValid(in [][]float64, out [][]int16, _ Params) {
for i := range out[0] {
m, any := int16(0), false
for _, layer := range in {
if v := layer[i]; valid(v) {
any = true
m = max(m, int16(v))
}
}
if !any {
m = NoData
}
out[0][i] = m
}
}
// kernelFuncZoning implements TZ_functional_zoning_go.md §4. Inputs: E, then
// S1..S5 (irrigation, snow, crop, pasture, tourism). Outputs: V1, V2, V3.
func kernelFuncZoning(in [][]float64, out [][]int16, p Params) {
e := in[0]
// The int16 conversions below cannot overflow (gosec G115): thresholds and
// tie_priority codes are limited to 1..5 by Params.Validate, and pixel
// values are converted only after valid() has bounded them to 1..5.
rules := [3]func(e, s int16) bool{
func(e, s int16) bool {
return (e <= 2 && s >= int16(p.V1MinSE12)) || (e == 3 && s >= int16(p.V1MinSE3)) // #nosec G115 -- 1..5, see above
},
func(e, s int16) bool {
return e <= 2 || (e == 3 && s >= int16(p.V2MinSE3)) || (e == 4 && s >= int16(p.V2MinSE4)) // #nosec G115 -- 1..5, see above
},
func(e, s int16) bool { return e <= 4 },
}
var s [5]int16
var base [5]bool
for i := range out[0] {
if !valid(e[i]) {
out[0][i], out[1][i], out[2][i] = NoData, NoData, NoData
continue
}
ev := int16(e[i])
anyBase := false
for k := 0; k < 5; k++ {
base[k] = valid(in[k+1][i])
if base[k] {
s[k] = int16(in[k+1][i])
anyBase = true
}
}
if !anyBase {
out[0][i], out[1][i], out[2][i] = NoData, NoData, NoData
continue
}
for v := 0; v < 3; v++ {
var cand [5]bool
maxS, anyCand := int16(0), false
for k := 0; k < 5; k++ {
cand[k] = base[k] && rules[v](ev, s[k])
if cand[k] {
anyCand = true
if s[k] > maxS {
maxS = s[k]
}
}
}
if !anyCand {
out[v][i] = 1 // regulated
continue
}
sector := int16(0)
for _, code := range p.TiePriority {
k := code - 1
if cand[k] && s[k] == maxS {
sector = int16(code) // #nosec G115 -- tie_priority code is 1..5, see above
break
}
}
out[v][i] = sector + 1
}
}
}

View File

@@ -0,0 +1,123 @@
package landscape
import (
"reflect"
"testing"
)
func run(k Kernel, nOut int, in ...[]float64) [][]int16 {
out := make([][]int16, nOut)
for j := range out {
out[j] = make([]int16, len(in[0]))
}
k(in, out, DefaultParams())
return out
}
func TestClassify(t *testing.T) {
cases := map[float32]int16{1.0: 1, 1.8: 1, 1.81: 2, 2.6: 2, 2.75: 3, 3.4: 3, 3.5: 4, 4.2: 4, 4.25: 5, 5.0: 5}
for x, want := range cases {
if got := classify(x); got != want {
t.Errorf("classify(%v)=%d want %d", x, got, want)
}
}
if classify(0.9) != NoData || classify(5.1) != NoData {
t.Error("out of range must be NoData")
}
}
func TestKernelZIntegrated(t *testing.T) {
// pixel0: all valid (1,2,3,4) -> 2.5 -> class 2 ; pixel1: one nodata -> NoData ; pixel2: all 5 -> 5 -> class 5
got := run(kernelZIntegrated, 1,
[]float64{1, -9999, 5}, []float64{2, 2, 5}, []float64{3, 3, 5}, []float64{4, 4, 5})
if want := []int16{2, NoData, 5}; !reflect.DeepEqual(got[0], want) {
t.Fatalf("got %v want %v", got[0], want)
}
}
func TestKernelUIntegrated_Applicability(t *testing.T) {
// order: water, soil, crop, veg
// p0: all valid 1,1,5,5 -> (0.25+0.25+1.25+1.25)/1.0 = 3.0 -> class 3
// p1: crop invalid: 1,1,-,5 -> (0.25+0.25+1.25)/0.75 = 2.333 -> class 2
// p2: crop and veg invalid -> NoData
// p3: water invalid -> NoData
got := run(kernelUIntegrated, 1,
[]float64{1, 1, 1, -9999}, []float64{1, 1, 1, 1}, []float64{5, 0, 0, 5}, []float64{5, 5, 0, 5})
if want := []int16{3, 2, NoData, NoData}; !reflect.DeepEqual(got[0], want) {
t.Fatalf("got %v want %v", got[0], want)
}
}
func TestKernelZMax_RequiresAllValid(t *testing.T) {
got := run(kernelZMax, 1, []float64{1, 1}, []float64{4, 4}, []float64{2, -9999}, []float64{3, 3})
if want := []int16{4, NoData}; !reflect.DeepEqual(got[0], want) {
t.Fatalf("got %v want %v", got[0], want)
}
}
func TestKernelUMax_IgnoresInvalidCropVeg(t *testing.T) {
// water, soil, crop, veg ; p0: 1,2,-,5 -> 5 ; p1: 3,2,0,0 -> NoData (needs crop or veg) ; p2: 3,2,1,0 -> 3
got := run(kernelUMax, 1, []float64{1, 3, 3}, []float64{2, 2, 2}, []float64{0, 0, 1}, []float64{5, 0, 0})
if want := []int16{5, NoData, 3}; !reflect.DeepEqual(got[0], want) {
t.Fatalf("got %v want %v", got[0], want)
}
}
func TestKernelE(t *testing.T) {
got := run(kernelE, 1, []float64{2, 5, -9999}, []float64{4, 1, 3})
if want := []int16{4, 5, NoData}; !reflect.DeepEqual(got[0], want) {
t.Fatalf("got %v want %v", got[0], want)
}
}
func TestKernelCConflicts(t *testing.T) {
// 5 P then E; p0: E=3 -> min(P,3); p1: E invalid -> all NoData; p2: P2 invalid
e := []float64{3, 0, 4}
got := run(kernelCConflicts, 5,
[]float64{5, 5, 5}, []float64{1, 1, 7}, []float64{3, 3, 3}, []float64{4, 4, 4}, []float64{2, 2, 2}, e)
want := [][]int16{{3, NoData, 4}, {1, NoData, NoData}, {3, NoData, 3}, {3, NoData, 4}, {2, NoData, 2}}
if !reflect.DeepEqual(got, want) {
t.Fatalf("got %v want %v", got, want)
}
}
func TestKernelMaxAnyValid(t *testing.T) {
got := run(kernelMaxAnyValid, 1, []float64{1, 0, 0}, []float64{4, 0, 2}, []float64{-9999, 0, 0})
if want := []int16{4, NoData, 2}; !reflect.DeepEqual(got[0], want) {
t.Fatalf("got %v want %v", got[0], want)
}
}
// TZ §7 edge cases. Inputs: E, then C1..C5 (irrigation, snow, crop, pasture, tourism).
func TestKernelFuncZoning_TZEdgeCases(t *testing.T) {
e := []float64{0, 3, 5, 4, 2, 2, 3}
s1 := []float64{3, 0, 3, 5, 3, 2, 4}
s2 := []float64{3, 0, 3, 5, 3, 2, 4}
s3 := []float64{3, 0, 3, 2, 3, 2, 5}
s4 := []float64{3, 0, 3, 2, 3, 2, 1}
s5 := []float64{3, 0, 3, 2, 3, 2, 1}
got := run(kernelFuncZoning, 3, e, s1, s2, s3, s4, s5)
// pixel: 0 E invalid -> NoData ; 1 all S invalid -> NoData ; 2 E=5 -> regulated(1) everywhere ;
// 3 E=4, S=5 for sectors 1,2 -> V1 regulated, V2/V3 sector 1 -> class 2 ;
// 4 all S equal (3), E=2 -> irrigation (2) in all ; 5 E=2, S=2 -> V1 regulated (S<3), V2 best=irrigation (2), V3 irrigation (2)
// 6 E=3: S=[4,4,5,1,1] -> V1 (S>=4): winners 1,2,3 max 5 -> crop (4); V2 (S>=3) -> crop (4); V3 -> crop (4)
want := [][]int16{
{NoData, NoData, 1, 1, 2, 1, 4}, // V1
{NoData, NoData, 1, 2, 2, 2, 4}, // V2
{NoData, NoData, 1, 2, 2, 2, 4}, // V3
}
if !reflect.DeepEqual(got, want) {
t.Fatalf("got %v want %v", got, want)
}
}
func TestKernelFuncZoning_TiePriorityIsConfigurable(t *testing.T) {
in := [][]float64{{2}, {3}, {3}, {3}, {3}, {3}}
out := [][]int16{{0}, {0}, {0}}
p := DefaultParams()
p.TiePriority = []int{5, 4, 3, 2, 1}
kernelFuncZoning(in, out, p)
if out[0][0] != 6 || out[1][0] != 6 || out[2][0] != 6 {
t.Fatalf("expected tourism (6) everywhere, got %v", out)
}
}

View File

@@ -0,0 +1,81 @@
package landscape
import (
"errors"
"fmt"
)
// Params are the run-time knobs of the graph (TZ §6): window size, tie
// priority for functional zoning, and the V1/V2 suitability thresholds.
type Params struct {
WindowSize int `json:"window_size"`
TiePriority []int `json:"tie_priority"`
V1MinSE12 int `json:"v1_min_s_e12"`
V1MinSE3 int `json:"v1_min_s_e3"`
V2MinSE3 int `json:"v2_min_s_e3"`
V2MinSE4 int `json:"v2_min_s_e4"`
}
// DefaultParams returns the TZ defaults.
func DefaultParams() Params {
return Params{WindowSize: 2048, TiePriority: []int{1, 2, 3, 4, 5}, V1MinSE12: 3, V1MinSE3: 4, V2MinSE3: 3, V2MinSE4: 5}
}
// WithDefaults fills zero-valued fields from DefaultParams.
func (p Params) WithDefaults() Params {
d := DefaultParams()
if p.WindowSize == 0 {
p.WindowSize = d.WindowSize
}
if len(p.TiePriority) == 0 {
p.TiePriority = d.TiePriority
}
if p.V1MinSE12 == 0 {
p.V1MinSE12 = d.V1MinSE12
}
if p.V1MinSE3 == 0 {
p.V1MinSE3 = d.V1MinSE3
}
if p.V2MinSE3 == 0 {
p.V2MinSE3 = d.V2MinSE3
}
if p.V2MinSE4 == 0 {
p.V2MinSE4 = d.V2MinSE4
}
return p
}
// Validate checks ranges: window 256..8192, thresholds 1..5, tie_priority a
// permutation of 1..5.
func (p Params) Validate() error {
var errs []error
if p.WindowSize < 256 || p.WindowSize > 8192 {
errs = append(errs, fmt.Errorf("window_size must be in 256..8192"))
}
for name, v := range map[string]int{"v1_min_s_e12": p.V1MinSE12, "v1_min_s_e3": p.V1MinSE3, "v2_min_s_e3": p.V2MinSE3, "v2_min_s_e4": p.V2MinSE4} {
if v < 1 || v > 5 {
errs = append(errs, fmt.Errorf("%s must be in 1..5", name))
}
}
seen := map[int]bool{}
for _, k := range p.TiePriority {
if k < 1 || k > 5 || seen[k] {
errs = append(errs, fmt.Errorf("tie_priority must be a permutation of 1..5"))
break
}
seen[k] = true
}
if len(p.TiePriority) != 5 && len(seen) == len(p.TiePriority) {
errs = append(errs, fmt.Errorf("tie_priority must list all of 1..5"))
}
return errors.Join(errs...)
}
// ForNode keeps only the fields that influence the node's result (window size
// never does), so fingerprints do not change when irrelevant knobs change.
func (p Params) ForNode(code string) Params {
if code == "30" {
return Params{TiePriority: p.TiePriority, V1MinSE12: p.V1MinSE12, V1MinSE3: p.V1MinSE3, V2MinSE3: p.V2MinSE3, V2MinSE4: p.V2MinSE4}
}
return Params{}
}

View File

@@ -0,0 +1,58 @@
package landscape
import (
"reflect"
"testing"
)
func TestDefaultParamsValidate(t *testing.T) {
if err := DefaultParams().Validate(); err != nil {
t.Fatalf("DefaultParams().Validate() = %v, want nil", err)
}
}
func TestWithDefaultsFillsZeroFields(t *testing.T) {
got := Params{}.WithDefaults()
want := DefaultParams()
if !reflect.DeepEqual(got, want) {
t.Fatalf("Params{}.WithDefaults() = %+v, want %+v", got, want)
}
}
func TestValidate_TiePriorityDuplicateInvalid(t *testing.T) {
p := DefaultParams()
p.TiePriority = []int{1, 1, 2, 3, 4}
if err := p.Validate(); err == nil {
t.Fatal("expected error for tie_priority with duplicate values")
}
}
func TestValidate_WindowSizeOutOfRangeInvalid(t *testing.T) {
p := DefaultParams()
p.WindowSize = 100
if err := p.Validate(); err == nil {
t.Fatal("expected error for window_size below 256")
}
}
func TestForNode_NonFuncZoningNodeIsZero(t *testing.T) {
p := DefaultParams()
got := p.ForNode("00")
if !reflect.DeepEqual(got, Params{}) {
t.Fatalf("ForNode(00) = %+v, want zero value", got)
}
}
func TestForNode_FuncZoningHasNoWindowSize(t *testing.T) {
p := DefaultParams()
got := p.ForNode("30")
if got.WindowSize != 0 {
t.Fatalf("ForNode(30).WindowSize = %d, want 0", got.WindowSize)
}
if !reflect.DeepEqual(got.TiePriority, p.TiePriority) {
t.Fatalf("ForNode(30).TiePriority = %v, want %v", got.TiePriority, p.TiePriority)
}
if got.V1MinSE12 != p.V1MinSE12 || got.V1MinSE3 != p.V1MinSE3 || got.V2MinSE3 != p.V2MinSE3 || got.V2MinSE4 != p.V2MinSE4 {
t.Fatalf("ForNode(30) thresholds = %+v, want matching %+v", got, p)
}
}

View File

@@ -0,0 +1,94 @@
package rabbitmq
import (
"context"
"encoding/json"
"fmt"
"log/slog"
"time"
"github.com/google/uuid"
amqp "github.com/rabbitmq/amqp091-go"
)
const (
// LandscapeRunRoutingKey routes Landscape Program run jobs.
LandscapeRunRoutingKey = "landscape.run"
// LandscapeRunQueue receives Landscape Program run jobs.
LandscapeRunQueue = "gis.landscape.run"
// markRunFailedTimeout bounds the best-effort terminal status write made
// after Execute failed, on a context detached from the delivery's.
markRunFailedTimeout = 10 * time.Second
)
// LandscapeJob is the message body for a Landscape Program run job.
type LandscapeJob struct {
RunID uuid.UUID `json:"run_id"`
}
// LandscapeJobPublisher publishes Landscape Program run jobs; it implements
// service.LandscapeJobEnqueuer.
type LandscapeJobPublisher struct {
pub *Publisher
}
// NewLandscapeJobPublisher returns a LandscapeJobPublisher.
func NewLandscapeJobPublisher(pub *Publisher) *LandscapeJobPublisher {
return &LandscapeJobPublisher{pub: pub}
}
// EnqueueRun publishes an execution job for the given run.
func (p *LandscapeJobPublisher) EnqueueRun(ctx context.Context, runID uuid.UUID) error {
body, err := json.Marshal(LandscapeJob{RunID: runID})
if err != nil {
return err
}
return p.pub.Publish(ctx, LandscapeRunRoutingKey, body)
}
// LandscapeProcessor executes Landscape Program runs; implemented by the service.
type LandscapeProcessor interface {
Execute(ctx context.Context, runID uuid.UUID) error
// MarkRunFailed records a terminal failure on the run so it does not remain
// stuck in an in-progress status.
MarkRunFailed(ctx context.Context, runID uuid.UUID, reason string) error
}
// landscapeHandler executes one Landscape Program run per delivery.
type landscapeHandler struct {
proc LandscapeProcessor
log *slog.Logger
}
func (h landscapeHandler) Handle(ctx context.Context, d amqp.Delivery) error {
var job LandscapeJob
if err := json.Unmarshal(d.Body, &job); err != nil {
return fmt.Errorf("decode landscape run job: %w", err)
}
h.log.Info("processing landscape run", "run_id", job.RunID)
err := h.proc.Execute(ctx, job.RunID)
if err != nil {
// The delivery is nacked without requeue (see Consumer.dispatch), so the
// run will not be retried. Record the failure on the run itself;
// otherwise it stays "running" forever. This covers the errors Execute
// returns before it can mark the run itself (a missing run, a failed
// status write), since Execute already marks the run failed for every
// node failure. Best-effort: log if marking fails, but still return the
// original error so the delivery is nacked.
// Detached from ctx: Execute may well have failed *because* ctx was
// cancelled (shutdown), and the write must still land or the run stays
// "running" forever.
mctx, cancel := context.WithTimeout(context.WithoutCancel(ctx), markRunFailedTimeout)
if markErr := h.proc.MarkRunFailed(mctx, job.RunID, err.Error()); markErr != nil {
h.log.Error("mark landscape run failed", "run_id", job.RunID, "error", markErr)
}
cancel()
}
return err
}
// NewLandscapeRunHandler returns a handler that executes Landscape Program runs.
func NewLandscapeRunHandler(p LandscapeProcessor, log *slog.Logger) Handler {
return landscapeHandler{proc: p, log: log}
}

View File

@@ -0,0 +1,90 @@
package rabbitmq
import (
"context"
"encoding/json"
"errors"
"testing"
"github.com/google/uuid"
amqp "github.com/rabbitmq/amqp091-go"
)
// fakeLandscapeProcessor records MarkRunFailed calls and returns a preset error
// from Execute.
type fakeLandscapeProcessor struct {
runErr error
executed []uuid.UUID
failedID uuid.UUID
failedWith string
failedCnt int
}
func (f *fakeLandscapeProcessor) Execute(_ context.Context, runID uuid.UUID) error {
f.executed = append(f.executed, runID)
return f.runErr
}
func (f *fakeLandscapeProcessor) MarkRunFailed(_ context.Context, runID uuid.UUID, reason string) error {
f.failedCnt++
f.failedID = runID
f.failedWith = reason
return nil
}
func landscapeDelivery(t *testing.T, id uuid.UUID) amqp.Delivery {
t.Helper()
body, err := json.Marshal(LandscapeJob{RunID: id})
if err != nil {
t.Fatalf("marshal job: %v", err)
}
return amqp.Delivery{Body: body}
}
func TestLandscapeHandler_MarksRunFailedOnExecuteError(t *testing.T) {
id := uuid.New()
fp := &fakeLandscapeProcessor{runErr: errors.New("node 00: grid mismatch")}
h := NewLandscapeRunHandler(fp, discardLogger())
err := h.Handle(context.Background(), landscapeDelivery(t, id))
if err == nil {
t.Fatal("expected the run error to be returned so the delivery is nacked")
}
if len(fp.executed) != 1 || fp.executed[0] != id {
t.Fatalf("Execute called with %v, want [%s]", fp.executed, id)
}
if fp.failedCnt != 1 {
t.Fatalf("expected MarkRunFailed to be called once, got %d", fp.failedCnt)
}
if fp.failedID != id {
t.Fatalf("MarkRunFailed called with wrong id: got %s want %s", fp.failedID, id)
}
if fp.failedWith != "node 00: grid mismatch" {
t.Fatalf("MarkRunFailed called with wrong reason: %q", fp.failedWith)
}
}
func TestLandscapeHandler_DoesNotMarkFailedOnSuccess(t *testing.T) {
fp := &fakeLandscapeProcessor{runErr: nil}
h := NewLandscapeRunHandler(fp, discardLogger())
if err := h.Handle(context.Background(), landscapeDelivery(t, uuid.New())); err != nil {
t.Fatalf("unexpected error: %v", err)
}
if fp.failedCnt != 0 {
t.Fatalf("expected MarkRunFailed not to be called on success, got %d", fp.failedCnt)
}
}
func TestLandscapeHandler_UndecodableBodyIsNacked(t *testing.T) {
fp := &fakeLandscapeProcessor{}
h := NewLandscapeRunHandler(fp, discardLogger())
err := h.Handle(context.Background(), amqp.Delivery{Body: []byte("not json")})
if err == nil {
t.Fatal("expected a decode error so the delivery is nacked")
}
if len(fp.executed) != 0 {
t.Fatalf("Execute must not run for an undecodable body, got %v", fp.executed)
}
}

View File

@@ -112,7 +112,9 @@ func gpkgScan(db *sql.DB, table string, names []string, limit int) []map[string]
for i, n := range names {
quoted[i] = quoteIdent(n)
}
query := fmt.Sprintf("SELECT %s FROM %s", strings.Join(quoted, ", "), quoteIdent(table))
// Identifiers cannot be bound as parameters; every one is quoted with
// quoteIdent (embedded quotes doubled) and the LIMIT is an int.
query := fmt.Sprintf("SELECT %s FROM %s", strings.Join(quoted, ", "), quoteIdent(table)) // #nosec G201 -- identifiers quoted by quoteIdent
if limit >= 0 {
query += fmt.Sprintf(" LIMIT %d", limit)
}

View File

@@ -25,7 +25,8 @@ func NewGDALConverter() *GDALConverter {
// ToCOG converts the source raster to a Cloud-Optimized GeoTIFF at dst. The COG
// driver builds internal tiling and overviews.
func (c *GDALConverter) ToCOG(ctx context.Context, src, dst string) error {
cmd := exec.CommandContext(ctx, "gdal_translate",
// Fixed binary, no shell; src/dst are service-generated temp-file paths.
cmd := exec.CommandContext(ctx, "gdal_translate", // #nosec G204 -- see above
"-of", "COG",
"-co", "COMPRESS="+c.compression,
src, dst,
@@ -41,7 +42,7 @@ func (c *GDALConverter) ToCOG(ctx context.Context, src, dst string) error {
// Footprint returns the raster's footprint as a GeoJSON polygon in EPSG:4326, or
// nil if the raster has no spatial reference.
func (c *GDALConverter) Footprint(ctx context.Context, src string) ([]byte, error) {
out, err := exec.CommandContext(ctx, "gdalinfo", "-json", src).Output()
out, err := exec.CommandContext(ctx, "gdalinfo", "-json", src).Output() // #nosec G204 -- fixed binary, no shell; src is a service temp file
if err != nil {
return nil, fmt.Errorf("gdalinfo: %w", err)
}
@@ -58,18 +59,22 @@ func (c *GDALConverter) Footprint(ctx context.Context, src string) ([]byte, erro
return info.Wgs84Extent, nil
}
// VectorGeometry reads every feature of a vector file and returns their combined
// geometry as a GeoJSON GeometryCollection reprojected to EPSG:4326, or nil if
// the file has no features. The caller (PostGIS) dissolves the collection into
// the union of all features. Zipped ESRI shapefiles are read in place via GDAL's
// /vsizip/ virtual filesystem; GeoJSON and GeoPackage are read directly.
func (c *GDALConverter) VectorGeometry(ctx context.Context, src string) ([]byte, error) {
// VectorFeatures reads every feature of a vector file and returns their
// geometries, reprojected to EPSG:4326, in file order (nil if the file has no
// features). A feature with no geometry yields a nil entry at its index, so
// the result stays aligned with attribute rows read from the same file (the
// shapefile format itself has no other way to pair a .shp geometry record with
// its .dbf attribute row than matching by position). Zipped ESRI shapefiles
// are read in place via GDAL's /vsizip/ virtual filesystem; GeoJSON and
// GeoPackage are read directly.
func (c *GDALConverter) VectorFeatures(ctx context.Context, src string) ([]json.RawMessage, error) {
input := src
if strings.EqualFold(filepath.Ext(src), ".zip") {
input = "/vsizip/" + src
}
cmd := exec.CommandContext(ctx, "ogr2ogr",
// Fixed binary, no shell; input is a service-generated temp-file path.
cmd := exec.CommandContext(ctx, "ogr2ogr", // #nosec G204 -- see above
"-f", "GeoJSON",
"-t_srs", "EPSG:4326",
"/vsistdout/", input,
@@ -89,20 +94,16 @@ func (c *GDALConverter) VectorGeometry(ctx context.Context, src string) ([]byte,
if err := json.Unmarshal(out, &fc); err != nil {
return nil, fmt.Errorf("parse ogr2ogr output: %w", err)
}
geoms := make([]json.RawMessage, 0, len(fc.Features))
for _, f := range fc.Features {
if len(f.Geometry) == 0 || string(f.Geometry) == "null" {
continue
}
geoms = append(geoms, f.Geometry)
}
if len(geoms) == 0 {
if len(fc.Features) == 0 {
return nil, nil
}
return json.Marshal(struct {
Type string `json:"type"`
Geometries []json.RawMessage `json:"geometries"`
}{Type: "GeometryCollection", Geometries: geoms})
geoms := make([]json.RawMessage, len(fc.Features))
for i, f := range fc.Features {
if len(f.Geometry) == 0 || string(f.Geometry) == "null" {
continue
}
geoms[i] = f.Geometry
}
return geoms, nil
}

140
internal/raster/godal.go Normal file
View File

@@ -0,0 +1,140 @@
package raster
import (
"context"
"fmt"
"math"
"sync"
"github.com/airbusgeo/godal"
)
var registerOnce sync.Once
// GodalIO implements IO with the GDAL bindings (github.com/airbusgeo/godal).
// GDAL datasets are not goroutine-safe: callers must not share a Reader or
// Writer between goroutines without external synchronisation.
type GodalIO struct{}
// NewGodalIO registers GDAL drivers (once) and returns a GodalIO.
func NewGodalIO() *GodalIO {
registerOnce.Do(godal.RegisterAll)
return &GodalIO{}
}
// DefaultWriterOptions returns the Landscape Program output profile.
func DefaultWriterOptions() WriterOptions {
return WriterOptions{NoData: -9999, TileSize: 512, Compression: "DEFLATE"}
}
type godalReader struct {
ds *godal.Dataset
grid Grid
}
func (g *GodalIO) Open(_ context.Context, path string) (Reader, error) {
ds, err := godal.Open(path)
if err != nil {
return nil, fmt.Errorf("open raster %s: %w", path, err)
}
st := ds.Structure()
if st.NBands < 1 {
ds.Close()
return nil, fmt.Errorf("open raster %s: no bands", path)
}
gt, err := ds.GeoTransform()
if err != nil {
ds.Close()
return nil, fmt.Errorf("open raster %s: geotransform: %w", path, err)
}
return &godalReader{ds: ds, grid: Grid{Width: st.SizeX, Height: st.SizeY, GeoTransform: gt, ProjectionWKT: ds.Projection()}}, nil
}
func (r *godalReader) Grid() Grid { return r.grid }
func (r *godalReader) NoData() (float64, bool) { return r.ds.Bands()[0].NoData() }
func (r *godalReader) ReadFloat64(w Window, dst []float64) error {
if len(dst) < w.Width*w.Height {
return fmt.Errorf("read window: dst has %d elements, need %d", len(dst), w.Width*w.Height)
}
return r.ds.Bands()[0].Read(w.X, w.Y, dst[:w.Width*w.Height], w.Width, w.Height)
}
func (r *godalReader) Close() error { return r.ds.Close() }
type godalWriter struct{ ds *godal.Dataset }
func (g *GodalIO) Create(_ context.Context, path string, grid Grid, o WriterOptions) (Writer, error) {
return createGTiff(path, grid, o, "IF_SAFER")
}
// createGTiff writes a tiled Int16 GeoTIFF. bigTIFF is the value of the
// BIGTIFF creation option ("IF_SAFER" in production); it is a parameter so
// tests can force a BigTIFF header.
func createGTiff(path string, grid Grid, o WriterOptions, bigTIFF string) (Writer, error) {
tile := fmt.Sprint(o.TileSize)
ds, err := godal.Create(godal.GTiff, path, 1, godal.Int16, grid.Width, grid.Height,
godal.CreationOption("TILED=YES", "BLOCKXSIZE="+tile, "BLOCKYSIZE="+tile,
"COMPRESS="+o.Compression, "PREDICTOR=2", "BIGTIFF="+bigTIFF))
if err != nil {
return nil, fmt.Errorf("create raster %s: %w", path, err)
}
if err := ds.SetGeoTransform(grid.GeoTransform); err != nil {
ds.Close()
return nil, fmt.Errorf("create raster %s: geotransform: %w", path, err)
}
if grid.ProjectionWKT != "" {
if err := ds.SetProjection(grid.ProjectionWKT); err != nil {
ds.Close()
return nil, fmt.Errorf("create raster %s: projection: %w", path, err)
}
}
if err := ds.Bands()[0].SetNoData(float64(o.NoData)); err != nil {
ds.Close()
return nil, fmt.Errorf("create raster %s: nodata: %w", path, err)
}
return &godalWriter{ds: ds}, nil
}
func (w *godalWriter) WriteInt16(win Window, src []int16) error {
if len(src) < win.Width*win.Height {
return fmt.Errorf("write window: src has %d elements, need %d", len(src), win.Width*win.Height)
}
return w.ds.Bands()[0].Write(win.X, win.Y, src[:win.Width*win.Height], win.Width, win.Height)
}
func (w *godalWriter) Close() error { return w.ds.Close() }
// SameGrid reports whether two grids describe the same pixels: equal size,
// geotransform within 1e-9 (absolute), and the same CRS (GDAL IsSame; an
// empty projection only matches an empty projection).
//
// ProjectionWKT is parsed with godal.NewSpatialRef rather than
// NewSpatialRefFromWKT: despite the field name, callers may pass any user
// projection string GDAL's SetProjection accepts (e.g. "EPSG:32642"), not
// only WKT, and NewSpatialRefFromWKT rejects non-WKT input.
func (g *GodalIO) SameGrid(a, b Grid) bool {
if a.Width != b.Width || a.Height != b.Height {
return false
}
for i := range a.GeoTransform {
if math.Abs(a.GeoTransform[i]-b.GeoTransform[i]) > 1e-9 {
return false
}
}
if a.ProjectionWKT == "" || b.ProjectionWKT == "" {
return a.ProjectionWKT == b.ProjectionWKT
}
sa, err := godal.NewSpatialRef(a.ProjectionWKT)
if err != nil {
return false
}
defer sa.Close()
sb, err := godal.NewSpatialRef(b.ProjectionWKT)
if err != nil {
return false
}
defer sb.Close()
return sa.IsSame(sb)
}

View File

@@ -0,0 +1,103 @@
package raster
import (
"bytes"
"context"
"io"
"os"
"path/filepath"
"testing"
"gis/internal/domain"
)
func TestGodalIO_RoundTrip(t *testing.T) {
rio := NewGodalIO()
ctx := context.Background()
path := filepath.Join(t.TempDir(), "out.tif")
grid := Grid{Width: 10, Height: 7, GeoTransform: [6]float64{100, 30, 0, 200, 0, -30}, ProjectionWKT: "EPSG:32642"}
w, err := rio.Create(ctx, path, grid, DefaultWriterOptions())
if err != nil {
t.Fatal(err)
}
full := make([]int16, 10*7)
for i := range full {
full[i] = int16(i%6) - 1 // -1..4 (-1 stands in for garbage, not nodata)
}
full[0] = -9999
if err := w.WriteInt16(Window{0, 0, 10, 7}, full); err != nil {
t.Fatal(err)
}
if err := w.Close(); err != nil {
t.Fatal(err)
}
r, err := rio.Open(ctx, path)
if err != nil {
t.Fatal(err)
}
defer r.Close()
g := r.Grid()
if g.Width != 10 || g.Height != 7 || g.GeoTransform != grid.GeoTransform || g.ProjectionWKT == "" {
t.Fatalf("grid mismatch: %+v", g)
}
if nd, ok := r.NoData(); !ok || nd != -9999 {
t.Fatalf("nodata = %v %v", nd, ok)
}
dst := make([]float64, 3*2)
if err := r.ReadFloat64(Window{X: 8, Y: 5, Width: 2, Height: 2}, dst[:4]); err != nil {
t.Fatal(err)
}
// pixel (8,5) has index 58 -> 58%6-1 = 3 ; (9,5)=59 -> 4 ; (8,6)=68 -> 1 ; (9,6)=69 -> 2
want := []float64{3, 4, 1, 2}
for i, v := range want {
if dst[i] != v {
t.Fatalf("pixel %d = %v want %v", i, dst[i], v)
}
}
if !rio.SameGrid(g, grid) {
t.Fatal("SameGrid should be true for identical grids")
}
other := grid
other.GeoTransform[0] += 1
if rio.SameGrid(g, other) {
t.Fatal("SameGrid should be false for shifted origin")
}
}
// TestGodalIO_BigTIFFPassesContentValidation guards the upload magic-byte
// check against the pipeline's own output: GDAL writes a BigTIFF header
// (version 43) whenever BIGTIFF is YES or IF_SAFER kicks in for a large
// raster, and domain.ValidateFileContent must accept it.
func TestGodalIO_BigTIFFPassesContentValidation(t *testing.T) {
path := filepath.Join(t.TempDir(), "big.tif")
grid := Grid{Width: 8, Height: 8, GeoTransform: [6]float64{100, 30, 0, 200, 0, -30}, ProjectionWKT: "EPSG:32642"}
w, err := createGTiff(path, grid, DefaultWriterOptions(), "YES")
if err != nil {
t.Fatal(err)
}
if err := w.WriteInt16(Window{0, 0, 8, 8}, make([]int16, 8*8)); err != nil {
t.Fatal(err)
}
if err := w.Close(); err != nil {
t.Fatal(err)
}
f, err := os.Open(path)
if err != nil {
t.Fatal(err)
}
defer f.Close()
head := make([]byte, 4)
if _, err := io.ReadFull(f, head); err != nil {
t.Fatal(err)
}
if !bytes.HasPrefix(head, []byte("II+\x00")) && !bytes.HasPrefix(head, []byte("MM\x00+")) {
t.Fatalf("BIGTIFF=YES did not produce a BigTIFF header: % x", head)
}
if err := domain.ValidateFileContent(".tif", head); err != nil {
t.Fatalf("ValidateFileContent rejected our own BigTIFF output: %v", err)
}
}

77
internal/raster/io.go Normal file
View File

@@ -0,0 +1,77 @@
package raster
import "context"
// Grid describes a raster's pixel grid: its size, geotransform, and
// projection. Two grids describe the same pixels when IO.SameGrid reports
// true for them.
type Grid struct {
Width, Height int
GeoTransform [6]float64
ProjectionWKT string
}
// Window is a rectangular block of pixels within a raster, in pixel/line
// coordinates with (0,0) at the top-left corner.
type Window struct {
X, Y, Width, Height int
}
// Reader reads pixel data and metadata from an existing raster.
type Reader interface {
// Grid returns the raster's pixel grid.
Grid() Grid
// NoData returns the raster's nodata value and whether one is set.
NoData() (float64, bool)
// ReadFloat64 reads the pixels in w into dst, row-major, converting the
// underlying pixel type to float64. len(dst) must be at least
// w.Width*w.Height.
ReadFloat64(w Window, dst []float64) error
// Close releases the underlying dataset handle.
Close() error
}
// WriterOptions configures a raster created by IO.Create.
type WriterOptions struct {
// NoData is the nodata value written to the output band.
NoData int16
// TileSize is the internal tile size (both block dimensions) in pixels.
TileSize int
// Compression is the GDAL COMPRESS creation option value.
Compression string
}
// Writer writes pixel data to a raster created by IO.Create.
type Writer interface {
// WriteInt16 writes src, row-major, into the pixels described by w.
// len(src) must be at least w.Width*w.Height.
WriteInt16(w Window, src []int16) error
// Close flushes and releases the underlying dataset handle.
Close() error
}
// IO is an engine-agnostic block I/O abstraction over raster storage. It lets
// callers open and create rasters without depending on a particular GDAL
// binding.
type IO interface {
// Open opens the raster at path for reading.
Open(ctx context.Context, path string) (Reader, error)
// Create creates a new single-band int16 raster at path with the given
// grid and writer options, ready for WriteInt16 calls.
Create(ctx context.Context, path string, g Grid, o WriterOptions) (Writer, error)
// SameGrid reports whether a and b describe the same pixel grid: equal
// size, geotransform, and coordinate reference system.
SameGrid(a, b Grid) bool
}
// Windows tiles a w×h raster into row-major windows of at most size×size.
func Windows(w, h, size int) []Window {
var out []Window
for y := 0; y < h; y += size {
wh := min(size, h-y)
for x := 0; x < w; x += size {
out = append(out, Window{X: x, Y: y, Width: min(size, w-x), Height: wh})
}
}
return out
}

View File

@@ -0,0 +1,61 @@
package raster
import (
"reflect"
"testing"
)
func TestWindows(t *testing.T) {
tests := []struct {
name string
w, h, size int
want []Window
}{
{
name: "10x10 size4 tiles into 9 windows",
w: 10, h: 10, size: 4,
want: []Window{
{X: 0, Y: 0, Width: 4, Height: 4},
{X: 4, Y: 0, Width: 4, Height: 4},
{X: 8, Y: 0, Width: 2, Height: 4},
{X: 0, Y: 4, Width: 4, Height: 4},
{X: 4, Y: 4, Width: 4, Height: 4},
{X: 8, Y: 4, Width: 2, Height: 4},
{X: 0, Y: 8, Width: 4, Height: 2},
{X: 4, Y: 8, Width: 4, Height: 2},
{X: 8, Y: 8, Width: 2, Height: 2},
},
},
{
name: "5x3 size8 fits in one window",
w: 5, h: 3, size: 8,
want: []Window{
{X: 0, Y: 0, Width: 5, Height: 3},
},
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
got := Windows(tt.w, tt.h, tt.size)
if !reflect.DeepEqual(got, tt.want) {
t.Fatalf("Windows(%d, %d, %d) = %+v, want %+v", tt.w, tt.h, tt.size, got, tt.want)
}
if len(got) > 0 {
last := got[len(got)-1]
wantLast := tt.want[len(tt.want)-1]
if last != wantLast {
t.Fatalf("last window = %+v, want %+v", last, wantLast)
}
}
})
}
}
func TestDefaultWriterOptions(t *testing.T) {
got := DefaultWriterOptions()
want := WriterOptions{NoData: -9999, TileSize: 512, Compression: "DEFLATE"}
if got != want {
t.Fatalf("DefaultWriterOptions() = %+v, want %+v", got, want)
}
}

View File

@@ -27,21 +27,22 @@ func NewDatasetRepository(pool *pgxpool.Pool) *DatasetRepository {
// datasetColumns lists the dataset columns for SELECT and RETURNING. The
// geometry is exposed as GeoJSON (jsonb) rather than its raw EWKB form, and a
// bounding box array is derived for raster datasets only.
const datasetColumns = `id, category_id, name, description, unit, filename, storage_key, cog_storage_key, file_type, size_bytes, content_type, properties, meta, automated, status, attribute_columns, kato_column, year_columns, parse_error, ST_AsGeoJSON(geometry)::jsonb AS geometry,
const datasetColumns = `id, category_id, dg_id, to_char(start_date, 'YYYY-MM-DD') AS start_date, to_char(end_date, 'YYYY-MM-DD') AS end_date, name, description, unit, filename, storage_key, cog_storage_key, file_type, size_bytes, content_type, properties, meta, automated, status, attribute_columns, kato_column, year_columns, parse_error, ST_AsGeoJSON(geometry)::jsonb AS geometry,
CASE WHEN file_type = 'raster' AND geometry IS NOT NULL
THEN ARRAY[ST_XMin(geometry), ST_YMin(geometry), ST_XMax(geometry), ST_YMax(geometry)]
ELSE NULL END AS bbox,
geojson,
geojson, sld_storage_key, sld_filename, sld_content_type,
created_at, updated_at`
func scanDataset(row pgx.Row) (domain.Dataset, error) {
var d domain.Dataset
err := row.Scan(
&d.ID, &d.CategoryID, &d.Name, &d.Description, &d.Unit,
&d.ID, &d.CategoryID, &d.DatasetGroupID, &d.StartDate, &d.EndDate, &d.Name, &d.Description, &d.Unit,
&d.Filename, &d.StorageKey, &d.CogStorageKey, &d.FileType, &d.SizeBytes, &d.ContentType,
&d.Properties, &d.Meta, &d.Automated, &d.Status,
&d.AttributeColumns, &d.KatoColumn, &d.YearColumns, &d.ParseError,
&d.Geometry, &d.BBox, &d.GeoJSON, &d.CreatedAt, &d.UpdatedAt,
&d.Geometry, &d.BBox, &d.GeoJSON, &d.SldStorageKey, &d.SldFilename, &d.SldContentType,
&d.CreatedAt, &d.UpdatedAt,
)
return d, err
}
@@ -58,11 +59,28 @@ func nullableJSON(raw json.RawMessage) any {
// Create inserts a new dataset and returns the stored row.
func (r *DatasetRepository) Create(ctx context.Context, d domain.Dataset) (domain.Dataset, error) {
row := r.pool.QueryRow(ctx,
`INSERT INTO datasets (category_id, name, description, unit, filename, storage_key, file_type, size_bytes, content_type, properties, meta, automated, status)
VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13)
`INSERT INTO datasets (category_id, name, description, unit, filename, storage_key, file_type, size_bytes, content_type, properties, meta, automated, status, dg_id, start_date, end_date)
VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13, $14, $15::date, $16::date)
RETURNING `+datasetColumns,
d.CategoryID, d.Name, d.Description, d.Unit, d.Filename, d.StorageKey, d.FileType, d.SizeBytes, d.ContentType,
nullableJSON(d.Properties), nullableJSON(d.Meta), d.Automated, d.Status,
d.DatasetGroupID, d.StartDate, d.EndDate,
)
out, err := scanDataset(row)
return out, mapError(err)
}
// UpdateFields overwrites a dataset's editable metadata columns. Nil pointers
// clear the nullable columns. Returns domain.ErrNotFound if the dataset does
// not exist, or domain.ErrConflict if the category/group does not exist.
func (r *DatasetRepository) UpdateFields(ctx context.Context, id uuid.UUID, f domain.DatasetFields) (domain.Dataset, error) {
row := r.pool.QueryRow(ctx,
`UPDATE datasets
SET category_id = $2, name = $3, description = $4, unit = $5,
dg_id = $6, start_date = $7::date, end_date = $8::date, updated_at = now()
WHERE id = $1
RETURNING `+datasetColumns,
id, f.CategoryID, f.Name, f.Description, f.Unit, f.DatasetGroupID, f.StartDate, f.EndDate,
)
out, err := scanDataset(row)
return out, mapError(err)
@@ -211,6 +229,38 @@ func (r *DatasetRepository) SetGeoJSON(ctx context.Context, id uuid.UUID, geojso
return nil
}
// SetSLD stores the dataset's SLD (Styled Layer Descriptor) style file
// metadata, replacing any previous one, and returns the updated row.
func (r *DatasetRepository) SetSLD(ctx context.Context, id uuid.UUID, key, filename, contentType string) (domain.Dataset, error) {
row := r.pool.QueryRow(ctx,
`UPDATE datasets
SET sld_storage_key = $2, sld_filename = $3, sld_content_type = $4, updated_at = now()
WHERE id = $1
RETURNING `+datasetColumns,
id, key, filename, contentType,
)
out, err := scanDataset(row)
return out, mapError(err)
}
// ClearGeometryAndGeoJSON nulls out a dataset's stored geometry and persisted
// GeoJSON. Used before re-enqueuing a dataset's processing job so a stale
// (possibly incorrect) geometry/GeoJSON is never served while the job is
// pending; the async worker repopulates both once it completes.
func (r *DatasetRepository) ClearGeometryAndGeoJSON(ctx context.Context, id uuid.UUID) error {
tag, err := r.pool.Exec(ctx,
`UPDATE datasets SET geometry = NULL, geojson = NULL, updated_at = now() WHERE id = $1`,
id,
)
if err != nil {
return mapError(err)
}
if tag.RowsAffected() == 0 {
return domain.ErrNotFound
}
return nil
}
// ReplaceObservations atomically replaces all observations for a dataset.
func (r *DatasetRepository) ReplaceObservations(ctx context.Context, datasetID uuid.UUID, obs []domain.Observation) error {
tx, err := r.pool.Begin(ctx)
@@ -226,15 +276,23 @@ func (r *DatasetRepository) ReplaceObservations(ctx context.Context, datasetID u
if len(obs) > 0 {
rows := make([][]any, len(obs))
for i, o := range obs {
d, err := time.Parse("2006-01-02", o.Date)
d, err := time.Parse("2006-01-02", o.StartDate)
if err != nil {
return fmt.Errorf("invalid observation date %q: %w", o.Date, err)
return fmt.Errorf("invalid observation start_date %q: %w", o.StartDate, err)
}
rows[i] = []any{datasetID, o.KatoCode, d, o.Value, o.ValueText}
var endDate any
if o.EndDate != nil {
dt, err := time.Parse("2006-01-02", *o.EndDate)
if err != nil {
return fmt.Errorf("invalid observation end_date %q: %w", *o.EndDate, err)
}
endDate = dt
}
rows[i] = []any{datasetID, o.KatoCode, d, endDate, o.Value, o.ValueText}
}
_, err := tx.CopyFrom(ctx,
pgx.Identifier{"dataset_observations"},
[]string{"dataset_id", "kato_code", "date", "value", "value_text"},
[]string{"dataset_id", "kato_code", "start_date", "end_date", "value", "value_text"},
pgx.CopyFromRows(rows),
)
if err != nil {
@@ -245,16 +303,16 @@ func (r *DatasetRepository) ReplaceObservations(ctx context.Context, datasetID u
return mapError(tx.Commit(ctx))
}
const observationColumns = `id, dataset_id, kato_code, to_char(date, 'YYYY-MM-DD') AS date, value, value_text`
const observationColumns = `id, dataset_id, kato_code, to_char(start_date, 'YYYY-MM-DD') AS start_date, to_char(end_date, 'YYYY-MM-DD') AS end_date, value, value_text`
func scanObservation(row pgx.Row) (domain.Observation, error) {
var o domain.Observation
err := row.Scan(&o.ID, &o.DatasetID, &o.KatoCode, &o.Date, &o.Value, &o.ValueText)
err := row.Scan(&o.ID, &o.DatasetID, &o.KatoCode, &o.StartDate, &o.EndDate, &o.Value, &o.ValueText)
return o, err
}
// ListObservations returns a page of observations for a dataset, optionally
// filtered by KATO code, ordered by (kato_code, date).
// filtered by KATO code, ordered by (kato_code, start_date).
func (r *DatasetRepository) ListObservations(ctx context.Context, datasetID uuid.UUID, katoCode *string, limit, offset int) ([]domain.Observation, error) {
base := `SELECT ` + observationColumns + ` FROM dataset_observations WHERE dataset_id = $1`
@@ -264,11 +322,11 @@ func (r *DatasetRepository) ListObservations(ctx context.Context, datasetID uuid
)
if katoCode != nil {
rows, err = r.pool.Query(ctx,
base+` AND kato_code = $2 ORDER BY kato_code, date LIMIT $3 OFFSET $4`,
base+` AND kato_code = $2 ORDER BY kato_code, start_date LIMIT $3 OFFSET $4`,
datasetID, *katoCode, limit, offset)
} else {
rows, err = r.pool.Query(ctx,
base+` ORDER BY kato_code, date LIMIT $2 OFFSET $3`,
base+` ORDER BY kato_code, start_date LIMIT $2 OFFSET $3`,
datasetID, limit, offset)
}
if err != nil {
@@ -310,12 +368,25 @@ func (r *DatasetRepository) GetByID(ctx context.Context, id uuid.UUID) (domain.D
return out, mapError(err)
}
const datasetSummaryColumns = `id, category_id, name, description, unit, file_type, size_bytes, status, created_at, updated_at`
// LatestInGroup returns the freshest dataset of a group: greatest end_date
// (NULL end_date sorts last), ties broken by created_at DESC. status and
// fileType optionally restrict candidates. Returns domain.ErrNotFound when no
// dataset matches.
func (r *DatasetRepository) LatestInGroup(ctx context.Context, groupID uuid.UUID, status *string, fileType *domain.FileType) (domain.Dataset, error) {
where, args := datasetFilterClause(domain.DatasetFilter{DatasetGroupID: &groupID, Status: status, FileType: fileType}, nil)
row := r.pool.QueryRow(ctx,
`SELECT `+datasetColumns+` FROM datasets`+where+
` ORDER BY end_date DESC NULLS LAST, created_at DESC LIMIT 1`, args...)
out, err := scanDataset(row)
return out, mapError(err)
}
const datasetSummaryColumns = `id, category_id, dg_id, to_char(start_date, 'YYYY-MM-DD') AS start_date, to_char(end_date, 'YYYY-MM-DD') AS end_date, name, description, unit, file_type, size_bytes, status, created_at, updated_at`
func scanDatasetSummary(row pgx.Row) (domain.DatasetSummary, error) {
var d domain.DatasetSummary
err := row.Scan(
&d.ID, &d.CategoryID, &d.Name, &d.Description, &d.Unit,
&d.ID, &d.CategoryID, &d.DatasetGroupID, &d.StartDate, &d.EndDate, &d.Name, &d.Description, &d.Unit,
&d.FileType, &d.SizeBytes, &d.Status, &d.CreatedAt, &d.UpdatedAt,
)
return d, err
@@ -342,6 +413,22 @@ func datasetFilterClause(f domain.DatasetFilter, args []any) (string, []any) {
args = append(args, *f.Status)
conds = append(conds, fmt.Sprintf("status = $%d", len(args)))
}
if f.DatasetGroupID != nil {
args = append(args, *f.DatasetGroupID)
conds = append(conds, fmt.Sprintf("dg_id = $%d", len(args)))
}
for _, b := range []struct {
col, op string
val *string
}{
{"start_date", ">=", f.StartDateFrom}, {"start_date", "<=", f.StartDateTo},
{"end_date", ">=", f.EndDateFrom}, {"end_date", "<=", f.EndDateTo},
} {
if b.val != nil {
args = append(args, *b.val)
conds = append(conds, fmt.Sprintf("%s %s $%d::date", b.col, b.op, len(args)))
}
}
if len(conds) == 0 {
return "", args
}

View File

@@ -0,0 +1,115 @@
package postgres
import (
"context"
"errors"
"testing"
"time"
"gis/internal/domain"
"github.com/google/uuid"
)
// TestDatasetRepository_GroupsPeriodsAndLatest exercises the Postgres-only
// parts of the dataset/group repositories added with dataset periods: the
// ::date casts on *string parameters, the date filter clause, and the
// "freshest in group" ordering (end_date DESC NULLS LAST, created_at DESC).
func TestDatasetRepository_GroupsPeriodsAndLatest(t *testing.T) {
pool := testPool(t)
ctx := context.Background()
groups := NewDatasetGroupRepository(pool)
datasets := NewDatasetRepository(pool)
categories := NewCategoryRepository(pool)
suffix := uuid.NewString()[:8]
cat, err := categories.Create(ctx, domain.Category{Code: "dbtest-" + suffix, Name: "db test"})
if err != nil {
t.Fatalf("create category: %v", err)
}
t.Cleanup(func() { _, _ = pool.Exec(ctx, `DELETE FROM categories WHERE id = $1`, cat.ID) })
group, err := groups.Create(ctx, domain.DatasetGroup{Code: "dbtest-group-" + suffix, Name: "db test group"})
if err != nil {
t.Fatalf("create group: %v", err)
}
t.Cleanup(func() { _, _ = pool.Exec(ctx, `DELETE FROM dataset_groups WHERE id = $1`, group.ID) })
s := func(v string) *string { return &v }
mk := func(name string, start, end *string) domain.Dataset {
d, err := datasets.Create(ctx, domain.Dataset{
CategoryID: cat.ID, DatasetGroupID: &group.ID, Name: name, Filename: name + ".tif",
StorageKey: "dbtest/" + suffix + "/" + name, FileType: domain.FileTypeRaster,
ContentType: "image/tiff", Status: domain.DatasetStatusReady,
StartDate: start, EndDate: end,
})
if err != nil {
t.Fatalf("create dataset %s: %v", name, err)
}
t.Cleanup(func() { _, _ = pool.Exec(ctx, `DELETE FROM datasets WHERE id = $1`, d.ID) })
time.Sleep(5 * time.Millisecond) // distinct created_at for the tie-break
return d
}
old := mk("old", s("2019-01-01"), s("2019-12-31"))
fresh := mk("fresh", s("2020-01-01"), s("2020-12-31"))
undated := mk("undated", nil, nil)
if old.StartDate == nil || *old.StartDate != "2019-01-01" || *old.EndDate != "2019-12-31" || old.DatasetGroupID == nil {
t.Fatalf("dates/group not round-tripped: %+v", old)
}
latest, err := datasets.LatestInGroup(ctx, group.ID, nil, nil)
if err != nil {
t.Fatalf("LatestInGroup: %v", err)
}
if latest.ID != fresh.ID {
t.Fatalf("latest = %s, want fresh %s (undated %s must sort last)", latest.Name, fresh.ID, undated.ID)
}
ft := domain.FileTypeVector
if _, err := datasets.LatestInGroup(ctx, group.ID, nil, &ft); !errors.Is(err, domain.ErrNotFound) {
t.Fatalf("no vector in group: err = %v, want ErrNotFound", err)
}
page, err := datasets.ListSummaries(ctx, domain.DatasetFilter{
DatasetGroupID: &group.ID, EndDateFrom: s("2020-01-01"), EndDateTo: s("2020-12-31"),
}, 10, 0)
if err != nil {
t.Fatalf("ListSummaries with date bounds: %v", err)
}
if len(page) != 1 || page[0].ID != fresh.ID {
t.Fatalf("date filter returned %+v, want only fresh", page)
}
n, err := datasets.Count(ctx, domain.DatasetFilter{DatasetGroupID: &group.ID, StartDateFrom: s("2019-06-01")})
if err != nil || n != 1 {
t.Fatalf("Count(start_date_from) = %d, %v; want 1", n, err)
}
updated, err := datasets.UpdateFields(ctx, undated.ID, domain.DatasetFields{
CategoryID: cat.ID, Name: "undated-now-dated", DatasetGroupID: &group.ID,
StartDate: s("2021-01-01"), EndDate: s("2021-12-31"),
})
if err != nil {
t.Fatalf("UpdateFields: %v", err)
}
if updated.EndDate == nil || *updated.EndDate != "2021-12-31" {
t.Fatalf("UpdateFields did not store the period: %+v", updated)
}
if _, err := datasets.UpdateFields(ctx, undated.ID, domain.DatasetFields{
CategoryID: cat.ID, Name: "bad", StartDate: s("2021-12-31"), EndDate: s("2021-01-01"),
}); !errors.Is(err, domain.ErrValidation) {
t.Fatalf("reversed period must hit the CHECK constraint: err = %v", err)
}
if err := groups.Delete(ctx, group.ID); !errors.Is(err, domain.ErrConflict) {
t.Fatalf("deleting a referenced group: err = %v, want ErrConflict", err)
}
g2, err := groups.SetSLD(ctx, group.ID, "k", "style.sld", "application/xml")
if err != nil || g2.SldStorageKey == nil || *g2.SldStorageKey != "k" {
t.Fatalf("SetSLD: %+v, %v", g2, err)
}
byCode, err := groups.GetByCode(ctx, group.Code)
if err != nil || byCode.ID != group.ID {
t.Fatalf("GetByCode: %+v, %v", byCode, err)
}
}

View File

@@ -0,0 +1,117 @@
package postgres
import (
"context"
"fmt"
"gis/internal/domain"
"github.com/google/uuid"
"github.com/jackc/pgx/v5"
"github.com/jackc/pgx/v5/pgxpool"
)
// DatasetGroupRepository persists dataset groups in Postgres.
type DatasetGroupRepository struct{ pool *pgxpool.Pool }
// NewDatasetGroupRepository returns a DatasetGroupRepository backed by pool.
func NewDatasetGroupRepository(pool *pgxpool.Pool) *DatasetGroupRepository {
return &DatasetGroupRepository{pool: pool}
}
const datasetGroupColumns = `id, code, name, description, sld_storage_key, sld_filename, sld_content_type, created_at, updated_at`
func scanDatasetGroup(row pgx.Row) (domain.DatasetGroup, error) {
var g domain.DatasetGroup
err := row.Scan(&g.ID, &g.Code, &g.Name, &g.Description, &g.SldStorageKey, &g.SldFilename, &g.SldContentType, &g.CreatedAt, &g.UpdatedAt)
return g, err
}
// Create inserts a new dataset group and returns the stored row.
func (r *DatasetGroupRepository) Create(ctx context.Context, g domain.DatasetGroup) (domain.DatasetGroup, error) {
row := r.pool.QueryRow(ctx,
`INSERT INTO dataset_groups (code, name, description) VALUES ($1, $2, $3) RETURNING `+datasetGroupColumns,
g.Code, g.Name, g.Description)
out, err := scanDatasetGroup(row)
return out, mapError(err)
}
// GetByID returns the group with the given id.
func (r *DatasetGroupRepository) GetByID(ctx context.Context, id uuid.UUID) (domain.DatasetGroup, error) {
out, err := scanDatasetGroup(r.pool.QueryRow(ctx, `SELECT `+datasetGroupColumns+` FROM dataset_groups WHERE id = $1`, id))
return out, mapError(err)
}
// GetByCode returns the group with the given (unique) code.
func (r *DatasetGroupRepository) GetByCode(ctx context.Context, code string) (domain.DatasetGroup, error) {
out, err := scanDatasetGroup(r.pool.QueryRow(ctx, `SELECT `+datasetGroupColumns+` FROM dataset_groups WHERE code = $1`, code))
return out, mapError(err)
}
// List returns a page of groups ordered by code; code (exact) is optional.
func (r *DatasetGroupRepository) List(ctx context.Context, code *string, limit, offset int) ([]domain.DatasetGroup, error) {
where, args := groupWhere(code)
args = append(args, limit, offset)
rows, err := r.pool.Query(ctx, fmt.Sprintf(`SELECT %s FROM dataset_groups%s ORDER BY code LIMIT $%d OFFSET $%d`,
datasetGroupColumns, where, len(args)-1, len(args)), args...)
if err != nil {
return nil, mapError(err)
}
defer rows.Close()
out := make([]domain.DatasetGroup, 0)
for rows.Next() {
g, err := scanDatasetGroup(rows)
if err != nil {
return nil, mapError(err)
}
out = append(out, g)
}
return out, mapError(rows.Err())
}
// Count returns how many groups match the optional code filter.
func (r *DatasetGroupRepository) Count(ctx context.Context, code *string) (int, error) {
where, args := groupWhere(code)
var n int
err := r.pool.QueryRow(ctx, `SELECT count(*) FROM dataset_groups`+where, args...).Scan(&n)
return n, mapError(err)
}
// groupWhere builds the optional exact-code WHERE clause and its arguments.
func groupWhere(code *string) (string, []any) {
if code == nil {
return "", nil
}
return " WHERE code = $1", []any{*code}
}
// Update replaces a group's editable fields and returns the stored row.
func (r *DatasetGroupRepository) Update(ctx context.Context, g domain.DatasetGroup) (domain.DatasetGroup, error) {
row := r.pool.QueryRow(ctx,
`UPDATE dataset_groups SET code = $2, name = $3, description = $4, updated_at = now() WHERE id = $1 RETURNING `+datasetGroupColumns,
g.ID, g.Code, g.Name, g.Description)
out, err := scanDatasetGroup(row)
return out, mapError(err)
}
// Delete removes a group. Returns domain.ErrNotFound if absent and
// domain.ErrConflict (via the FK) if datasets still reference it.
func (r *DatasetGroupRepository) Delete(ctx context.Context, id uuid.UUID) error {
tag, err := r.pool.Exec(ctx, `DELETE FROM dataset_groups WHERE id = $1`, id)
if err != nil {
return mapError(err)
}
if tag.RowsAffected() == 0 {
return domain.ErrNotFound
}
return nil
}
// SetSLD points the group at a stored SLD style object.
func (r *DatasetGroupRepository) SetSLD(ctx context.Context, id uuid.UUID, key, filename, contentType string) (domain.DatasetGroup, error) {
row := r.pool.QueryRow(ctx,
`UPDATE dataset_groups SET sld_storage_key = $2, sld_filename = $3, sld_content_type = $4, updated_at = now() WHERE id = $1 RETURNING `+datasetGroupColumns,
id, key, filename, contentType)
out, err := scanDatasetGroup(row)
return out, mapError(err)
}

View File

@@ -9,14 +9,14 @@ import (
)
// ListAllObservations returns every observation for a dataset ordered by
// (kato_code, date). Unlike ListObservations it is unpaginated: it backs GeoJSON
// (kato_code, start_date). Unlike ListObservations it is unpaginated: it backs GeoJSON
// assembly, where all of a dataset's observations are needed at once.
func (r *DatasetRepository) ListAllObservations(ctx context.Context, datasetID uuid.UUID) ([]domain.Observation, error) {
rows, err := r.pool.Query(ctx,
`SELECT `+observationColumns+`
FROM dataset_observations
WHERE dataset_id = $1
ORDER BY kato_code, date`,
ORDER BY kato_code, start_date`,
datasetID)
if err != nil {
return nil, mapError(err)

View File

@@ -0,0 +1,239 @@
package postgres
import (
"context"
"encoding/json"
"gis/internal/domain"
"github.com/google/uuid"
"github.com/jackc/pgx/v5"
"github.com/jackc/pgx/v5/pgxpool"
)
// LandscapeRepository persists Landscape Program runs in Postgres.
type LandscapeRepository struct {
pool *pgxpool.Pool
}
// NewLandscapeRepository returns a LandscapeRepository backed by the given pool.
func NewLandscapeRepository(pool *pgxpool.Pool) *LandscapeRepository {
return &LandscapeRepository{pool: pool}
}
const runColumns = `id, status, from_stage, params, overrides, error, created_at, started_at, finished_at`
// scanRun scans a run row. overrides is stored as jsonb but scanned into
// []byte and unmarshalled manually so an empty/NULL value maps to an empty
// map rather than nil.
func scanRun(row pgx.Row) (domain.LPRun, error) {
var r domain.LPRun
var overrides []byte
err := row.Scan(&r.ID, &r.Status, &r.FromStage, &r.Params, &overrides, &r.Error, &r.CreatedAt, &r.StartedAt, &r.FinishedAt)
if err != nil {
return r, err
}
r.Overrides = map[string]uuid.UUID{}
if len(overrides) > 0 {
if err := json.Unmarshal(overrides, &r.Overrides); err != nil {
return r, err
}
}
return r, nil
}
const nodeColumns = `id, run_id, node_code, stage, status, fingerprint, input_datasets, output_datasets, error, started_at, finished_at`
func scanNode(row pgx.Row) (domain.LPRunNode, error) {
var n domain.LPRunNode
err := row.Scan(&n.ID, &n.RunID, &n.NodeCode, &n.Stage, &n.Status, &n.Fingerprint, &n.Inputs, &n.Outputs, &n.Error, &n.StartedAt, &n.FinishedAt)
return n, err
}
// CreateRun inserts a new run and its nodes (status pending) in one
// transaction, and returns the run with its nodes populated (in the order
// they were given).
func (r *LandscapeRepository) CreateRun(ctx context.Context, run domain.LPRun) (domain.LPRun, error) {
tx, err := r.pool.Begin(ctx)
if err != nil {
return domain.LPRun{}, mapError(err)
}
defer tx.Rollback(ctx)
overrides := run.Overrides
if overrides == nil {
overrides = map[string]uuid.UUID{}
}
overridesJSON, err := json.Marshal(overrides)
if err != nil {
return domain.LPRun{}, err
}
row := tx.QueryRow(ctx,
`INSERT INTO lp_runs (status, from_stage, params, overrides)
VALUES ($1, $2, $3, $4)
RETURNING `+runColumns,
run.Status, run.FromStage, run.Params, overridesJSON,
)
out, err := scanRun(row)
if err != nil {
return domain.LPRun{}, mapError(err)
}
nodes := make([]domain.LPRunNode, 0, len(run.Nodes))
for _, n := range run.Nodes {
row := tx.QueryRow(ctx,
`INSERT INTO lp_run_nodes (run_id, node_code, stage, status)
VALUES ($1, $2, $3, $4)
RETURNING `+nodeColumns,
out.ID, n.NodeCode, n.Stage, domain.LPNodePending,
)
node, err := scanNode(row)
if err != nil {
return domain.LPRun{}, mapError(err)
}
nodes = append(nodes, node)
}
out.Nodes = nodes
if err := tx.Commit(ctx); err != nil {
return domain.LPRun{}, mapError(err)
}
return out, nil
}
// GetRun returns the run with the given id, with its nodes ordered by
// node_code. Returns domain.ErrNotFound if the run does not exist.
func (r *LandscapeRepository) GetRun(ctx context.Context, id uuid.UUID) (domain.LPRun, error) {
row := r.pool.QueryRow(ctx, `SELECT `+runColumns+` FROM lp_runs WHERE id = $1`, id)
out, err := scanRun(row)
if err != nil {
return domain.LPRun{}, mapError(err)
}
rows, err := r.pool.Query(ctx,
`SELECT `+nodeColumns+` FROM lp_run_nodes WHERE run_id = $1 ORDER BY node_code`, id)
if err != nil {
return domain.LPRun{}, mapError(err)
}
defer rows.Close()
nodes := make([]domain.LPRunNode, 0)
for rows.Next() {
n, err := scanNode(rows)
if err != nil {
return domain.LPRun{}, mapError(err)
}
nodes = append(nodes, n)
}
if err := rows.Err(); err != nil {
return domain.LPRun{}, mapError(err)
}
out.Nodes = nodes
return out, nil
}
// ListRuns returns a page of runs (without nodes), newest first.
func (r *LandscapeRepository) ListRuns(ctx context.Context, limit, offset int) ([]domain.LPRun, error) {
rows, err := r.pool.Query(ctx,
`SELECT `+runColumns+` FROM lp_runs ORDER BY created_at DESC LIMIT $1 OFFSET $2`, limit, offset)
if err != nil {
return nil, mapError(err)
}
defer rows.Close()
runs := make([]domain.LPRun, 0)
for rows.Next() {
run, err := scanRun(rows)
if err != nil {
return nil, mapError(err)
}
runs = append(runs, run)
}
return runs, mapError(rows.Err())
}
// CountRuns returns the total number of runs.
func (r *LandscapeRepository) CountRuns(ctx context.Context) (int, error) {
var n int
err := r.pool.QueryRow(ctx, `SELECT count(*) FROM lp_runs`).Scan(&n)
return n, mapError(err)
}
// SetRunStatus updates a run's status and error. Transitioning to running
// sets started_at; transitioning to succeeded or failed sets finished_at.
// Returns domain.ErrNotFound if the run does not exist.
func (r *LandscapeRepository) SetRunStatus(ctx context.Context, id uuid.UUID, status domain.LPRunStatus, errMsg *string) error {
tag, err := r.pool.Exec(ctx,
// $2 is used both as the assigned value (varchar column) and in the CASE
// comparisons (text literals); the explicit ::text casts keep Postgres
// from deducing two different types for one parameter (SQLSTATE 42P08).
`UPDATE lp_runs SET status = $2::text, error = $3,
started_at = CASE WHEN $2::text = 'running' THEN now() ELSE started_at END,
finished_at = CASE WHEN $2::text IN ('succeeded','failed') THEN now() ELSE finished_at END
WHERE id = $1`,
id, status, errMsg,
)
if err != nil {
return mapError(err)
}
if tag.RowsAffected() == 0 {
return domain.ErrNotFound
}
return nil
}
// SetNodeStarted marks a run's node as running, storing its fingerprint and
// resolved inputs, and clears any previous error. Returns domain.ErrNotFound
// if no such node exists.
func (r *LandscapeRepository) SetNodeStarted(ctx context.Context, runID uuid.UUID, code string, fingerprint string, inputs []domain.LPDatasetRef) error {
tag, err := r.pool.Exec(ctx,
`UPDATE lp_run_nodes
SET status = 'running', fingerprint = $3, input_datasets = $4, started_at = now(), error = NULL
WHERE run_id = $1 AND node_code = $2`,
runID, code, fingerprint, inputs,
)
if err != nil {
return mapError(err)
}
if tag.RowsAffected() == 0 {
return domain.ErrNotFound
}
return nil
}
// SetNodeFinished marks a run's node as finished with the given status,
// storing its outputs and error. Returns domain.ErrNotFound if no such node
// exists.
func (r *LandscapeRepository) SetNodeFinished(ctx context.Context, runID uuid.UUID, code string, status domain.LPNodeStatus, outputs []domain.LPDatasetRef, errMsg *string) error {
tag, err := r.pool.Exec(ctx,
`UPDATE lp_run_nodes
SET status = $3, output_datasets = $4, error = $5, finished_at = now()
WHERE run_id = $1 AND node_code = $2`,
runID, code, status, outputs, errMsg,
)
if err != nil {
return mapError(err)
}
if tag.RowsAffected() == 0 {
return domain.ErrNotFound
}
return nil
}
// FindReusableOutputs returns the outputs of the most recently succeeded node
// with the same code and fingerprint, excluding excludeRun, so its outputs
// may be reused instead of recomputed. Returns domain.ErrNotFound if none.
func (r *LandscapeRepository) FindReusableOutputs(ctx context.Context, code, fingerprint string, excludeRun uuid.UUID) ([]domain.LPDatasetRef, error) {
var outputs []domain.LPDatasetRef
err := r.pool.QueryRow(ctx,
`SELECT output_datasets FROM lp_run_nodes
WHERE node_code = $1 AND fingerprint = $2 AND status = 'succeeded' AND run_id <> $3 AND output_datasets IS NOT NULL
ORDER BY finished_at DESC LIMIT 1`,
code, fingerprint, excludeRun,
).Scan(&outputs)
if err != nil {
return nil, mapError(err)
}
return outputs, nil
}

View File

@@ -0,0 +1,151 @@
package postgres
import (
"context"
"errors"
"os"
"testing"
"time"
"gis/internal/domain"
"github.com/google/uuid"
"github.com/jackc/pgx/v5/pgxpool"
)
// testPool connects to the database named by TEST_DB_URL (migrations already
// applied) or skips the test. Repository SQL is otherwise only exercised by
// hand, so these tests are the guard against Postgres-only failures such as
// parameter type deduction (SQLSTATE 42P08) that no stub can reproduce.
func testPool(t *testing.T) *pgxpool.Pool {
t.Helper()
url := os.Getenv("TEST_DB_URL")
if url == "" {
t.Skip("TEST_DB_URL not set; skipping database test")
}
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
defer cancel()
pool, err := Connect(ctx, url)
if err != nil {
t.Fatalf("connect: %v", err)
}
t.Cleanup(pool.Close)
return pool
}
func TestLandscapeRepository_SetRunStatus_AllTransitions(t *testing.T) {
pool := testPool(t)
repo := NewLandscapeRepository(pool)
ctx := context.Background()
run, err := repo.CreateRun(ctx, domain.LPRun{
Status: domain.LPRunQueued,
Params: []byte(`{}`),
Nodes: []domain.LPRunNode{{NodeCode: "00", Stage: 0, Status: domain.LPNodePending}},
})
if err != nil {
t.Fatalf("CreateRun: %v", err)
}
t.Cleanup(func() { _, _ = pool.Exec(ctx, `DELETE FROM lp_runs WHERE id = $1`, run.ID) })
if err := repo.SetRunStatus(ctx, run.ID, domain.LPRunRunning, nil); err != nil {
t.Fatalf("SetRunStatus(running): %v", err)
}
got, err := repo.GetRun(ctx, run.ID)
if err != nil {
t.Fatalf("GetRun: %v", err)
}
if got.Status != domain.LPRunRunning || got.StartedAt == nil || got.FinishedAt != nil {
t.Fatalf("after running: status=%s started=%v finished=%v", got.Status, got.StartedAt, got.FinishedAt)
}
msg := "boom"
if err := repo.SetRunStatus(ctx, run.ID, domain.LPRunFailed, &msg); err != nil {
t.Fatalf("SetRunStatus(failed): %v", err)
}
got, err = repo.GetRun(ctx, run.ID)
if err != nil {
t.Fatalf("GetRun: %v", err)
}
if got.Status != domain.LPRunFailed || got.FinishedAt == nil || got.Error == nil || *got.Error != msg {
t.Fatalf("after failed: status=%s finished=%v error=%v", got.Status, got.FinishedAt, got.Error)
}
if err := repo.SetRunStatus(ctx, run.ID, domain.LPRunSucceeded, nil); err != nil {
t.Fatalf("SetRunStatus(succeeded): %v", err)
}
}
func TestLandscapeRepository_NodeLifecycleAndReuse(t *testing.T) {
pool := testPool(t)
repo := NewLandscapeRepository(pool)
ctx := context.Background()
newRun := func() domain.LPRun {
run, err := repo.CreateRun(ctx, domain.LPRun{
Status: domain.LPRunQueued,
Params: []byte(`{"window_size":2048}`),
Overrides: map[string]uuid.UUID{"lp-e": uuid.New()},
Nodes: []domain.LPRunNode{{NodeCode: "00", Stage: 0, Status: domain.LPNodePending}},
})
if err != nil {
t.Fatalf("CreateRun: %v", err)
}
t.Cleanup(func() { _, _ = pool.Exec(ctx, `DELETE FROM lp_runs WHERE id = $1`, run.ID) })
return run
}
inputs := []domain.LPDatasetRef{{Group: "lp-z-wr", DatasetID: uuid.New()}}
outputs := []domain.LPDatasetRef{{Group: "lp-z-integrated", DatasetID: uuid.New()}}
first := newRun()
if err := repo.SetNodeStarted(ctx, first.ID, "00", "fp-1", inputs); err != nil {
t.Fatalf("SetNodeStarted: %v", err)
}
if err := repo.SetNodeFinished(ctx, first.ID, "00", domain.LPNodeSucceeded, outputs, nil); err != nil {
t.Fatalf("SetNodeFinished: %v", err)
}
got, err := repo.GetRun(ctx, first.ID)
if err != nil {
t.Fatalf("GetRun: %v", err)
}
if len(got.Nodes) != 1 || got.Nodes[0].Status != domain.LPNodeSucceeded ||
len(got.Nodes[0].Inputs) != 1 || got.Nodes[0].Inputs[0] != inputs[0] ||
len(got.Nodes[0].Outputs) != 1 || got.Nodes[0].Outputs[0] != outputs[0] ||
got.Nodes[0].StartedAt == nil || got.Nodes[0].FinishedAt == nil {
t.Fatalf("node row not round-tripped: %+v", got.Nodes)
}
if len(got.Overrides) != 1 {
t.Fatalf("overrides not round-tripped: %+v", got.Overrides)
}
second := newRun()
reused, err := repo.FindReusableOutputs(ctx, "00", "fp-1", second.ID)
if err != nil {
t.Fatalf("FindReusableOutputs: %v", err)
}
if len(reused) != 1 || reused[0] != outputs[0] {
t.Fatalf("reused = %+v, want %+v", reused, outputs)
}
if _, err := repo.FindReusableOutputs(ctx, "00", "fp-other", second.ID); !errors.Is(err, domain.ErrNotFound) {
t.Fatalf("unknown fingerprint: err = %v, want ErrNotFound", err)
}
if _, err := repo.FindReusableOutputs(ctx, "00", "fp-1", first.ID); !errors.Is(err, domain.ErrNotFound) {
t.Fatalf("own run must be excluded: err = %v, want ErrNotFound", err)
}
failMsg := "grid mismatch"
if err := repo.SetNodeFinished(ctx, second.ID, "00", domain.LPNodeFailed, nil, &failMsg); err != nil {
t.Fatalf("SetNodeFinished(failed): %v", err)
}
runs, err := repo.ListRuns(ctx, 10, 0)
if err != nil {
t.Fatalf("ListRuns: %v", err)
}
if len(runs) < 2 || runs[0].ID != second.ID {
t.Fatalf("ListRuns must be newest first: %+v", runs)
}
if n, err := repo.CountRuns(ctx); err != nil || n < 2 {
t.Fatalf("CountRuns = %d, %v", n, err)
}
}

View File

@@ -22,10 +22,23 @@ import (
// maxParseBytes caps how much of a file is read into memory for parsing.
const maxParseBytes = 256 << 20 // 256 MiB
var (
// ErrNotRaster is returned when a raster-only operation is requested for a
// non-raster dataset.
ErrNotRaster = errors.New("dataset is not a raster")
// ErrCOGNotReady is returned when a raster's Cloud-Optimized GeoTIFF has not
// been produced yet.
ErrCOGNotReady = errors.New("cog is not ready")
// ErrSLDNotFound is returned when a dataset has no SLD style file uploaded.
ErrSLDNotFound = fmt.Errorf("%w: dataset has no sld", domain.ErrNotFound)
)
// DatasetRepository is the persistence behaviour DatasetService needs.
type DatasetRepository interface {
Create(ctx context.Context, d domain.Dataset) (domain.Dataset, error)
GetByID(ctx context.Context, id uuid.UUID) (domain.Dataset, error)
UpdateFields(ctx context.Context, id uuid.UUID, f domain.DatasetFields) (domain.Dataset, error)
LatestInGroup(ctx context.Context, groupID uuid.UUID, status *string, fileType *domain.FileType) (domain.Dataset, error)
ListSummaries(ctx context.Context, filter domain.DatasetFilter, limit, offset int) ([]domain.DatasetSummary, error)
Count(ctx context.Context, filter domain.DatasetFilter) (int, error)
Delete(ctx context.Context, id uuid.UUID) error
@@ -35,7 +48,9 @@ type DatasetRepository interface {
MarkConverted(ctx context.Context, id uuid.UUID, cogKey string, footprint []byte) error
SetProperties(ctx context.Context, id uuid.UUID, properties, geometry []byte) error
SetGeoJSON(ctx context.Context, id uuid.UUID, geojson []byte) error
ClearGeometryAndGeoJSON(ctx context.Context, id uuid.UUID) error
SaveMapping(ctx context.Context, id uuid.UUID, katoColumn string, years []domain.YearColumn) (domain.Dataset, error)
SetSLD(ctx context.Context, id uuid.UUID, key, filename, contentType string) (domain.Dataset, error)
ReplaceObservations(ctx context.Context, datasetID uuid.UUID, obs []domain.Observation) error
ListObservations(ctx context.Context, datasetID uuid.UUID, katoCode *string, limit, offset int) ([]domain.Observation, error)
CountObservations(ctx context.Context, datasetID uuid.UUID, katoCode *string) (int, error)
@@ -71,6 +86,12 @@ type categoryReader interface {
GetByCode(ctx context.Context, code string) (domain.Category, error)
}
// groupReader lets the dataset service verify a dataset group exists before a
// dataset is filed under it, and resolve its shared SLD as a fallback.
type groupReader interface {
GetByID(ctx context.Context, id uuid.UUID) (domain.DatasetGroup, error)
}
// JobEnqueuer schedules asynchronous dataset jobs.
type JobEnqueuer interface {
EnqueueParse(ctx context.Context, datasetID uuid.UUID) error
@@ -86,14 +107,16 @@ type ColumnParser func(filename string, data []byte) ([]domain.AttributeColumn,
type RowParser func(filename string, data []byte) ([]map[string]string, error)
// RasterConverter converts a raster file to a Cloud-Optimized GeoTIFF, reads its
// footprint, and dissolves a vector file's features into a single geometry. It
// operates on local file paths.
// footprint, and reads a vector file's per-feature geometry. It operates on
// local file paths.
type RasterConverter interface {
ToCOG(ctx context.Context, srcPath, dstPath string) error
Footprint(ctx context.Context, srcPath string) ([]byte, error)
// VectorGeometry returns the combined geometry of a vector file's features as
// GeoJSON in EPSG:4326 (nil if the file has no features).
VectorGeometry(ctx context.Context, srcPath string) ([]byte, error)
// VectorFeatures returns each feature's geometry as GeoJSON in EPSG:4326, in
// file order (nil if the file has no features). A feature with no geometry
// yields a nil entry at its index, keeping the result aligned by position
// with attribute rows read from the same file via RowParser.
VectorFeatures(ctx context.Context, srcPath string) ([]json.RawMessage, error)
}
// UploadInput carries everything needed to store a new dataset.
@@ -109,6 +132,11 @@ type UploadInput struct {
ContentType string
Size int64
Reader io.Reader
// DatasetGroupID optionally files the dataset under a dataset group.
DatasetGroupID *uuid.UUID
// StartDate / EndDate is the period the dataset describes (YYYY-MM-DD).
StartDate *string
EndDate *string
}
// DatasetService implements dataset business rules and object storage handling.
@@ -116,6 +144,7 @@ type DatasetService struct {
repo DatasetRepository
store ObjectStore
categories categoryReader
groups groupReader
jobs JobEnqueuer
parseColumns ColumnParser
parseRows RowParser
@@ -123,12 +152,14 @@ type DatasetService struct {
}
// NewDatasetService wires the dataset repository, object store, category reader
// (for parent validation), the job enqueuer, the column/row parsers, and the
// raster converter.
// (for parent validation), the dataset-group reader (for group validation and
// the SLD fallback), the job enqueuer, the column/row parsers, and the raster
// converter.
func NewDatasetService(
repo DatasetRepository,
store ObjectStore,
categories categoryReader,
groups groupReader,
jobs JobEnqueuer,
parseColumns ColumnParser,
parseRows RowParser,
@@ -138,6 +169,7 @@ func NewDatasetService(
repo: repo,
store: store,
categories: categories,
groups: groups,
jobs: jobs,
parseColumns: parseColumns,
parseRows: parseRows,
@@ -171,10 +203,13 @@ func (s *DatasetService) Upload(ctx context.Context, in UploadInput) (domain.Dat
}
content := io.MultiReader(bytes.NewReader(head), in.Reader)
if _, err := s.categories.GetByID(ctx, in.CategoryID); err != nil {
if errors.Is(err, domain.ErrNotFound) {
return domain.Dataset{}, fmt.Errorf("%w: category does not exist", domain.ErrValidation)
}
if err := s.ensureCategoryExists(ctx, in.CategoryID); err != nil {
return domain.Dataset{}, err
}
if err := s.ensureGroupExists(ctx, in.DatasetGroupID); err != nil {
return domain.Dataset{}, err
}
if err := domain.ValidateDatePeriod(in.StartDate, in.EndDate); err != nil {
return domain.Dataset{}, err
}
@@ -191,24 +226,31 @@ func (s *DatasetService) Upload(ctx context.Context, in UploadInput) (domain.Dat
status = domain.DatasetStatusParsing
}
storageKey := fmt.Sprintf("%s/%s", uuid.New().String(), in.Filename)
objectName, err := safeObjectName(in.Filename)
if err != nil {
return domain.Dataset{}, err
}
storageKey := fmt.Sprintf("%s/%s", uuid.New().String(), objectName)
if err := s.store.Put(ctx, storageKey, content, in.Size, in.ContentType); err != nil {
return domain.Dataset{}, err
}
dataset, err := s.repo.Create(ctx, domain.Dataset{
CategoryID: in.CategoryID,
Name: name,
Description: in.Description,
Unit: in.Unit,
Meta: in.Meta,
Automated: in.Automated,
Status: status,
Filename: in.Filename,
StorageKey: storageKey,
FileType: in.FileType,
SizeBytes: in.Size,
ContentType: in.ContentType,
CategoryID: in.CategoryID,
DatasetGroupID: in.DatasetGroupID,
StartDate: in.StartDate,
EndDate: in.EndDate,
Name: name,
Description: in.Description,
Unit: in.Unit,
Meta: in.Meta,
Automated: in.Automated,
Status: status,
Filename: in.Filename,
StorageKey: storageKey,
FileType: in.FileType,
SizeBytes: in.Size,
ContentType: in.ContentType,
})
if err != nil {
// Compensate: the row was not written, so the object would be orphaned.
@@ -225,6 +267,160 @@ func (s *DatasetService) Upload(ctx context.Context, in UploadInput) (domain.Dat
return dataset, nil
}
// UploadSLDInput carries an SLD (Styled Layer Descriptor) style file to attach
// to an existing dataset.
type UploadSLDInput struct {
Filename string
ContentType string
Size int64
Reader io.Reader
}
// UploadSLD validates and stores an SLD style file for a dataset, replacing
// any previously stored one. Applies to every file_type.
func (s *DatasetService) UploadSLD(ctx context.Context, id uuid.UUID, in UploadSLDInput) (domain.Dataset, error) {
dataset, err := s.repo.GetByID(ctx, id)
if err != nil {
return domain.Dataset{}, err
}
ext := strings.ToLower(filepath.Ext(in.Filename))
if !domain.ExtensionAllowedForSLD(ext) {
return domain.Dataset{}, fmt.Errorf("%w: sld file must have one of these extensions: %s",
domain.ErrValidation, strings.Join(domain.AllowedSLDExtensions, ", "))
}
head := make([]byte, 512)
n, err := io.ReadFull(in.Reader, head)
if err != nil && err != io.EOF && err != io.ErrUnexpectedEOF {
return domain.Dataset{}, fmt.Errorf("read sld upload: %w", err)
}
head = head[:n]
if err := domain.ValidateFileContent(ext, head); err != nil {
return domain.Dataset{}, fmt.Errorf("%w: %s", domain.ErrValidation, err)
}
content := io.MultiReader(bytes.NewReader(head), in.Reader)
key, err := deriveSLDKey(dataset.StorageKey, in.Filename)
if err != nil {
return domain.Dataset{}, err
}
if err := s.store.Put(ctx, key, content, in.Size, in.ContentType); err != nil {
return domain.Dataset{}, err
}
updated, err := s.repo.SetSLD(ctx, id, key, in.Filename, in.ContentType)
if err != nil {
// Compensate: the row was not updated, so the object would be orphaned.
_ = s.store.Remove(ctx, key)
return domain.Dataset{}, err
}
if dataset.SldStorageKey != nil && *dataset.SldStorageKey != key {
_ = s.store.Remove(ctx, *dataset.SldStorageKey) // best-effort: drop the replaced file
}
return updated, nil
}
// DownloadSLD returns the dataset metadata and a reader for its stored SLD
// style file. Fails with ErrSLDNotFound when none has been uploaded. The
// caller must close the reader.
func (s *DatasetService) DownloadSLD(ctx context.Context, id uuid.UUID) (domain.Dataset, io.ReadCloser, error) {
dataset, err := s.repo.GetByID(ctx, id)
if err != nil {
return domain.Dataset{}, nil, err
}
if dataset.SldStorageKey == nil {
// Fall back to the dataset group's shared style, if the group has one.
if dataset.DatasetGroupID == nil {
return domain.Dataset{}, nil, ErrSLDNotFound
}
group, err := s.groups.GetByID(ctx, *dataset.DatasetGroupID)
if err != nil {
if errors.Is(err, domain.ErrNotFound) {
return domain.Dataset{}, nil, ErrSLDNotFound
}
return domain.Dataset{}, nil, err
}
if group.SldStorageKey == nil {
return domain.Dataset{}, nil, ErrSLDNotFound
}
dataset.SldStorageKey = group.SldStorageKey
dataset.SldFilename = group.SldFilename
dataset.SldContentType = group.SldContentType
}
obj, err := s.store.Get(ctx, *dataset.SldStorageKey)
if err != nil {
return domain.Dataset{}, nil, err
}
return dataset, obj, nil
}
// CreateRasterInput describes a raster file already on local disk that should
// become a dataset (used by the Landscape Program to publish results).
type CreateRasterInput struct {
CategoryID uuid.UUID
DatasetGroupID *uuid.UUID
Name string
Filename string // stored filename, e.g. "lp-z-integrated.tif"
Path string // local path to read from
StartDate *string
EndDate *string
Meta json.RawMessage
Automated bool
}
// CreateRasterFromFile stores a local GeoTIFF as a new raster dataset and
// enqueues the standard COG conversion, exactly like an uploaded raster.
func (s *DatasetService) CreateRasterFromFile(ctx context.Context, in CreateRasterInput) (domain.Dataset, error) {
f, err := os.Open(in.Path)
if err != nil {
return domain.Dataset{}, err
}
defer f.Close()
info, err := f.Stat()
if err != nil {
return domain.Dataset{}, err
}
return s.Upload(ctx, UploadInput{
CategoryID: in.CategoryID,
DatasetGroupID: in.DatasetGroupID,
Name: in.Name,
Meta: in.Meta,
Automated: in.Automated,
Filename: in.Filename,
FileType: domain.FileTypeRaster,
ContentType: "image/tiff",
Size: info.Size(),
Reader: f,
StartDate: in.StartDate,
EndDate: in.EndDate,
})
}
// safeObjectName reduces a user-supplied filename to a single path element so
// it can never escape the object-key prefix it is appended to. Rejects names
// that carry no usable base ("", ".", "..", "/").
func safeObjectName(name string) (string, error) {
base := path.Base(filepath.ToSlash(strings.TrimSpace(name)))
switch base {
case "", ".", "..", "/":
return "", fmt.Errorf("%w: invalid filename", domain.ErrValidation)
}
return base, nil
}
// deriveSLDKey places a dataset's SLD style file alongside the original under
// an sld/ prefix, named after the uploaded SLD file. The filename is
// sanitised, so the key always stays under the dataset's own prefix.
func deriveSLDKey(storageKey, filename string) (string, error) {
name, err := safeObjectName(filename)
if err != nil {
return "", err
}
return path.Join(path.Dir(storageKey), "sld", name), nil
}
// enqueueProcessing schedules the appropriate async job for a dataset based on
// its file type: vector_with_kato is parsed for column selection, plain vector
// has its attribute table extracted into properties, and raster is converted to
@@ -244,26 +440,35 @@ func (s *DatasetService) enqueueProcessing(ctx context.Context, d domain.Dataset
// Reprocess re-enqueues the processing job for an existing dataset, restarting
// its asynchronous pipeline from the step appropriate to its file type. Useful
// for retrying after a transient failure or a worker that was behind the schema.
// for retrying after a transient failure or a worker that was behind the
// schema. Its stored geometry and GeoJSON are cleared first so a stale (e.g.
// previously miscomputed) value is never served while the job is pending; the
// worker repopulates both once the job completes.
func (s *DatasetService) Reprocess(ctx context.Context, id uuid.UUID) (domain.Dataset, error) {
dataset, err := s.repo.GetByID(ctx, id)
if err != nil {
return domain.Dataset{}, err
}
if err := s.repo.ClearGeometryAndGeoJSON(ctx, id); err != nil {
return domain.Dataset{}, fmt.Errorf("clear stale geometry/geojson: %w", err)
}
if err := s.enqueueProcessing(ctx, dataset); err != nil {
return domain.Dataset{}, fmt.Errorf("enqueue processing: %w", err)
}
return dataset, nil
}
// ReprocessAll re-enqueues the processing job for every dataset, restarting each
// one's asynchronous pipeline from the step appropriate to its file type. It
// paginates through all datasets and continues past individual failures,
// returning how many jobs were enqueued and a per-dataset map of any failures.
func (s *DatasetService) ReprocessAll(ctx context.Context) (enqueued int, failures map[uuid.UUID]error, err error) {
// ReprocessAll re-enqueues the processing job for every dataset matching
// fileType (nil matches every file type), restarting each one's asynchronous
// pipeline from the step appropriate to its file type. Each dataset's stored
// geometry and GeoJSON are cleared first (see Reprocess). It paginates through
// all matching datasets and continues past individual failures, returning how
// many jobs were enqueued and a per-dataset map of any failures.
func (s *DatasetService) ReprocessAll(ctx context.Context, fileType *domain.FileType) (enqueued int, failures map[uuid.UUID]error, err error) {
failures = make(map[uuid.UUID]error)
filter := domain.DatasetFilter{FileType: fileType}
for offset := 0; ; {
summaries, err := s.repo.ListSummaries(ctx, domain.DatasetFilter{}, MaxPageSize, offset)
summaries, err := s.repo.ListSummaries(ctx, filter, MaxPageSize, offset)
if err != nil {
return enqueued, failures, err
}
@@ -271,6 +476,10 @@ func (s *DatasetService) ReprocessAll(ctx context.Context) (enqueued int, failur
break
}
for _, sum := range summaries {
if e := s.repo.ClearGeometryAndGeoJSON(ctx, sum.ID); e != nil {
failures[sum.ID] = e
continue
}
if e := s.enqueueProcessing(ctx, domain.Dataset{ID: sum.ID, FileType: sum.FileType}); e != nil {
failures[sum.ID] = e
continue
@@ -285,12 +494,15 @@ func (s *DatasetService) ReprocessAll(ctx context.Context) (enqueued int, failur
return enqueued, failures, nil
}
// ExtractProperties reads a plain vector dataset's attribute table and spatial
// geometry and stores them (the attribute table as a JSON array of row objects
// in the properties column, the dissolved feature geometry in the geometry
// column), then marks the dataset ready. Invoked by the worker. Parse failures
// are recorded; storage failures are returned for retry. Geometry extraction is
// best-effort: a failure leaves geometry unset rather than failing the job.
// ExtractProperties reads a plain vector dataset's attribute table and
// per-feature geometry and stores them: the dissolved (unioned) geometry in
// the geometry column, and a GeoJSON FeatureCollection with one Feature per
// original polygon (its own geometry paired with its own attribute row, by
// position - the same pairing a shapefile's .shp/.dbf records have) as the
// persisted GeoJSON. Then marks the dataset ready. Invoked by the worker.
// Parse failures are recorded; storage failures are returned for retry.
// Geometry extraction is best-effort: a failure leaves geometry unset rather
// than failing the job.
func (s *DatasetService) ExtractProperties(ctx context.Context, id uuid.UUID) error {
dataset, err := s.repo.GetByID(ctx, id)
if err != nil {
@@ -317,20 +529,27 @@ func (s *DatasetService) ExtractProperties(ctx context.Context, id uuid.UUID) er
}
}
geometry := s.vectorGeometry(ctx, dataset.Filename, data)
if err := s.repo.SetProperties(ctx, id, properties, geometry); err != nil {
geoms := s.vectorFeatureGeometries(ctx, dataset.Filename, data)
if err := s.repo.SetProperties(ctx, id, properties, dissolvedGeometryJSON(geoms)); err != nil {
return err
}
// Assemble and persist the GeoJSON served by the .geojson endpoint from the
// now-canonical (unioned) geometry and attribute table.
return s.generateGeoJSON(ctx, id)
fc := domain.FeatureCollection{
Type: domain.GeoJSONFeatureCollection,
Features: buildVectorFeatures(rows, geoms),
}
raw, err := json.Marshal(fc)
if err != nil {
return err
}
return s.repo.SetGeoJSON(ctx, id, raw)
}
// vectorGeometry dissolves a vector file's features into a single GeoJSON
// geometry via the converter. It writes the in-memory bytes to a temp file
// (preserving the extension so the converter detects the format) because the
// converter operates on file paths. Best-effort: any failure yields nil.
func (s *DatasetService) vectorGeometry(ctx context.Context, filename string, data []byte) []byte {
// vectorFeatureGeometries reads a vector file's per-feature geometries via the
// converter. It writes the in-memory bytes to a temp file (preserving the
// extension so the converter detects the format) because the converter
// operates on file paths. Best-effort: any failure yields nil.
func (s *DatasetService) vectorFeatureGeometries(ctx context.Context, filename string, data []byte) []json.RawMessage {
ext := strings.ToLower(filepath.Ext(filename))
f, err := os.CreateTemp("", "gis-vec-*"+ext)
if err != nil {
@@ -344,8 +563,61 @@ func (s *DatasetService) vectorGeometry(ctx context.Context, filename string, da
if err := f.Close(); err != nil {
return nil
}
geom, _ := s.converter.VectorGeometry(ctx, f.Name())
return geom
geoms, _ := s.converter.VectorFeatures(ctx, f.Name())
return geoms
}
// dissolvedGeometryJSON wraps a vector file's per-feature geometries into a
// single GeoJSON GeometryCollection for storage in the geometry column, where
// PostGIS dissolves it into the union of all features (nil entries and an
// empty/nil input are dropped, yielding nil when no geometry survives).
func dissolvedGeometryJSON(geoms []json.RawMessage) []byte {
present := make([]json.RawMessage, 0, len(geoms))
for _, g := range geoms {
if len(g) == 0 || string(g) == "null" {
continue
}
present = append(present, g)
}
if len(present) == 0 {
return nil
}
b, err := json.Marshal(struct {
Type string `json:"type"`
Geometries []json.RawMessage `json:"geometries"`
}{Type: "GeometryCollection", Geometries: present})
if err != nil {
return nil
}
return b
}
// buildVectorFeatures pairs each row with the geometry at the same position
// (rows and geoms both come from reading the same file in file order, which
// is how a shapefile's .dbf attribute records correspond to its .shp geometry
// records) into one Feature per polygon. A row or geometry beyond the other
// slice's length, or a feature with no geometry, is skipped.
func buildVectorFeatures(rows []map[string]string, geoms []json.RawMessage) []domain.Feature {
n := len(geoms)
if len(rows) < n {
n = len(rows)
}
features := make([]domain.Feature, 0, n)
for i := 0; i < n; i++ {
if len(geoms[i]) == 0 || string(geoms[i]) == "null" {
continue
}
props := make(map[string]any, len(rows[i]))
for k, v := range rows[i] {
props[k] = v
}
features = append(features, domain.Feature{
Type: domain.GeoJSONFeature,
Geometry: geoms[i],
Properties: props,
})
}
return features
}
// hasAttributeData reports whether any row carries at least one attribute.
@@ -420,7 +692,7 @@ func (s *DatasetService) downloadToTemp(ctx context.Context, key, pattern string
// uploadFile streams a local file to the object store.
func (s *DatasetService) uploadFile(ctx context.Context, key, filePath, contentType string) error {
f, err := os.Open(filePath)
f, err := os.Open(filePath) // #nosec G304 -- filePath is a temp file this service created
if err != nil {
return err
}
@@ -517,8 +789,18 @@ func (s *DatasetService) SaveMapping(ctx context.Context, id uuid.UUID, in Mappi
if _, ok := known[yc.Column]; !ok {
return domain.Dataset{}, fmt.Errorf("%w: year column %q is not among the detected columns", domain.ErrValidation, yc.Column)
}
if _, err := time.Parse("2006-01-02", yc.Date); err != nil {
return domain.Dataset{}, fmt.Errorf("%w: invalid date %q for column %q (want YYYY-MM-DD)", domain.ErrValidation, yc.Date, yc.Column)
from, err := time.Parse("2006-01-02", yc.StartDate)
if err != nil {
return domain.Dataset{}, fmt.Errorf("%w: invalid start_date %q for column %q (want YYYY-MM-DD)", domain.ErrValidation, yc.StartDate, yc.Column)
}
if yc.EndDate != nil {
to, err := time.Parse("2006-01-02", *yc.EndDate)
if err != nil {
return domain.Dataset{}, fmt.Errorf("%w: invalid end_date %q for column %q (want YYYY-MM-DD)", domain.ErrValidation, *yc.EndDate, yc.Column)
}
if to.Before(from) {
return domain.Dataset{}, fmt.Errorf("%w: end_date %q is before start_date %q for column %q", domain.ErrValidation, *yc.EndDate, yc.StartDate, yc.Column)
}
}
}
@@ -563,8 +845,8 @@ func (s *DatasetService) Extract(ctx context.Context, id uuid.UUID) error {
return err // transient
}
geometry := s.vectorGeometry(ctx, dataset.Filename, data)
if err := s.repo.MarkReady(ctx, id, geometry); err != nil {
geoms := s.vectorFeatureGeometries(ctx, dataset.Filename, data)
if err := s.repo.MarkReady(ctx, id, dissolvedGeometryJSON(geoms)); err != nil {
return err
}
// Assemble and persist the GeoJSON served by the .geojson endpoint by joining
@@ -573,8 +855,10 @@ func (s *DatasetService) Extract(ctx context.Context, id uuid.UUID) error {
}
// buildObservations unpivots rows into observations. Rows without a KATO code
// are skipped; duplicate (kato, date) pairs keep the last value. Numeric cells
// populate Value, others ValueText.
// are skipped; duplicate (kato, start_date, end_date) tuples keep the last
// value. Numeric cells populate Value, others ValueText. A year column's
// EndDate (nil for a point-in-time mapping) carries through to the observation
// unchanged.
func buildObservations(datasetID uuid.UUID, katoColumn string, years []domain.YearColumn, rows []map[string]string) []domain.Observation {
obs := make([]domain.Observation, 0, len(rows)*len(years))
index := make(map[string]int)
@@ -585,7 +869,7 @@ func buildObservations(datasetID uuid.UUID, katoColumn string, years []domain.Ye
continue
}
for _, yc := range years {
o := domain.Observation{DatasetID: datasetID, KatoCode: kato, Date: yc.Date}
o := domain.Observation{DatasetID: datasetID, KatoCode: kato, StartDate: yc.StartDate, EndDate: yc.EndDate}
if raw := strings.TrimSpace(row[yc.Column]); raw != "" {
if f, err := strconv.ParseFloat(raw, 64); err == nil {
o.Value = &f
@@ -593,7 +877,7 @@ func buildObservations(datasetID uuid.UUID, katoColumn string, years []domain.Ye
o.ValueText = &raw
}
}
key := kato + "\x00" + yc.Date
key := kato + "\x00" + yc.StartDate + "\x00" + endDateKey(yc.EndDate)
if i, ok := index[key]; ok {
obs[i] = o
} else {
@@ -605,6 +889,14 @@ func buildObservations(datasetID uuid.UUID, katoColumn string, years []domain.Ye
return obs
}
// endDateKey normalizes an optional period end date for use in a dedup key.
func endDateKey(endDate *string) string {
if endDate == nil {
return ""
}
return *endDate
}
// ObservationPage is a page of observations with pagination metadata.
type ObservationPage struct {
Items []domain.Observation
@@ -703,7 +995,11 @@ func (s *DatasetService) generateGeoJSON(ctx context.Context, id uuid.UUID) erro
// joins the districts table on KATO code: one Feature is emitted per KATO, its
// boundary taken from the districts table and its observation values nested
// under a `data` object (keyed by date) alongside `kato` and `name`. KATO codes
// with no matching district are skipped.
// with no matching district are skipped. When any observation carries a period
// (EndDate set), its end date is additionally exposed under a `periods` object
// (keyed by the same start date used in `data`); `periods` is omitted entirely
// when every observation is point-in-time, so a dataset mapped without periods
// produces byte-identical output to before periods existed.
func (s *DatasetService) buildGeoJSON(ctx context.Context, id uuid.UUID) (json.RawMessage, error) {
dataset, err := s.repo.GetByID(ctx, id)
if err != nil {
@@ -730,8 +1026,8 @@ func (s *DatasetService) buildGeoJSON(ctx context.Context, id uuid.UUID) (json.R
if err != nil {
return nil, err
}
grouped, order := groupObservationsByKato(obs)
features, err := s.districtFeatures(ctx, grouped, order)
grouped, periods, order := groupObservationsByKato(obs)
features, err := s.districtFeatures(ctx, grouped, periods, order)
if err != nil {
return nil, err
}
@@ -764,9 +1060,12 @@ func (s *DatasetService) loadGeoJSONDataset(ctx context.Context, id uuid.UUID) (
// districtFeatures builds one Feature per KATO from the districts table, with
// the grouped observation values (keyed by date) nested under a `data` object
// alongside `kato` and `name` in each Feature's properties. KATO codes with no
// alongside `kato` and `name` in each Feature's properties. When periods holds
// an entry for the KATO (at least one observation had a period), its date->
// end_date map is additionally nested under a `periods` object; a KATO with no
// period observations gets no `periods` key at all. KATO codes with no
// matching district are skipped. order drives the deterministic feature order.
func (s *DatasetService) districtFeatures(ctx context.Context, grouped map[string]map[string]any, order []string) ([]domain.Feature, error) {
func (s *DatasetService) districtFeatures(ctx context.Context, grouped map[string]map[string]any, periods map[string]map[string]string, order []string) ([]domain.Feature, error) {
districts, err := s.repo.DistrictGeometriesByKato(ctx, order)
if err != nil {
return nil, err
@@ -777,10 +1076,14 @@ func (s *DatasetService) districtFeatures(ctx context.Context, grouped map[strin
if !ok {
continue // skip KATO codes with no district boundary
}
props := map[string]any{"kato": kato, "name": dist.Name, "data": grouped[kato]}
if p, ok := periods[kato]; ok {
props["periods"] = p
}
features = append(features, domain.Feature{
Type: domain.GeoJSONFeature,
Geometry: dist.Geometry,
Properties: map[string]any{"kato": kato, "name": dist.Name, "data": grouped[kato]},
Properties: props,
})
}
return features, nil
@@ -823,11 +1126,15 @@ func vectorFeatureProperties(raw json.RawMessage) map[string]any {
// groupObservationsByKato groups observations by KATO code into date->value
// maps. The value is the numeric Value when present, otherwise ValueText,
// otherwise nil (an empty cell). order lists KATO codes in first-seen order so
// the resulting feature order is deterministic.
func groupObservationsByKato(obs []domain.Observation) (map[string]map[string]any, []string) {
grouped := make(map[string]map[string]any)
order := make([]string, 0)
// otherwise nil (an empty cell). It also groups, per KATO, a start_date->end_date map
// for observations that carry a period (EndDate set); a KATO with no period
// observations gets no entry in periods at all, so callers can tell "no
// periods" apart from "empty periods". order lists KATO codes in first-seen
// order so the resulting feature order is deterministic.
func groupObservationsByKato(obs []domain.Observation) (grouped map[string]map[string]any, periods map[string]map[string]string, order []string) {
grouped = make(map[string]map[string]any)
periods = make(map[string]map[string]string)
order = make([]string, 0)
for _, o := range obs {
values, ok := grouped[o.KatoCode]
if !ok {
@@ -835,9 +1142,18 @@ func groupObservationsByKato(obs []domain.Observation) (map[string]map[string]an
grouped[o.KatoCode] = values
order = append(order, o.KatoCode)
}
values[o.Date] = observationValue(o)
values[o.StartDate] = observationValue(o)
if o.EndDate != nil {
p, ok := periods[o.KatoCode]
if !ok {
p = make(map[string]string)
periods[o.KatoCode] = p
}
p[o.StartDate] = *o.EndDate
}
}
return grouped, order
return grouped, periods, order
}
// observationValue returns the typed cell value: numeric Value, else ValueText,
@@ -858,6 +1174,147 @@ func (s *DatasetService) Get(ctx context.Context, id uuid.UUID) (domain.Dataset,
return s.repo.GetByID(ctx, id)
}
// DatasetUpdateInput carries the full set of editable metadata fields for a
// PUT replace. Description and Unit being nil clears those (nullable)
// columns.
type DatasetUpdateInput struct {
CategoryID uuid.UUID
Name string
Description *string
Unit *string
// DatasetGroupID, StartDate and EndDate are nullable: nil clears them.
DatasetGroupID *uuid.UUID
StartDate *string
EndDate *string
}
// Update replaces a dataset's editable metadata (category, name, description,
// unit). File-related and lifecycle fields are untouched.
func (s *DatasetService) Update(ctx context.Context, id uuid.UUID, in DatasetUpdateInput) (domain.Dataset, error) {
if err := s.ensureCategoryExists(ctx, in.CategoryID); err != nil {
return domain.Dataset{}, err
}
if err := s.ensureGroupExists(ctx, in.DatasetGroupID); err != nil {
return domain.Dataset{}, err
}
if err := domain.ValidateDatePeriod(in.StartDate, in.EndDate); err != nil {
return domain.Dataset{}, err
}
return s.repo.UpdateFields(ctx, id, domain.DatasetFields{
CategoryID: in.CategoryID,
Name: in.Name,
Description: in.Description,
Unit: in.Unit,
DatasetGroupID: in.DatasetGroupID,
StartDate: in.StartDate,
EndDate: in.EndDate,
})
}
// DatasetPatchInput carries a sparse set of metadata field changes. A nil
// CategoryID/Name leaves that column unchanged. Description/Unit are only
// applied when their paired Set flag is true; a nil value with Set true
// clears the (nullable) column.
type DatasetPatchInput struct {
CategoryID *uuid.UUID
Name *string
Description *string
DescriptionSet bool
Unit *string
UnitSet bool
DatasetGroupID *uuid.UUID
DatasetGroupIDSet bool
StartDate *string
StartDateSet bool
EndDate *string
EndDateSet bool
}
// Patch applies a sparse update to a dataset's editable metadata (category,
// name, description, unit). Fields absent from the input are left unchanged.
func (s *DatasetService) Patch(ctx context.Context, id uuid.UUID, in DatasetPatchInput) (domain.Dataset, error) {
current, err := s.repo.GetByID(ctx, id)
if err != nil {
return domain.Dataset{}, err
}
categoryID := current.CategoryID
if in.CategoryID != nil {
categoryID = *in.CategoryID
if err := s.ensureCategoryExists(ctx, categoryID); err != nil {
return domain.Dataset{}, err
}
}
name := current.Name
if in.Name != nil {
name = *in.Name
}
description := current.Description
if in.DescriptionSet {
description = in.Description
}
unit := current.Unit
if in.UnitSet {
unit = in.Unit
}
groupID := current.DatasetGroupID
if in.DatasetGroupIDSet {
groupID = in.DatasetGroupID
if err := s.ensureGroupExists(ctx, groupID); err != nil {
return domain.Dataset{}, err
}
}
startDate := current.StartDate
if in.StartDateSet {
startDate = in.StartDate
}
endDate := current.EndDate
if in.EndDateSet {
endDate = in.EndDate
}
if err := domain.ValidateDatePeriod(startDate, endDate); err != nil {
return domain.Dataset{}, err
}
return s.repo.UpdateFields(ctx, id, domain.DatasetFields{
CategoryID: categoryID,
Name: name,
Description: description,
Unit: unit,
DatasetGroupID: groupID,
StartDate: startDate,
EndDate: endDate,
})
}
// ensureCategoryExists validates that categoryID references an existing
// category, mirroring the check performed on upload.
func (s *DatasetService) ensureCategoryExists(ctx context.Context, categoryID uuid.UUID) error {
if _, err := s.categories.GetByID(ctx, categoryID); err != nil {
if errors.Is(err, domain.ErrNotFound) {
return fmt.Errorf("%w: category does not exist", domain.ErrValidation)
}
return err
}
return nil
}
// ensureGroupExists validates that a non-nil dataset group id references an
// existing group. A nil id (no group) is always valid.
func (s *DatasetService) ensureGroupExists(ctx context.Context, id *uuid.UUID) error {
if id == nil {
return nil
}
if _, err := s.groups.GetByID(ctx, *id); err != nil {
if errors.Is(err, domain.ErrNotFound) {
return fmt.Errorf("%w: dataset group %s not found", domain.ErrValidation, id)
}
return err
}
return nil
}
// Status-wait bounds and polling cadence for long polling.
const (
DefaultStatusWait = 25 * time.Second
@@ -958,6 +1415,27 @@ func (s *DatasetService) Download(ctx context.Context, id uuid.UUID) (domain.Dat
return dataset, obj, nil
}
// COG returns the dataset metadata and a reader for its Cloud-Optimized GeoTIFF.
// It fails with ErrNotRaster for non-raster datasets and ErrCOGNotReady when the
// raster has not been converted yet. The caller must close the reader.
func (s *DatasetService) COG(ctx context.Context, id uuid.UUID) (domain.Dataset, io.ReadCloser, error) {
dataset, err := s.repo.GetByID(ctx, id)
if err != nil {
return domain.Dataset{}, nil, err
}
if dataset.FileType != domain.FileTypeRaster {
return domain.Dataset{}, nil, ErrNotRaster
}
if dataset.CogStorageKey == nil {
return domain.Dataset{}, nil, ErrCOGNotReady
}
obj, err := s.store.Get(ctx, *dataset.CogStorageKey)
if err != nil {
return domain.Dataset{}, nil, err
}
return dataset, obj, nil
}
// Delete removes the dataset row and its stored object.
func (s *DatasetService) Delete(ctx context.Context, id uuid.UUID) error {
dataset, err := s.repo.GetByID(ctx, id)

View File

@@ -0,0 +1,189 @@
package service
import (
"bytes"
"context"
"fmt"
"io"
"path"
"path/filepath"
"strings"
"gis/internal/domain"
"github.com/google/uuid"
)
// DatasetGroupRepository is the persistence behaviour DatasetGroupService needs.
type DatasetGroupRepository interface {
Create(ctx context.Context, g domain.DatasetGroup) (domain.DatasetGroup, error)
GetByID(ctx context.Context, id uuid.UUID) (domain.DatasetGroup, error)
GetByCode(ctx context.Context, code string) (domain.DatasetGroup, error)
List(ctx context.Context, code *string, limit, offset int) ([]domain.DatasetGroup, error)
Count(ctx context.Context, code *string) (int, error)
Update(ctx context.Context, g domain.DatasetGroup) (domain.DatasetGroup, error)
Delete(ctx context.Context, id uuid.UUID) error
SetSLD(ctx context.Context, id uuid.UUID, key, filename, contentType string) (domain.DatasetGroup, error)
}
// DatasetGroupInput carries the mutable fields of a group.
type DatasetGroupInput struct {
Code string
Name string
Description string
}
// DatasetGroupPage is a page of groups.
type DatasetGroupPage struct {
Items []domain.DatasetGroup
Page int
PageSize int
Total int
}
// DatasetGroupService implements dataset-group business rules.
type DatasetGroupService struct {
repo DatasetGroupRepository
datasets DatasetRepository
store ObjectStore
}
// NewDatasetGroupService wires the group repository, the dataset repository
// (for the "latest" lookup) and the object store (for the group SLD).
func NewDatasetGroupService(repo DatasetGroupRepository, datasets DatasetRepository, store ObjectStore) *DatasetGroupService {
return &DatasetGroupService{repo: repo, datasets: datasets, store: store}
}
// errInvalidGroupCode is the validation error returned for a non-slug code.
func errInvalidGroupCode() error {
return fmt.Errorf("%w: code must be a slug (lowercase latin letters, digits, and dashes)", domain.ErrValidation)
}
// Create persists a new group after validating its code.
func (s *DatasetGroupService) Create(ctx context.Context, in DatasetGroupInput) (domain.DatasetGroup, error) {
if !domain.ValidSlug(in.Code) {
return domain.DatasetGroup{}, errInvalidGroupCode()
}
return s.repo.Create(ctx, domain.DatasetGroup{Code: in.Code, Name: in.Name, Description: in.Description})
}
// Get returns one group by id.
func (s *DatasetGroupService) Get(ctx context.Context, id uuid.UUID) (domain.DatasetGroup, error) {
return s.repo.GetByID(ctx, id)
}
// List returns a page of groups; page/pageSize are clamped like datasets.
func (s *DatasetGroupService) List(ctx context.Context, code *string, page, pageSize int) (DatasetGroupPage, error) {
if page < 1 {
page = 1
}
if pageSize < 1 {
pageSize = DefaultPageSize
}
if pageSize > MaxPageSize {
pageSize = MaxPageSize
}
items, err := s.repo.List(ctx, code, pageSize, (page-1)*pageSize)
if err != nil {
return DatasetGroupPage{}, err
}
total, err := s.repo.Count(ctx, code)
if err != nil {
return DatasetGroupPage{}, err
}
return DatasetGroupPage{Items: items, Page: page, PageSize: pageSize, Total: total}, nil
}
// Update replaces a group's editable fields.
func (s *DatasetGroupService) Update(ctx context.Context, id uuid.UUID, in DatasetGroupInput) (domain.DatasetGroup, error) {
if !domain.ValidSlug(in.Code) {
return domain.DatasetGroup{}, errInvalidGroupCode()
}
return s.repo.Update(ctx, domain.DatasetGroup{ID: id, Code: in.Code, Name: in.Name, Description: in.Description})
}
// Delete removes a group (409 via ErrConflict while datasets reference it) and
// then best-effort removes its SLD object.
func (s *DatasetGroupService) Delete(ctx context.Context, id uuid.UUID) error {
g, err := s.repo.GetByID(ctx, id)
if err != nil {
return err
}
if err := s.repo.Delete(ctx, id); err != nil {
return err
}
if g.SldStorageKey != nil {
_ = s.store.Remove(ctx, *g.SldStorageKey) // best-effort: the row is gone
}
return nil
}
// Latest returns the freshest dataset of the group (greatest end_date, NULLs
// last, then newest), optionally restricted by status and file type.
func (s *DatasetGroupService) Latest(ctx context.Context, id uuid.UUID, status *string, fileType *domain.FileType) (domain.Dataset, error) {
if _, err := s.repo.GetByID(ctx, id); err != nil {
return domain.Dataset{}, err
}
return s.datasets.LatestInGroup(ctx, id, status, fileType)
}
// UploadSLD validates and stores the group's shared SLD, replacing any previous one.
func (s *DatasetGroupService) UploadSLD(ctx context.Context, id uuid.UUID, in UploadSLDInput) (domain.DatasetGroup, error) {
g, err := s.repo.GetByID(ctx, id)
if err != nil {
return domain.DatasetGroup{}, err
}
ext := strings.ToLower(filepath.Ext(in.Filename))
if !domain.ExtensionAllowedForSLD(ext) {
return domain.DatasetGroup{}, fmt.Errorf("%w: sld file must have one of these extensions: %s",
domain.ErrValidation, strings.Join(domain.AllowedSLDExtensions, ", "))
}
head := make([]byte, 512)
n, err := io.ReadFull(in.Reader, head)
if err != nil && err != io.EOF && err != io.ErrUnexpectedEOF {
return domain.DatasetGroup{}, fmt.Errorf("read sld upload: %w", err)
}
head = head[:n]
if err := domain.ValidateFileContent(ext, head); err != nil {
return domain.DatasetGroup{}, fmt.Errorf("%w: %s", domain.ErrValidation, err)
}
content := io.MultiReader(bytes.NewReader(head), in.Reader)
name, err := safeObjectName(in.Filename)
if err != nil {
return domain.DatasetGroup{}, err
}
key := path.Join("groups", g.ID.String(), "sld", name)
if err := s.store.Put(ctx, key, content, in.Size, in.ContentType); err != nil {
return domain.DatasetGroup{}, err
}
updated, err := s.repo.SetSLD(ctx, id, key, in.Filename, in.ContentType)
if err != nil {
// Compensate: the row was not updated, so the object would be orphaned.
_ = s.store.Remove(ctx, key)
return domain.DatasetGroup{}, err
}
if g.SldStorageKey != nil && *g.SldStorageKey != key {
_ = s.store.Remove(ctx, *g.SldStorageKey) // best-effort: drop the replaced file
}
return updated, nil
}
// DownloadSLD returns the group and a reader for its SLD; ErrSLDNotFound when
// none has been uploaded. The caller must close the reader.
func (s *DatasetGroupService) DownloadSLD(ctx context.Context, id uuid.UUID) (domain.DatasetGroup, io.ReadCloser, error) {
g, err := s.repo.GetByID(ctx, id)
if err != nil {
return domain.DatasetGroup{}, nil, err
}
if g.SldStorageKey == nil {
return domain.DatasetGroup{}, nil, ErrSLDNotFound
}
obj, err := s.store.Get(ctx, *g.SldStorageKey)
if err != nil {
return domain.DatasetGroup{}, nil, err
}
return g, obj, nil
}

View File

@@ -0,0 +1,283 @@
package service
import (
"context"
"errors"
"strings"
"testing"
"time"
"gis/internal/domain"
"github.com/google/uuid"
)
// stubGroupRepo is an in-memory DatasetGroupRepository for tests. It also
// satisfies the dataset service's groupReader dependency.
type stubGroupRepo struct {
store map[uuid.UUID]domain.DatasetGroup
}
func newStubGroupRepo() *stubGroupRepo {
return &stubGroupRepo{store: map[uuid.UUID]domain.DatasetGroup{}}
}
func (r *stubGroupRepo) Create(_ context.Context, g domain.DatasetGroup) (domain.DatasetGroup, error) {
for _, e := range r.store {
if e.Code == g.Code {
return domain.DatasetGroup{}, domain.ErrConflict
}
}
g.ID = uuid.New()
r.store[g.ID] = g
return g, nil
}
func (r *stubGroupRepo) GetByID(_ context.Context, id uuid.UUID) (domain.DatasetGroup, error) {
g, ok := r.store[id]
if !ok {
return domain.DatasetGroup{}, domain.ErrNotFound
}
return g, nil
}
func (r *stubGroupRepo) GetByCode(_ context.Context, code string) (domain.DatasetGroup, error) {
for _, g := range r.store {
if g.Code == code {
return g, nil
}
}
return domain.DatasetGroup{}, domain.ErrNotFound
}
func (r *stubGroupRepo) List(_ context.Context, code *string, _, _ int) ([]domain.DatasetGroup, error) {
out := []domain.DatasetGroup{}
for _, g := range r.store {
if code == nil || g.Code == *code {
out = append(out, g)
}
}
return out, nil
}
func (r *stubGroupRepo) Count(_ context.Context, code *string) (int, error) {
l, _ := r.List(context.Background(), code, 0, 0)
return len(l), nil
}
func (r *stubGroupRepo) Update(_ context.Context, g domain.DatasetGroup) (domain.DatasetGroup, error) {
if _, ok := r.store[g.ID]; !ok {
return domain.DatasetGroup{}, domain.ErrNotFound
}
r.store[g.ID] = g
return g, nil
}
func (r *stubGroupRepo) Delete(_ context.Context, id uuid.UUID) error {
if _, ok := r.store[id]; !ok {
return domain.ErrNotFound
}
delete(r.store, id)
return nil
}
func (r *stubGroupRepo) SetSLD(_ context.Context, id uuid.UUID, key, filename, contentType string) (domain.DatasetGroup, error) {
g, ok := r.store[id]
if !ok {
return domain.DatasetGroup{}, domain.ErrNotFound
}
g.SldStorageKey, g.SldFilename, g.SldContentType = &key, &filename, &contentType
r.store[id] = g
return g, nil
}
func TestDatasetGroupService_Create_RequiresSlug(t *testing.T) {
svc := NewDatasetGroupService(newStubGroupRepo(), newStubDatasetRepo(), newStubStore())
_, err := svc.Create(context.Background(), DatasetGroupInput{Code: "Bad Code", Name: "x"})
if !errors.Is(err, domain.ErrValidation) {
t.Fatalf("want ErrValidation, got %v", err)
}
}
func TestDatasetGroupService_Latest_PrefersGreatestEndDateThenCreatedAt(t *testing.T) {
groups := newStubGroupRepo()
datasets := newStubDatasetRepo()
svc := NewDatasetGroupService(groups, datasets, newStubStore())
g, err := svc.Create(context.Background(), DatasetGroupInput{Code: "pop", Name: "Population"})
if err != nil {
t.Fatal(err)
}
s := func(v string) *string { return &v }
base := time.Date(2024, 1, 1, 0, 0, 0, 0, time.UTC)
old := domain.Dataset{ID: uuid.New(), DatasetGroupID: &g.ID, EndDate: s("2019-12-31"), Status: domain.DatasetStatusReady, CreatedAt: base}
fresh := domain.Dataset{ID: uuid.New(), DatasetGroupID: &g.ID, EndDate: s("2020-12-31"), Status: domain.DatasetStatusReady, CreatedAt: base.Add(time.Hour)}
undated := domain.Dataset{ID: uuid.New(), DatasetGroupID: &g.ID, Status: domain.DatasetStatusReady, CreatedAt: base.Add(2 * time.Hour)}
for _, d := range []domain.Dataset{old, fresh, undated} {
datasets.store[d.ID] = d
}
got, err := svc.Latest(context.Background(), g.ID, nil, nil)
if err != nil {
t.Fatal(err)
}
if got.ID != fresh.ID {
t.Fatalf("want %s, got %s", fresh.ID, got.ID)
}
}
func TestDatasetGroupService_Latest_NotFoundWhenEmpty(t *testing.T) {
groups := newStubGroupRepo()
svc := NewDatasetGroupService(groups, newStubDatasetRepo(), newStubStore())
g, err := svc.Create(context.Background(), DatasetGroupInput{Code: "pop", Name: "Population"})
if err != nil {
t.Fatal(err)
}
if _, err := svc.Latest(context.Background(), g.ID, nil, nil); !errors.Is(err, domain.ErrNotFound) {
t.Fatalf("want ErrNotFound, got %v", err)
}
}
func TestDatasetGroupService_Latest_UnknownGroupIsNotFound(t *testing.T) {
svc := NewDatasetGroupService(newStubGroupRepo(), newStubDatasetRepo(), newStubStore())
if _, err := svc.Latest(context.Background(), uuid.New(), nil, nil); !errors.Is(err, domain.ErrNotFound) {
t.Fatalf("want ErrNotFound, got %v", err)
}
}
func TestDatasetGroupService_UploadSLD_RejectsBadExtension(t *testing.T) {
groups := newStubGroupRepo()
store := newStubStore()
svc := NewDatasetGroupService(groups, newStubDatasetRepo(), store)
g, err := svc.Create(context.Background(), DatasetGroupInput{Code: "pop", Name: "Population"})
if err != nil {
t.Fatal(err)
}
_, err = svc.UploadSLD(context.Background(), g.ID, UploadSLDInput{Filename: "style.txt", ContentType: "text/plain", Size: 3, Reader: strings.NewReader("<x>")})
if !errors.Is(err, domain.ErrValidation) {
t.Fatalf("want ErrValidation, got %v", err)
}
if len(store.put) != 0 {
t.Fatalf("nothing should be stored on validation failure, got %v", store.put)
}
}
func TestDatasetGroupService_UploadSLD_StoresAndReplaces(t *testing.T) {
ctx := context.Background()
groups := newStubGroupRepo()
store := newStubStore()
svc := NewDatasetGroupService(groups, newStubDatasetRepo(), store)
g, err := svc.Create(ctx, DatasetGroupInput{Code: "pop", Name: "Population"})
if err != nil {
t.Fatal(err)
}
updated, err := svc.UploadSLD(ctx, g.ID, UploadSLDInput{
Filename: "style.sld", ContentType: "application/xml", Size: 3, Reader: strings.NewReader("<x>"),
})
if err != nil {
t.Fatal(err)
}
wantKey := "groups/" + g.ID.String() + "/sld/style.sld"
if updated.SldStorageKey == nil || *updated.SldStorageKey != wantKey {
t.Fatalf("want key %s, got %v", wantKey, updated.SldStorageKey)
}
// A second upload under a different name replaces the previous object.
if _, err := svc.UploadSLD(ctx, g.ID, UploadSLDInput{
Filename: "other.sld", ContentType: "application/xml", Size: 3, Reader: strings.NewReader("<y>"),
}); err != nil {
t.Fatal(err)
}
if len(store.removed) != 1 || store.removed[0] != wantKey {
t.Fatalf("want previous sld removed, got %v", store.removed)
}
}
func TestDatasetGroupService_DownloadSLD_NotFoundWhenAbsent(t *testing.T) {
svc := NewDatasetGroupService(newStubGroupRepo(), newStubDatasetRepo(), newStubStore())
g, err := svc.Create(context.Background(), DatasetGroupInput{Code: "pop", Name: "Population"})
if err != nil {
t.Fatal(err)
}
if _, _, err := svc.DownloadSLD(context.Background(), g.ID); !errors.Is(err, ErrSLDNotFound) {
t.Fatalf("want ErrSLDNotFound, got %v", err)
}
}
func TestDatasetGroupService_Delete_RemovesSLDObject(t *testing.T) {
ctx := context.Background()
store := newStubStore()
svc := NewDatasetGroupService(newStubGroupRepo(), newStubDatasetRepo(), store)
g, err := svc.Create(ctx, DatasetGroupInput{Code: "pop", Name: "Population"})
if err != nil {
t.Fatal(err)
}
if _, err := svc.UploadSLD(ctx, g.ID, UploadSLDInput{
Filename: "style.sld", ContentType: "application/xml", Size: 3, Reader: strings.NewReader("<x>"),
}); err != nil {
t.Fatal(err)
}
if err := svc.Delete(ctx, g.ID); err != nil {
t.Fatal(err)
}
wantKey := "groups/" + g.ID.String() + "/sld/style.sld"
if len(store.removed) != 1 || store.removed[0] != wantKey {
t.Fatalf("want sld object removed, got %v", store.removed)
}
if _, err := svc.Get(ctx, g.ID); !errors.Is(err, domain.ErrNotFound) {
t.Fatalf("want ErrNotFound after delete, got %v", err)
}
}
func TestDatasetGroupService_List_ClampsPageSize(t *testing.T) {
svc := NewDatasetGroupService(newStubGroupRepo(), newStubDatasetRepo(), newStubStore())
if _, err := svc.Create(context.Background(), DatasetGroupInput{Code: "pop", Name: "Population"}); err != nil {
t.Fatal(err)
}
page, err := svc.List(context.Background(), nil, 0, 10_000)
if err != nil {
t.Fatal(err)
}
if page.Page != 1 || page.PageSize != MaxPageSize {
t.Fatalf("want page 1 and page size %d, got %d/%d", MaxPageSize, page.Page, page.PageSize)
}
if page.Total != 1 {
t.Fatalf("want total 1, got %d", page.Total)
}
}
func TestDatasetGroupService_Update_RequiresSlug(t *testing.T) {
svc := NewDatasetGroupService(newStubGroupRepo(), newStubDatasetRepo(), newStubStore())
g, err := svc.Create(context.Background(), DatasetGroupInput{Code: "pop", Name: "Population"})
if err != nil {
t.Fatal(err)
}
if _, err := svc.Update(context.Background(), g.ID, DatasetGroupInput{Code: "Bad Code", Name: "x"}); !errors.Is(err, domain.ErrValidation) {
t.Fatalf("want ErrValidation, got %v", err)
}
}
func TestDatasetGroupService_UploadSLD_RejectsPathTraversal(t *testing.T) {
ctx := context.Background()
groups := newStubGroupRepo()
store := newStubStore()
svc := NewDatasetGroupService(groups, newStubDatasetRepo(), store)
g, err := svc.Create(ctx, DatasetGroupInput{Code: "pop", Name: "Population"})
if err != nil {
t.Fatal(err)
}
updated, err := svc.UploadSLD(ctx, g.ID, UploadSLDInput{
Filename: "../../other/sld/evil.sld", ContentType: "application/xml", Size: 3, Reader: strings.NewReader("<x>"),
})
if err != nil {
t.Fatal(err)
}
wantPrefix := "groups/" + g.ID.String() + "/sld/"
if updated.SldStorageKey == nil || !strings.HasPrefix(*updated.SldStorageKey, wantPrefix) ||
!strings.HasSuffix(*updated.SldStorageKey, "evil.sld") {
t.Fatalf("sld key escaped the group prefix: %v", updated.SldStorageKey)
}
if len(store.put) != 1 || !strings.HasPrefix(store.put[0], wantPrefix) {
t.Fatalf("object stored outside the group sld prefix: %v", store.put)
}
}

File diff suppressed because it is too large Load Diff

View File

@@ -0,0 +1,808 @@
package service
import (
"context"
"crypto/sha256"
"encoding/hex"
"encoding/json"
"errors"
"fmt"
"io"
"log/slog"
"os"
"path/filepath"
"sort"
"sync"
"time"
"gis/internal/domain"
"gis/internal/landscape"
"gis/internal/raster"
"github.com/google/uuid"
"golang.org/x/sync/errgroup"
)
// ComputeFunc runs one graph node over its input rasters, writing one raster
// per output group into outDir and returning the produced paths by group. It
// mirrors landscape.Compute so tests can substitute a stub.
type ComputeFunc func(ctx context.Context, rio raster.IO, node landscape.Node, inputs map[string]string, outDir string, p landscape.Params) (map[string]string, error)
// ComputeWithWorkersFunc is ComputeFunc plus an explicit per-node worker cap.
// It mirrors landscape.ComputeWithWorkers.
type ComputeWithWorkersFunc func(ctx context.Context, rio raster.IO, node landscape.Node, inputs map[string]string, outDir string, p landscape.Params, workers int) (map[string]string, error)
// Default bounds on a run's parallelism, mirroring config.LandscapeConfig.
const (
// DefaultMaxWorkers is the total window-worker budget of one run.
DefaultMaxWorkers = 8
// DefaultMaxConcurrentNodes is how many nodes of a stage run at a time.
DefaultMaxConcurrentNodes = 2
)
// terminalWriteTimeout bounds a terminal status write made on a context that
// has already been cancelled (see terminalCtx).
const terminalWriteTimeout = 10 * time.Second
// LandscapeOptions bounds a run's parallelism, and with it the worker's peak
// memory: MaxConcurrentNodes nodes run at a time, and they share the
// MaxWorkers window-worker budget. Zero or negative values fall back to the
// defaults.
type LandscapeOptions struct {
MaxWorkers int
MaxConcurrentNodes int
}
func (o LandscapeOptions) withDefaults() LandscapeOptions {
if o.MaxWorkers < 1 {
o.MaxWorkers = DefaultMaxWorkers
}
if o.MaxConcurrentNodes < 1 {
o.MaxConcurrentNodes = DefaultMaxConcurrentNodes
}
return o
}
// workersPerNode splits the total worker budget over the concurrently running
// nodes, so nodes x workers stays at MaxWorkers. Always at least 1.
func (o LandscapeOptions) workersPerNode() int {
return max(1, o.MaxWorkers/o.MaxConcurrentNodes)
}
// LandscapeRepository is the persistence behaviour LandscapeService needs.
type LandscapeRepository interface {
CreateRun(ctx context.Context, run domain.LPRun) (domain.LPRun, error)
GetRun(ctx context.Context, id uuid.UUID) (domain.LPRun, error)
ListRuns(ctx context.Context, limit, offset int) ([]domain.LPRun, error)
CountRuns(ctx context.Context) (int, error)
SetRunStatus(ctx context.Context, id uuid.UUID, status domain.LPRunStatus, errMsg *string) error
SetNodeStarted(ctx context.Context, runID uuid.UUID, code string, fingerprint string, inputs []domain.LPDatasetRef) error
SetNodeFinished(ctx context.Context, runID uuid.UUID, code string, status domain.LPNodeStatus, outputs []domain.LPDatasetRef, errMsg *string) error
FindReusableOutputs(ctx context.Context, code, fingerprint string, excludeRun uuid.UUID) ([]domain.LPDatasetRef, error)
}
// LandscapeJobEnqueuer schedules asynchronous Landscape Program runs.
type LandscapeJobEnqueuer interface {
EnqueueRun(ctx context.Context, runID uuid.UUID) error
}
// datasetPublisher is the slice of DatasetService the Landscape Program needs:
// uploading a slot input and publishing a computed result.
type datasetPublisher interface {
Upload(ctx context.Context, in UploadInput) (domain.Dataset, error)
CreateRasterFromFile(ctx context.Context, in CreateRasterInput) (domain.Dataset, error)
}
// DatasetService is what the landscape service publishes through in production.
var _ datasetPublisher = (*DatasetService)(nil)
// SlotInfo describes one leaf input slot and the dataset currently filling it.
type SlotInfo struct {
Code string `json:"code"`
Title string `json:"title"`
GroupID uuid.UUID `json:"group_id"`
Latest *domain.DatasetSummary `json:"latest"`
}
// ResultInfo describes one computed output group and its freshest ready dataset.
type ResultInfo struct {
Group string `json:"group"`
Title string `json:"title"`
Node string `json:"node"`
GroupID uuid.UUID `json:"group_id"`
Latest *domain.DatasetSummary `json:"latest"`
CogURL *string `json:"cog_url"`
SldURL *string `json:"sld_url"`
}
// SlotUploadInput carries a raster upload destined for one input slot.
type SlotUploadInput struct {
Filename string
ContentType string
Size int64
Reader io.Reader
Name string
StartDate *string
EndDate *string
}
// RunInput carries the options of a new Landscape Program run.
type RunInput struct {
// FromStage forces recomputation from this stage on (0..3); nil recomputes
// only what changed.
FromStage *int
// Overrides pins a specific dataset for a group code instead of resolving
// the freshest ready one.
Overrides map[string]uuid.UUID
Params landscape.Params
}
// LPRunPage is a page of runs with pagination metadata.
type LPRunPage struct {
Items []domain.LPRun
Page int
PageSize int
Total int
}
// LandscapeService plans and executes Landscape Program runs.
type LandscapeService struct {
runs LandscapeRepository
datasets DatasetRepository
groups DatasetGroupRepository
categories categoryReader
publisher datasetPublisher
store ObjectStore
jobs LandscapeJobEnqueuer
rio raster.IO
// compute is the legacy worker-count-agnostic entry point, kept so an
// existing ComputeFunc still wires up; runNode always calls
// computeWithWorkers, which wraps compute when one was provided.
compute ComputeFunc
computeWithWorkers ComputeWithWorkersFunc
opts LandscapeOptions
log *slog.Logger
}
// NewLandscapeService wires the run repository, the dataset and group
// repositories (input resolution), the category reader (the "landscape"
// category every result is filed under), the dataset publisher (result
// datasets and slot uploads), the object store (input downloads), the job
// enqueuer, and the raster IO / compute function the graph runs on. Pass a nil
// compute (the production wiring) to run landscape.ComputeWithWorkers with the
// per-node share of opts.MaxWorkers; a non-nil ComputeFunc is honoured for
// backward compatibility but never sees the worker cap. opts bounds the run's
// parallelism; zero fields fall back to the package defaults.
func NewLandscapeService(
runs LandscapeRepository,
datasets DatasetRepository,
groups DatasetGroupRepository,
categories categoryReader,
publisher datasetPublisher,
store ObjectStore,
jobs LandscapeJobEnqueuer,
rio raster.IO,
compute ComputeFunc,
log *slog.Logger,
opts LandscapeOptions,
) *LandscapeService {
// Default to the workers-aware entry point; wrap a supplied legacy
// ComputeFunc (which has no worker parameter) when one is given.
withWorkers := ComputeWithWorkersFunc(landscape.ComputeWithWorkers)
if compute != nil {
withWorkers = func(ctx context.Context, rio raster.IO, node landscape.Node, inputs map[string]string, outDir string, p landscape.Params, _ int) (map[string]string, error) {
return compute(ctx, rio, node, inputs, outDir, p)
}
}
return &LandscapeService{
runs: runs,
datasets: datasets,
groups: groups,
categories: categories,
publisher: publisher,
store: store,
jobs: jobs,
rio: rio,
compute: compute,
computeWithWorkers: withWorkers,
opts: opts.withDefaults(),
log: log,
}
}
// Slots lists every leaf input slot with its group and latest dataset (any
// status), so the UI can show what has been uploaded.
func (s *LandscapeService) Slots(ctx context.Context) ([]SlotInfo, error) {
out := make([]SlotInfo, 0, len(landscape.Slots))
for _, slot := range landscape.Slots {
g, err := s.groups.GetByCode(ctx, slot.Code)
if err != nil {
return nil, fmt.Errorf("dataset group %q: %w", slot.Code, err)
}
info := SlotInfo{Code: slot.Code, Title: slot.Title, GroupID: g.ID}
latest, err := s.datasets.LatestInGroup(ctx, g.ID, nil, nil)
switch {
case err == nil:
summary := summaryOf(latest)
info.Latest = &summary
case errors.Is(err, domain.ErrNotFound):
// An empty slot is expected before the first upload.
default:
return nil, err
}
out = append(out, info)
}
return out, nil
}
// UploadToSlot stores a raster under the slot's dataset group in the landscape
// category, through the regular upload path (COG conversion is enqueued).
// Returns domain.ErrNotFound when the code is not a known slot.
func (s *LandscapeService) UploadToSlot(ctx context.Context, code string, in SlotUploadInput) (domain.Dataset, error) {
slot, ok := landscape.SlotByCode(code)
if !ok {
return domain.Dataset{}, fmt.Errorf("%w: unknown landscape slot %q", domain.ErrNotFound, code)
}
g, err := s.groups.GetByCode(ctx, slot.Code)
if err != nil {
return domain.Dataset{}, fmt.Errorf("dataset group %q: %w", slot.Code, err)
}
cat, err := s.categories.GetByCode(ctx, landscape.CategoryCode)
if err != nil {
return domain.Dataset{}, fmt.Errorf("category %q: %w", landscape.CategoryCode, err)
}
name := in.Name
if name == "" {
name = in.Filename
}
return s.publisher.Upload(ctx, UploadInput{
CategoryID: cat.ID,
DatasetGroupID: &g.ID,
Name: name,
Filename: in.Filename,
FileType: domain.FileTypeRaster,
ContentType: in.ContentType,
Size: in.Size,
Reader: in.Reader,
StartDate: in.StartDate,
EndDate: in.EndDate,
})
}
// CreateRun validates the params and overrides, persists a queued run with one
// pending node per graph node, and publishes the worker job.
func (s *LandscapeService) CreateRun(ctx context.Context, in RunInput) (domain.LPRun, error) {
params := in.Params.WithDefaults()
if err := params.Validate(); err != nil {
return domain.LPRun{}, fmt.Errorf("%w: %s", domain.ErrValidation, err)
}
if in.FromStage != nil {
if *in.FromStage < 0 || *in.FromStage > maxStage() {
return domain.LPRun{}, fmt.Errorf("%w: from_stage must be in 0..%d", domain.ErrValidation, maxStage())
}
}
overrides := map[string]uuid.UUID{}
for group, id := range in.Overrides {
if !knownGroup(group) {
return domain.LPRun{}, fmt.Errorf("%w: unknown landscape group %q in overrides", domain.ErrValidation, group)
}
d, err := s.datasets.GetByID(ctx, id)
if err != nil {
if errors.Is(err, domain.ErrNotFound) {
return domain.LPRun{}, fmt.Errorf("%w: override for %q: dataset %s does not exist", domain.ErrValidation, group, id)
}
return domain.LPRun{}, err
}
if d.FileType != domain.FileTypeRaster {
return domain.LPRun{}, fmt.Errorf("%w: override for %q: dataset %s is not a raster", domain.ErrValidation, group, id)
}
if d.Status != domain.DatasetStatusReady {
return domain.LPRun{}, fmt.Errorf("%w: override for %q: dataset %s is not ready", domain.ErrValidation, group, id)
}
overrides[group] = id
}
// lp_runs.params is NOT NULL, and the kernels misbehave on a partially
// filled Params, so the defaulted-and-validated value is what we persist.
paramsJSON, err := json.Marshal(params)
if err != nil {
return domain.LPRun{}, err
}
nodes := make([]domain.LPRunNode, 0, len(landscape.Nodes))
for _, n := range landscape.Nodes {
nodes = append(nodes, domain.LPRunNode{NodeCode: n.Code, Stage: n.Stage, Status: domain.LPNodePending})
}
run, err := s.runs.CreateRun(ctx, domain.LPRun{
Status: domain.LPRunQueued,
FromStage: in.FromStage,
Params: paramsJSON,
Overrides: overrides,
Nodes: nodes,
})
if err != nil {
return domain.LPRun{}, err
}
if err := s.jobs.EnqueueRun(ctx, run.ID); err != nil {
msg := fmt.Sprintf("failed to enqueue: %s", err)
if setErr := s.runs.SetRunStatus(ctx, run.ID, domain.LPRunFailed, &msg); setErr != nil {
s.log.Error("mark landscape run failed", "run_id", run.ID, "error", setErr)
}
return domain.LPRun{}, fmt.Errorf("enqueue landscape run: %w", err)
}
return run, nil
}
// GetRun returns one run with its nodes.
func (s *LandscapeService) GetRun(ctx context.Context, id uuid.UUID) (domain.LPRun, error) {
return s.runs.GetRun(ctx, id)
}
// ListRuns returns a page of runs, newest first.
func (s *LandscapeService) ListRuns(ctx context.Context, page, pageSize int) (LPRunPage, error) {
if page < 1 {
page = 1
}
if pageSize < 1 {
pageSize = DefaultPageSize
}
if pageSize > MaxPageSize {
pageSize = MaxPageSize
}
items, err := s.runs.ListRuns(ctx, pageSize, (page-1)*pageSize)
if err != nil {
return LPRunPage{}, err
}
total, err := s.runs.CountRuns(ctx)
if err != nil {
return LPRunPage{}, err
}
return LPRunPage{Items: items, Page: page, PageSize: pageSize, Total: total}, nil
}
// Results lists every output group with its freshest ready dataset and the URLs
// to render it.
func (s *LandscapeService) Results(ctx context.Context) ([]ResultInfo, error) {
ready := domain.DatasetStatusReady
fileType := domain.FileTypeRaster
out := make([]ResultInfo, 0, len(landscape.Outputs))
for _, o := range landscape.Outputs {
g, err := s.groups.GetByCode(ctx, o.Code)
if err != nil {
return nil, fmt.Errorf("dataset group %q: %w", o.Code, err)
}
info := ResultInfo{Group: o.Code, Title: o.Title, GroupID: g.ID}
if node, ok := landscape.NodeForOutput(o.Code); ok {
info.Node = node.Code
}
latest, err := s.datasets.LatestInGroup(ctx, g.ID, &ready, &fileType)
switch {
case err == nil:
summary := summaryOf(latest)
info.Latest = &summary
cog := "/datasets/" + latest.ID.String() + ".cog"
info.CogURL = &cog
if g.SldStorageKey != nil {
sld := "/dataset-groups/" + g.ID.String() + "/sld"
info.SldURL = &sld
}
case errors.Is(err, domain.ErrNotFound):
// Nothing computed for this group yet.
default:
return nil, err
}
out = append(out, info)
}
return out, nil
}
// MarkRunFailed records a terminal failure on a run so it does not stay stuck
// in an in-progress status.
func (s *LandscapeService) MarkRunFailed(ctx context.Context, runID uuid.UUID, reason string) error {
wctx, cancel := terminalCtx(ctx)
defer cancel()
return s.runs.SetRunStatus(wctx, runID, domain.LPRunFailed, &reason)
}
// Execute runs the whole graph for one run: stages in order, nodes within a
// stage concurrently. The first failing node fails the run and later stages are
// not started. It is the worker entry point.
func (s *LandscapeService) Execute(ctx context.Context, runID uuid.UUID) error {
run, err := s.runs.GetRun(ctx, runID)
if err != nil {
return err
}
// A redelivered or stale message must not rerun a finished run. "running"
// is allowed through so a run whose worker crashed mid-flight still
// recovers on redelivery.
if run.Status != domain.LPRunQueued && run.Status != domain.LPRunRunning {
s.log.Info("landscape run already terminal, ignoring redelivery", "run_id", runID, "status", run.Status)
return nil
}
var params landscape.Params
if len(run.Params) > 0 {
if err := json.Unmarshal(run.Params, &params); err != nil {
return s.failRun(ctx, runID, fmt.Errorf("decode run params: %w", err))
}
}
params = params.WithDefaults()
cat, err := s.categories.GetByCode(ctx, landscape.CategoryCode)
if err != nil {
return s.failRun(ctx, runID, fmt.Errorf("category %q: %w", landscape.CategoryCode, err))
}
if err := s.runs.SetRunStatus(ctx, runID, domain.LPRunRunning, nil); err != nil {
return err
}
// Re-read so started_at is available for result dataset names; on failure
// keep the pre-transition copy and fall back to time.Now() for the names.
if started, err := s.runs.GetRun(ctx, runID); err == nil {
run = started
} else {
s.log.Warn("re-read landscape run after marking it running", "run_id", runID, "error", err)
}
state := &runState{produced: map[string]uuid.UUID{}}
for _, stage := range landscape.Stages() {
// WithContext so the first failing node cancels its siblings: Compute
// honours cancellation, so a doomed stage stops reading rasters
// instead of running to completion. Terminal status writes use
// terminalCtx, so they still land on the cancelled context.
g, gctx := errgroup.WithContext(ctx)
// Each node's Compute spins its own worker pool (each worker holding a
// window buffer per input band), so peak memory is the product of the
// two limits. Bound the node fan-out here and split the worker budget
// over it so that product stays at opts.MaxWorkers.
g.SetLimit(s.opts.MaxConcurrentNodes)
for _, node := range stage {
g.Go(func() error { return s.runNode(gctx, run, node, params, cat, state) })
}
// Wait for every node of the stage: each one records its own outcome,
// and a sibling's failure must not leave a node half-recorded.
if err := g.Wait(); err != nil {
return s.failRun(ctx, runID, err)
}
}
return s.runs.SetRunStatus(ctx, runID, domain.LPRunSucceeded, nil)
}
// terminalCtx derives a bounded context that outlives the cancellation of ctx.
// Terminal status writes must happen even when the run failed *because* the
// context was cancelled (shutdown, a sibling node failing the stage); on the
// original context they would be refused and leave the run stuck in "running".
func terminalCtx(ctx context.Context) (context.Context, context.CancelFunc) {
return context.WithTimeout(context.WithoutCancel(ctx), terminalWriteTimeout)
}
// failRun marks the run failed with cause's message and returns cause.
func (s *LandscapeService) failRun(ctx context.Context, runID uuid.UUID, cause error) error {
msg := cause.Error()
wctx, cancel := terminalCtx(ctx)
defer cancel()
if err := s.runs.SetRunStatus(wctx, runID, domain.LPRunFailed, &msg); err != nil {
s.log.Error("mark landscape run failed", "run_id", runID, "error", err)
}
return cause
}
// runState carries what the run has produced so far, shared by the concurrent
// nodes of a stage.
type runState struct {
mu sync.Mutex
produced map[string]uuid.UUID
}
func (r *runState) get(group string) (uuid.UUID, bool) {
r.mu.Lock()
defer r.mu.Unlock()
id, ok := r.produced[group]
return id, ok
}
func (r *runState) record(refs []domain.LPDatasetRef) {
r.mu.Lock()
defer r.mu.Unlock()
for _, ref := range refs {
r.produced[ref.Group] = ref.DatasetID
}
}
// runNode resolves a node's inputs, reuses a previous result when the
// fingerprint matches, and otherwise computes and publishes its outputs.
func (s *LandscapeService) runNode(ctx context.Context, run domain.LPRun, node landscape.Node, params landscape.Params, cat domain.Category, state *runState) error {
refs, inputs, err := s.resolveInputs(ctx, run, node, state)
if err != nil {
return s.finishNodeFailed(ctx, run.ID, node, err)
}
fp := fingerprint(node, refs, params)
if err := s.runs.SetNodeStarted(ctx, run.ID, node.Code, fp, refs); err != nil {
// Best-effort, so the node is not left pending while the run is failed.
return s.finishNodeFailed(ctx, run.ID, node, err)
}
if run.FromStage == nil || node.Stage < *run.FromStage {
reused, ok, err := s.reusableOutputs(ctx, run.ID, node, fp)
if err != nil {
return s.finishNodeFailed(ctx, run.ID, node, err)
}
if ok {
state.record(reused)
return s.runs.SetNodeFinished(ctx, run.ID, node.Code, domain.LPNodeSkipped, reused, nil)
}
}
outputs, err := s.computeNode(ctx, run, node, params, cat, refs, inputs)
if err != nil {
return s.finishNodeFailed(ctx, run.ID, node, err)
}
state.record(outputs)
if err := s.runs.SetNodeFinished(ctx, run.ID, node.Code, domain.LPNodeSucceeded, outputs, nil); err != nil {
// Best-effort, so the node is not left running while the run is failed.
return s.finishNodeFailed(ctx, run.ID, node, err)
}
return nil
}
// finishNodeFailed records the failure on the node and returns the cause.
func (s *LandscapeService) finishNodeFailed(ctx context.Context, runID uuid.UUID, node landscape.Node, cause error) error {
msg := cause.Error()
wctx, cancel := terminalCtx(ctx)
defer cancel()
if err := s.runs.SetNodeFinished(wctx, runID, node.Code, domain.LPNodeFailed, nil, &msg); err != nil {
s.log.Error("mark landscape node failed", "run_id", runID, "node", node.Code, "error", err)
}
return cause
}
// resolveInputs picks one dataset per input group: an explicit override first,
// then whatever this run already produced for the group, then the freshest
// ready raster of the group.
func (s *LandscapeService) resolveInputs(ctx context.Context, run domain.LPRun, node landscape.Node, state *runState) ([]domain.LPDatasetRef, []domain.Dataset, error) {
ready := domain.DatasetStatusReady
fileType := domain.FileTypeRaster
refs := make([]domain.LPDatasetRef, 0, len(node.Inputs))
datasets := make([]domain.Dataset, 0, len(node.Inputs))
for _, group := range node.Inputs {
var (
d domain.Dataset
err error
)
switch id, ok := s.pinnedInput(run, group, state); {
case ok:
d, err = s.datasets.GetByID(ctx, id)
default:
g, gerr := s.groups.GetByCode(ctx, group)
if gerr != nil {
return nil, nil, fmt.Errorf("node %s: dataset group %q: %w", node.Code, group, gerr)
}
d, err = s.datasets.LatestInGroup(ctx, g.ID, &ready, &fileType)
}
if err != nil {
if errors.Is(err, domain.ErrNotFound) {
return nil, nil, fmt.Errorf("node %s: no ready raster dataset in group %q", node.Code, group)
}
return nil, nil, err
}
refs = append(refs, domain.LPDatasetRef{Group: group, DatasetID: d.ID})
datasets = append(datasets, d)
}
return refs, datasets, nil
}
// pinnedInput returns the dataset already decided for a group: the run's
// override, or what an earlier node of this run produced.
func (s *LandscapeService) pinnedInput(run domain.LPRun, group string, state *runState) (uuid.UUID, bool) {
if id, ok := run.Overrides[group]; ok {
return id, true
}
return state.get(group)
}
// reusableOutputs reports whether a previous succeeded node with the same
// fingerprint left outputs that all still exist and are ready.
func (s *LandscapeService) reusableOutputs(ctx context.Context, runID uuid.UUID, node landscape.Node, fp string) ([]domain.LPDatasetRef, bool, error) {
refs, err := s.runs.FindReusableOutputs(ctx, node.Code, fp, runID)
if err != nil {
if errors.Is(err, domain.ErrNotFound) {
return nil, false, nil
}
return nil, false, err
}
if len(refs) != len(node.Outputs) {
return nil, false, nil
}
for _, ref := range refs {
d, err := s.datasets.GetByID(ctx, ref.DatasetID)
if err != nil {
if errors.Is(err, domain.ErrNotFound) {
return nil, false, nil
}
return nil, false, err
}
if d.Status != domain.DatasetStatusReady {
return nil, false, nil
}
}
return refs, true, nil
}
// computeNode downloads the node's inputs into a temp dir, computes it, and
// publishes one raster dataset per output group. Temp files are always removed.
func (s *LandscapeService) computeNode(
ctx context.Context,
run domain.LPRun,
node landscape.Node,
params landscape.Params,
cat domain.Category,
refs []domain.LPDatasetRef,
inputs []domain.Dataset,
) ([]domain.LPDatasetRef, error) {
// landscape.Compute does not create outDir, so we own the whole directory.
dir, err := os.MkdirTemp("", "lp-"+node.Code+"-*")
if err != nil {
return nil, fmt.Errorf("node %s: temp dir: %w", node.Code, err)
}
defer os.RemoveAll(dir)
paths := make(map[string]string, len(inputs))
for i, d := range inputs {
group := refs[i].Group
// Downstream nodes read the original object, never the COG, so they
// never wait for the conversion job.
path := filepath.Join(dir, "in-"+group+".tif")
if err := s.download(ctx, d.StorageKey, path); err != nil {
return nil, fmt.Errorf("node %s: download %q: %w", node.Code, group, err)
}
paths[group] = path
}
outPaths, err := s.computeWithWorkers(ctx, s.rio, node, paths, dir, params, s.opts.workersPerNode())
if err != nil {
return nil, err
}
start, end := periodOf(inputs)
date := time.Now()
if run.StartedAt != nil {
date = *run.StartedAt
}
meta, err := json.Marshal(map[string]any{
"lp_run_id": run.ID,
"lp_node": node.Code,
"inputs": refs,
})
if err != nil {
return nil, err
}
outputs := make([]domain.LPDatasetRef, 0, len(node.Outputs))
for _, group := range node.Outputs {
path, ok := outPaths[group]
if !ok || path == "" {
return nil, fmt.Errorf("node %s: compute produced no output for group %q", node.Code, group)
}
g, err := s.groups.GetByCode(ctx, group)
if err != nil {
return nil, fmt.Errorf("node %s: dataset group %q: %w", node.Code, group, err)
}
title := group
if o, ok := landscape.OutputByCode(group); ok {
title = o.Title
}
d, err := s.publisher.CreateRasterFromFile(ctx, CreateRasterInput{
CategoryID: cat.ID,
DatasetGroupID: &g.ID,
Name: title + " " + date.Format("2006-01-02"),
Filename: group + ".tif",
Path: path,
StartDate: start,
EndDate: end,
Meta: meta,
Automated: true,
})
if err != nil {
return nil, fmt.Errorf("node %s: publish %q: %w", node.Code, group, err)
}
outputs = append(outputs, domain.LPDatasetRef{Group: group, DatasetID: d.ID})
}
return outputs, nil
}
// download copies an object from the store to a local path.
func (s *LandscapeService) download(ctx context.Context, key, path string) error {
obj, err := s.store.Get(ctx, key)
if err != nil {
return err
}
defer obj.Close()
f, err := os.Create(path) // #nosec G304 -- path is inside the node's os.MkdirTemp directory
if err != nil {
return err
}
if _, err := io.Copy(f, obj); err != nil {
f.Close()
return err
}
return f.Close()
}
// fingerprint is a stable hash of the node, its resolved inputs and the
// node-relevant params.
func fingerprint(node landscape.Node, inputs []domain.LPDatasetRef, p landscape.Params) string {
sorted := append([]domain.LPDatasetRef(nil), inputs...)
sort.Slice(sorted, func(i, j int) bool { return sorted[i].Group < sorted[j].Group })
payload, _ := json.Marshal(struct {
Node string `json:"node"`
Inputs []domain.LPDatasetRef `json:"inputs"`
Params landscape.Params `json:"params"`
}{node.Code, sorted, p.ForNode(node.Code)})
sum := sha256.Sum256(payload)
return hex.EncodeToString(sum[:])
}
// periodOf returns the min start_date and max end_date over the inputs, or nil
// when every input leaves the respective bound unset.
func periodOf(inputs []domain.Dataset) (*string, *string) {
var start, end *string
for _, d := range inputs {
if d.StartDate != nil && (start == nil || *d.StartDate < *start) {
start = d.StartDate
}
if d.EndDate != nil && (end == nil || *d.EndDate > *end) {
end = d.EndDate
}
}
return start, end
}
// knownGroup reports whether a group code names an input slot or a node output.
func knownGroup(code string) bool {
if _, ok := landscape.SlotByCode(code); ok {
return true
}
_, ok := landscape.OutputByCode(code)
return ok
}
// maxStage returns the highest stage number in the graph.
func maxStage() int {
highest := 0
for _, n := range landscape.Nodes {
if n.Stage > highest {
highest = n.Stage
}
}
return highest
}
// summaryOf projects a dataset onto the summary shape used in list responses.
func summaryOf(d domain.Dataset) domain.DatasetSummary {
return domain.DatasetSummary{
ID: d.ID,
CategoryID: d.CategoryID,
DatasetGroupID: d.DatasetGroupID,
StartDate: d.StartDate,
EndDate: d.EndDate,
Name: d.Name,
Description: d.Description,
Unit: d.Unit,
FileType: d.FileType,
SizeBytes: d.SizeBytes,
Status: d.Status,
CreatedAt: d.CreatedAt,
UpdatedAt: d.UpdatedAt,
}
}

File diff suppressed because it is too large Load Diff

View File

@@ -0,0 +1,63 @@
package http
import (
"fmt"
"strings"
)
// fallbackDownloadName is used when nothing printable survives sanitisation.
const fallbackDownloadName = "download"
// contentDisposition builds a Content-Disposition header value for a stored,
// user-supplied filename following RFC 6266: an ASCII-only quoted fallback in
// filename= and the exact UTF-8 name, percent-encoded per RFC 8187, in
// filename*=. Neither parameter can carry quotes, backslashes, control
// characters (CR/LF) or parameter separators taken from the filename, so a
// crafted upload name cannot inject extra header parameters or headers.
func contentDisposition(disposition, filename string) string {
name := strings.Map(func(r rune) rune {
if r < 0x20 || r == 0x7f {
return -1 // drop control characters, including CR and LF
}
return r
}, filename)
name = strings.TrimSpace(name)
if name == "" {
name = fallbackDownloadName
}
ascii := strings.Map(func(r rune) rune {
if r > 0x7e || r == '"' || r == '\\' {
return '_'
}
return r
}, name)
return fmt.Sprintf(`%s; filename="%s"; filename*=UTF-8''%s`, disposition, ascii, encodeRFC8187(name))
}
// encodeRFC8187 percent-encodes every byte of s that is not an attr-char
// (RFC 8187 §3.2.1).
func encodeRFC8187(s string) string {
const hex = "0123456789ABCDEF"
var b strings.Builder
for i := 0; i < len(s); i++ {
c := s[i]
if isAttrChar(c) {
b.WriteByte(c)
continue
}
b.WriteByte('%')
b.WriteByte(hex[c>>4])
b.WriteByte(hex[c&0x0f])
}
return b.String()
}
func isAttrChar(c byte) bool {
switch {
case 'a' <= c && c <= 'z', 'A' <= c && c <= 'Z', '0' <= c && c <= '9':
return true
}
return strings.IndexByte("!#$&+-.^_`|~", c) >= 0
}

View File

@@ -0,0 +1,57 @@
package http
import (
"mime"
"strings"
"testing"
)
func TestContentDisposition(t *testing.T) {
tests := []struct {
name string
filename string
want string
}{
{"plain", "roads.geojson", `attachment; filename="roads.geojson"; filename*=UTF-8''roads.geojson`},
{"space", "my map.tif", `attachment; filename="my map.tif"; filename*=UTF-8''my%20map.tif`},
{"quote injection", `a.tif"; filename="evil.exe`, `attachment; filename="a.tif_; filename=_evil.exe"; filename*=UTF-8''a.tif%22%3B%20filename%3D%22evil.exe`},
{"crlf injection", "a.tif\r\nSet-Cookie: x=1", `attachment; filename="a.tifSet-Cookie: x=1"; filename*=UTF-8''a.tifSet-Cookie%3A%20x%3D1`},
{"backslash", `..\..\a.sld`, `attachment; filename=".._.._a.sld"; filename*=UTF-8''..%5C..%5Ca.sld`},
{"cyrillic", "карта.tif", `attachment; filename="_____.tif"; filename*=UTF-8''%D0%BA%D0%B0%D1%80%D1%82%D0%B0.tif`},
{"empty", "", `attachment; filename="download"; filename*=UTF-8''download`},
{"only controls", "\r\n\t", `attachment; filename="download"; filename*=UTF-8''download`},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
got := contentDisposition("attachment", tt.filename)
if got != tt.want {
t.Fatalf("contentDisposition(%q)\n got %s\nwant %s", tt.filename, got, tt.want)
}
if strings.ContainsAny(got, "\r\n") {
t.Fatalf("header value contains CR/LF: %q", got)
}
// The value must parse as a single disposition whose decoded
// filename is the sanitised original.
disp, params, err := mime.ParseMediaType(got)
if err != nil {
t.Fatalf("ParseMediaType(%q): %v", got, err)
}
if disp != "attachment" {
t.Fatalf("disposition = %q, want attachment", disp)
}
if len(params) != 1 {
t.Fatalf("params = %v, want exactly one decoded filename", params)
}
})
}
}
func TestContentDispositionDecodesUTF8Name(t *testing.T) {
_, params, err := mime.ParseMediaType(contentDisposition("inline", "карта 2026.cog.tif"))
if err != nil {
t.Fatal(err)
}
if got := params["filename"]; got != "карта 2026.cog.tif" {
t.Fatalf("decoded filename = %q", got)
}
}

View File

@@ -0,0 +1,235 @@
package http
import (
"io"
"net/http"
"strings"
"gis/internal/domain"
"gis/internal/service"
"gis/pkg/httputil"
"github.com/go-chi/chi/v5"
"github.com/go-playground/validator/v10"
)
// DatasetGroupHandler serves the /dataset-groups routes.
type DatasetGroupHandler struct {
svc *service.DatasetGroupService
validate *validator.Validate
}
// NewDatasetGroupHandler returns a DatasetGroupHandler.
func NewDatasetGroupHandler(svc *service.DatasetGroupService, validate *validator.Validate) *DatasetGroupHandler {
return &DatasetGroupHandler{svc: svc, validate: validate}
}
// Register mounts the dataset-group routes on r.
func (h *DatasetGroupHandler) Register(r chi.Router) {
r.Get("/", h.list)
r.Post("/", h.create)
r.Get("/{id}", h.get)
r.Put("/{id}", h.update)
r.Delete("/{id}", h.delete)
r.Get("/{id}/latest", h.latest)
r.Post("/{id}/sld", h.uploadSLD)
r.Get("/{id}/sld", h.downloadSLD)
}
type datasetGroupRequest struct {
Code string `json:"code" validate:"required,max=255,slug"`
Name string `json:"name" validate:"required,max=255"`
Description string `json:"description" validate:"max=2000"`
}
func (r datasetGroupRequest) toInput() service.DatasetGroupInput {
return service.DatasetGroupInput{Code: r.Code, Name: r.Name, Description: r.Description}
}
func (h *DatasetGroupHandler) create(w http.ResponseWriter, r *http.Request) {
req, err := httputil.DecodeJSON[datasetGroupRequest](w, r)
if err != nil {
httputil.WriteError(w, http.StatusBadRequest, "invalid request body")
return
}
if err := h.validate.Struct(req); err != nil {
httputil.WriteValidationErrors(w, err)
return
}
group, err := h.svc.Create(r.Context(), req.toInput())
if err != nil {
respondDomainError(w, err)
return
}
httputil.WriteJSON(w, http.StatusCreated, group)
}
func (h *DatasetGroupHandler) list(w http.ResponseWriter, r *http.Request) {
var code *string
if v := strings.TrimSpace(r.URL.Query().Get("code")); v != "" {
code = &v
}
page, ok := parsePositiveIntQuery(w, r, "page", 1)
if !ok {
return
}
pageSize, ok := parsePositiveIntQuery(w, r, "page_size", service.DefaultPageSize)
if !ok {
return
}
result, err := h.svc.List(r.Context(), code, page, pageSize)
if err != nil {
respondDomainError(w, err)
return
}
httputil.WriteJSON(w, http.StatusOK, newPaginated(result.Items, result.Page, result.PageSize, result.Total))
}
func (h *DatasetGroupHandler) get(w http.ResponseWriter, r *http.Request) {
id, ok := parseUUIDParam(w, r, "id")
if !ok {
return
}
group, err := h.svc.Get(r.Context(), id)
if err != nil {
respondDomainError(w, err)
return
}
httputil.WriteJSON(w, http.StatusOK, group)
}
func (h *DatasetGroupHandler) update(w http.ResponseWriter, r *http.Request) {
id, ok := parseUUIDParam(w, r, "id")
if !ok {
return
}
req, err := httputil.DecodeJSON[datasetGroupRequest](w, r)
if err != nil {
httputil.WriteError(w, http.StatusBadRequest, "invalid request body")
return
}
if err := h.validate.Struct(req); err != nil {
httputil.WriteValidationErrors(w, err)
return
}
group, err := h.svc.Update(r.Context(), id, req.toInput())
if err != nil {
respondDomainError(w, err)
return
}
httputil.WriteJSON(w, http.StatusOK, group)
}
// delete removes a group. A group still referenced by datasets yields 409.
func (h *DatasetGroupHandler) delete(w http.ResponseWriter, r *http.Request) {
id, ok := parseUUIDParam(w, r, "id")
if !ok {
return
}
if err := h.svc.Delete(r.Context(), id); err != nil {
respondDomainError(w, err)
return
}
w.WriteHeader(http.StatusNoContent)
}
// latest returns the freshest dataset of the group (greatest end_date, NULLs
// last, then newest), optionally narrowed by ?status= and ?file_type=.
func (h *DatasetGroupHandler) latest(w http.ResponseWriter, r *http.Request) {
id, ok := parseUUIDParam(w, r, "id")
if !ok {
return
}
var status *string
if v := strings.TrimSpace(r.URL.Query().Get("status")); v != "" {
if !domain.ValidDatasetStatus(v) {
httputil.WriteError(w, http.StatusBadRequest, "invalid status")
return
}
status = &v
}
var fileType *domain.FileType
if v := strings.TrimSpace(r.URL.Query().Get("file_type")); v != "" {
ft := domain.FileType(v)
if !ft.Valid() {
httputil.WriteError(w, http.StatusBadRequest, "file_type must be one of: vector_with_kato, vector, raster")
return
}
fileType = &ft
}
dataset, err := h.svc.Latest(r.Context(), id, status, fileType)
if err != nil {
respondDomainError(w, err)
return
}
httputil.WriteJSON(w, http.StatusOK, dataset)
}
func (h *DatasetGroupHandler) uploadSLD(w http.ResponseWriter, r *http.Request) {
id, ok := parseUUIDParam(w, r, "id")
if !ok {
return
}
if !parseMultipartForm(w, r) {
return
}
file, header, err := r.FormFile("file")
if err != nil {
httputil.WriteError(w, http.StatusBadRequest, "file is required")
return
}
defer file.Close()
contentType := header.Header.Get("Content-Type")
if contentType == "" {
contentType = "application/xml"
}
group, err := h.svc.UploadSLD(r.Context(), id, service.UploadSLDInput{
Filename: header.Filename,
ContentType: contentType,
Size: header.Size,
Reader: file,
})
if err != nil {
respondDomainError(w, err)
return
}
httputil.WriteJSON(w, http.StatusOK, group)
}
func (h *DatasetGroupHandler) downloadSLD(w http.ResponseWriter, r *http.Request) {
id, ok := parseUUIDParam(w, r, "id")
if !ok {
return
}
group, obj, err := h.svc.DownloadSLD(r.Context(), id)
if err != nil {
respondDomainError(w, err)
return
}
defer obj.Close()
contentType := "application/xml"
if group.SldContentType != nil && *group.SldContentType != "" {
contentType = *group.SldContentType
}
filename := "style.sld"
if group.SldFilename != nil && *group.SldFilename != "" {
filename = *group.SldFilename
}
w.Header().Set("Content-Type", contentType)
w.Header().Set("Content-Disposition", contentDisposition("attachment", filename))
if _, err := io.Copy(w, obj); err != nil {
// Headers are already sent; nothing useful to return to the client.
return
}
}

View File

@@ -2,6 +2,7 @@ package http
import (
"encoding/json"
"errors"
"io"
"net/http"
"strconv"
@@ -17,9 +18,31 @@ import (
"github.com/google/uuid"
)
// maxUploadBytes caps the in-memory portion of a multipart upload (64 MiB).
// maxUploadBytes caps the in-memory portion of a multipart upload (64 MiB);
// larger parts spill to temporary files. The total request size is capped by
// the limitRequestBody middleware (HTTP_MAX_UPLOAD_BYTES).
const maxUploadBytes = 64 << 20
// parseMultipartForm parses a multipart upload and writes the error response
// itself when that fails: 413 when the body exceeds the router's size cap, 400
// otherwise. It reports whether the handler may continue.
func parseMultipartForm(w http.ResponseWriter, r *http.Request) bool {
// #nosec G120 -- r.Body is wrapped in http.MaxBytesReader by the
// limitRequestBody middleware that NewRouter installs on every route.
err := r.ParseMultipartForm(maxUploadBytes)
if err == nil {
return true
}
var tooLarge *http.MaxBytesError
if errors.As(err, &tooLarge) {
httputil.WriteError(w, http.StatusRequestEntityTooLarge,
"request body exceeds the "+strconv.FormatInt(tooLarge.Limit, 10)+" byte limit")
return false
}
httputil.WriteError(w, http.StatusBadRequest, "request must be multipart/form-data")
return false
}
// DatasetHandler serves the /datasets routes.
type DatasetHandler struct {
svc *service.DatasetService
@@ -36,9 +59,14 @@ func (h *DatasetHandler) Register(r chi.Router) {
r.Get("/", h.list)
r.Post("/", h.upload)
r.Get("/{id}", h.get)
r.Put("/{id}", h.update)
r.Patch("/{id}", h.patch)
r.Get("/{id}.geojson", h.geojson)
r.Get("/{id}.cog", h.cog)
r.Get("/{id}/status", h.status)
r.Get("/{id}/download", h.download)
r.Post("/{id}/sld", h.uploadSLD)
r.Get("/{id}/sld", h.downloadSLD)
r.Post("/{id}/mapping", h.mapping)
r.Get("/{id}/observations", h.observations)
r.Delete("/{id}", h.delete)
@@ -74,8 +102,9 @@ func (h *DatasetHandler) status(w http.ResponseWriter, r *http.Request) {
}
type yearColumnInput struct {
Column string `json:"column" validate:"required"`
Date string `json:"date" validate:"required,datetime=2006-01-02"`
Column string `json:"column" validate:"required"`
StartDate string `json:"start_date" validate:"required,datetime=2006-01-02"`
EndDate *string `json:"end_date" validate:"omitempty,datetime=2006-01-02"`
}
type mappingRequest struct {
@@ -102,7 +131,7 @@ func (h *DatasetHandler) mapping(w http.ResponseWriter, r *http.Request) {
in := service.MappingInput{KatoColumn: req.KatoColumn}
for _, yc := range req.YearColumns {
in.YearColumns = append(in.YearColumns, domain.YearColumn{Column: yc.Column, Date: yc.Date})
in.YearColumns = append(in.YearColumns, domain.YearColumn{Column: yc.Column, StartDate: yc.StartDate, EndDate: yc.EndDate})
}
dataset, err := h.svc.SaveMapping(r.Context(), id, in)
@@ -114,8 +143,7 @@ func (h *DatasetHandler) mapping(w http.ResponseWriter, r *http.Request) {
}
func (h *DatasetHandler) upload(w http.ResponseWriter, r *http.Request) {
if err := r.ParseMultipartForm(maxUploadBytes); err != nil {
httputil.WriteError(w, http.StatusBadRequest, "request must be multipart/form-data")
if !parseMultipartForm(w, r) {
return
}
@@ -143,6 +171,17 @@ func (h *DatasetHandler) upload(w http.ResponseWriter, r *http.Request) {
return
}
// The dataset group is optional; the period is validated by the service.
var groupID *uuid.UUID
if v := optionalFormValue(r, "dg_id"); v != nil {
gid, err := uuid.Parse(*v)
if err != nil {
httputil.WriteError(w, http.StatusUnprocessableEntity, "dg_id must be a valid UUID")
return
}
groupID = &gid
}
file, header, err := r.FormFile("file")
if err != nil {
httputil.WriteError(w, http.StatusBadRequest, "file is required")
@@ -156,17 +195,20 @@ func (h *DatasetHandler) upload(w http.ResponseWriter, r *http.Request) {
}
dataset, err := h.svc.Upload(r.Context(), service.UploadInput{
CategoryID: categoryID,
Name: r.FormValue("name"),
Description: optionalFormValue(r, "description"),
Unit: optionalFormValue(r, "unit"),
Meta: meta,
Automated: automated,
Filename: header.Filename,
FileType: fileType,
ContentType: contentType,
Size: header.Size,
Reader: file,
CategoryID: categoryID,
Name: r.FormValue("name"),
Description: optionalFormValue(r, "description"),
Unit: optionalFormValue(r, "unit"),
Meta: meta,
Automated: automated,
Filename: header.Filename,
FileType: fileType,
ContentType: contentType,
Size: header.Size,
Reader: file,
DatasetGroupID: groupID,
StartDate: optionalFormValue(r, "start_date"),
EndDate: optionalFormValue(r, "end_date"),
})
if err != nil {
respondDomainError(w, err)
@@ -219,7 +261,32 @@ func (h *DatasetHandler) list(w http.ResponseWriter, r *http.Request) {
categoryCode = &v
}
filter := domain.DatasetFilter{CategoryID: categoryID}
groupID, ok := parseOptionalUUIDQuery(w, r, "dg_id")
if !ok {
return
}
filter := domain.DatasetFilter{CategoryID: categoryID, DatasetGroupID: groupID}
for _, b := range []struct {
name string
dst **string
}{
{"start_date_from", &filter.StartDateFrom},
{"start_date_to", &filter.StartDateTo},
{"end_date_from", &filter.EndDateFrom},
{"end_date_to", &filter.EndDateTo},
} {
v := strings.TrimSpace(r.URL.Query().Get(b.name))
if v == "" {
continue
}
if _, err := time.Parse("2006-01-02", v); err != nil {
httputil.WriteError(w, http.StatusBadRequest, b.name+" must be YYYY-MM-DD")
return
}
*b.dst = &v
}
if v := strings.TrimSpace(r.URL.Query().Get("file_type")); v != "" {
ft := domain.FileType(v)
@@ -304,6 +371,222 @@ func (h *DatasetHandler) get(w http.ResponseWriter, r *http.Request) {
httputil.WriteJSON(w, http.StatusOK, dataset)
}
// datasetPutRequest is the full JSON body for PUT /datasets/{id}: every
// editable field is required. Description and Unit are pointers so an empty
// string is distinguishable from an absent field; a blank value clears the
// (nullable) column. File-related and lifecycle fields are not editable here.
type datasetPutRequest struct {
CategoryID string `json:"category_id" validate:"required,uuid"`
Name string `json:"name" validate:"required,max=255"`
Description *string `json:"description" validate:"required"`
Unit *string `json:"unit" validate:"required"`
// DgID, StartDate and EndDate are optional; omitting them (or sending null
// or a blank string) clears the corresponding nullable column.
DgID *string `json:"dg_id" validate:"omitempty,uuid"`
StartDate *string `json:"start_date" validate:"omitempty,datetime=2006-01-02"`
EndDate *string `json:"end_date" validate:"omitempty,datetime=2006-01-02"`
}
// update replaces a dataset's editable metadata (category, name, description,
// unit, dataset group, period).
func (h *DatasetHandler) update(w http.ResponseWriter, r *http.Request) {
id, ok := parseUUIDParam(w, r, "id")
if !ok {
return
}
req, err := httputil.DecodeJSON[datasetPutRequest](w, r)
if err != nil {
httputil.WriteError(w, http.StatusBadRequest, "invalid request body")
return
}
if err := h.validate.Struct(req); err != nil {
httputil.WriteValidationErrors(w, err)
return
}
categoryID, err := uuid.Parse(req.CategoryID)
if err != nil {
httputil.WriteError(w, http.StatusUnprocessableEntity, "category_id must be a valid UUID")
return
}
var groupID *uuid.UUID
if raw := optionalPtr(req.DgID); raw != nil {
gid, err := uuid.Parse(*raw)
if err != nil {
httputil.WriteError(w, http.StatusUnprocessableEntity, "dg_id must be a valid UUID")
return
}
groupID = &gid
}
dataset, err := h.svc.Update(r.Context(), id, service.DatasetUpdateInput{
CategoryID: categoryID,
Name: strings.TrimSpace(req.Name),
Description: normalizeOptional(*req.Description),
Unit: normalizeOptional(*req.Unit),
DatasetGroupID: groupID,
StartDate: optionalPtr(req.StartDate),
EndDate: optionalPtr(req.EndDate),
})
if err != nil {
respondDomainError(w, err)
return
}
httputil.WriteJSON(w, http.StatusOK, dataset)
}
// datasetPatchFields lists the JSON keys accepted by PATCH /datasets/{id}.
var datasetPatchFields = map[string]struct{}{
"category_id": {}, "name": {}, "description": {}, "unit": {},
"dg_id": {}, "start_date": {}, "end_date": {},
}
// patch applies a sparse update to a dataset's editable metadata (category,
// name, description, unit, dg_id, start_date, end_date): only fields present in
// the body are changed. Sending an explicit null for a nullable field clears
// that column.
func (h *DatasetHandler) patch(w http.ResponseWriter, r *http.Request) {
id, ok := parseUUIDParam(w, r, "id")
if !ok {
return
}
raw, err := httputil.DecodeJSON[map[string]json.RawMessage](w, r)
if err != nil {
httputil.WriteError(w, http.StatusBadRequest, "invalid request body")
return
}
if len(raw) == 0 {
httputil.WriteError(w, http.StatusBadRequest, "request body must include at least one field")
return
}
var in service.DatasetPatchInput
for key, val := range raw {
if _, ok := datasetPatchFields[key]; !ok {
httputil.WriteError(w, http.StatusUnprocessableEntity, "unknown field: "+key)
return
}
switch key {
case "category_id":
var s string
if err := json.Unmarshal(val, &s); err != nil || s == "" {
httputil.WriteError(w, http.StatusUnprocessableEntity, "category_id must be a valid UUID")
return
}
cid, err := uuid.Parse(s)
if err != nil {
httputil.WriteError(w, http.StatusUnprocessableEntity, "category_id must be a valid UUID")
return
}
in.CategoryID = &cid
case "name":
var s string
if err := json.Unmarshal(val, &s); err != nil || strings.TrimSpace(s) == "" {
httputil.WriteError(w, http.StatusUnprocessableEntity, "name must not be blank")
return
}
name := strings.TrimSpace(s)
in.Name = &name
case "description":
desc, err := decodeOptionalPatchString(val)
if err != nil {
httputil.WriteError(w, http.StatusUnprocessableEntity, "description must be a string or null")
return
}
in.Description, in.DescriptionSet = desc, true
case "unit":
unit, err := decodeOptionalPatchString(val)
if err != nil {
httputil.WriteError(w, http.StatusUnprocessableEntity, "unit must be a string or null")
return
}
in.Unit, in.UnitSet = unit, true
case "dg_id":
s, err := decodeOptionalPatchString(val)
if err != nil {
httputil.WriteError(w, http.StatusUnprocessableEntity, "dg_id must be a valid UUID or null")
return
}
if s != nil {
gid, err := uuid.Parse(*s)
if err != nil {
httputil.WriteError(w, http.StatusUnprocessableEntity, "dg_id must be a valid UUID or null")
return
}
in.DatasetGroupID = &gid
}
in.DatasetGroupIDSet = true
case "start_date":
date, err := decodeOptionalPatchDate(val)
if err != nil {
httputil.WriteError(w, http.StatusUnprocessableEntity, "start_date must be YYYY-MM-DD or null")
return
}
in.StartDate, in.StartDateSet = date, true
case "end_date":
date, err := decodeOptionalPatchDate(val)
if err != nil {
httputil.WriteError(w, http.StatusUnprocessableEntity, "end_date must be YYYY-MM-DD or null")
return
}
in.EndDate, in.EndDateSet = date, true
}
}
dataset, err := h.svc.Patch(r.Context(), id, in)
if err != nil {
respondDomainError(w, err)
return
}
httputil.WriteJSON(w, http.StatusOK, dataset)
}
// optionalPtr normalizes an optional request field: a nil or blank value
// yields nil, so an omitted or empty field clears a nullable column.
func optionalPtr(v *string) *string {
if v == nil {
return nil
}
return normalizeOptional(*v)
}
// normalizeOptional trims s and returns nil when blank, so an empty string
// clears a nullable column rather than being stored verbatim.
func normalizeOptional(s string) *string {
s = strings.TrimSpace(s)
if s == "" {
return nil
}
return &s
}
// decodeOptionalPatchString unmarshals a PATCH field that may be a JSON string
// or null, returning nil (clear the column) for null or a blank string.
func decodeOptionalPatchString(raw json.RawMessage) (*string, error) {
if string(raw) == "null" {
return nil, nil
}
var s string
if err := json.Unmarshal(raw, &s); err != nil {
return nil, err
}
return normalizeOptional(s), nil
}
// decodeOptionalPatchDate unmarshals a PATCH date field that may be a
// YYYY-MM-DD JSON string or null, returning nil (clear the column) for null or
// a blank string.
func decodeOptionalPatchDate(raw json.RawMessage) (*string, error) {
v, err := decodeOptionalPatchString(raw)
if err != nil || v == nil {
return nil, err
}
if _, err := time.Parse("2006-01-02", *v); err != nil {
return nil, err
}
return v, nil
}
// geojson returns the dataset's pre-assembled GeoJSON FeatureCollection (RFC
// 7946), generated and stored at processing time. A vector_with_kato dataset's
// collection joins the districts table on KATO code (one feature per KATO with
@@ -321,7 +604,45 @@ func (h *DatasetHandler) geojson(w http.ResponseWriter, r *http.Request) {
}
w.Header().Set("Content-Type", "application/geo+json")
w.WriteHeader(http.StatusOK)
_, _ = w.Write(fc)
_, _ = w.Write(fc) // #nosec G705 -- JSON body served as application/geo+json with nosniff, never as HTML
}
// cog streams the raster dataset's Cloud-Optimized GeoTIFF. It responds 422 for
// a non-raster dataset and 409 when the raster's COG has not been produced yet.
func (h *DatasetHandler) cog(w http.ResponseWriter, r *http.Request) {
id, ok := parseUUIDParam(w, r, "id")
if !ok {
return
}
dataset, obj, err := h.svc.COG(r.Context(), id)
if err != nil {
switch {
case errors.Is(err, service.ErrNotRaster):
httputil.WriteError(w, http.StatusUnprocessableEntity, "dataset is not a raster")
case errors.Is(err, service.ErrCOGNotReady):
httputil.WriteError(w, http.StatusConflict, "cog is not ready")
default:
respondDomainError(w, err)
}
return
}
defer obj.Close()
w.Header().Set("Content-Type", "image/tiff")
w.Header().Set("Content-Disposition", contentDisposition("inline", cogFilename(dataset.Filename)))
w.WriteHeader(http.StatusOK)
if _, err := io.Copy(w, obj); err != nil {
// Headers are already sent; nothing useful to return to the client.
return
}
}
// cogFilename derives a .cog.tif download name from the source filename.
func cogFilename(filename string) string {
if i := strings.LastIndex(filename, "."); i > 0 {
filename = filename[:i]
}
return filename + ".cog.tif"
}
func (h *DatasetHandler) download(w http.ResponseWriter, r *http.Request) {
@@ -337,7 +658,7 @@ func (h *DatasetHandler) download(w http.ResponseWriter, r *http.Request) {
defer obj.Close()
w.Header().Set("Content-Type", dataset.ContentType)
w.Header().Set("Content-Disposition", `attachment; filename="`+dataset.Filename+`"`)
w.Header().Set("Content-Disposition", contentDisposition("attachment", dataset.Filename))
if dataset.SizeBytes > 0 {
w.Header().Set("Content-Length", strconv.FormatInt(dataset.SizeBytes, 10))
}
@@ -347,6 +668,73 @@ func (h *DatasetHandler) download(w http.ResponseWriter, r *http.Request) {
}
}
// uploadSLD stores (or replaces) a dataset's SLD (Styled Layer Descriptor)
// style file. Multipart upload, mirroring POST /datasets. Applies to every
// file_type.
func (h *DatasetHandler) uploadSLD(w http.ResponseWriter, r *http.Request) {
id, ok := parseUUIDParam(w, r, "id")
if !ok {
return
}
if !parseMultipartForm(w, r) {
return
}
file, header, err := r.FormFile("file")
if err != nil {
httputil.WriteError(w, http.StatusBadRequest, "file is required")
return
}
defer file.Close()
contentType := header.Header.Get("Content-Type")
if contentType == "" {
contentType = "application/xml"
}
dataset, err := h.svc.UploadSLD(r.Context(), id, service.UploadSLDInput{
Filename: header.Filename,
ContentType: contentType,
Size: header.Size,
Reader: file,
})
if err != nil {
respondDomainError(w, err)
return
}
httputil.WriteJSON(w, http.StatusOK, dataset)
}
// downloadSLD streams a dataset's stored SLD style file. Returns 404 if none
// has been uploaded yet.
func (h *DatasetHandler) downloadSLD(w http.ResponseWriter, r *http.Request) {
id, ok := parseUUIDParam(w, r, "id")
if !ok {
return
}
dataset, obj, err := h.svc.DownloadSLD(r.Context(), id)
if err != nil {
respondDomainError(w, err)
return
}
defer obj.Close()
contentType := "application/xml"
if dataset.SldContentType != nil && *dataset.SldContentType != "" {
contentType = *dataset.SldContentType
}
filename := "style.sld"
if dataset.SldFilename != nil && *dataset.SldFilename != "" {
filename = *dataset.SldFilename
}
w.Header().Set("Content-Type", contentType)
w.Header().Set("Content-Disposition", contentDisposition("attachment", filename))
if _, err := io.Copy(w, obj); err != nil {
// Headers are already sent; nothing useful to return to the client.
return
}
}
func (h *DatasetHandler) delete(w http.ResponseWriter, r *http.Request) {
id, ok := parseUUIDParam(w, r, "id")
if !ok {

View File

@@ -2,6 +2,7 @@ package http
import (
"context"
"log/slog"
"net/http"
"gis/pkg/httputil"
@@ -13,11 +14,17 @@ type ReadinessCheck func(ctx context.Context) error
// HealthHandler serves liveness and readiness probes.
type HealthHandler struct {
checks map[string]ReadinessCheck
log *slog.Logger
}
// NewHealthHandler builds a HealthHandler with the given named readiness checks.
func NewHealthHandler(checks map[string]ReadinessCheck) *HealthHandler {
return &HealthHandler{checks: checks}
// Failed checks are logged with their error; the response only names the
// failing component, so connection details never reach the client.
func NewHealthHandler(checks map[string]ReadinessCheck, log *slog.Logger) *HealthHandler {
if log == nil {
log = slog.Default()
}
return &HealthHandler{checks: checks, log: log}
}
// Live reports that the process is up.
@@ -32,7 +39,8 @@ func (h *HealthHandler) Ready(w http.ResponseWriter, r *http.Request) {
ready := true
for name, check := range h.checks {
if err := check(r.Context()); err != nil {
results[name] = "error: " + err.Error()
h.log.Warn("readiness check failed", "component", name, "error", err)
results[name] = "unavailable"
ready = false
continue
}

View File

@@ -0,0 +1,185 @@
package http
import (
"errors"
"io"
"net/http"
"gis/internal/landscape"
"gis/internal/service"
"gis/pkg/httputil"
"github.com/go-chi/chi/v5"
"github.com/go-playground/validator/v10"
"github.com/google/uuid"
)
// LandscapeHandler serves the /landscape routes.
type LandscapeHandler struct {
svc *service.LandscapeService
validate *validator.Validate
}
// NewLandscapeHandler returns a LandscapeHandler.
func NewLandscapeHandler(svc *service.LandscapeService, validate *validator.Validate) *LandscapeHandler {
return &LandscapeHandler{svc: svc, validate: validate}
}
// Register mounts the Landscape Program routes on r.
func (h *LandscapeHandler) Register(r chi.Router) {
r.Get("/slots", h.slots)
r.Post("/slots/{code}", h.uploadSlot)
r.Get("/runs", h.listRuns)
r.Post("/runs", h.createRun)
r.Get("/runs/{id}", h.getRun)
r.Get("/results", h.results)
}
// runRequest is the body of POST /landscape/runs. Overrides are decoded as
// strings so a malformed id is a 422 instead of a 400 on the whole body.
type runRequest struct {
FromStage *int `json:"from_stage" validate:"omitempty,gte=0,lte=3"`
Overrides map[string]string `json:"overrides" validate:"omitempty,dive,uuid"`
Params *landscape.Params `json:"params"`
}
// slots lists every leaf input slot with the dataset currently filling it.
func (h *LandscapeHandler) slots(w http.ResponseWriter, r *http.Request) {
slots, err := h.svc.Slots(r.Context())
if err != nil {
respondDomainError(w, err)
return
}
httputil.WriteJSON(w, http.StatusOK, slots)
}
// uploadSlot stores a raster in the slot's dataset group under the landscape
// category. An unknown slot code yields 404.
func (h *LandscapeHandler) uploadSlot(w http.ResponseWriter, r *http.Request) {
code := chi.URLParam(r, "code")
if !parseMultipartForm(w, r) {
return
}
file, header, err := r.FormFile("file")
if err != nil {
httputil.WriteError(w, http.StatusBadRequest, "file is required")
return
}
defer file.Close()
contentType := header.Header.Get("Content-Type")
if contentType == "" {
contentType = "image/tiff"
}
dataset, err := h.svc.UploadToSlot(r.Context(), code, service.SlotUploadInput{
Filename: header.Filename,
ContentType: contentType,
Size: header.Size,
Reader: file,
Name: r.FormValue("name"),
StartDate: optionalFormValue(r, "start_date"),
EndDate: optionalFormValue(r, "end_date"),
})
if err != nil {
respondDomainError(w, err)
return
}
httputil.WriteJSON(w, http.StatusCreated, dataset)
}
// decodeRunRequest reads the run body. The whole body is optional — every field
// has a default — so an empty body is the zero request (a default incremental
// run), not an error. A malformed body writes a 400 and reports false.
func decodeRunRequest(w http.ResponseWriter, r *http.Request) (runRequest, bool) {
req, err := httputil.DecodeJSON[runRequest](w, r)
switch {
case errors.Is(err, io.EOF):
return runRequest{}, true
case err != nil:
httputil.WriteError(w, http.StatusBadRequest, "invalid request body")
return runRequest{}, false
}
return req, true
}
// createRun queues a Landscape Program run and returns it with 202 Accepted;
// the worker executes it asynchronously.
func (h *LandscapeHandler) createRun(w http.ResponseWriter, r *http.Request) {
req, ok := decodeRunRequest(w, r)
if !ok {
return
}
if err := h.validate.Struct(req); err != nil {
httputil.WriteValidationErrors(w, err)
return
}
overrides := make(map[string]uuid.UUID, len(req.Overrides))
for group, raw := range req.Overrides {
id, err := uuid.Parse(raw)
if err != nil {
httputil.WriteError(w, http.StatusUnprocessableEntity, "overrides["+group+"] must be a valid UUID")
return
}
overrides[group] = id
}
in := service.RunInput{FromStage: req.FromStage, Overrides: overrides}
if req.Params != nil {
in.Params = *req.Params
}
run, err := h.svc.CreateRun(r.Context(), in)
if err != nil {
respondDomainError(w, err)
return
}
httputil.WriteJSON(w, http.StatusAccepted, run)
}
// listRuns returns a page of runs, newest first.
func (h *LandscapeHandler) listRuns(w http.ResponseWriter, r *http.Request) {
page, ok := parsePositiveIntQuery(w, r, "page", 1)
if !ok {
return
}
pageSize, ok := parsePositiveIntQuery(w, r, "page_size", service.DefaultPageSize)
if !ok {
return
}
result, err := h.svc.ListRuns(r.Context(), page, pageSize)
if err != nil {
respondDomainError(w, err)
return
}
httputil.WriteJSON(w, http.StatusOK, newPaginated(result.Items, result.Page, result.PageSize, result.Total))
}
// getRun returns one run with its per-node execution state.
func (h *LandscapeHandler) getRun(w http.ResponseWriter, r *http.Request) {
id, ok := parseUUIDParam(w, r, "id")
if !ok {
return
}
run, err := h.svc.GetRun(r.Context(), id)
if err != nil {
respondDomainError(w, err)
return
}
httputil.WriteJSON(w, http.StatusOK, run)
}
// results lists every output group with its freshest ready dataset and the URLs
// to render it.
func (h *LandscapeHandler) results(w http.ResponseWriter, r *http.Request) {
results, err := h.svc.Results(r.Context())
if err != nil {
respondDomainError(w, err)
return
}
httputil.WriteJSON(w, http.StatusOK, results)
}

View File

@@ -0,0 +1,191 @@
package http
import (
"net/http"
"net/http/httptest"
"sort"
"strings"
"testing"
"github.com/go-chi/chi/v5"
"github.com/go-playground/validator/v10"
)
// newLandscapeTestHandler builds a handler with a nil service: every case below
// is rejected while parsing or validating the request, before the service is
// reached.
func newLandscapeTestHandler() *LandscapeHandler {
return NewLandscapeHandler(nil, validator.New(validator.WithRequiredStructEnabled()))
}
func postRun(t *testing.T, body string) *httptest.ResponseRecorder {
t.Helper()
req := httptest.NewRequest(http.MethodPost, "/landscape/runs", strings.NewReader(body))
req.Header.Set("Content-Type", "application/json")
rec := httptest.NewRecorder()
newLandscapeTestHandler().createRun(rec, req)
return rec
}
func TestCreateRunInvalidJSON(t *testing.T) {
rec := postRun(t, `{"from_stage":`)
if rec.Code != http.StatusBadRequest {
t.Fatalf("status = %d, want %d (body %s)", rec.Code, http.StatusBadRequest, rec.Body.String())
}
}
func TestCreateRunUnknownField(t *testing.T) {
// DecodeJSON rejects unknown fields, so a typo is a 400 rather than a
// silently ignored option.
rec := postRun(t, `{"from_stages":1}`)
if rec.Code != http.StatusBadRequest {
t.Fatalf("status = %d, want %d (body %s)", rec.Code, http.StatusBadRequest, rec.Body.String())
}
}
func TestCreateRunFromStageOutOfRange(t *testing.T) {
rec := postRun(t, `{"from_stage":7}`)
if rec.Code != http.StatusUnprocessableEntity {
t.Fatalf("status = %d, want %d (body %s)", rec.Code, http.StatusUnprocessableEntity, rec.Body.String())
}
if !strings.Contains(rec.Body.String(), "FromStage") {
t.Fatalf("body %s does not name the offending field", rec.Body.String())
}
}
func TestCreateRunOverrideNotUUID(t *testing.T) {
rec := postRun(t, `{"overrides":{"lp-z-lcr":"not-a-uuid"}}`)
if rec.Code != http.StatusUnprocessableEntity {
t.Fatalf("status = %d, want %d (body %s)", rec.Code, http.StatusUnprocessableEntity, rec.Body.String())
}
}
func TestUploadSlotRequiresMultipart(t *testing.T) {
req := httptest.NewRequest(http.MethodPost, "/landscape/slots/lp-z-lcr", strings.NewReader(`{}`))
req.Header.Set("Content-Type", "application/json")
rec := httptest.NewRecorder()
newLandscapeTestHandler().uploadSlot(rec, req)
if rec.Code != http.StatusBadRequest {
t.Fatalf("status = %d, want %d (body %s)", rec.Code, http.StatusBadRequest, rec.Body.String())
}
}
func TestListRunsInvalidPage(t *testing.T) {
req := httptest.NewRequest(http.MethodGet, "/landscape/runs?page=0", nil)
rec := httptest.NewRecorder()
newLandscapeTestHandler().listRuns(rec, req)
if rec.Code != http.StatusBadRequest {
t.Fatalf("status = %d, want %d (body %s)", rec.Code, http.StatusBadRequest, rec.Body.String())
}
}
func TestGetRunInvalidID(t *testing.T) {
req := httptest.NewRequest(http.MethodGet, "/landscape/runs/nope", nil)
rec := httptest.NewRecorder()
newLandscapeTestHandler().getRun(rec, req)
if rec.Code != http.StatusBadRequest {
t.Fatalf("status = %d, want %d (body %s)", rec.Code, http.StatusBadRequest, rec.Body.String())
}
}
// TestDecodeRunRequestEmptyBody pins the "all defaults" call: POST
// /landscape/runs with no body at all is a valid request, not a 400. Tested at
// the helper level because the 202 path needs a live service.
func TestDecodeRunRequestEmptyBody(t *testing.T) {
for _, body := range []string{"", "{}"} {
req := httptest.NewRequest(http.MethodPost, "/landscape/runs", strings.NewReader(body))
rec := httptest.NewRecorder()
got, ok := decodeRunRequest(rec, req)
if !ok {
t.Fatalf("body %q: decodeRunRequest reported failure with status %d", body, rec.Code)
}
if got.FromStage != nil || len(got.Overrides) != 0 || got.Params != nil {
t.Fatalf("body %q: got %+v, want the zero request", body, got)
}
if rec.Code != http.StatusOK || rec.Body.Len() != 0 {
t.Fatalf("body %q: helper wrote a response (status %d, body %s)", body, rec.Code, rec.Body.String())
}
// The zero request must also survive validation, or an empty body
// would still be rejected one step later.
if err := validator.New(validator.WithRequiredStructEnabled()).Struct(got); err != nil {
t.Fatalf("body %q: zero request fails validation: %v", body, err)
}
}
}
func TestDecodeRunRequestMalformedBody(t *testing.T) {
req := httptest.NewRequest(http.MethodPost, "/landscape/runs", strings.NewReader(`{`))
rec := httptest.NewRecorder()
if _, ok := decodeRunRequest(rec, req); ok {
t.Fatal("decodeRunRequest accepted a truncated body")
}
if rec.Code != http.StatusBadRequest {
t.Fatalf("status = %d, want %d", rec.Code, http.StatusBadRequest)
}
}
// TestRegisterMountsRoutes walks the route table of a router with the handler
// registered, so every one of the six routes is checked — including GET /slots
// and GET /results, which cannot be exercised without a live service.
func TestRegisterMountsRoutes(t *testing.T) {
r := chi.NewRouter()
r.Route("/landscape", newLandscapeTestHandler().Register)
got := map[string]bool{}
err := chi.Walk(r, func(method, route string, _ http.Handler, _ ...func(http.Handler) http.Handler) error {
got[method+" "+strings.TrimSuffix(route, "/")] = true
return nil
})
if err != nil {
t.Fatalf("walk routes: %v", err)
}
want := []string{
"GET /landscape/slots",
"POST /landscape/slots/{code}",
"GET /landscape/runs",
"POST /landscape/runs",
"GET /landscape/runs/{id}",
"GET /landscape/results",
}
for _, w := range want {
if !got[w] {
t.Errorf("route %q is not mounted; mounted routes: %v", w, keys(got))
}
}
if len(got) != len(want) {
t.Errorf("mounted %d routes, want %d: %v", len(got), len(want), keys(got))
}
}
// TestRoutedRequestReachesHandler sends a real request through the router so the
// URL parameter wiring (chi.URLParam) is exercised, not just the pattern.
func TestRoutedRequestReachesHandler(t *testing.T) {
r := chi.NewRouter()
r.Route("/landscape", newLandscapeTestHandler().Register)
req := httptest.NewRequest(http.MethodGet, "/landscape/runs/nope", nil)
rec := httptest.NewRecorder()
r.ServeHTTP(rec, req)
if rec.Code != http.StatusBadRequest {
t.Fatalf("status = %d, want %d (body %s)", rec.Code, http.StatusBadRequest, rec.Body.String())
}
if !strings.Contains(rec.Body.String(), "invalid id") {
t.Fatalf("body %s does not report the bad id", rec.Body.String())
}
}
func keys(m map[string]bool) []string {
out := make([]string, 0, len(m))
for k := range m {
out = append(out, k)
}
sort.Strings(out)
return out
}

View File

@@ -31,3 +31,29 @@ func requestLogger(log *slog.Logger) func(http.Handler) http.Handler {
})
}
}
// limitRequestBody caps every request body at maxBytes (http.MaxBytesReader),
// so a client cannot stream an unbounded upload to memory or to the temporary
// files multipart parsing spills to. A non-positive maxBytes disables the cap.
// JSON endpoints apply their own, smaller cap in httputil.DecodeJSON.
func limitRequestBody(maxBytes int64) func(http.Handler) http.Handler {
return func(next http.Handler) http.Handler {
if maxBytes <= 0 {
return next
}
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
r.Body = http.MaxBytesReader(w, r.Body, maxBytes)
next.ServeHTTP(w, r)
})
}
}
// securityHeaders sets response headers that are safe for every route.
// nosniff stops browsers from MIME-sniffing user-supplied content (downloads,
// GeoJSON) into an executable type.
func securityHeaders(next http.Handler) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.Header().Set("X-Content-Type-Options", "nosniff")
next.ServeHTTP(w, r)
})
}

View File

@@ -0,0 +1,131 @@
package http
import (
"bytes"
"context"
"errors"
"io"
"log/slog"
"mime/multipart"
"net/http"
"net/http/httptest"
"strings"
"testing"
"github.com/go-playground/validator/v10"
)
// multipartBody builds a multipart/form-data body with one file part of
// size bytes.
func multipartBody(t *testing.T, size int) (*bytes.Buffer, string) {
t.Helper()
var buf bytes.Buffer
mw := multipart.NewWriter(&buf)
part, err := mw.CreateFormFile("file", "big.tif")
if err != nil {
t.Fatal(err)
}
if _, err := part.Write(bytes.Repeat([]byte{'x'}, size)); err != nil {
t.Fatal(err)
}
if err := mw.Close(); err != nil {
t.Fatal(err)
}
return &buf, mw.FormDataContentType()
}
func TestLimitRequestBodyRejectsOversizedUpload(t *testing.T) {
body, ct := multipartBody(t, 4096)
req := httptest.NewRequest(http.MethodPost, "/landscape/slots/lp-z-lcr", body)
req.Header.Set("Content-Type", ct)
rec := httptest.NewRecorder()
h := limitRequestBody(1024)(http.HandlerFunc(newLandscapeTestHandler().uploadSlot))
h.ServeHTTP(rec, req)
if rec.Code != http.StatusRequestEntityTooLarge {
t.Fatalf("status = %d, want %d (body %s)", rec.Code, http.StatusRequestEntityTooLarge, rec.Body.String())
}
if !strings.Contains(rec.Body.String(), "1024 byte limit") {
t.Fatalf("body %s does not state the limit", rec.Body.String())
}
}
func TestLimitRequestBodyPassesSmallBody(t *testing.T) {
var got []byte
h := limitRequestBody(1024)(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
var err error
got, err = io.ReadAll(r.Body)
if err != nil {
t.Errorf("read: %v", err)
}
}))
h.ServeHTTP(httptest.NewRecorder(), httptest.NewRequest(http.MethodPost, "/", strings.NewReader("hello")))
if string(got) != "hello" {
t.Fatalf("body = %q, want hello", got)
}
}
func TestLimitRequestBodyDisabled(t *testing.T) {
h := limitRequestBody(0)(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if _, err := io.ReadAll(r.Body); err != nil {
t.Errorf("read with cap disabled: %v", err)
}
}))
h.ServeHTTP(httptest.NewRecorder(), httptest.NewRequest(http.MethodPost, "/", bytes.NewReader(make([]byte, 1<<20))))
}
func newTestRouter() http.Handler {
v := validator.New(validator.WithRequiredStructEnabled())
return NewRouter(RouterDeps{
Logger: slog.New(slog.NewTextHandler(io.Discard, nil)),
Health: NewHealthHandler(nil, nil),
Categories: NewCategoryHandler(nil, v),
Datasets: NewDatasetHandler(nil, v),
DatasetGroups: NewDatasetGroupHandler(nil, v),
Landscape: NewLandscapeHandler(nil, v),
MaxBodyBytes: 1024,
})
}
func TestRouterSetsNosniff(t *testing.T) {
rec := httptest.NewRecorder()
newTestRouter().ServeHTTP(rec, httptest.NewRequest(http.MethodGet, "/healthz", nil))
if got := rec.Header().Get("X-Content-Type-Options"); got != "nosniff" {
t.Fatalf("X-Content-Type-Options = %q, want nosniff", got)
}
}
func TestRouterCapsUploadBody(t *testing.T) {
body, ct := multipartBody(t, 4096)
req := httptest.NewRequest(http.MethodPost, "/landscape/slots/lp-z-lcr", body)
req.Header.Set("Content-Type", ct)
rec := httptest.NewRecorder()
newTestRouter().ServeHTTP(rec, req)
if rec.Code != http.StatusRequestEntityTooLarge {
t.Fatalf("status = %d, want %d (body %s)", rec.Code, http.StatusRequestEntityTooLarge, rec.Body.String())
}
}
func TestReadyDoesNotLeakCheckErrors(t *testing.T) {
h := NewHealthHandler(map[string]ReadinessCheck{
"postgres": func(context.Context) error {
return errors.New("dial tcp 10.1.2.3:5432: connect: connection refused (user=gis)")
},
"s3": func(context.Context) error { return nil },
}, slog.New(slog.NewTextHandler(io.Discard, nil)))
rec := httptest.NewRecorder()
h.Ready(rec, httptest.NewRequest(http.MethodGet, "/readyz", nil))
if rec.Code != http.StatusServiceUnavailable {
t.Fatalf("status = %d, want %d", rec.Code, http.StatusServiceUnavailable)
}
body := rec.Body.String()
if strings.Contains(body, "10.1.2.3") || strings.Contains(body, "dial tcp") {
t.Fatalf("readiness body leaks internal error details: %s", body)
}
if !strings.Contains(body, `"postgres":"unavailable"`) || !strings.Contains(body, `"s3":"ok"`) {
t.Fatalf("unexpected readiness body: %s", body)
}
}

View File

@@ -10,21 +10,32 @@ import (
// RouterDeps holds the handlers and dependencies the router mounts.
type RouterDeps struct {
Logger *slog.Logger
Health *HealthHandler
Categories *CategoryHandler
Datasets *DatasetHandler
Logger *slog.Logger
Health *HealthHandler
Categories *CategoryHandler
Datasets *DatasetHandler
// DatasetGroups serves the /dataset-groups routes.
DatasetGroups *DatasetGroupHandler
// Landscape serves the /landscape routes.
Landscape *LandscapeHandler
OpenAPISpec []byte
// MaxBodyBytes caps every request body (uploads included); <= 0 disables
// the cap.
MaxBodyBytes int64
}
// NewRouter builds the application's HTTP handler with middleware and routes.
func NewRouter(deps RouterDeps) http.Handler {
r := chi.NewRouter()
// middleware.RealIP is deliberately not used: it rewrites r.RemoteAddr
// from client-controlled headers (X-Forwarded-For, X-Real-IP,
// True-Client-IP) and is deprecated upstream as an IP-spoofing vector.
r.Use(middleware.RequestID)
r.Use(middleware.RealIP)
r.Use(requestLogger(deps.Logger))
r.Use(middleware.Recoverer)
r.Use(securityHeaders)
r.Use(limitRequestBody(deps.MaxBodyBytes))
// Health/readiness endpoints.
r.Get("/healthz", deps.Health.Live)
@@ -37,6 +48,8 @@ func NewRouter(deps RouterDeps) http.Handler {
// Domain routes. Add future feature routers here.
r.Route("/categories", deps.Categories.Register)
r.Route("/datasets", deps.Datasets.Register)
r.Route("/dataset-groups", deps.DatasetGroups.Register)
r.Route("/landscape", deps.Landscape.Register)
return r
}

View File

@@ -0,0 +1,12 @@
-- +goose Up
-- Optional SLD (Styled Layer Descriptor) style file for a dataset. Applies to
-- every file_type; a dataset may have at most one (uploading again replaces
-- it).
ALTER TABLE datasets ADD COLUMN sld_storage_key TEXT;
ALTER TABLE datasets ADD COLUMN sld_filename VARCHAR(255);
ALTER TABLE datasets ADD COLUMN sld_content_type VARCHAR(255);
-- +goose Down
ALTER TABLE datasets DROP COLUMN sld_storage_key;
ALTER TABLE datasets DROP COLUMN sld_filename;
ALTER TABLE datasets DROP COLUMN sld_content_type;

View File

@@ -0,0 +1,15 @@
-- +goose Up
-- Optional period end date, so a year_column mapping can represent a date
-- range (date..date_to) instead of a single point in time. NULL preserves the
-- pre-existing point-in-time behavior; the unique index treats a NULL date_to
-- as equal to date itself so point observations keep deduplicating exactly as
-- before.
ALTER TABLE dataset_observations ADD COLUMN date_to DATE;
DROP INDEX uq_dataset_observations;
CREATE UNIQUE INDEX uq_dataset_observations ON dataset_observations (dataset_id, kato_code, date, COALESCE(date_to, date));
-- +goose Down
DROP INDEX uq_dataset_observations;
ALTER TABLE dataset_observations DROP COLUMN date_to;
CREATE UNIQUE INDEX uq_dataset_observations ON dataset_observations (dataset_id, kato_code, date);

View File

@@ -0,0 +1,67 @@
-- +goose Up
-- Dataset groups: "одноимённые" datasets versioned by period. A group may carry a
-- shared SLD used as a fallback for its datasets.
CREATE TABLE dataset_groups (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
code VARCHAR(255) NOT NULL UNIQUE,
name VARCHAR(255) NOT NULL,
description TEXT NOT NULL DEFAULT '',
sld_storage_key TEXT,
sld_filename VARCHAR(255),
sld_content_type VARCHAR(255),
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
updated_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
ALTER TABLE datasets ADD COLUMN dg_id UUID REFERENCES dataset_groups (id) ON DELETE RESTRICT;
ALTER TABLE datasets ADD COLUMN start_date DATE;
ALTER TABLE datasets ADD COLUMN end_date DATE;
ALTER TABLE datasets ADD CONSTRAINT chk_datasets_period
CHECK (start_date IS NULL OR end_date IS NULL OR end_date >= start_date);
CREATE INDEX idx_datasets_dg_id_end_date ON datasets (dg_id, end_date DESC NULLS LAST, created_at DESC);
CREATE INDEX idx_datasets_start_date ON datasets (start_date);
CREATE INDEX idx_datasets_end_date ON datasets (end_date);
-- Period columns are renamed everywhere: observations, and the year_columns
-- mapping JSON ({column, date, date_to} -> {column, start_date, end_date}).
ALTER TABLE dataset_observations RENAME COLUMN date TO start_date;
ALTER TABLE dataset_observations RENAME COLUMN date_to TO end_date;
DROP INDEX uq_dataset_observations;
DROP INDEX idx_dataset_observations_kato_date;
CREATE UNIQUE INDEX uq_dataset_observations ON dataset_observations (dataset_id, kato_code, start_date, COALESCE(end_date, start_date));
CREATE INDEX idx_dataset_observations_kato_date ON dataset_observations (kato_code, start_date);
UPDATE datasets
SET year_columns = (
SELECT jsonb_agg(
(elem - 'date' - 'date_to')
|| jsonb_build_object('start_date', elem->'date')
|| CASE WHEN elem ? 'date_to' AND elem->'date_to' IS NOT NULL
THEN jsonb_build_object('end_date', elem->'date_to') ELSE '{}'::jsonb END)
FROM jsonb_array_elements(year_columns) AS elem)
WHERE year_columns IS NOT NULL AND jsonb_typeof(year_columns) = 'array' AND jsonb_array_length(year_columns) > 0;
-- +goose Down
UPDATE datasets
SET year_columns = (
SELECT jsonb_agg(
(elem - 'start_date' - 'end_date')
|| jsonb_build_object('date', elem->'start_date')
|| CASE WHEN elem ? 'end_date' AND elem->'end_date' IS NOT NULL
THEN jsonb_build_object('date_to', elem->'end_date') ELSE '{}'::jsonb END)
FROM jsonb_array_elements(year_columns) AS elem)
WHERE year_columns IS NOT NULL AND jsonb_typeof(year_columns) = 'array' AND jsonb_array_length(year_columns) > 0;
DROP INDEX uq_dataset_observations;
DROP INDEX idx_dataset_observations_kato_date;
ALTER TABLE dataset_observations RENAME COLUMN start_date TO date;
ALTER TABLE dataset_observations RENAME COLUMN end_date TO date_to;
CREATE UNIQUE INDEX uq_dataset_observations ON dataset_observations (dataset_id, kato_code, date, COALESCE(date_to, date));
CREATE INDEX idx_dataset_observations_kato_date ON dataset_observations (kato_code, date);
DROP INDEX idx_datasets_dg_id_end_date;
DROP INDEX idx_datasets_start_date;
DROP INDEX idx_datasets_end_date;
ALTER TABLE datasets DROP CONSTRAINT chk_datasets_period;
ALTER TABLE datasets DROP COLUMN dg_id;
ALTER TABLE datasets DROP COLUMN start_date;
ALTER TABLE datasets DROP COLUMN end_date;
DROP TABLE dataset_groups;

View File

@@ -0,0 +1,51 @@
-- +goose Up
-- Landscape Program: seed the category and dataset groups (input slots and
-- node outputs; codes must match internal/landscape/graph.go), and store runs.
INSERT INTO categories (code, name, description)
VALUES ('landscape', 'Ландшафтная программа', 'Входные и результирующие растры Ландшафтной программы')
ON CONFLICT (code) DO NOTHING;
INSERT INTO dataset_groups (code, name) VALUES
('lp-z-biodiversity', 'Z Biodiversity'), ('lp-z-lcr', 'Z LCR'), ('lp-z-soil-protection', 'Z Soil Protection'), ('lp-z-wr', 'Z Water Resources'),
('lp-u-surf-water', 'U Surface Water'), ('lp-u-soil', 'U Soil'), ('lp-u-crop', 'U Crop'), ('lp-u-veg', 'U Vegetation'),
('lp-p-irrigation', 'P Irrigation'), ('lp-p-snow', 'P Snow'), ('lp-p-crop', 'P Crop'), ('lp-p-pasture', 'P Pasture'), ('lp-p-tourism', 'P Tourism'),
('lp-z-integrated', 'Z Integrated'), ('lp-u-integrated', 'U Integrated'), ('lp-z-max', 'Z Max'), ('lp-u-max', 'U Max'),
('lp-e', 'E Ecological Constraint'),
('lp-c-irrigation', 'C Irrigation'), ('lp-c-snow', 'C Snow'), ('lp-c-crop', 'C Crop'), ('lp-c-pasture', 'C Pasture'), ('lp-c-tourism', 'C Tourism'),
('lp-p-max', 'P Max'),
('lp-v1-conservative', 'V1 Conservative'), ('lp-v2-balanced', 'V2 Balanced'), ('lp-v3-development', 'V3 Development'), ('lp-c-max', 'C Max')
ON CONFLICT (code) DO NOTHING;
CREATE TABLE lp_runs (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
status VARCHAR(32) NOT NULL,
from_stage SMALLINT,
params JSONB NOT NULL,
overrides JSONB NOT NULL DEFAULT '{}'::jsonb,
error TEXT,
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
started_at TIMESTAMPTZ,
finished_at TIMESTAMPTZ
);
CREATE INDEX idx_lp_runs_created_at ON lp_runs (created_at DESC);
CREATE TABLE lp_run_nodes (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
run_id UUID NOT NULL REFERENCES lp_runs (id) ON DELETE CASCADE,
node_code VARCHAR(8) NOT NULL,
stage SMALLINT NOT NULL,
status VARCHAR(32) NOT NULL,
fingerprint TEXT NOT NULL DEFAULT '',
input_datasets JSONB NOT NULL DEFAULT '[]'::jsonb,
output_datasets JSONB,
error TEXT,
started_at TIMESTAMPTZ,
finished_at TIMESTAMPTZ,
UNIQUE (run_id, node_code)
);
CREATE INDEX idx_lp_run_nodes_lookup ON lp_run_nodes (node_code, fingerprint, status, finished_at DESC);
-- +goose Down
DROP TABLE lp_run_nodes;
DROP TABLE lp_runs;
-- Seeded category/groups are left in place (datasets may reference them).

2
test/acceptance/.gitignore vendored Normal file
View File

@@ -0,0 +1,2 @@
results/
__pycache__/

320
test/acceptance/README.md Normal file
View File

@@ -0,0 +1,320 @@
# Пакет приёмочных испытаний (Этап 8)
Пакет позволяет Заказчику самостоятельно воспроизвести то, что заявлено в
документах Этапа 8, на изолированном испытательном стенде, развёрнутом из
исходного кода этой ветки:
| № | Что проверяется | Скрипт | Документ |
|---|---|---|---|
| 1 | 89 функциональных проверок испытательного стенда (F01–F38, N01–N38, L02–L05, L07–L15) | `functional.sh` | Приложение 7 «Протокол функционального тестирования», раздел 5 |
| 2 | Попиксельная сверка расчёта Ландшафтной программы (ЛП) с независимой эталонной реализацией | `lp_gen.py`, `lp_run.sh`, `lp_verify.py` | ЛП техописание (правила пикселя, граф узлов); Приложение 7, L07/L08/L12 |
| 3 | Нагрузочные испытания: 1000 одновременных клиентов, сценарии S1–S6 | `load.sh` (+ `seed.sh`) | Приложение 3 «Протокол нагрузочного тестирования» |
| 4 | Повторный анализ безопасности: 0 уязвимостей High/Critical | `scan.sh` | Приложение 4 «Отчет об устранении уязвимостей» |
Все скрипты — bash (macOS и Linux, `set -euo pipefail`, совместимы с bash 3.2),
curl, jq и Python 3. Результаты прогонов сохраняются в `test/acceptance/results/`
(каталог не версионируется).
> **Внимание.** `functional.sh`, `lp_run.sh`, `seed.sh` и сценарий S5 из
> `load.sh` **создают данные** (в том числе заполняют слоты ЛП). Их следует
> запускать только на изолированном стенде `gis-acceptance`. На рабочем стенде
> https://dssgis.dwh.kz допускается только `lp_verify.py` — он выполняет
> исключительно GET-запросы (`lp_run.sh` и `seed.sh` отказываются работать с
> этим адресом).
## Состав пакета
| Файл | Назначение |
|---|---|
| `compose.acceptance.yml` | Надстройка над `deployments/docker-compose.yml` для изолированного стенда (проект `gis-acceptance`) |
| `stand.sh` | Развёртывание (`up`: сборка, миграции, справочник районов, ожидание `/readyz`) и удаление (`down`) стенда |
| `functional.sh` | 89 функциональных проверок, построчный вывод `ID \| описание \| PASS/FAIL \| detail` |
| `lp_gen.py` | Генерация 13 синтетических входных растров ЛП и одного невыровненного растра |
| `lp_run.sh` | Загрузка входов в 13 слотов → прогон `{"from_stage":0}` → `lp_verify.py` |
| `lp_verify.py` | Независимая (numpy) реализация всех узлов ЛП и попиксельная сверка 15 выходов; только GET |
| `load.sh` | Сценарии нагрузки S1–S6 (hey 0.1.5), сводная таблица, CSV/JSON |
| `seed.sh` | Наполнение стенда данными для нагрузочных испытаний (`seed.sh clean` — удаление) |
| `scan.sh` | govulncheck (исходный код и бинарный файл), gosec, trivy fs, trivy image, golangci-lint |
| `data/` | Тестовые векторные данные (участки, население и поголовье КРС по 14 районам СКО с реальными кодами КАТО), SLD-стили |
| `lib/` | Вспомогательные скрипты: генерация NDVI, анализ вывода hey, Dockerfile генератора нагрузки, пересоздание `data/` |
| `requirements.txt` | Python-зависимости (`numpy`, `rasterio`) |
## Требования
* Docker Engine с Docker Compose v2.24+ (используется тег `!override` для портов).
* `curl`, `jq`, Python 3.10+ с `numpy` и `rasterio`:
```sh
python3 -m venv .venv && .venv/bin/pip install -r test/acceptance/requirements.txt
export PYTHON=$PWD/.venv/bin/python # скрипты используют $PYTHON (по умолчанию python3)
```
* Для `scan.sh`: Go (версия toolchain берётся из `go.mod`), `govulncheck`,
`gosec`, `trivy`; `golangci-lint` — по желанию.
## 0. Испытательный стенд
```sh
git clone https://dssgit.dwh.kz/infureal/gis.git && cd gis && git checkout stage8/security-fixes
test/acceptance/stand.sh up # сборка образа, запуск, migrate up, import-districts, ожидание /readyz
```
`stand.sh up` выполняет те же шаги, что и вручную:
```sh
C="docker compose -p gis-acceptance -f deployments/docker-compose.yml -f test/acceptance/compose.acceptance.yml"
$C up -d --build --wait postgres minio rabbitmq
$C run --rm --no-deps app migrate up # 11 миграций 00001…00011
$C run --rm --no-deps app import-districts /stubs/districts.geojson # 14 районов СКО
$C up -d app worker
curl http://localhost:18080/readyz # {"components":{"postgres":"ok","rabbitmq":"ok","s3":"ok"},"ready":true}
```
Порты стенда: API — 18080, PostgreSQL — 15432, MinIO — 19000 (API) / 19001
(консоль), RabbitMQ — 15672 (только веб-интерфейс управления). Сеть контейнеров —
`gis-acceptance_default`. Учётные данные — значения по умолчанию из
`deployments/docker-compose.yml` (стенд изолированный).
Переменные окружения стенда:
| Переменная | По умолчанию | Назначение |
|---|---|---|
| `POSTGIS_PLATFORM` | `linux/amd64` | Образ `postgis/postgis:17-3.5` публикуется только для amd64. На Apple Silicon (arm64) он запускается в эмуляции, на amd64-хостах — нативно, поэтому значение по умолчанию работает на обеих платформах |
| `MINIO_IMAGE` | `minio/minio:latest` | Если образ `minio/minio` недоступен в реестре, подойдёт любой образ с сервером `minio` и клиентом `mc`, например `MINIO_IMAGE=cgr.dev/chainguard/minio:latest` |
| `APP_IMAGE` | `gis-acceptance-app:local` | Тег локально собранного образа приложения |
| `HTTP_MAX_UPLOAD_BYTES` | `2147483648` | Предельный размер тела запроса (2 ГиБ) |
| `LP_MAX_WORKERS` / `LP_MAX_NODES` | `8` / `2` | Параллелизм расчёта ЛП |
Удаление стенда вместе с томами: `test/acceptance/stand.sh down`
(= `docker compose -p gis-acceptance down -v`).
## 1. Функциональные проверки — `functional.sh`
```sh
test/acceptance/functional.sh # стенд http://localhost:18080
BASE_URL=http://host:18080 LOCAL_CHECKS=0 test/acceptance/functional.sh # удалённый испытательный стенд
```
| Переменная | По умолчанию | Назначение |
|---|---|---|
| `BASE_URL` | `http://localhost:18080` | Адрес API |
| `LOCAL_CHECKS` | `auto` | `1` — подтверждение в PostGIS и MinIO через контейнеры проекта (`docker exec … psql`, `mc` в контейнере MinIO): F35 (часть MinIO), F36, F37, F38, ресурсы worker в L15. `0` — эти проверки пропускаются (`SKIP`). `auto` — `1`, если запущены контейнеры проекта `gis-acceptance` |
| `COMPOSE_PROJECT` | `gis-acceptance` | Проект Compose для `LOCAL_CHECKS` |
| `KEEP_DATA` | `0` | `1` — не удалять созданные данные |
| `WAIT_TIMEOUT` | `300` | Ожидание асинхронной обработки, с |
Скрипт создаёт собственные тестовые данные (все коды имеют уникальный префикс
`acc<время>-`), никогда не изменяет и не удаляет существовавшие объекты и в
конце удаляет всё созданное: наборы данных (в том числе входы и выходы ЛП своих
прогонов), группы, категории. Записи прогонов ЛП остаются — API не
предусматривает их удаления. Порядок и смысл проверок совпадают с таблицами
раздела 5 Приложения 7.
Ожидаемый вывод (фрагмент):
```
F01 | Создание корневой категории POST /categories | PASS | 201; acc1792084680-ecology «Экология», parent_id null
...
N38 | Загрузка 100 МБ мусора с сигнатурой TIFF | PASS | 201 (104 857 604 байт за 0.5 с) → status failed, parse_error от gdal_translate; /readyz 200, сервис доступен
...
L07 | Попиксельная сверка 15 выходов с эталонной реализацией | PASS | 15 выходов × 6 000 000 пикселей; «TOTAL mismatching pixels: 0 -> PASS» …
...
ИТОГО: 89 проверок — PASS 89, FAIL 0, SKIP 0
RESULT: 89/89 PASS
```
Код возврата ненулевой при любом `FAIL`. Журнал и вспомогательные выводы
(`L07_lp_verify.txt`, `L08_points.txt`, `L12_lp_verify.txt`, `F36.txt`) —
в `results/functional-<время>/`.
### Соответствие Приложению 7 и уточнения методики
| Проверки | Раздел Приложения 7 | Как проверяется в пакете |
|---|---|---|
| F01–F09 | 5.2 Категории | HTTP-код и поля ответа, контроль цикла, дубликат, удаление |
| F10–F34, N01–N13 | 5.3 Импорт/экспорт, SLD, группы, фильтры | Загрузка `data/plots_nko.geojson` (vector), `data/population_kato.geojson` и `data/livestock_kato.gpkg` (vector_with_kato), растра NDVI 600×400 EPSG:4326 Float32 (генерируется `lib/helpers.py`) |
| N14–N38, F35 | 5.4 Негативные проверки | 12 некорректных загрузок, 10 эндпоинтов с несуществующим id, 4 — с некорректным id, некорректные тела JSON, параметры ЛП, 100 МБ мусора |
| F36–F38 | 5.5 PostGIS и MinIO | SQL `ST_GeometryType/ST_SRID/ST_IsValid`, соединение наблюдений с `districts`, `mc stat` объектов |
| L02–L15 | 5.6 Ландшафтная программа | 13 растров `lp_gen.py`, прогоны, `lp_verify.py`, инкрементальный пересчёт, `from_stage`, параметры, невыровненная сетка |
Уточнения относительно протокола (смысл и ожидаемые результаты проверок не
изменены):
* **Коды объектов** имеют префикс прогона (`acc…-ecology` вместо `ecology`),
чтобы повторный запуск не конфликтовал с данными прошлых прогонов.
* **F33, F34** — выборки ограничены собственной категорией (`category_id`), чтобы
результат не зависел от других данных стенда; во втором фильтре F34
(`file_type + status`) добавлен `category_id`.
* **F12** — охват проверяется по координатам выгруженного GeoJSON (вместо
`ogrinfo`); **F22, L05** — формат COG проверяется через rasterio/GDAL
(`LAYOUT=COG`, сжатие, размер, EPSG, тип, NoData) вместо `gdalinfo`;
**L08** — 9 точек проверяются `lp_verify.py --points` вместо `gdallocationinfo`.
* **F38** — проверяется наличие всех 7 объектов, созданных скриптом (4 исходных
файла, COG, SLD набора, SLD группы), а не общее число объектов в бакете.
* **F16, F17** — эталонные значения берутся из `data/population_kato.geojson`
(значения, приведённые в протоколе, в файле сохранены).
* **L05** — перед чтением `/landscape/results` скрипт дожидается статуса `ready`
всех 15 выходов: конвертация выходов в COG выполняется worker'ом асинхронно
после завершения прогона, а `/landscape/results` возвращает только готовые слои.
* **N38 и ограничение размера тела запроса.** В этой ветке (исправления по
Приложению 4) тело запроса ограничено `HTTP_MAX_UPLOAD_BYTES` (по умолчанию
2 ГиБ); загрузка сверх предела отклоняется с **413** `request body exceeds
the … byte limit`. Файл N38 (100 МБ) меньше предела, поэтому ожидание N38
не изменилось: 201, затем `status = failed`. Проверить 413 можно вручную,
запустив стенд с `HTTP_MAX_UPLOAD_BYTES=1048576` и загрузив файл больше 1 МиБ.
* **`/readyz`** в этой ветке не раскрывает текст ошибок зависимостей: при сбое
компонент помечается `"unavailable"` (ранее — `"error: <текст>"`). На исправном
стенде ответ не изменился: `200 {"components":{"postgres":"ok","rabbitmq":"ok","s3":"ok"},"ready":true}`;
N38 проверяет `200` и `ready = true`.
* **L15** — PASS определяется успешным полным прогоном (202, `succeeded`, 9 узлов
`succeeded`); время CPU и `memory.peak` worker (cgroup v2) выводятся
справочно и включают COG-конвертацию выходов.
## 2. Ландшафтная программа: попиксельная сверка
### Входные данные — `lp_gen.py`
```sh
$PYTHON test/acceptance/lp_gen.py --out /tmp/lp-in [--width 3000 --height 2000]
```
13 растров `<код слота>.tif`: Int16, EPSG:32642, пиксель 100 м, 3000×2000,
классы 1…5 из гладких тригонометрических полей (у каждого слота своё),
рамка NoData −9999 шириной 40 пикселей. Особые случаи для правил пикселя:
в `lp-u-crop` — блок NoData 200×200 (U считается по water, soil и veg),
в `lp-u-veg` — блок NoData 100×100, наполовину перекрывающий его (там нет ни
crop, ни veg → U = NoData), в `lp-p-snow` — блоки со значениями 0 и 6
(не NoData, но вне 1…5 → недопустимы). Плюс `misaligned-lp-u-veg.tif`
(1500×1000, 200 м) для проверки L13. Генерация детерминирована.
### Прогон на стенде — `lp_run.sh`, режим (a)
```sh
test/acceptance/lp_run.sh # генерация → 13 слотов → {"from_stage":0} → lp_verify.py
```
Входы и выходы остаются на стенде, прогон можно перепроверить:
`$PYTHON test/acceptance/lp_verify.py --base-url http://localhost:18080 --run <run-id>`.
### Проверка любого стенда (только чтение) — `lp_verify.py`, режим (b)
```sh
$PYTHON test/acceptance/lp_verify.py --base-url https://dssgis.dwh.kz --run <run-id> [--cache DIR]
```
Скрипт выполняет только GET-запросы:
1. `GET /landscape/runs/{id}` — параметры прогона, для каждого узла `inputs` и
`outputs` (группа → набор данных);
2. `GET /datasets/{id}.cog` — 13 входов (наборы слотов, которые прогон
фактически использовал) и 15 выходов (`--inputs-from download` читает
входы через `/datasets/{id}/download` — исходный файл, который читал узел).
Ожидаемые значения рассчитываются по всей цепочке **только из 13 входов**
(E — из пересчитанных Z и U, C и V — из пересчитанного E), поэтому ошибка в
промежуточном слое не может скрыть ошибку в последующем. Дополнительно
проверяется согласованность цепочки самого прогона (узел 10 читал выход узла
00 этого прогона и т. д.) и совпадение сетки всех 28 растров. Растры
обрабатываются окнами 512×4096 пикселей (выровнены по тайлам COG 512), поэтому
расход памяти ограничен (~0,5 ГБ) и для растров рабочего стенда 22219×14977.
Правила пикселя (источник истины — `internal/landscape/kernels.go`, они же —
в ЛП техописании): значение допустимо при `1 ≤ v ≤ 5`; `classify` в float32
(`≤1.8→1, ≤2.6→2, ≤3.4→3, ≤4.2→4, ≤5.0→5`); z_integrated — среднее четырёх Z с
весом 0,25; u_integrated — water ∧ soil ∧ (crop ∨ veg), веса 0,25 только у
допустимых crop/veg; z_max/u_max — максимум; E = max(Z, U); C_j = min(P_j, E);
p_max/c_max — максимум по допустимым; func_zoning V1–V3 с порогами
`v1_min_s_e12`, `v1_min_s_e3`, `v2_min_s_e3`, `v2_min_s_e4` и `tie_priority`
из параметров прогона.
Вывод: по каждому из 15 выходов — число пикселей, допустимых (по эталону) и
**число несовпадающих пикселей** (с координатами первого расхождения), итоговая
строка `TOTAL mismatching pixels: N -> PASS/FAIL`. Код возврата: 0 — полное
совпадение, 1 — есть расхождения, 2 — ошибка данных (прогон не `succeeded`,
не найден вход/выход, разные сетки).
## 3. Нагрузочные испытания — `load.sh`
```sh
test/acceptance/load.sh # S1–S6, генератор внутри сети стенда
SCENARIOS="S1 S3 S6" test/acceptance/load.sh # часть сценариев
GEN=host TARGET_URL=http://localhost:18080 test/acceptance/load.sh # hey на хосте, через порт стенда
NETWORK=bridge TARGET_URL=https://host BASE_URL=https://host SCENARIOS="S1 S3" test/acceptance/load.sh
```
Генератор — hey 0.1.5, собирается из исходного кода в образ
`gis-acceptance-hey:0.1.5` (`lib/hey.Dockerfile`, на тех же закреплённых образах
Go/Alpine, что и приложение) и по умолчанию запускается **внутри сети стенда**
(`docker run --network gis-acceptance_default …`, цель `http://app:8080`).
Перед нагрузкой `seed.sh` (идемпотентно) наполняет стенд: 25 vector, 25
vector_with_kato (с маппингом КАТО, 14 районов × 3 года) и 50 raster в
категории `acc-load`; если прогонов ЛП ещё нет, выполняется `lp_run.sh`.
На удалённом стенде без `ALLOW_WRITES=1` данные не создаются: id для
эндпоинтов определяются GET-запросами, сценарий S5 пропускается.
Эндпоинты (табл. 5 Приложения 3): `/datasets?page_size=20`, `/datasets/{id}`,
`/datasets/{id}.geojson`, `/datasets/{id}/observations`, `/categories`,
`/dataset-groups/{id}/latest`, `/landscape/slots`, `/landscape/results`,
`/landscape/runs/{id}`, `/readyz`.
| Сценарий | Параметры hey |
|---|---|
| S1 | `-n 100 -c 1` последовательно по 10 эндпоинтам (1000 запросов) |
| S2 | 1000 запросов, 100 клиентов: 10 параллельных процессов `-n 100 -c 10` |
| S3.1 | `-n 1000 -c 1000 /datasets?page_size=20`, 3 прогона |
| S3.2 | `-n 1000 -c 1000 /landscape/results`, 3 прогона |
| S4 | `-z 60s -q 1`, 1000 клиентов в 8 процессах: 300 `/datasets?page_size=20`, 150 `/datasets/{id}`, по 100 `.geojson`, `/observations`, `/categories`, `/latest`, `/landscape/results`, 50 `/readyz` |
| S5 | `-z 30s -c 10` × 10 эндпоинтов + 20 параллельных `POST /datasets` (10 GeoJSON + 10 GeoTIFF), все загрузки должны дойти до `ready` (затем удаляются) |
| S6 | Повтор S1 и `GET /readyz` после нагрузки |
S3.3 протокола (1000 одновременных запросов через опубликованный порт)
воспроизводится запуском `GEN=host TARGET_URL=http://localhost:18080 SCENARIOS=S3`.
Вывод: сводная таблица (запросы, клиенты, длительность, RPS, доля успешных
ответов, min/avg/P50/P90/P95/P99/max в мс, коды ответов, транспортные ошибки,
итог по критериям), таблицы по эндпоинтам/прогонам для S2–S5, результаты
загрузок S5, пиковое потребление CPU/памяти контейнерами (docker stats во время
S4/S5) и критерий K4. Как и в протоколе, перцентили сценариев из нескольких
процессов/прогонов — средневзвешенные по числу ответов. Файлы:
`results/load-<время>/summary.csv`, `summary.json`, `detail.csv`, `raw/*.txt`
(исходные выводы hey). Критерии (табл. 8 Приложения 3): K1 — успешных ответов
≥ 99 %, K2 — ответов 5xx нет, K3 — P95 ≤ 2000 мс, K4 — после нагрузки `/readyz`
200 и P50 S6 не хуже P50 S1 (допуск ×2 + 1 мс). Код возврата 1 при
невыполнении любого критерия.
Генератор и стенд работают на одной машине и делят CPU, поэтому абсолютные
значения RPS зависят от оборудования (протокол фиксирует их как нижнюю оценку).
## 4. Анализ безопасности — `scan.sh`
```sh
test/acceptance/scan.sh
```
| Инструмент | Что проверяет | Что считается High/Critical |
|---|---|---|
| `govulncheck ./...` | Уязвимости Go-модулей и стандартной библиотеки, достижимые из кода | Каждая уязвимость с цепочкой вызовов из кода (у govulncheck нет шкалы severity) |
| `govulncheck -mode=binary` | Бинарный файл `/usr/local/bin/gis` из собранного образа | Каждая уязвимость с затронутыми символами в бинарнике |
| `gosec ./...` | Статический анализ кода | Severity `HIGH` |
| `trivy fs` | Зависимости (`go.mod`), секреты, конфигурации (Dockerfile, compose) | `HIGH`, `CRITICAL` |
| `trivy image` | Образ приложения (Alpine + GDAL + бинарник), собирается из `build/package/Dockerfile` | `HIGH`, `CRITICAL` |
| `golangci-lint` | Качество кода (если установлен) | Справочно: число замечаний (шкалы severity нет) |
Скрипт печатает таблицу «инструмент | цель | High | Critical | примечания» и
итог `RESULT: 0 High / 0 Critical -> PASS`; код возврата 1 при наличии
High/Critical. Полные отчёты (JSON) — `results/scan-<время>/`. Без Docker:
`SKIP_IMAGE=1` (бинарник собирается локально с cgo и GDAL, `trivy image`
пропускается).
## Порядок полного воспроизведения
```sh
test/acceptance/stand.sh up
test/acceptance/functional.sh | tee functional.txt # RESULT: 89/89 PASS
test/acceptance/lp_run.sh | tee lp_local.txt # TOTAL mismatching pixels: 0 -> PASS
$PYTHON test/acceptance/lp_verify.py --base-url https://dssgis.dwh.kz --run <run-id> | tee lp_prod.txt
test/acceptance/load.sh | tee load.txt # критерии K1–K4 выполнены
test/acceptance/scan.sh | tee scan.txt # RESULT: 0 High / 0 Critical -> PASS
test/acceptance/stand.sh down
```
Тестовые векторные данные `data/` пересоздаются командой
`python3 test/acceptance/lib/make_fixtures.py` (нужен `ogr2ogr`).

View File

@@ -0,0 +1,55 @@
# Isolated acceptance-test stand (Приложения 3, 4, 7 к отчёту Этапа 8).
#
# Overlay on deployments/docker-compose.yml. Always use the project name
# gis-acceptance so the stand never collides with a development or production
# stack on the same host:
#
# docker compose -p gis-acceptance \
# -f deployments/docker-compose.yml \
# -f test/acceptance/compose.acceptance.yml up -d --build
#
# Host ports: app 18080, postgres 15432, minio 19000 (API) / 19001 (console),
# rabbitmq 15672 (management UI only; AMQP stays inside the docker network).
#
# Environment knobs (all optional):
# POSTGIS_PLATFORM platform of postgis/postgis (default linux/amd64: the image
# has no arm64 build; Apple Silicon runs it under emulation,
# amd64 hosts run it natively, so the default works on both)
# MINIO_IMAGE MinIO image (default minio/minio:latest). Any image that
# ships the `minio` server and the `mc` client works, e.g.
# cgr.dev/chainguard/minio:latest.
# APP_IMAGE tag for the locally built application image
# HTTP_MAX_UPLOAD_BYTES, LP_MAX_WORKERS, LP_MAX_NODES — passed to app/worker.
name: gis-acceptance
services:
app:
image: ${APP_IMAGE:-gis-acceptance-app:local}
environment:
HTTP_MAX_UPLOAD_BYTES: ${HTTP_MAX_UPLOAD_BYTES:-2147483648}
LP_MAX_WORKERS: ${LP_MAX_WORKERS:-8}
LP_MAX_NODES: ${LP_MAX_NODES:-2}
ports: !override
- "18080:8080"
worker:
image: ${APP_IMAGE:-gis-acceptance-app:local}
environment:
HTTP_MAX_UPLOAD_BYTES: ${HTTP_MAX_UPLOAD_BYTES:-2147483648}
LP_MAX_WORKERS: ${LP_MAX_WORKERS:-8}
LP_MAX_NODES: ${LP_MAX_NODES:-2}
postgres:
platform: ${POSTGIS_PLATFORM:-linux/amd64}
ports: !override
- "15432:5432"
minio:
image: ${MINIO_IMAGE:-minio/minio:latest}
ports: !override
- "19000:9000"
- "19001:9001"
rabbitmq:
ports: !override
- "15672:15672"

View File

@@ -0,0 +1,19 @@
<?xml version="1.0" encoding="UTF-8"?>
<sld:StyledLayerDescriptor version="1.0.0" xmlns:sld="http://www.opengis.net/sld" xmlns:ogc="http://www.opengis.net/ogc">
<sld:NamedLayer>
<sld:Name>group</sld:Name>
<sld:UserStyle>
<sld:Title>Group fallback style</sld:Title>
<sld:FeatureTypeStyle>
<sld:Rule>
<sld:RasterSymbolizer>
<sld:ColorMap>
<sld:ColorMapEntry color="#f7fbff" quantity="0"/>
<sld:ColorMapEntry color="#08306b" quantity="100000"/>
</sld:ColorMap>
</sld:RasterSymbolizer>
</sld:Rule>
</sld:FeatureTypeStyle>
</sld:UserStyle>
</sld:NamedLayer>
</sld:StyledLayerDescriptor>

Binary file not shown.

View File

@@ -0,0 +1,145 @@
{
"type": "FeatureCollection",
"features": [
{
"type": "Feature",
"properties": {
"plot_id": 1,
"name": "Участок 1",
"area_ha": 1680.0
},
"geometry": {
"type": "Polygon",
"coordinates": [
[
[
69.0,
53.6
],
[
69.6,
53.6
],
[
69.6,
54.0
],
[
69.0,
54.0
],
[
69.0,
53.6
]
]
]
}
},
{
"type": "Feature",
"properties": {
"plot_id": 2,
"name": "Участок 2",
"area_ha": 2100.0
},
"geometry": {
"type": "Polygon",
"coordinates": [
[
[
69.9,
54.2
],
[
70.5,
54.2
],
[
70.5,
54.7
],
[
69.9,
54.7
],
[
69.9,
54.2
]
]
]
}
},
{
"type": "Feature",
"properties": {
"plot_id": 3,
"name": "Участок 3",
"area_ha": 1400.0
},
"geometry": {
"type": "Polygon",
"coordinates": [
[
[
70.6,
53.7
],
[
71.1,
53.7
],
[
71.1,
54.1
],
[
70.6,
54.1
],
[
70.6,
53.7
]
]
]
}
},
{
"type": "Feature",
"properties": {
"plot_id": 4,
"name": "Участок 4",
"area_ha": 1750.0
},
"geometry": {
"type": "Polygon",
"coordinates": [
[
[
71.2,
54.5
],
[
71.7,
54.5
],
[
71.7,
55.0
],
[
71.2,
55.0
],
[
71.2,
54.5
]
]
]
}
}
]
}

View File

@@ -0,0 +1,243 @@
{
"type": "FeatureCollection",
"features": [
{
"type": "Feature",
"properties": {
"kato": "591000000",
"district": "Петропавловск Г.А.",
"Y2022": 47567.6,
"Y2023": 48311.2,
"Y2024": 49120.5
},
"geometry": {
"type": "Point",
"coordinates": [
69.14998,
54.89148
]
}
},
{
"type": "Feature",
"properties": {
"kato": "593200000",
"district": "Айыртауский район",
"Y2022": 21802.7,
"Y2023": 22615.1,
"Y2024": 23427.5
},
"geometry": {
"type": "Point",
"coordinates": [
68.01276,
53.19148
]
}
},
{
"type": "Feature",
"properties": {
"kato": "593400000",
"district": "Акжарский район",
"Y2022": 23605.4,
"Y2023": 24417.8,
"Y2024": 25230.2
},
"geometry": {
"type": "Point",
"coordinates": [
71.74765,
53.78951
]
}
},
{
"type": "Feature",
"properties": {
"kato": "595800000",
"district": "Аккайынский район",
"Y2022": 25264.0,
"Y2023": 26076.4,
"Y2024": 26888.8
},
"geometry": {
"type": "Point",
"coordinates": [
69.55798,
54.45755
]
}
},
{
"type": "Feature",
"properties": {
"kato": "594200000",
"district": "Есильский район",
"Y2022": 27066.7,
"Y2023": 27879.1,
"Y2024": 28691.5
},
"geometry": {
"type": "Point",
"coordinates": [
68.32766,
54.11517
]
}
},
{
"type": "Feature",
"properties": {
"kato": "594600000",
"district": "Жамбылский район",
"Y2022": 28869.4,
"Y2023": 29681.8,
"Y2024": 30494.2
},
"geometry": {
"type": "Point",
"coordinates": [
66.98273,
54.4391
]
}
},
{
"type": "Feature",
"properties": {
"kato": "595000000",
"district": "Кызылжарский район",
"Y2022": 46862.7,
"Y2023": 48268.6,
"Y2024": 49205.8
},
"geometry": {
"type": "Point",
"coordinates": [
69.25907,
54.89353
]
}
},
{
"type": "Feature",
"properties": {
"kato": "595200000",
"district": "Мамлютский район",
"Y2022": 32330.7,
"Y2023": 33143.1,
"Y2024": 33955.5
},
"geometry": {
"type": "Point",
"coordinates": [
68.21021,
54.81946
]
}
},
{
"type": "Feature",
"properties": {
"kato": "596000000",
"district": "Тайыншинский район",
"Y2022": 34133.4,
"Y2023": 34945.8,
"Y2024": 35758.2
},
"geometry": {
"type": "Point",
"coordinates": [
69.89444,
53.80885
]
}
},
{
"type": "Feature",
"properties": {
"kato": "596200000",
"district": "Тимирязевский район",
"Y2022": 35792.0,
"Y2023": 36604.4,
"Y2024": 37416.8
},
"geometry": {
"type": "Point",
"coordinates": [
66.56047,
53.66856
]
}
},
{
"type": "Feature",
"properties": {
"kato": "596400000",
"district": "Уалихановский район",
"Y2022": 37594.7,
"Y2023": 38407.1,
"Y2024": 39219.5
},
"geometry": {
"type": "Point",
"coordinates": [
72.82078,
53.36512
]
}
},
{
"type": "Feature",
"properties": {
"kato": "593600000",
"district": "Район Магжана Жумабаева",
"Y2022": 39253.3,
"Y2023": 40065.7,
"Y2024": 40878.1
},
"geometry": {
"type": "Point",
"coordinates": [
70.5439,
54.63954
]
}
},
{
"type": "Feature",
"properties": {
"kato": "595600000",
"district": "Район Шал акына",
"Y2022": 41056.0,
"Y2023": 41868.4,
"Y2024": 42680.8
},
"geometry": {
"type": "Point",
"coordinates": [
67.40455,
53.80847
]
}
},
{
"type": "Feature",
"properties": {
"kato": "596600000",
"district": "Район им. Габита Мусрепова",
"Y2022": 42858.7,
"Y2023": 43671.1,
"Y2024": 44483.5
},
"geometry": {
"type": "Point",
"coordinates": [
67.02892,
52.78419
]
}
}
]
}

View File

@@ -0,0 +1,20 @@
<?xml version="1.0" encoding="UTF-8"?>
<sld:StyledLayerDescriptor version="1.0.0" xmlns:sld="http://www.opengis.net/sld" xmlns:ogc="http://www.opengis.net/ogc">
<sld:NamedLayer>
<sld:Name>ndvi</sld:Name>
<sld:UserStyle>
<sld:Title>NDVI gradient</sld:Title>
<sld:FeatureTypeStyle>
<sld:Rule>
<sld:RasterSymbolizer>
<sld:ColorMap>
<sld:ColorMapEntry color="#a50026" quantity="0.0"/>
<sld:ColorMapEntry color="#ffffbf" quantity="0.5"/>
<sld:ColorMapEntry color="#006837" quantity="1.0"/>
</sld:ColorMap>
</sld:RasterSymbolizer>
</sld:Rule>
</sld:FeatureTypeStyle>
</sld:UserStyle>
</sld:NamedLayer>
</sld:StyledLayerDescriptor>

View File

@@ -0,0 +1 @@
not an SLD file

758
test/acceptance/functional.sh Executable file
View File

@@ -0,0 +1,758 @@
#!/usr/bin/env bash
# Functional acceptance test of the GIS service on a test stand — the 89 checks
# of section 5 of «Протокол функционального тестирования» (Приложение 7):
# F01–F38 positive scenarios (categories, datasets, groups, SLD, filters,
# PostGIS / MinIO confirmation)
# N01–N38 negative scenarios
# L02–L05, L07–L15 Landscape Program (ЛП) on synthetic rasters, incl. the
# pixel-by-pixel reference check (lp_verify.py)
#
# Usage:
# test/acceptance/functional.sh # stand at http://localhost:18080
# BASE_URL=http://host:port test/acceptance/functional.sh
#
# Environment:
# BASE_URL API address (default http://localhost:18080)
# LOCAL_CHECKS 1 = confirm in PostGIS/MinIO through the docker containers of
# the compose project (F35 MinIO part, F36, F37, F38, L15
# resource figures); 0 = skip those (reported as SKIP);
# auto (default) = 1 when the gis-acceptance containers run
# COMPOSE_PROJECT compose project of the stand (default gis-acceptance)
# PYTHON python3 with numpy + rasterio (default python3)
# KEEP_DATA 1 = do not delete the created test data at the end
# WAIT_TIMEOUT seconds to wait for asynchronous processing (default 300)
#
# Output: one line per check "ID | описание | PASS/FAIL/SKIP | detail", then a
# summary. Exit code 0 only when there is no FAIL. Every object the script
# creates carries a unique run prefix and is deleted at the end (unless
# KEEP_DATA=1); pre-existing objects are never modified or deleted. The script
# writes the ЛП slots, so run it on an isolated test stand, never on production.
set -euo pipefail
HERE="$(cd "$(dirname "$0")" && pwd)"
DATA="$HERE/data"
BASE_URL="${BASE_URL:-http://localhost:18080}"
BASE_URL="${BASE_URL%/}"
COMPOSE_PROJECT="${COMPOSE_PROJECT:-gis-acceptance}"
PYTHON="${PYTHON:-python3}"
KEEP_DATA="${KEEP_DATA:-0}"
WAIT_TIMEOUT="${WAIT_TIMEOUT:-300}"
LOCAL_CHECKS="${LOCAL_CHECKS:-auto}"
for bin in curl jq "$PYTHON"; do
command -v "$bin" >/dev/null 2>&1 || { echo "missing required tool: $bin" >&2; exit 2; }
done
"$PYTHON" -c 'import numpy, rasterio' 2>/dev/null || { echo "$PYTHON needs numpy and rasterio (pip install -r $HERE/requirements.txt)" >&2; exit 2; }
container_of() { # service -> container id of the compose project (empty if none)
docker ps -q --filter "label=com.docker.compose.project=$COMPOSE_PROJECT" \
--filter "label=com.docker.compose.service=$1" 2>/dev/null | head -n1
}
if [ "$LOCAL_CHECKS" = auto ]; then
LOCAL_CHECKS=0
if command -v docker >/dev/null 2>&1 && [ -n "$(container_of postgres)" ] && [ -n "$(container_of minio)" ]; then
LOCAL_CHECKS=1
fi
fi
PG_C=""; MINIO_C=""; WORKER_C=""
if [ "$LOCAL_CHECKS" = 1 ]; then
PG_C="$(container_of postgres)"; MINIO_C="$(container_of minio)"; WORKER_C="$(container_of worker)"
fi
STAMP="$(date +%Y%m%d-%H%M%S)"
P="acc$(date +%s)" # unique slug prefix of every object this run creates
TMP="$(mktemp -d "${TMPDIR:-/tmp}/gis-func.XXXXXX")"
RESULTS="$HERE/results/functional-$STAMP"
mkdir -p "$RESULTS"
BODY="$TMP/body"
JSON=(-H 'Content-Type: application/json')
# ------------------------------------------------------------------ reporting
PASSN=0; FAILN=0; SKIPN=0
CUR_ID=""; CUR_DESC=""; CUR_OK=1; CUR_MSG=""
begin() { CUR_ID="$1"; CUR_DESC="$2"; CUR_OK=1; CUR_MSG=""; }
bad() { CUR_OK=0; CUR_MSG="${CUR_MSG:+$CUR_MSG; }$1"; }
line() { printf '%-4s | %s | %s | %s\n' "$1" "$2" "$3" "$4" | tee -a "$RESULTS/functional.txt"; }
end() { # detail
if [ "$CUR_OK" = 1 ]; then PASSN=$((PASSN + 1)); line "$CUR_ID" "$CUR_DESC" PASS "$1"
else FAILN=$((FAILN + 1)); line "$CUR_ID" "$CUR_DESC" FAIL "$CUR_MSG"; fi
}
skip() { SKIPN=$((SKIPN + 1)); line "$CUR_ID" "$CUR_DESC" SKIP "$1"; }
snippet() { head -c 220 "$BODY" 2>/dev/null | tr '\n\r' ' '; }
# ----------------------------------------------------------------------- HTTP
CODE=000; TIME=0
http() { # METHOD PATH [curl args...] -> $CODE, $TIME (s), body in $BODY
local m="$1" p="$2" out; shift 2
out="$(curl -sS -o "$BODY" -w '%{http_code} %{time_total}' -X "$m" "$BASE_URL$p" "$@" 2>"$TMP/curl.err")" || out="000 0"
CODE="${out%% *}"; TIME="${out#* }"
}
jb() { jq -r "$1" "$BODY" 2>/dev/null || true; }
jbc() { jq -c "${1:-.}" "$BODY" 2>/dev/null || snippet; }
need_code() { [ "$CODE" = "$1" ] || bad "HTTP $CODE (ожидалось $1): $(snippet)"; }
need_eq() { [ "$1" = "$2" ] || bad "$3: «$1» ≠ «$2»"; }
need_has() { case "$1" in *"$2"*) ;; *) bad "$3: нет «$2» в «${1:0:200}»";; esac; }
need_true() { [ "$1" = true ] || bad "$2"; }
sha256() { if command -v sha256sum >/dev/null 2>&1; then sha256sum "$1" | cut -d' ' -f1; else shasum -a 256 "$1" | cut -d' ' -f1; fi; }
# dataset status polling: wait_ds ID STATUS... -> $WS (final status)
WS=""
wait_ds() {
local id="$1"; shift
local deadline=$(( $(date +%s) + WAIT_TIMEOUT ))
while :; do
WS="$(curl -sS "$BASE_URL/datasets/$id" | jq -r '.status' 2>/dev/null || echo "?")"
for s in "$@"; do [ "$WS" = "$s" ] && return 0; done
[ "$WS" = failed ] && return 0
[ "$(date +%s)" -ge "$deadline" ] && return 0
sleep 1
done
}
# run polling: wait_run ID -> $RS final status; run JSON in $TMP/run-ID.json
RS=""
wait_run() {
local id="$1" deadline=$(( $(date +%s) + WAIT_TIMEOUT ))
while :; do
curl -sS "$BASE_URL/landscape/runs/$id" -o "$TMP/run-$id.json" || true
RS="$(jq -r '.status' "$TMP/run-$id.json" 2>/dev/null || echo "?")"
case "$RS" in succeeded|failed) return 0;; esac
[ "$(date +%s)" -ge "$deadline" ] && return 0
sleep 1
done
}
node_statuses() { jq -r '[.nodes[] | "\(.node_code):\(.status)"] | sort | join(" ")' "$TMP/run-$1.json" 2>/dev/null || true; }
wait_outputs() { # RUN: wait until every output dataset of the run is ready (COG done)
local id; for id in $(run_outputs "$1" | jq -r '.[].dataset_id'); do wait_ds "$id" ready; done
}
run_outputs() { jq -c '[.nodes[] | .outputs[]?] | sort_by(.group)' "$TMP/run-$1.json" 2>/dev/null || echo '[]'; }
# ------------------------------------------------------------- cleanup ledger
DATASETS=""; GRPS=""; CATS=""; RUNS=""
track_ds() { [ -n "$1" ] && [ "$1" != null ] && DATASETS="$DATASETS $1"; return 0; }
cleanup() {
local rc=$?
if [ "$KEEP_DATA" = 1 ]; then
echo "KEEP_DATA=1: test data kept (prefix $P)"
else
local ids="$DATASETS" r d
for r in $RUNS; do # outputs computed by our runs (never reused foreign ones)
[ -f "$TMP/run-$r.json" ] || continue
ids="$ids $(jq -r '[.nodes[] | select(.status=="succeeded") | .outputs[]?.dataset_id] | join(" ")' "$TMP/run-$r.json" 2>/dev/null || true)"
done
for d in $ids; do curl -sS -o /dev/null -X DELETE "$BASE_URL/datasets/$d" || true; done
for d in $GRPS; do curl -sS -o /dev/null -X DELETE "$BASE_URL/dataset-groups/$d" || true; done
for d in $CATS; do curl -sS -o /dev/null -X DELETE "$BASE_URL/categories/$d" || true; done
echo "cleanup: removed test datasets, groups and categories with prefix $P (ЛП run records are kept: the API has no run deletion)"
fi
rm -rf "$TMP"
exit "$rc"
}
trap cleanup EXIT
echo "functional acceptance test: $BASE_URL prefix=$P local_checks=$LOCAL_CHECKS log=$RESULTS"
: > "$RESULTS/functional.txt"
# ====================================================================== 5.2
begin F01 "Создание корневой категории POST /categories"
http POST /categories "${JSON[@]}" -d "{\"code\":\"$P-ecology\",\"name\":\"Экология\",\"description\":\"Приёмочный тест\"}"
need_code 201; CAT_ECO="$(jb .id)"; need_eq "$(jb .parent_id)" null "parent_id"
[ -n "$CAT_ECO" ] && [ "$CAT_ECO" != null ] && CATS="$CAT_ECO $CATS"
end "201; $P-ecology «Экология», parent_id null"
begin F02 "Создание дочерних категорий (parent_id)"
http POST /categories "${JSON[@]}" -d "{\"code\":\"$P-ecology-soil\",\"name\":\"Почвы\",\"parent_id\":\"$CAT_ECO\"}"
need_code 201; CAT_SOIL="$(jb .id)"; need_eq "$(jb .parent_id)" "$CAT_ECO" "parent_id soil"
[ -n "$CAT_SOIL" ] && [ "$CAT_SOIL" != null ] && CATS="$CAT_SOIL $CATS"
c1="$CODE"
http POST /categories "${JSON[@]}" -d "{\"code\":\"$P-ecology-soil-humus\",\"name\":\"Гумус\",\"parent_id\":\"$CAT_SOIL\"}"
need_code 201; CAT_HUM="$(jb .id)"; need_eq "$(jb .parent_id)" "$CAT_SOIL" "parent_id humus"
[ -n "$CAT_HUM" ] && [ "$CAT_HUM" != null ] && CATS="$CAT_HUM $CATS"
end "$c1 / $CODE: трёхуровневая иерархия ecology → soil → humus"
begin F03 "Список категорий с фильтром parent_id"
http GET "/categories?parent_id=$CAT_ECO"
need_code 200; need_eq "$(jb 'length')" 1 "число дочерних"; need_eq "$(jb '.[0].id')" "$CAT_SOIL" "дочерняя категория"
end "200; возвращена дочерняя категория $P-ecology-soil"
begin F04 "Получение категории по id"
http GET "/categories/$CAT_SOIL"
need_code 200
need_eq "$(jb '[has("id"),has("parent_id"),has("code"),has("name"),has("description")]|all')" true "состав полей"
SOIL_UPDATED="$(jb .updated_at)"
end "200; поля id, parent_id, code, name, description"
begin F05 "Обновление категории (PUT)"
sleep 1
http PUT "/categories/$CAT_SOIL" "${JSON[@]}" -d "{\"code\":\"$P-ecology-soil\",\"name\":\"Почвы и грунты\",\"description\":\"Обновлено\",\"parent_id\":\"$CAT_ECO\"}"
need_code 200; need_eq "$(jb .name)" "Почвы и грунты" name; need_eq "$(jb .description)" "Обновлено" description
[ "$(jb .updated_at)" != "$SOIL_UPDATED" ] || bad "updated_at не изменился"
end "200; name «Почвы и грунты», description «Обновлено», updated_at обновлён"
begin F06 "Цикл в иерархии (корень → внук)"
http PUT "/categories/$CAT_ECO" "${JSON[@]}" -d "{\"code\":\"$P-ecology\",\"name\":\"Экология\",\"parent_id\":\"$CAT_HUM\"}"
need_code 422; need_has "$(jb .error)" "own ancestor" сообщение
end "422 «$(jb .error)»"
begin F07 "Категория-родитель самой себя"
http PUT "/categories/$CAT_ECO" "${JSON[@]}" -d "{\"code\":\"$P-ecology\",\"name\":\"Экология\",\"parent_id\":\"$CAT_ECO\"}"
need_code 422; need_has "$(jb .error)" "own ancestor" сообщение
end "422 «$(jb .error)»"
begin F08 "Дубликат кода категории"
http POST /categories "${JSON[@]}" -d "{\"code\":\"$P-ecology\",\"name\":\"Дубликат\"}"
need_code 409
[ "$CODE" = 201 ] && CATS="$(jb .id) $CATS"
end "409 «$(jb .error)»"
begin F09 "Удаление категории"
http POST /categories "${JSON[@]}" -d "{\"code\":\"$P-tmp\",\"name\":\"Временная\"}"
need_code 201; tmpc="$(jb .id)"; c1="$CODE"
http DELETE "/categories/$tmpc"; need_code 204; c2="$CODE"
http GET "/categories/$tmpc"; need_code 404
end "$c1 (создана) → $c2 (удалена) → повторный GET $CODE"
# ====================================================================== 5.3
begin F10 "Создание группы наборов POST /dataset-groups"
http POST /dataset-groups "${JSON[@]}" -d "{\"code\":\"$P-population-nko\",\"name\":\"Численность населения СКО\",\"description\":\"Приёмочный тест\"}"
need_code 201; GRP="$(jb .id)"
[ -n "$GRP" ] && [ "$GRP" != null ] && GRPS="$GRP $GRPS"
end "201; группа $P-population-nko «Численность населения СКО»"
begin F11 "Загрузка набора vector (GeoJSON)"
http POST /datasets -F "file=@$DATA/plots_nko.geojson" -F file_type=vector -F "category_id=$CAT_ECO" \
-F "name=Участки СКО" -F "description=Полигоны участков" -F "unit=ha"
need_code 201; DS_PLOTS="$(jb .id)"; track_ds "$DS_PLOTS"; s0="$(jb .status)"; key="$(jb .storage_key)"
need_has "$key" "/plots_nko.geojson" storage_key
wait_ds "$DS_PLOTS" ready; need_eq "$WS" ready status
end "201, status $s0 → $WS; storage_key …/plots_nko.geojson"
begin F12 "Выгрузка vector в GeoJSON"
http GET "/datasets/$DS_PLOTS.geojson"
need_code 200; need_eq "$(jb .type)" FeatureCollection type; need_eq "$(jb '.features|length')" 4 "число объектов"
need_eq "$(jb '[.features[].geometry.type]|unique|join(",")')" Polygon "тип геометрии"
ext="$(jb '[.. | arrays | select(length==2 and (.[0]|type)=="number")] as $p | "\([$p[][0]]|min),\([$p[][1]]|min) – \([$p[][0]]|max),\([$p[][1]]|max)"')"
need_eq "$(jb '[.. | arrays | select(length==2 and (.[0]|type)=="number")] as $p | ([$p[][0]]|min)==69 and ([$p[][1]]|min)==53.6 and ([$p[][0]]|max)==71.7 and ([$p[][1]]|max)==55')" true "охват (69.0, 53.6) – (71.7, 55.0): $ext"
end "200; FeatureCollection, 4 объекта Polygon; охват ($ext)"
begin F13 "Загрузка vector_with_kato (GeoJSON, реальные коды КАТО)"
http POST /datasets -F "file=@$DATA/population_kato.geojson" -F file_type=vector_with_kato -F "category_id=$CAT_ECO" \
-F "name=Численность населения" -F unit=чел -F "dg_id=$GRP" -F start_date=2022-01-01 -F end_date=2024-12-31
need_code 201; DS_POP="$(jb .id)"; track_ds "$DS_POP"; s0="$(jb .status)"; need_eq "$(jb .dg_id)" "$GRP" dg_id
wait_ds "$DS_POP" awaiting_mapping; need_eq "$WS" awaiting_mapping status
end "201, status $s0 → $WS, dg_id = группа $P-population-nko"
begin F14 "Обнаружение колонок"
http GET "/datasets/$DS_POP"
need_code 200; need_eq "$(jb .status)" awaiting_mapping status
cols="$(jb '[.attribute_columns[].name]|join(",")')"
need_eq "$cols" "kato,district,Y2022,Y2023,Y2024" колонки
need_eq "$(jb '[.attribute_columns[]|(.samples|length)>0]|all')" true "примеры значений"
end "200; колонки $cols с примерами значений"
MAP="$TMP/map.json"
mapping() { http POST "/datasets/$DS_POP/mapping" "${JSON[@]}" -d "$1"; }
begin N01 "Маппинг: несуществующая колонка года"
mapping '{"kato_column":"kato","year_columns":[{"column":"Y1999","start_date":"2022-01-01"}]}'
need_code 422; need_has "$(jb .error)" 'year column "Y1999" is not among the detected columns' сообщение
end "422 «$(jb .error)»"
begin N02 "Маппинг: несуществующая колонка КАТО"
mapping '{"kato_column":"KATO_X","year_columns":[{"column":"Y2022","start_date":"2022-01-01"}]}'
need_code 422; need_has "$(jb .error)" 'kato_column "KATO_X" is not among the detected columns' сообщение
end "422 «$(jb .error)»"
begin N03 "Маппинг: end_date раньше start_date"
mapping '{"kato_column":"kato","year_columns":[{"column":"Y2022","start_date":"2022-12-31","end_date":"2022-01-01"}]}'
need_code 422; need_has "$(jb .error)" 'is before start_date' сообщение
end "422 «$(jb .error)»"
begin N04 "Маппинг: некорректная дата 2022-13-45"
mapping '{"kato_column":"kato","year_columns":[{"column":"Y2022","start_date":"2022-13-45"}]}'
need_code 422; need_eq "$(jb .errors.StartDate)" "is invalid" errors.StartDate
end "422 $(jbc)"
begin N05 "Маппинг: пустой список year_columns"
mapping '{"kato_column":"kato","year_columns":[]}'
need_code 422; need_has "$(jb .errors.YearColumns)" "at least 1" errors.YearColumns
end "422 $(jbc)"
begin F15 "Сохранение маппинга КАТО и годовых колонок с периодами"
mapping '{"kato_column":"kato","year_columns":[{"column":"Y2022","start_date":"2022-01-01","end_date":"2022-12-31"},{"column":"Y2023","start_date":"2023-01-01","end_date":"2023-12-31"},{"column":"Y2024","start_date":"2024-01-01","end_date":"2024-12-31"}]}'
need_code 200; s0="$(jb .status)"; need_eq "$(jb '.year_columns|length')" 3 year_columns; need_eq "$(jb .kato_column)" kato kato_column
wait_ds "$DS_POP" ready; need_eq "$WS" ready status
end "200, status $s0 → $WS; 3 колонки Y2022–Y2024 с периодами"
fixval() { jq -r --arg k "$1" --arg y "$2" '.features[]|select(.properties.kato==$k)|.properties[$y]' "$DATA/population_kato.geojson"; }
begin F16 "Наблюдения (long format)"
http GET "/datasets/$DS_POP/observations?page_size=100"
need_code 200; need_eq "$(jb .total)" 42 total
exp="$(fixval 591000000 Y2022)"
got="$(jb '.data[]|select(.kato_code=="591000000" and .start_date=="2022-01-01")|.value')"
need_eq "$got" "$exp" "591000000 / 2022"
end "200; total $(jb .total) (14 районов × 3 года), 591000000 / 2022 → $got"
begin F17 "Фильтр наблюдений по kato_code"
http GET "/datasets/$DS_POP/observations?kato_code=595000000"
need_code 200; need_eq "$(jb .total)" 3 total
got="$(jb '[.data|sort_by(.start_date)[]|.value]|join(" / ")')"
exp="$(fixval 595000000 Y2022) / $(fixval 595000000 Y2023) / $(fixval 595000000 Y2024)"
need_eq "$got" "$exp" "значения 595000000"
end "200; для 595000000 — $(jb .total) наблюдения: $got"
begin F18 "GeoJSON vector_with_kato: соединение с districts"
http GET "/datasets/$DS_POP.geojson"
need_code 200; need_eq "$(jb '.features|length')" 14 "число объектов"
need_eq "$(jb '[.features[].geometry.type]|unique|join(",")')" MultiPolygon "тип геометрии"
need_eq "$(jb '.features[0].properties|keys|join(",")')" "data,kato,name,periods" "свойства"
end "200; 14 объектов MultiPolygon, свойства data, kato, name, periods"
begin F19 "Загрузка vector_with_kato (GeoPackage)"
http POST /datasets -F "file=@$DATA/livestock_kato.gpkg" -F file_type=vector_with_kato -F "category_id=$CAT_ECO" \
-F "name=Поголовье КРС" -F unit=голов
need_code 201; DS_LIVE="$(jb .id)"; track_ds "$DS_LIVE"; c1="$CODE"
wait_ds "$DS_LIVE" awaiting_mapping; need_eq "$WS" awaiting_mapping status
http GET "/datasets/$DS_LIVE"; cols="$(jb '[.attribute_columns[].name]|join(",")')"
need_eq "$cols" "fid,KATO_CODE,district,H_2022,H_2023,H_2024" колонки
http POST "/datasets/$DS_LIVE/mapping" "${JSON[@]}" -d '{"kato_column":"KATO_CODE","year_columns":[{"column":"H_2022","start_date":"2022-01-01","end_date":"2022-12-31"},{"column":"H_2023","start_date":"2023-01-01","end_date":"2023-12-31"},{"column":"H_2024","start_date":"2024-01-01","end_date":"2024-12-31"}]}'
need_code 200; c2="$CODE"
wait_ds "$DS_LIVE" ready; need_eq "$WS" ready status
end "$c1 → awaiting_mapping; колонки $cols; маппинг $c2, статус $WS"
NDVI="$TMP/ndvi_nko.tif"
"$PYTHON" "$HERE/lib/helpers.py" ndvi "$NDVI"
begin F20 "Загрузка растра GeoTIFF (600×400, EPSG:4326, Float32)"
http POST /datasets -F "file=@$NDVI" -F file_type=raster -F "category_id=$CAT_ECO" -F "name=NDVI СКО" -F unit=NDVI
need_code 201; DS_NDVI="$(jb .id)"; track_ds "$DS_NDVI"; s0="$(jb .status)"
wait_ds "$DS_NDVI" ready; need_eq "$WS" ready status
end "201, status $s0 → $WS"
begin F21 "Растр: статус, bbox"
http GET "/datasets/$DS_NDVI"
need_code 200; need_eq "$(jb .status)" ready status
need_has "$(jb .cog_storage_key)" "/cog/ndvi_nko.tif" cog_storage_key
bbox="$(jb '.bbox|map(.*1000|round/1000)|join(", ")')"
need_eq "$bbox" "68.5, 51.5, 74.5, 55.5" bbox
end "200; ready, cog_storage_key …/cog/ndvi_nko.tif, bbox [$bbox]"
begin F22 "Получение COG и проверка LAYOUT=COG"
ctype="$(curl -sS -o "$TMP/ndvi.cog.tif" -w '%{http_code} %{content_type}' "$BASE_URL/datasets/$DS_NDVI.cog" || echo 000)"
need_eq "$ctype" "200 image/tiff" "код и Content-Type"
ri="$("$PYTHON" "$HERE/lib/helpers.py" rinfo "$TMP/ndvi.cog.tif" 2>&1 || true)"
need_eq "$(echo "$ri" | jq -r '"\(.driver) \(.width)x\(.height) EPSG:\(.epsg) \(.layout) \(.compression)"' 2>/dev/null)" "GTiff 600x400 EPSG:4326 COG DEFLATE" "формат"
end "$ctype, $(wc -c < "$TMP/ndvi.cog.tif" | tr -d ' ') байт; $(echo "$ri" | jq -r '"\(.driver), \(.width)×\(.height), EPSG:\(.epsg), LAYOUT=\(.layout), \(.compression)"' 2>/dev/null)"
begin F23 "Скачивание оригинала /download"
dl="$(curl -sS -o "$TMP/ndvi.dl.tif" -w '%{http_code}' "$BASE_URL/datasets/$DS_NDVI/download" || echo 000)"
need_eq "$dl" 200 "HTTP"
h1="$(sha256 "$NDVI")"; h2="$(sha256 "$TMP/ndvi.dl.tif")"
need_eq "$h2" "$h1" "SHA-256"
end "$dl, $(wc -c < "$TMP/ndvi.dl.tif" | tr -d ' ') байт; SHA-256 совпадает (${h1:0:8}…${h1: -5})"
begin F24 "Загрузка SLD для набора"
http GET "/datasets/$DS_NDVI/sld"; need_code 404; c0="$CODE"
http POST "/datasets/$DS_NDVI/sld" -F "file=@$DATA/style.sld"; need_code 200; need_eq "$(jb .sld_filename)" style.sld sld_filename; c1="$CODE"
http GET "/datasets/$DS_NDVI/sld"; need_code 200; need_has "$(cat "$BODY")" "NDVI gradient" "SLD"
end "до загрузки GET /sld → $c0; POST → $c1, sld_filename style.sld; GET /sld → $CODE («NDVI gradient»)"
begin F25 "Общий SLD группы и наследование набором"
http POST "/dataset-groups/$GRP/sld" -F "file=@$DATA/group_style.sld"; need_code 200; c1="$CODE"
http GET "/dataset-groups/$GRP/sld"; need_code 200; need_has "$(cat "$BODY")" "Group fallback style" "SLD группы"
http GET "/datasets/$DS_POP/sld"; need_code 200; need_has "$(cat "$BODY")" "Group fallback style" "SLD набора без собственного"
end "POST группы → $c1; GET /sld группы и набора без собственного SLD → «Group fallback style»"
begin N06 "SLD: недопустимое расширение файла"
http POST "/datasets/$DS_NDVI/sld" -F "file=@$DATA/style.txt"
need_code 422; need_has "$(jb .error)" "sld file must have one of these extensions" сообщение
end "422 «$(jb .error)»"
begin F26 "PATCH набора: dg_id и период"
http PATCH "/datasets/$DS_NDVI" "${JSON[@]}" -d "{\"dg_id\":\"$GRP\",\"start_date\":\"2026-01-01\",\"end_date\":\"2026-03-31\",\"description\":\"NDVI за I квартал 2026\"}"
need_code 200; need_eq "$(jb .dg_id)" "$GRP" dg_id; need_eq "$(jb .start_date)" 2026-01-01 start_date
need_eq "$(jb .end_date)" 2026-03-31 end_date; need_eq "$(jb .description)" "NDVI за I квартал 2026" description
end "200; dg_id, start_date 2026-01-01, end_date 2026-03-31, description обновлены"
begin F27 "PUT набора: полная замена метаданных"
http PUT "/datasets/$DS_PLOTS" "${JSON[@]}" -d "{\"category_id\":\"$CAT_ECO\",\"name\":\"Участки СКО (ред.)\",\"description\":\"Полигоны участков\",\"unit\":\"га\",\"dg_id\":\"$GRP\",\"start_date\":\"2025-01-01\",\"end_date\":\"2025-12-31\"}"
need_code 200; need_eq "$(jb .name)" "Участки СКО (ред.)" name; need_eq "$(jb .unit)" "га" unit
need_eq "$(jb .dg_id)" "$GRP" dg_id; need_eq "$(jb .end_date)" 2025-12-31 end_date
end "200; name «Участки СКО (ред.)», unit га, dg_id, период 2025"
begin N07 "PATCH: end_date раньше start_date"
http PATCH "/datasets/$DS_PLOTS" "${JSON[@]}" -d '{"start_date":"2025-12-31","end_date":"2025-01-01"}'
need_code 422; need_has "$(jb .error)" 'end_date "2025-01-01" is before start_date "2025-12-31"' сообщение
end "422 «$(jb .error)»"
begin N08 "PATCH: нередактируемое поле file_type"
http PATCH "/datasets/$DS_PLOTS" "${JSON[@]}" -d '{"file_type":"raster"}'
need_code 422; need_eq "$(jb .error)" "unknown field: file_type" сообщение
end "422 «$(jb .error)»"
begin N09 "PUT: отсутствуют обязательные поля"
http PUT "/datasets/$DS_PLOTS" "${JSON[@]}" -d '{"name":"Только имя"}'
need_code 422
need_eq "$(jb '[.errors.CategoryID,.errors.Description,.errors.Unit]|join(",")')" "is required,is required,is required" errors
end "422 CategoryID, Description, Unit «is required»"
begin F28 "Свежайший набор группы /latest (по max end_date, фильтры file_type, status)"
http GET "/dataset-groups/$GRP/latest"; need_code 200; need_eq "$(jb .id)" "$DS_NDVI" "latest"; l1="$(jb .end_date)"
http GET "/dataset-groups/$GRP/latest?file_type=vector_with_kato"; need_code 200; need_eq "$(jb .id)" "$DS_POP" "latest vector_with_kato"; l2="$(jb .end_date)"
http GET "/dataset-groups/$GRP/latest?status=failed"; need_code 404
end "200; NDVI (end_date $l1); с file_type=vector_with_kato — набор населения ($l2); status=failed → $CODE"
begin F29 "Обновление группы"
http PUT "/dataset-groups/$GRP" "${JSON[@]}" -d "{\"code\":\"$P-population-nko\",\"name\":\"Население и NDVI СКО\",\"description\":\"Приёмочный тест\"}"
need_code 200; need_eq "$(jb .name)" "Население и NDVI СКО" name
end "200; name «Население и NDVI СКО»"
begin F30 "Удаление группы, на которую ссылаются наборы"
http DELETE "/dataset-groups/$GRP"
need_code 409
end "409 «$(jb .error)»"
begin F31 "Список групп с фильтром code"
http GET "/dataset-groups?code=$P-population-nko"
need_code 200; need_eq "$(jb .total)" 1 total; need_eq "$(jb '.data[0].id')" "$GRP" группа
end "200; total $(jb .total)"
begin F32 "Удаление пустой группы"
http POST /dataset-groups "${JSON[@]}" -d "{\"code\":\"$P-empty\",\"name\":\"Пустая\"}"
need_code 201; eg="$(jb .id)"; c1="$CODE"
http DELETE "/dataset-groups/$eg"; need_code 204
[ "$CODE" = 204 ] || GRPS="$eg $GRPS"
end "$c1 → $CODE"
begin N10 "Группа: код не slug («Bad Code!»)"
http POST /dataset-groups "${JSON[@]}" -d '{"code":"Bad Code!","name":"Плохой код"}'
need_code 422; need_eq "$(jb .errors.Code)" "is invalid" errors.Code
[ "$CODE" = 201 ] && GRPS="$(jb .id) $GRPS"
end "422 $(jbc)"
begin F33 "Список наборов: пагинация"
http GET "/datasets?category_id=$CAT_ECO&page_size=2&page=1"
need_code 200; need_eq "$(jb .total)" 4 total; need_eq "$(jb .page_size)" 2 page_size; need_eq "$(jb .total_pages)" 2 total_pages
p1="$(jb '[.data[].id]|join(" ")')"
http GET "/datasets?category_id=$CAT_ECO&page_size=2&page=2"
need_code 200; p2="$(jb '[.data[].id]|join(" ")')"
all="$(printf '%s\n' $p1 $p2 | sort -u | wc -l | tr -d ' ')"
need_eq "$all" 4 "уникальных записей на 2 страницах"
end "200; total 4, page_size 2, total_pages 2; страницы 1 и 2 без пересечений"
begin F34 "Фильтры dg_id + end_date_from, file_type + status, category_id + start_date_to"
http GET "/datasets?dg_id=$GRP&end_date_from=2025-01-01"; need_code 200; n1="$(jb .total)"
need_eq "$(jb '[.data[].id]|sort|join(" ")')" "$(printf '%s\n' "$DS_PLOTS" "$DS_NDVI" | sort | tr '\n' ' ' | sed 's/ $//')" "dg_id+end_date_from"
http GET "/datasets?category_id=$CAT_ECO&file_type=vector_with_kato&status=ready"; need_code 200; n2="$(jb .total)"
need_eq "$(jb '[.data[].id]|sort|join(" ")')" "$(printf '%s\n' "$DS_POP" "$DS_LIVE" | sort | tr '\n' ' ' | sed 's/ $//')" "file_type+status"
http GET "/datasets?category_id=$CAT_ECO&start_date_to=2024-01-01"; need_code 200; n3="$(jb .total)"
need_eq "$(jb '[.data[].id]|join(" ")')" "$DS_POP" "category_id+start_date_to"
end "200; $n1 / $n2 / $n3 записи соответственно, состав соответствует условиям"
begin N11 "Фильтр: некорректная дата 2025-31-12"
http GET "/datasets?start_date_from=2025-31-12"
need_code 400; need_eq "$(jb .error)" "start_date_from must be YYYY-MM-DD" сообщение
end "400 «$(jb .error)»"
begin N12 "Пагинация: page_size=0"
http GET "/datasets?page_size=0"
need_code 400; need_eq "$(jb .error)" "invalid page_size" сообщение
end "400 «$(jb .error)»"
begin N13 "Фильтр: file_type=shapefile"
http GET "/datasets?file_type=shapefile"
need_code 400; need_has "$(jb .error)" "file_type must be one of" сообщение
end "400 «$(jb .error)»"
# ====================================================================== 5.4
printf 'just text\n' > "$TMP/notes.txt"
printf 'this is not a tiff\n' > "$TMP/fake.tif"
printf 'not json at all\n' > "$TMP/bad.geojson"
up() { http POST /datasets "$@"; [ "$CODE" = 201 ] && track_ds "$(jb .id)"; return 0; }
negup() { # ID desc expected-code expected-substring curl-args...
local id="$1" desc="$2" ec="$3" sub="$4"; shift 4
begin "$id" "$desc"; up "$@"; need_code "$ec"; need_has "$(jb .error)" "$sub" сообщение
end "$CODE «$(jb .error)»"
}
negup N14 "Загрузка: расширение .txt" 422 'extension ".txt" is not allowed for file_type "vector"' \
-F "file=@$TMP/notes.txt" -F file_type=vector -F "category_id=$CAT_SOIL" -F name=x
negup N15 "Загрузка: .tif без TIFF-сигнатуры" 422 "file is not a valid TIFF/BigTIFF GeoTIFF" \
-F "file=@$TMP/fake.tif" -F file_type=raster -F "category_id=$CAT_SOIL" -F name=x
negup N16 "Загрузка: .geojson не JSON" 422 "file is not valid GeoJSON" \
-F "file=@$TMP/bad.geojson" -F file_type=vector -F "category_id=$CAT_SOIL" -F name=x
negup N17 "Загрузка: неизвестный file_type" 422 "file_type must be one of: vector_with_kato, vector, raster" \
-F "file=@$DATA/plots_nko.geojson" -F file_type=shapefile -F "category_id=$CAT_SOIL" -F name=x
negup N18 "Загрузка: .tif для типа vector" 422 'extension ".tif" is not allowed for file_type "vector"' \
-F "file=@$NDVI" -F file_type=vector -F "category_id=$CAT_SOIL" -F name=x
negup N19 "Загрузка: нет файла" 400 "file is required" \
-F file_type=vector -F "category_id=$CAT_SOIL" -F name=x
negup N20 "Загрузка: нет category_id" 422 "category_id must be a valid UUID" \
-F "file=@$DATA/plots_nko.geojson" -F file_type=vector -F name=x
begin N21 "Загрузка без name (поле необязательно по OpenAPI)"
up -F "file=@$DATA/plots_nko.geojson" -F file_type=vector -F "category_id=$CAT_SOIL"
need_code 201; DS_N21="$(jb .id)"; N21_KEY="$(jb .storage_key)"; need_eq "$(jb .name)" plots_nko.geojson name
end "201; name = имя файла ($(jb .name))"
negup N22 "Загрузка: end_date раньше start_date" 422 'end_date "2025-01-01" is before start_date "2025-12-31"' \
-F "file=@$DATA/plots_nko.geojson" -F file_type=vector -F "category_id=$CAT_SOIL" -F name=x -F start_date=2025-12-31 -F end_date=2025-01-01
negup N23 "Загрузка: несуществующая категория" 422 "category does not exist" \
-F "file=@$DATA/plots_nko.geojson" -F file_type=vector -F "category_id=00000000-0000-4000-8000-000000000000" -F name=x
negup N24 "Загрузка: невалидный JSON в meta" 422 "meta must be valid JSON" \
-F "file=@$DATA/plots_nko.geojson" -F file_type=vector -F "category_id=$CAT_SOIL" -F name=x -F 'meta={bad json'
begin N25 "Загрузка: не multipart"
http POST /datasets "${JSON[@]}" -d '{"file_type":"vector"}'
need_code 400; need_eq "$(jb .error)" "request must be multipart/form-data" сообщение
end "400 «$(jb .error)»"
begin N26 "Несуществующие id на 10 эндпоинтах"
X=7d3c9a52-1f4e-4b8a-9c2d-5e6f7a8b9c0d; codes=""
for p in "/datasets/$X" "/datasets/$X.geojson" "/datasets/$X.cog" "/datasets/$X/download" "/datasets/$X/observations" \
"/datasets/$X/sld" "/categories/$X" "/dataset-groups/$X" "/dataset-groups/$X/latest" "/landscape/runs/$X"; do
http GET "$p"; codes="$codes $CODE"; [ "$CODE" = 404 ] || bad "$p → $CODE"
done
end "404 на всех 10 эндпоинтах:$codes"
begin N27 "Некорректный формат id (4 эндпоинта)"
codes=""
for p in /datasets/not-a-uuid /categories/not-a-uuid /dataset-groups/not-a-uuid /landscape/runs/not-a-uuid; do
http GET "$p"; codes="$codes $CODE"; [ "$CODE" = 400 ] || bad "$p → $CODE"; [ "$(jb .error)" = "invalid id" ] || bad "$p: $(jb .error)"
done
end "400 «invalid id» на всех 4:$codes"
begin N28 "Битый JSON"
http POST /categories "${JSON[@]}" -d '{"code":"x","name":'
need_code 400; need_eq "$(jb .error)" "invalid request body" сообщение
end "400 «$(jb .error)»"
begin N29 "Мусорное тело 200 КБ"
head -c 204800 /dev/urandom > "$TMP/garbage.bin"
http POST /categories "${JSON[@]}" --data-binary "@$TMP/garbage.bin"
need_code 400; need_eq "$(jb .error)" "invalid request body" сообщение
end "400 «$(jb .error)»"
begin N30 "Неизвестное поле в JSON"
http POST /categories "${JSON[@]}" -d "{\"code\":\"$P-unknown\",\"name\":\"x\",\"color\":\"red\"}"
need_code 400; need_eq "$(jb .error)" "invalid request body" сообщение
[ "$CODE" = 201 ] && CATS="$(jb .id) $CATS"
end "400 «$(jb .error)»"
begin N37 "JSON-тело 2 200 048 байт (> 1 МиБ)"
{ printf '{"code":"%s-big","name":"' "$P"; head -c $((2200048 - 25 - ${#P})) /dev/zero | tr '\0' 'a'; printf '"}'; } > "$TMP/big.json"
http POST /categories "${JSON[@]}" --data-binary "@$TMP/big.json"
need_code 400; need_eq "$(jb .error)" "invalid request body" сообщение
[ "$CODE" = 201 ] && CATS="$(jb .id) $CATS"
end "400 «$(jb .error)» (тело $(wc -c < "$TMP/big.json" | tr -d ' ') байт)"
runreq() { http POST /landscape/runs "${JSON[@]}" -d "$1"; [ "$CODE" = 202 ] && RUNS="$RUNS $(jb .id)"; return 0; }
begin N31 "ЛП: from_stage=7"
runreq '{"from_stage":7}'; need_code 422; need_eq "$(jb .errors.FromStage)" "must be 3 or less" errors.FromStage
end "422 $(jbc)"
begin N32 "ЛП: tie_priority не перестановка"
runreq '{"params":{"tie_priority":[1,1,2,3,4]}}'; need_code 422; need_has "$(jb .error)" "tie_priority must be a permutation of 1..5" сообщение
end "422 «$(jb .error)»"
begin N33 "ЛП: override с некорректным UUID"
runreq '{"overrides":{"lp-z-lcr":"not-a-uuid"}}'; need_code 422; need_eq "$(jb '.errors["Overrides[lp-z-lcr]"]')" "must be a valid UUID" errors
end "422 $(jbc)"
begin N34 "ЛП: неизвестный слот"
http POST /landscape/slots/lp-unknown-slot -F "file=@$NDVI"; need_code 404
[ "$CODE" = 201 ] && track_ds "$(jb .id)"
end "404 «$(jb .error)»"
begin N35 "ЛП: не-GeoTIFF в слот"
http POST /landscape/slots/lp-z-lcr -F "file=@$DATA/plots_nko.geojson"; need_code 422
need_has "$(jb .error)" 'extension ".geojson" is not allowed for file_type "raster"' сообщение
[ "$CODE" = 201 ] && track_ds "$(jb .id)"
end "422 «$(jb .error)»"
begin N36 "ЛП: window_size=100"
runreq '{"params":{"window_size":100}}'; need_code 422; need_has "$(jb .error)" "window_size must be in 256..8192" сообщение
end "422 «$(jb .error)»"
begin N38 "Загрузка 100 МБ мусора с сигнатурой TIFF"
{ printf 'II*\000'; head -c 104857600 /dev/urandom; } > "$TMP/garbage.tif"
up -F "file=@$TMP/garbage.tif" -F file_type=raster -F "category_id=$CAT_SOIL" -F "name=$P garbage"
need_code 201; DS_G="$(jb .id)"; up_time="$TIME"
wait_ds "$DS_G" ready; need_eq "$WS" failed status
http GET "/datasets/$DS_G"; pe="$(jb .parse_error)"; need_has "$pe" gdal_translate parse_error
http GET /readyz; need_code 200; need_eq "$(jb .ready)" true "/readyz ready"
rm -f "$TMP/garbage.tif"
end "201 (104 857 604 байт за ${up_time} с) → status $WS, parse_error от gdal_translate; /readyz 200, сервис доступен"
begin F35 "Удаление набора (строка и объект в S3)"
http DELETE "/datasets/$DS_N21"; need_code 204; c1="$CODE"
http GET "/datasets/$DS_N21"; need_code 404; c2="$CODE"
detail="$c1; GET → $c2"
if [ "$LOCAL_CHECKS" = 1 ]; then
prefix="${N21_KEY%/*}"
n="$(docker exec "$MINIO_C" sh -c "MC_CONFIG_DIR=/tmp/.mc MC_HOST_local=http://\$MINIO_ROOT_USER:\$MINIO_ROOT_PASSWORD@127.0.0.1:9000 mc ls --recursive local/geofiles/$prefix/ 2>/dev/null | wc -l" | tr -d ' ' || echo "?")"
need_eq "$n" 0 "объектов в префиксе MinIO"
detail="$detail; объектов в префиксе MinIO $prefix/ — $n"
else
detail="$detail; проверка MinIO пропущена (LOCAL_CHECKS=0)"
fi
end "$detail"
# ====================================================================== 5.5
psql_q() { docker exec -i "$PG_C" psql -U gis -d gis -At -F '|' -c "$1"; }
begin F36 "Геометрии наборов в PostGIS: тип, SRID, ST_IsValid"
if [ "$LOCAL_CHECKS" = 1 ]; then
out="$(psql_q "SELECT id, ST_GeometryType(geometry), ST_SRID(geometry), ST_IsValid(geometry) FROM datasets WHERE id IN ('$DS_PLOTS','$DS_POP','$DS_LIVE','$DS_NDVI')" 2>&1 || true)"
echo "$out" > "$RESULTS/F36.txt"
for pair in "$DS_PLOTS:ST_MultiPolygon" "$DS_POP:ST_MultiPoint" "$DS_LIVE:ST_MultiPoint" "$DS_NDVI:ST_Polygon"; do
id="${pair%%:*}"; want="${pair#*:}"
row="$(echo "$out" | grep "^$id|" || true)"
need_eq "$(echo "$row" | cut -d'|' -f2-4)" "$want|4326|t" "геометрия $id"
done
end "участки ST_MultiPolygon; население и КРС ST_MultiPoint; NDVI ST_Polygon; у всех SRID 4326, valid = t"
else skip "LOCAL_CHECKS=0 (нет доступа к контейнеру PostGIS)"; fi
begin F37 "Наблюдения dataset_observations соединяются с districts по КАТО"
if [ "$LOCAL_CHECKS" = 1 ]; then
out="$(psql_q "SELECT count(*), count(DISTINCT o.kato_code), count(*) FILTER (WHERE d.kato IS NULL) FROM dataset_observations o LEFT JOIN districts d ON d.kato = o.kato_code WHERE o.dataset_id IN ('$DS_POP','$DS_LIVE')" 2>&1 || true)"
need_eq "$out" "84|14|0" "наблюдений|кодов КАТО|сирот"
kz="$(psql_q "SELECT d.name || ' (' || d.kato || ') — ' || string_agg(o.value::text, ' / ' ORDER BY o.start_date) FROM dataset_observations o JOIN districts d ON d.kato = o.kato_code WHERE o.dataset_id = '$DS_POP' AND o.kato_code = '595000000' GROUP BY d.name, d.kato" 2>&1 || true)"
end "84 наблюдения, 14 кодов КАТО, orphan_kato 0 ($out); $kz"
else skip "LOCAL_CHECKS=0 (нет доступа к контейнеру PostGIS)"; fi
begin F38 "Объекты в бакете geofiles MinIO"
if [ "$LOCAL_CHECKS" = 1 ]; then
keys=""
for d in "$DS_PLOTS" "$DS_POP" "$DS_LIVE" "$DS_NDVI"; do
http GET "/datasets/$d"; keys="$keys $(jb '[.storage_key, .cog_storage_key, .sld_storage_key] | map(select(. != null)) | join(" ")')"
done
http GET "/dataset-groups/$GRP"; keys="$keys $(jb .sld_storage_key)"
found=0; total=0
for k in $keys; do
total=$((total + 1))
if docker exec "$MINIO_C" sh -c "MC_CONFIG_DIR=/tmp/.mc MC_HOST_local=http://\$MINIO_ROOT_USER:\$MINIO_ROOT_PASSWORD@127.0.0.1:9000 mc stat local/geofiles/$k" >/dev/null 2>&1; then
found=$((found + 1))
else bad "нет объекта $k"; fi
done
du="$(docker exec "$MINIO_C" sh -c 'MC_CONFIG_DIR=/tmp/.mc MC_HOST_local=http://$MINIO_ROOT_USER:$MINIO_ROOT_PASSWORD@127.0.0.1:9000 mc du local/geofiles' 2>/dev/null | tr -s ' \t' ' ' || true)"
[ "$total" -eq 7 ] || bad "ожидалось 7 ключей (4 исходных файла, COG, SLD набора, SLD группы), получено $total"
end "$found/$total объектов найдено: plots_nko.geojson, population_kato.geojson, livestock_kato.gpkg, ndvi_nko.tif, cog/ndvi_nko.tif, sld/style.sld, groups/…/sld/group_style.sld; бакет: $du"
else skip "LOCAL_CHECKS=0 (нет доступа к контейнеру MinIO)"; fi
# ====================================================================== 5.6 ЛП
LPIN="$TMP/lpin"; LPCACHE="$TMP/lpcache"; mkdir -p "$LPCACHE"
"$PYTHON" "$HERE/lp_gen.py" --out "$LPIN" > "$RESULTS/lp_gen.txt"
SLOTS="lp-z-biodiversity lp-z-lcr lp-z-soil-protection lp-z-wr lp-u-surf-water lp-u-soil lp-u-crop lp-u-veg lp-p-irrigation lp-p-snow lp-p-crop lp-p-pasture lp-p-tourism"
slot_id_file() { echo "$TMP/slot-$1"; }
begin L02 "Загрузка 13 входных растров в слоты POST /landscape/slots/{code}"
codes=""
for s in $SLOTS; do
http POST "/landscape/slots/$s" -F "file=@$LPIN/$s.tif" -F "name=$P $s" -F start_date=2025-01-01 -F end_date=2025-12-31
codes="$codes $CODE"; [ "$CODE" = 201 ] || bad "$s → $CODE: $(snippet)"
id="$(jb .id)"; echo "$id" > "$(slot_id_file "$s")"; track_ds "$id"
done
for s in $SLOTS; do wait_ds "$(cat "$(slot_id_file "$s")")" ready; [ "$WS" = ready ] || bad "$s: status $WS"; done
http GET /landscape/slots
for s in $SLOTS; do
got="$(jq -r --arg s "$s" '.[]|select(.code==$s)|"\(.latest.id) \(.latest.status)"' "$BODY")"
need_eq "$got" "$(cat "$(slot_id_file "$s")") ready" "слот $s"
done
end "201 для всех 13 слотов; GET /landscape/slots — у всех слотов latest = загруженный растр, status ready"
begin L03 "Запуск полного прогона {\"from_stage\":0}"
runreq '{"from_stage":0}'; need_code 202; RUN1="$(jb .id)"; need_eq "$(jb .status)" queued status
need_eq "$(jq -S -c .params "$BODY")" '{"tie_priority":[1,2,3,4,5],"v1_min_s_e12":3,"v1_min_s_e3":4,"v2_min_s_e3":3,"v2_min_s_e4":5,"window_size":2048}' params
end "202; run ${RUN1:0:8} поставлен в очередь, params по умолчанию (window_size 2048, tie_priority [1..5], пороги 3/4/3/5)"
begin L04 "Завершение прогона"
wait_run "$RUN1"; need_eq "$RS" succeeded status; ns="$(node_statuses "$RUN1")"
need_eq "$ns" "00:succeeded 01:succeeded 02:succeeded 03:succeeded 10:succeeded 20:succeeded 21:succeeded 30:succeeded 31:succeeded" узлы
need_eq "$(jq -r .error "$TMP/run-$RUN1.json")" null error
end "$RS за $("$PYTHON" "$HERE/lib/helpers.py" durations "$TMP/run-$RUN1.json"); error null; все 9 узлов succeeded"
begin L05 "Результаты GET /landscape/results"
outs="$(run_outputs "$RUN1")"
# Results list only ready layers; the COG conversion of the outputs is asynchronous.
for id in $(jq -r '.[].dataset_id' <<< "$outs"); do wait_ds "$id" ready; [ "$WS" = ready ] || bad "выход $id: status $WS"; done
http GET /landscape/results; need_code 200; cp "$BODY" "$TMP/results.json"
need_eq "$(jb 'length')" 15 "число групп"
for g in $(jq -r '.[].group' <<< "$outs"); do
want="$(jq -r --arg g "$g" '.[]|select(.group==$g)|.dataset_id' <<< "$outs")"
got="$(jq -r --arg g "$g" '.[]|select(.group==$g)|"\(.latest.id) \(.latest.status) \(.cog_url)"' "$TMP/results.json")"
need_eq "$got" "$want ready /datasets/$want.cog" "результат $g"
done
first="$(jq -r '.[0].dataset_id' <<< "$outs")"
http GET "/datasets/$first"; need_eq "$(jb .automated)" true automated; need_eq "$(jb .meta.lp_run_id)" "$RUN1" meta.lp_run_id
[ "$(jb .meta.lp_node)" != null ] || bad "meta.lp_node"; [ "$(jb '.meta.inputs|length')" -ge 1 ] 2>/dev/null || bad "meta.inputs"
fmt_ok=0
for id in $(jq -r '.[].dataset_id' <<< "$outs"); do
curl -fsS -o "$LPCACHE/$id.cog.tif" "$BASE_URL/datasets/$id.cog" || rm -f "$LPCACHE/$id.cog.tif"
ri="$("$PYTHON" "$HERE/lib/helpers.py" rinfo "$LPCACHE/$id.cog.tif" 2>/dev/null || echo '{}')"
f="$(jq -r '"\(.width)x\(.height) EPSG:\(.epsg) \(.layout) \(.dtype) \(.nodata|floor)"' <<< "$ri" 2>/dev/null || true)"
if [ "$f" = "3000x2000 EPSG:32642 COG int16 -9999" ]; then fmt_ok=$((fmt_ok + 1)); else bad "COG $id: $f"; fi
done
end "200; 15 групп (узлы 00…31), все ready, cog_url /datasets/{id}.cog; выходной набор: automated true, meta.lp_run_id, lp_node, inputs. $fmt_ok/15 COG: 3000×2000, EPSG:32642, LAYOUT=COG, Int16, NoData −9999"
begin L07 "Попиксельная сверка 15 выходов с эталонной реализацией"
if "$PYTHON" "$HERE/lp_verify.py" --base-url "$BASE_URL" --run "$RUN1" --cache "$LPCACHE" > "$RESULTS/L07_lp_verify.txt" 2>&1; then :; else bad "lp_verify: $(tail -n 3 "$RESULTS/L07_lp_verify.txt" | tr '\n' ' ')"; fi
tot="$(grep 'TOTAL mismatching' "$RESULTS/L07_lp_verify.txt" || echo 'нет итога')"
end "15 выходов × 6 000 000 пикселей; «$tot» (журнал: results/functional-$STAMP/L07_lp_verify.txt)"
begin L08 "Выборочная проверка 9 пикселей (Z_int, U_int, E, C_k, V1–V3)"
PTS="20,20;1450,1000;1575,1000;350,350;520,320;500,1500;1000,700;2200,1500;2900,1900"
if "$PYTHON" "$HERE/lp_verify.py" --base-url "$BASE_URL" --run "$RUN1" --cache "$LPCACHE" --points "$PTS" > "$RESULTS/L08_points.txt" 2>&1; then :; else bad "$(tail -n 1 "$RESULTS/L08_points.txt")"; fi
end "$(tail -n 1 "$RESULTS/L08_points.txt"): NoData (20,20), дыра NoData в U crop (1450,1000), crop и veg NoData (1575,1000), P snow = 0 и 6"
begin L09 "Повторный прогон {} без изменений входов"
runreq '{}'; need_code 202; RUN2="$(jb .id)"; wait_run "$RUN2"; need_eq "$RS" succeeded status
ns="$(node_statuses "$RUN2")"
need_eq "$ns" "00:skipped 01:skipped 02:skipped 03:skipped 10:skipped 20:skipped 21:skipped 30:skipped 31:skipped" узлы
need_eq "$(run_outputs "$RUN2")" "$(run_outputs "$RUN1")" "выходные наборы"
end "202; run ${RUN2:0:8} $RS за $("$PYTHON" "$HERE/lib/helpers.py" durations "$TMP/run-$RUN2.json"); все 9 узлов skipped, выходные наборы совпадают с первым прогоном"
begin L10 "Прогон {\"from_stage\":3}"
runreq '{"from_stage":3}'; need_code 202; RUN3="$(jb .id)"; wait_run "$RUN3"; need_eq "$RS" succeeded status
ns="$(node_statuses "$RUN3")"
need_eq "$ns" "00:skipped 01:skipped 02:skipped 03:skipped 10:skipped 20:skipped 21:skipped 30:succeeded 31:succeeded" узлы
new="$(jq -r '[.nodes[]|select(.node_code=="30" or .node_code=="31")|.outputs[]?.dataset_id]|join(" ")' "$TMP/run-$RUN3.json" || true)"
for id in $new; do run_outputs "$RUN1" | grep -q "$id" && bad "узел стадии 3 переиспользовал $id"; done
end "202; run ${RUN3:0:8} $RS за $("$PYTHON" "$HERE/lib/helpers.py" durations "$TMP/run-$RUN3.json"); узлы 00–21 skipped, узлы 30 и 31 succeeded (новые выходные наборы)"
begin L11 "Прогон с параметрами tie_priority [5,4,3,2,1], пороги 4/5/4/4"
runreq '{"params":{"tie_priority":[5,4,3,2,1],"v1_min_s_e12":4,"v1_min_s_e3":5,"v2_min_s_e3":4,"v2_min_s_e4":4}}'
need_code 202; RUN4="$(jb .id)"; wait_run "$RUN4"; need_eq "$RS" succeeded status
ns="$(node_statuses "$RUN4")"
need_eq "$ns" "00:skipped 01:skipped 02:skipped 03:skipped 10:skipped 20:skipped 21:skipped 30:succeeded 31:skipped" узлы
end "202; run ${RUN4:0:8} $RS за $("$PYTHON" "$HERE/lib/helpers.py" durations "$TMP/run-$RUN4.json"); пересчитан только узел 30 (V1–V3), остальные skipped"
begin L12 "Сверка V1–V3 с пользовательскими параметрами"
wait_outputs "$RUN4"
if "$PYTHON" "$HERE/lp_verify.py" --base-url "$BASE_URL" --run "$RUN4" --cache "$LPCACHE" > "$RESULTS/L12_lp_verify.txt" 2>&1; then :; else bad "lp_verify: $(tail -n 3 "$RESULTS/L12_lp_verify.txt" | tr '\n' ' ')"; fi
v="$(grep -E '^lp-v[123]-' "$RESULTS/L12_lp_verify.txt" | awk '{printf "%s mismatch %s; ", $1, $6}' || true)"
end "$v; «$(grep 'TOTAL mismatching' "$RESULTS/L12_lp_verify.txt" || echo 'нет итога')» (все 15 выходов)"
begin L13 "Невыровненный растр (1500×1000, 200 м) в слоте lp-u-veg"
http POST /landscape/slots/lp-u-veg -F "file=@$LPIN/misaligned-lp-u-veg.tif" -F "name=$P misaligned" -F start_date=2025-01-01 -F end_date=2025-12-31
need_code 201; DS_MIS="$(jb .id)"; track_ds "$DS_MIS"; wait_ds "$DS_MIS" ready; need_eq "$WS" ready "status растра"
runreq '{}'; need_code 202; RUN5="$(jb .id)"; wait_run "$RUN5"; need_eq "$RS" failed status
err="$(jq -r .error "$TMP/run-$RUN5.json" || true)"
need_has "$err" "input lp-u-veg grid (1500x1000" "сообщение"; need_has "$err" "inputs must be aligned" "сообщение"
later="$(jq -r '[.nodes[]|select(.stage>=1)|.status]|unique|join(",")' "$TMP/run-$RUN5.json" || true)"
need_eq "$later" pending "узлы стадий 1–3"
end "run ${RUN5:0:8} $RS: «${err:0:160}…»; последующие узлы не выполняются ($later)"
begin L14 "Восстановление: удаление растра, повторный прогон"
http DELETE "/datasets/$DS_MIS"; need_code 204; c1="$CODE"
http GET /landscape/slots
need_eq "$(jq -r '.[]|select(.code=="lp-u-veg")|.latest.id' "$BODY")" "$(cat "$(slot_id_file lp-u-veg)")" "слот lp-u-veg"
runreq '{}'; need_code 202; RUN6="$(jb .id)"; wait_run "$RUN6"; need_eq "$RS" succeeded status
ns="$(node_statuses "$RUN6")"
need_eq "$ns" "00:skipped 01:skipped 02:skipped 03:skipped 10:skipped 20:skipped 21:skipped 30:skipped 31:skipped" узлы
end "DELETE $c1; в слоте снова исходный растр; run ${RUN6:0:8} $RS, все узлы skipped"
begin L15 "Повторный полный прогон с замером ресурсов worker"
cpu0=""
if [ "$LOCAL_CHECKS" = 1 ] && [ -n "$WORKER_C" ]; then
cpu0="$(docker exec "$WORKER_C" sh -c "awk '/usage_usec/{print \$2}' /sys/fs/cgroup/cpu.stat" 2>/dev/null || true)"
fi
runreq '{"from_stage":0}'; need_code 202; RUN7="$(jb .id)"; c1="$CODE"; wait_run "$RUN7"; need_eq "$RS" succeeded status
ns="$(node_statuses "$RUN7")"
need_eq "$ns" "00:succeeded 01:succeeded 02:succeeded 03:succeeded 10:succeeded 20:succeeded 21:succeeded 30:succeeded 31:succeeded" узлы
res=""
if [ -n "$cpu0" ]; then
cpu1="$(docker exec "$WORKER_C" sh -c "awk '/usage_usec/{print \$2}' /sys/fs/cgroup/cpu.stat" 2>/dev/null || echo "$cpu0")"
peak="$(docker exec "$WORKER_C" sh -c 'cat /sys/fs/cgroup/memory.peak 2>/dev/null || echo 0')"
res="; CPU worker $(awk -v a="$cpu0" -v b="$cpu1" 'BEGIN{printf "%.2f", (b-a)/1e6}') с (включая COG-конвертацию), memory.peak $(awk -v p="$peak" 'BEGIN{printf "%.2f", p/1e9}') ГБ (с учётом страничного кэша)"
fi
end "$c1; run ${RUN7:0:8} $RS за $("$PYTHON" "$HERE/lib/helpers.py" durations "$TMP/run-$RUN7.json")$res"
# ==================================================================== summary
TOTAL=$((PASSN + FAILN + SKIPN))
{
echo "-----------------------------------------------------------------"
echo "ИТОГО: $TOTAL проверок — PASS $PASSN, FAIL $FAILN, SKIP $SKIPN"
echo "RESULT: $PASSN/$TOTAL PASS"
echo "ЛП прогоны: RUN1=${RUN1:-} (from_stage 0), RUN4=${RUN4:-} (пользовательские параметры)"
} | tee -a "$RESULTS/functional.txt"
[ "$FAILN" -eq 0 ]

View File

@@ -0,0 +1,78 @@
#!/usr/bin/env python3
"""Small helpers for functional.sh (numpy + rasterio).
helpers.py ndvi OUT.tif write the 600x400 EPSG:4326 Float32 NDVI test raster
helpers.py rinfo FILE.tif print raster facts as JSON (driver, size, crs, dtype,
nodata, layout, compression, overviews)
helpers.py durations RUN.json print "total=<s> nodes=<min>..<max>ms" for a run
helpers.py elapsed DATASET.json print updated_at - created_at of a dataset, seconds
"""
import json
import sys
from datetime import datetime
def ndvi(out):
import numpy as np
import rasterio
from rasterio.transform import from_origin
w, h = 600, 400 # 0.01 deg pixels, bbox [68.5, 51.5, 74.5, 55.5]
y, x = np.mgrid[0:h, 0:w].astype(np.float32)
v = 0.45 + 0.35 * np.sin(x / 47.0) * np.cos(y / 31.0) + 0.1 * np.sin((x + y) / 13.0)
v = np.clip(v, -0.2, 1.0).astype(np.float32)
v[:5, :] = -9999
profile = dict(driver="GTiff", width=w, height=h, count=1, dtype="float32", crs="EPSG:4326",
transform=from_origin(68.5, 55.5, 0.01, 0.01), nodata=-9999)
with rasterio.open(out, "w", **profile) as dst:
dst.write(v, 1)
def rinfo(path):
import rasterio
with rasterio.open(path) as d:
st = d.tags(ns="IMAGE_STRUCTURE")
print(json.dumps({
"driver": d.driver, "width": d.width, "height": d.height,
"epsg": d.crs.to_epsg() if d.crs else None, "dtype": d.dtypes[0], "nodata": d.nodata,
"layout": st.get("LAYOUT"), "compression": st.get("COMPRESSION"),
"overviews": len(d.overviews(1)), "bounds": [round(b, 6) for b in d.bounds],
}))
def ts(s):
if not s:
return None
s = s.replace("Z", "+00:00")
if "." in s: # trim nanoseconds to microseconds for fromisoformat
head, rest = s.split(".", 1)
frac = "".join(ch for ch in rest if ch.isdigit())
tz = rest[len(frac):]
s = "%s.%s%s" % (head, frac[:6].ljust(6, "0"), tz)
return datetime.fromisoformat(s)
def durations(path):
with open(path, encoding="utf-8") as f:
run = json.load(f)
total = None
if run.get("started_at") and run.get("finished_at"):
total = (ts(run["finished_at"]) - ts(run["started_at"])).total_seconds()
nd = [(ts(n["finished_at"]) - ts(n["started_at"])).total_seconds() * 1000
for n in run.get("nodes", []) if n.get("started_at") and n.get("finished_at")]
out = "total=%.2fs" % total if total is not None else "total=?"
if nd:
out += " nodes=%d..%dms" % (min(nd), max(nd))
print(out)
def elapsed(path):
with open(path, encoding="utf-8") as f:
d = json.load(f)
print("%.2f" % (ts(d["updated_at"]) - ts(d["created_at"])).total_seconds())
if __name__ == "__main__":
cmd, arg = sys.argv[1], sys.argv[2]
{"ndvi": ndvi, "rinfo": rinfo, "durations": durations, "elapsed": elapsed}[cmd](arg)

View File

@@ -0,0 +1,11 @@
# Load generator image for load.sh: hey v0.1.5 (github.com/rakyll/hey) built
# from source on the same pinned Go/Alpine images as the application.
FROM golang:1.26.9-alpine3.24 AS build
RUN CGO_ENABLED=0 go install github.com/rakyll/hey@v0.1.5
FROM alpine:3.24
COPY --from=build /go/bin/hey /usr/local/bin/hey
RUN adduser -D -u 10001 hey
USER hey
# One-shot CLI container: nothing to health-check.
HEALTHCHECK NONE
ENTRYPOINT ["hey"]

View File

@@ -0,0 +1,200 @@
#!/usr/bin/env python3
"""Aggregates the hey outputs of load.sh into the tables of Приложение 3.
load_analyze.py RESULTS_DIR
RESULTS_DIR/manifest.csv rows: scenario,mode,label,clients,file
mode par = the files of the scenario ran at the same time (duration = max)
seq = they ran one after another (duration = sum)
Each file is a default (summary) hey output. Per scenario the program reports
requests, clients, duration, RPS, success %, status codes, transport errors,
min/avg/P50/P90/P95/P99/max (ms). As in the protocol, percentiles of a
scenario made of several hey processes/runs are averages weighted by the
number of responses; min/max are exact.
Criteria (Приложение 3, табл. 8): K1 success >= 99 %, K2 no 5xx, K3 P95 <=
2000 ms; K4 (S6 vs S1): /readyz 200 and S6 P50 <= 2 x S1 P50 + 1 ms.
Writes summary.csv, summary.json, detail.csv into RESULTS_DIR and prints the
tables. Exit code 1 when a criterion fails.
"""
import csv
import json
import os
import re
import sys
PCTS = ["10", "25", "50", "75", "90", "95", "99"]
def parse_hey(path):
txt = open(path, encoding="utf-8", errors="replace").read()
def num(label):
m = re.search(r"^\s*%s:\s+([0-9.]+)" % re.escape(label), txt, re.M)
return float(m.group(1)) if m else None
r = {"total_s": num("Total"), "slowest_s": num("Slowest"), "fastest_s": num("Fastest"),
"average_s": num("Average"), "rps": num("Requests/sec"), "codes": {}, "errors": 0, "pct": {}}
for p, v in re.findall(r"^\s*(\d+)%+ in ([0-9.]+) secs", txt, re.M):
if p != "0": # hey 0.1.5 prints "%%" and a bogus "0%% in 0.0000" row for tiny samples
r["pct"][p] = float(v)
sec = txt.split("Status code distribution:", 1)
if len(sec) == 2:
for code, n in re.findall(r"^\s*\[(\d+)\]\s+(\d+) responses", sec[1].split("Error distribution:")[0], re.M):
r["codes"][code] = r["codes"].get(code, 0) + int(n)
sec = txt.split("Error distribution:", 1)
if len(sec) == 2:
r["errors"] = sum(int(n) for n in re.findall(r"^\s*\[(\d+)\]\s", sec[1], re.M))
r["responses"] = sum(r["codes"].values())
r["requests"] = r["responses"] + r["errors"]
return r
def aggregate(rows, mode):
parts = [x["hey"] for x in rows]
resp = sum(p["responses"] for p in parts)
req = sum(p["requests"] for p in parts)
durs = [p["total_s"] or 0 for p in parts]
dur = max(durs) if mode == "par" else sum(durs)
codes = {}
for p in parts:
for c, n in p["codes"].items():
codes[c] = codes.get(c, 0) + n
def wavg(get):
num = sum(get(p) * p["responses"] for p in parts if get(p) is not None and p["responses"])
den = sum(p["responses"] for p in parts if get(p) is not None and p["responses"])
return num / den if den else None
ms = lambda v: None if v is None else v * 1000.0
ok2xx = sum(n for c, n in codes.items() if c.startswith("2"))
fast = [p["fastest_s"] for p in parts if p["fastest_s"] is not None]
slow = [p["slowest_s"] for p in parts if p["slowest_s"] is not None]
return {
"requests": req, "responses": resp, "errors": sum(p["errors"] for p in parts),
"clients": sum(int(x["clients"]) for x in rows) if mode == "par" else max(int(x["clients"]) for x in rows),
"duration_s": dur, "rps": (resp / dur) if dur else None,
"success_pct": (100.0 * ok2xx / req) if req else 0.0, "codes": codes,
"n5xx": sum(n for c, n in codes.items() if c.startswith("5")),
"min_ms": ms(min(fast)) if fast else None, "avg_ms": ms(wavg(lambda p: p["average_s"])),
**{"p%s_ms" % k: ms(wavg(lambda p, k=k: p["pct"].get(k))) for k in ("50", "90", "95", "99")},
"max_ms": ms(max(slow)) if slow else None,
}
def f1(v, nd=1):
return "-" if v is None else ("%.*f" % (nd, v))
def codes_str(c):
return ", ".join("%s: %d" % (k, v) for k, v in sorted(c.items())) or "-"
def main():
d = sys.argv[1]
rows = []
with open(os.path.join(d, "manifest.csv"), encoding="utf-8") as f:
for x in csv.DictReader(f):
x["hey"] = parse_hey(os.path.join(d, "raw", x["file"]))
rows.append(x)
order, by = [], {}
for x in rows:
if x["scenario"] not in by:
order.append(x["scenario"])
by[x["scenario"]] = []
by[x["scenario"]].append(x)
summary = []
for s in order:
agg = aggregate(by[s], by[s][0]["mode"])
agg["scenario"] = s
agg["k1"] = agg["success_pct"] >= 99.0
agg["k2"] = agg["n5xx"] == 0
agg["k3"] = agg["p95_ms"] is not None and agg["p95_ms"] <= 2000.0
summary.append(agg)
extra = {}
xp = os.path.join(d, "extra.json")
if os.path.exists(xp):
extra = json.load(open(xp, encoding="utf-8"))
k4 = None
s1 = next((x for x in summary if x["scenario"] == "S1"), None)
s6 = next((x for x in summary if x["scenario"] == "S6"), None)
if s1 and s6:
k4 = (extra.get("readyz_after") == 200 and s6["p50_ms"] is not None and s1["p50_ms"] is not None
and s6["p50_ms"] <= 2 * s1["p50_ms"] + 1.0)
hdr = ["Сцен.", "Запр.", "Клиентов", "Длит.,с", "RPS", "Успех,%", "min", "avg", "P50", "P90", "P95", "P99", "max", "Коды", "Ошибки", "Итог"]
lines = []
for a in summary:
ok = a["k1"] and a["k2"] and a["k3"]
lines.append([a["scenario"], str(a["requests"]), str(a["clients"]), f1(a["duration_s"]), f1(a["rps"], 0),
f1(a["success_pct"], 2), f1(a["min_ms"]), f1(a["avg_ms"]), f1(a["p50_ms"]), f1(a["p90_ms"]),
f1(a["p95_ms"]), f1(a["p99_ms"]), f1(a["max_ms"]), codes_str(a["codes"]), str(a["errors"]),
"PASS" if ok else "FAIL"])
print_table("Сводные результаты (время — мс)", hdr, lines)
for s in order:
if len(by[s]) > 1 and s not in ("S1", "S6"):
dl = []
for x in by[s]:
a = aggregate([x], "par")
dl.append([x["label"], x["clients"], str(a["responses"]), f1(a["rps"], 1), f1(a["p50_ms"]),
f1(a["p95_ms"]), f1(a["p99_ms"]), f1(a["max_ms"]), codes_str(a["codes"]), str(a["errors"])])
print_table("%s по эндпоинтам / прогонам (время — мс)" % s,
["Эндпоинт / прогон", "Клиентов", "Ответов", "RPS", "P50", "P95", "P99", "max", "Коды", "Ошибки"], dl)
if extra.get("uploads"):
u = extra["uploads"]
print("S5 загрузки: %d запросов POST /datasets, коды %s, время ответа %s–%s мс; обработано до ready: %d/%d, макс. время обработки %s с"
% (u["count"], u["codes"], f1(u.get("min_ms")), f1(u.get("max_ms")), u.get("ready", 0), u["count"], f1(u.get("max_processing_s"), 2)))
if extra.get("stats_peak"):
print("\nПиковое потребление ресурсов (docker stats, сценарии S4/S5):")
for name, v in sorted(extra["stats_peak"].items()):
print(" %-34s CPU %6.1f %% память %8.1f МиБ" % (name, v["cpu"], v["mem_mib"]))
if k4 is not None:
print("\nK4 (после нагрузки): /readyz %s; S6 P50/P95 = %s/%s мс против S1 %s/%s мс -> %s" % (
extra.get("readyz_after"), f1(s6["p50_ms"]), f1(s6["p95_ms"]), f1(s1["p50_ms"]), f1(s1["p95_ms"]),
"PASS" if k4 else "FAIL"))
with open(os.path.join(d, "summary.csv"), "w", newline="", encoding="utf-8") as f:
w = csv.writer(f)
w.writerow(["scenario", "requests", "clients", "duration_s", "rps", "success_pct", "min_ms", "avg_ms",
"p50_ms", "p90_ms", "p95_ms", "p99_ms", "max_ms", "codes", "errors", "result"])
for ln, a in zip(lines, summary):
w.writerow([a["scenario"], a["requests"], a["clients"], a["duration_s"], a["rps"], a["success_pct"],
a["min_ms"], a["avg_ms"], a["p50_ms"], a["p90_ms"], a["p95_ms"], a["p99_ms"], a["max_ms"],
codes_str(a["codes"]), a["errors"], ln[-1]])
with open(os.path.join(d, "detail.csv"), "w", newline="", encoding="utf-8") as f:
w = csv.writer(f)
w.writerow(["scenario", "label", "clients", "requests", "responses", "errors", "duration_s", "rps",
"min_ms", "avg_ms", "p50_ms", "p90_ms", "p95_ms", "p99_ms", "max_ms", "codes", "file"])
for x in rows:
a = aggregate([x], "par")
w.writerow([x["scenario"], x["label"], x["clients"], a["requests"], a["responses"], a["errors"],
a["duration_s"], a["rps"], a["min_ms"], a["avg_ms"], a["p50_ms"], a["p90_ms"], a["p95_ms"],
a["p99_ms"], a["max_ms"], codes_str(a["codes"]), x["file"]])
with open(os.path.join(d, "summary.json"), "w", encoding="utf-8") as f:
json.dump({"scenarios": summary, "k4": k4, "extra": extra}, f, ensure_ascii=False, indent=2)
failed = [a["scenario"] for a in summary if not (a["k1"] and a["k2"] and a["k3"])]
if k4 is False:
failed.append("K4")
print("\nКритерии K1 (успех >= 99 %%), K2 (5xx = 0), K3 (P95 <= 2000 мс)%s: %s" % (
", K4" if k4 is not None else "", "все выполнены" if not failed else "НЕ выполнены: " + ", ".join(failed)))
print("Файлы: %s/{summary.csv,summary.json,detail.csv,raw/}" % d)
return 1 if failed else 0
def print_table(title, hdr, lines):
widths = [max(len(str(r[i])) for r in [hdr] + lines) for i in range(len(hdr))]
print("\n" + title)
print(" | ".join(h.ljust(w) for h, w in zip(hdr, widths)))
print("-+-".join("-" * w for w in widths))
for r in lines:
print(" | ".join(str(c).ljust(w) for c, w in zip(r, widths)))
if __name__ == "__main__":
sys.exit(main())

View File

@@ -0,0 +1,150 @@
#!/usr/bin/env python3
"""Regenerates the committed vector/SLD fixtures in test/acceptance/data/.
The fixtures are committed, so this script is only needed to rebuild them.
Requires the GDAL command-line tool ogr2ogr (GeoPackage conversion).
python3 test/acceptance/lib/make_fixtures.py
Values are deterministic: district KATO codes and names come from
stubs/districts.geojson (14 districts of North Kazakhstan region), the
indicator values follow a fixed formula, so re-running yields identical files.
"""
import json
import os
import subprocess
import sys
import tempfile
HERE = os.path.dirname(os.path.abspath(__file__))
ROOT = os.path.abspath(os.path.join(HERE, "..", "..", ".."))
DATA = os.path.abspath(os.path.join(HERE, "..", "data"))
def ring_bbox(coords):
xs, ys = [], []
def walk(c):
if isinstance(c[0], (int, float)):
xs.append(c[0])
ys.append(c[1])
else:
for x in c:
walk(x)
walk(coords)
return min(xs), min(ys), max(xs), max(ys)
def population(i, kato, year):
# Values quoted in the protocol (Приложение 7, F16/F17) are kept verbatim.
fixed = {
("591000000", 2022): 47567.6,
("591000000", 2023): 48311.2,
("591000000", 2024): 49120.5,
("595000000", 2022): 46862.7,
("595000000", 2023): 48268.6,
("595000000", 2024): 49205.8,
}
if (kato, year) in fixed:
return fixed[(kato, year)]
return round(20000 + 1750.3 * i + 812.4 * (year - 2022) + (i * 37 % 11) * 13.1, 1)
def livestock(i, year):
return int(30000 + 2311 * i + 977 * (year - 2022) + (i * 53 % 17) * 41)
def main():
with open(os.path.join(ROOT, "stubs", "districts.geojson"), encoding="utf-8") as f:
districts = json.load(f)["features"]
os.makedirs(DATA, exist_ok=True)
# 1. plain vector: 4 plot polygons spanning lon 69.0..71.7, lat 53.6..55.0
plots = []
boxes = [
(69.0, 53.6, 69.6, 54.0),
(69.9, 54.2, 70.5, 54.7),
(70.6, 53.7, 71.1, 54.1),
(71.2, 54.5, 71.7, 55.0),
]
for n, (x0, y0, x1, y1) in enumerate(boxes, 1):
plots.append({
"type": "Feature",
"properties": {"plot_id": n, "name": "Участок %d" % n, "area_ha": round((x1 - x0) * (y1 - y0) * 7000, 1)},
"geometry": {"type": "Polygon", "coordinates": [[[x0, y0], [x1, y0], [x1, y1], [x0, y1], [x0, y0]]]},
})
write_json("plots_nko.geojson", {"type": "FeatureCollection", "features": plots})
# 2. vector_with_kato (GeoJSON): population per district, Y2022..Y2024
pop = []
live = []
for i, d in enumerate(districts):
p = d["properties"]
kato = str(p["kato"])
x0, y0, x1, y1 = ring_bbox(d["geometry"]["coordinates"])
pt = {"type": "Point", "coordinates": [round((x0 + x1) / 2, 5), round((y0 + y1) / 2, 5)]}
props = {"kato": kato, "district": p["name_ru"]}
for y in (2022, 2023, 2024):
props["Y%d" % y] = population(i, kato, y)
pop.append({"type": "Feature", "properties": props, "geometry": pt})
lprops = {"KATO_CODE": kato, "district": p["name_ru"]}
for y in (2022, 2023, 2024):
lprops["H_%d" % y] = livestock(i, y)
live.append({"type": "Feature", "properties": lprops, "geometry": pt})
write_json("population_kato.geojson", {"type": "FeatureCollection", "features": pop})
# 3. vector_with_kato (GeoPackage): cattle per district, H_2022..H_2024
with tempfile.TemporaryDirectory() as tmp:
src = os.path.join(tmp, "livestock.geojson")
with open(src, "w", encoding="utf-8") as f:
json.dump({"type": "FeatureCollection", "features": live}, f, ensure_ascii=False)
dst = os.path.join(DATA, "livestock_kato.gpkg")
if os.path.exists(dst):
os.remove(dst)
subprocess.run(["ogr2ogr", "-f", "GPKG", "-nln", "livestock", "-a_srs", "EPSG:4326", dst, src], check=True)
# 4. SLD styles
write_text("style.sld", sld("ndvi", "NDVI gradient", [("0.0", "#a50026"), ("0.5", "#ffffbf"), ("1.0", "#006837")]))
write_text("group_style.sld", sld("group", "Group fallback style", [("0", "#f7fbff"), ("100000", "#08306b")]))
write_text("style.txt", "not an SLD file\n")
print("fixtures written to", DATA)
def sld(name, title, stops):
entries = "\n".join(
' <sld:ColorMapEntry color="%s" quantity="%s"/>' % (c, q) for q, c in stops)
return """<?xml version="1.0" encoding="UTF-8"?>
<sld:StyledLayerDescriptor version="1.0.0" xmlns:sld="http://www.opengis.net/sld" xmlns:ogc="http://www.opengis.net/ogc">
<sld:NamedLayer>
<sld:Name>%s</sld:Name>
<sld:UserStyle>
<sld:Title>%s</sld:Title>
<sld:FeatureTypeStyle>
<sld:Rule>
<sld:RasterSymbolizer>
<sld:ColorMap>
%s
</sld:ColorMap>
</sld:RasterSymbolizer>
</sld:Rule>
</sld:FeatureTypeStyle>
</sld:UserStyle>
</sld:NamedLayer>
</sld:StyledLayerDescriptor>
""" % (name, title, entries)
def write_json(name, obj):
with open(os.path.join(DATA, name), "w", encoding="utf-8") as f:
json.dump(obj, f, ensure_ascii=False, indent=1)
f.write("\n")
def write_text(name, text):
with open(os.path.join(DATA, name), "w", encoding="utf-8") as f:
f.write(text)
if __name__ == "__main__":
sys.exit(main())

227
test/acceptance/load.sh Executable file
View File

@@ -0,0 +1,227 @@
#!/usr/bin/env bash
# Load test of the GIS API — scenarios S1–S6 of «Протокол нагрузочного
# тестирования» (Приложение 3), generator hey 0.1.5:
# S1 baseline: 10 endpoints one after another, hey -n 100 -c 1 each
# S2 mixed: 1000 requests, 100 concurrent clients over the 10 endpoints
# (10 parallel hey processes, -n 100 -c 10 each)
# S3.1 1000 simultaneous requests: hey -n 1000 -c 1000 /datasets?page_size=20, 3 runs
# S3.2 1000 simultaneous requests: hey -n 1000 -c 1000 /landscape/results, 3 runs
# S4 sustained ~1000 req/s for 60 s: 1000 clients at -q 1 split over 8
# endpoints (300/150/100/100/100/100/100/50), 8 parallel hey processes
# S5 30 s of reads (10 endpoints x 10 clients, -z 30s) while 20 uploads
# (10 GeoJSON + 10 GeoTIFF) run in parallel; uploads must reach ready
# S6 recovery: S1 again + /readyz
#
# By default the generator runs INSIDE the stand's docker network
# (docker run --network gis-acceptance_default gis-acceptance-hey:0.1.5,
# target http://app:8080), so host port forwarding is not measured.
#
# Environment:
# BASE_URL API address for setup calls (default http://localhost:18080)
# TARGET_URL address the generator loads (default http://app:8080 inside
# the network; any URL with GEN=host or NETWORK=bridge)
# GEN docker (default) | host (use a locally installed hey)
# NETWORK docker network of the generator (default gis-acceptance_default)
# SCENARIOS subset to run (default "S1 S2 S3 S4 S5 S6")
# S4_DURATION / S5_DURATION default 60s / 30s
# PYTHON python3 with numpy + rasterio (S5 raster uploads, seeding)
# ALLOW_WRITES 1 = allow seeding and S5 uploads on a non-local BASE_URL
#
# Output: summary table on stdout; test/acceptance/results/load-<timestamp>/
# {summary.csv, summary.json, detail.csv, raw/*.txt}. Exit code 1 when a
# criterion K1–K4 fails.
set -euo pipefail
HERE="$(cd "$(dirname "$0")" && pwd)"
BASE_URL="${BASE_URL:-http://localhost:18080}"; BASE_URL="${BASE_URL%/}"
GEN="${GEN:-docker}"
NETWORK="${NETWORK:-gis-acceptance_default}"
HEY_IMAGE="${HEY_IMAGE:-gis-acceptance-hey:0.1.5}"
SCENARIOS="${SCENARIOS:-S1 S2 S3 S4 S5 S6}"
S4_DURATION="${S4_DURATION:-60s}"
S5_DURATION="${S5_DURATION:-30s}"
PYTHON="${PYTHON:-python3}"
ALLOW_WRITES="${ALLOW_WRITES:-0}"
if [ -z "${TARGET_URL:-}" ]; then
if [ "$GEN" = docker ] && [ "$NETWORK" = gis-acceptance_default ]; then TARGET_URL=http://app:8080; else TARGET_URL="$BASE_URL"; fi
fi
TARGET_URL="${TARGET_URL%/}"
LOCAL=0
case "$BASE_URL" in http://localhost:*|http://127.0.0.1:*) LOCAL=1;; esac
case "$BASE_URL" in *dssgis.dwh.kz*) ALLOW_WRITES=0;; esac
for bin in curl jq python3; do command -v "$bin" >/dev/null || { echo "missing $bin" >&2; exit 2; }; done
STAMP="$(date +%Y%m%d-%H%M%S)"
OUT="$HERE/results/load-$STAMP"; RAW="$OUT/raw"; mkdir -p "$RAW"
echo "scenario,mode,label,clients,file" > "$OUT/manifest.csv"
EXTRA="$OUT/extra.json"; echo '{}' > "$EXTRA"
log() { echo "[$(date +%H:%M:%S)] $*"; }
extra_set() { local t; t="$(mktemp)"; jq "$1" "$EXTRA" > "$t" && mv "$t" "$EXTRA"; }
# ---------------------------------------------------------------- generator
if [ "$GEN" = docker ]; then
if ! docker image inspect "$HEY_IMAGE" >/dev/null 2>&1; then
log "building $HEY_IMAGE"
docker build -q -t "$HEY_IMAGE" -f "$HERE/lib/hey.Dockerfile" "$HERE/lib" >/dev/null
fi
NETARG="--network $NETWORK"; [ "$NETWORK" = bridge ] && NETARG=""
fi
# run_batch SCEN MODE "label|clients|file|hey args and path" ...
# MODE par: all hey processes start together; seq: one after another.
run_batch() {
local scen="$1" mode="$2"; shift 2
local script="" spec label clients file args sep=" & "
[ "$mode" = seq ] && sep=" ; "
for spec in "$@"; do
IFS='|' read -r label clients file args <<< "$spec"
echo "$scen,$mode,\"$label\",$clients,$file" >> "$OUT/manifest.csv"
script="${script}hey $args > /out/$file 2>&1${sep}"
done
script="${script} wait"
if [ "$GEN" = docker ]; then
# shellcheck disable=SC2086
docker run --rm $NETARG --ulimit nofile=65536:65536 -v "$RAW:/out" --entrypoint sh "$HEY_IMAGE" -c "$script"
else
( cd "$RAW" && ulimit -n 65536 2>/dev/null || true; sh -c "${script//\/out\//$RAW/}" )
fi
}
# --------------------------------------------------------------- endpoints
if [ "$LOCAL" = 1 ] || [ "$ALLOW_WRITES" = 1 ]; then
log "seed (idempotent): $BASE_URL"
BASE_URL="$BASE_URL" PYTHON="$PYTHON" "$HERE/seed.sh" | tail -n 4
# shellcheck disable=SC1091
. "$HERE/results/seed.env"
else
log "read-only discovery of endpoint ids on $BASE_URL"
VWK_ID="$(curl -fsS "$BASE_URL/datasets?file_type=vector_with_kato&status=ready&page_size=1" | jq -r '.data[0].id')"
GROUP_ID="$(curl -fsS "$BASE_URL/datasets/$VWK_ID" | jq -r '.dg_id // empty')"
CAT_ID=""
fi
RUN_ID="$(curl -fsS "$BASE_URL/landscape/runs?page_size=1" | jq -r '.data[0].id // empty')"
if [ -z "$RUN_ID" ] && { [ "$LOCAL" = 1 ] || [ "$ALLOW_WRITES" = 1 ]; }; then
log "no ЛП run yet: running lp_run.sh"
BASE_URL="$BASE_URL" PYTHON="$PYTHON" "$HERE/lp_run.sh" > "$OUT/lp_run.txt" 2>&1
RUN_ID="$(curl -fsS "$BASE_URL/landscape/runs?page_size=1" | jq -r '.data[0].id')"
fi
for v in VWK_ID GROUP_ID RUN_ID; do
eval "val=\${$v:-}"
# shellcheck disable=SC2154
[ -n "$val" ] && [ "$val" != null ] || { echo "cannot resolve $v on $BASE_URL" >&2; exit 2; }
done
EP_PATHS="/datasets?page_size=20 /datasets/$VWK_ID /datasets/$VWK_ID.geojson /datasets/$VWK_ID/observations /categories /dataset-groups/$GROUP_ID/latest /landscape/slots /landscape/results /landscape/runs/$RUN_ID /readyz"
label_of() { # path -> protocol label
echo "$1" | sed -e "s#$VWK_ID#{id}#g" -e "s#$GROUP_ID#{id}#g" -e "s#$RUN_ID#{id}#g"
}
fname() { echo "$1" | sed -e 's#^/##' -e 's#[/?=.{}]#_#g'; }
for p in $EP_PATHS; do # every endpoint must answer 200 before loading it
c="$(curl -sS -o /dev/null -w '%{http_code}' "$BASE_URL$p")"
[ "$c" = 200 ] || { echo "endpoint $p answers $c, fix the stand first" >&2; exit 2; }
done
log "target $TARGET_URL (generator: $GEN${NETARG:+, $NETARG}); results $OUT"
curl -fsS "$BASE_URL/datasets?status=ready&page_size=1" | jq -r '"ready datasets on the stand: \(.total)"'
stats_start() { # docker stats sampler for the compose project (local stand only)
[ "$LOCAL" = 1 ] || return 0
( while :; do docker stats --no-stream --format '{{.Name}},{{.CPUPerc}},{{.MemUsage}}' \
$(docker ps -q --filter label=com.docker.compose.project=gis-acceptance) >> "$OUT/stats.csv" 2>/dev/null || true
sleep 3; done ) &
STATS_PID=$!
}
stats_stop() { [ -n "${STATS_PID:-}" ] && kill "$STATS_PID" 2>/dev/null || true; STATS_PID=""; }
trap 'stats_stop' EXIT
baseline() { # S1 / S6
local scen="$1" specs=() p
for p in $EP_PATHS; do specs+=("$(label_of "$p")|1|$scen-$(fname "$p").txt|-n 100 -c 1 '$TARGET_URL$p'"); done
log "$scen: 10 endpoints x hey -n 100 -c 1 (sequential)"
run_batch "$scen" seq "${specs[@]}"
}
for S in $SCENARIOS; do
case "$S" in
S1) baseline S1 ;;
S2)
specs=()
for p in $EP_PATHS; do specs+=("$(label_of "$p")|10|S2-$(fname "$p").txt|-n 100 -c 10 '$TARGET_URL$p'"); done
log "S2: 1000 requests, 100 clients over 10 endpoints"
run_batch S2 par "${specs[@]}" ;;
S3)
for i in 1 2 3; do
log "S3.1 run $i: hey -n 1000 -c 1000 /datasets?page_size=20"
run_batch S3.1 seq "/datasets?page_size=20 №$i|1000|S3.1-run$i.txt|-n 1000 -c 1000 '$TARGET_URL/datasets?page_size=20'"
done
for i in 1 2 3; do
log "S3.2 run $i: hey -n 1000 -c 1000 /landscape/results"
run_batch S3.2 seq "/landscape/results №$i|1000|S3.2-run$i.txt|-n 1000 -c 1000 '$TARGET_URL/landscape/results'"
done ;;
S4)
specs=()
for pair in "/datasets?page_size=20:300" "/datasets/$VWK_ID:150" "/datasets/$VWK_ID.geojson:100" "/datasets/$VWK_ID/observations:100" \
"/categories:100" "/dataset-groups/$GROUP_ID/latest:100" "/landscape/results:100" "/readyz:50"; do
p="${pair%:*}"; c="${pair##*:}"
specs+=("$(label_of "$p")|$c|S4-$(fname "$p").txt|-z $S4_DURATION -q 1 -c $c '$TARGET_URL$p'")
done
log "S4: 1000 clients at 1 req/s each for $S4_DURATION (8 hey processes)"
stats_start; run_batch S4 par "${specs[@]}"; stats_stop ;;
S5)
if [ "$LOCAL" != 1 ] && [ "$ALLOW_WRITES" != 1 ]; then log "S5 skipped: uploads need a local stand or ALLOW_WRITES=1"; continue; fi
TMP5="$(mktemp -d "${TMPDIR:-/tmp}/gis-load.XXXXXX")"
"$PYTHON" "$HERE/lib/helpers.py" ndvi "$TMP5/ndvi_s5.tif"
specs=()
for p in $EP_PATHS; do specs+=("$(label_of "$p")|10|S5-$(fname "$p").txt|-z $S5_DURATION -c 10 '$TARGET_URL$p'"); done
log "S5: reads 10 x 10 clients for $S5_DURATION + 20 parallel uploads"
stats_start
run_batch S5 par "${specs[@]}" &
READ_PID=$!
sleep 5
for i in $(seq 1 20); do
if [ "$i" -le 10 ]; then f="$HERE/data/plots_nko.geojson"; t=vector; else f="$TMP5/ndvi_s5.tif"; t=raster; fi
( curl -sS -o "$TMP5/up$i.json" -w '%{http_code} %{time_total}\n' -X POST "$BASE_URL/datasets" \
-F "file=@$f" -F "file_type=$t" -F "category_id=$CAT_ID" -F "name=S5 upload $i" > "$TMP5/up$i.code"; date +%s > "$TMP5/up$i.t" ) &
done
for i in $(seq 1 20); do while [ ! -s "$TMP5/up$i.t" ]; do sleep 0.2; done; done
wait "$READ_PID" || true
stats_stop
# processing: every upload must reach ready
codes=""; ready=0; maxproc=0; mins=999999; maxs=0
for i in $(seq 1 20); do
read -r code tt < "$TMP5/up$i.code"; codes="$codes $code"
ms="$(awk -v t="$tt" 'BEGIN{printf "%.1f", t*1000}')"
mins="$(awk -v a="$mins" -v b="$ms" 'BEGIN{print (b<a)?b:a}')"; maxs="$(awk -v a="$maxs" -v b="$ms" 'BEGIN{print (b>a)?b:a}')"
id="$(jq -r '.id // empty' "$TMP5/up$i.json")"; [ -n "$id" ] || continue
for _ in $(seq 1 300); do
st="$(curl -fsS "$BASE_URL/datasets/$id" | jq -r .status)"
[ "$st" = ready ] || [ "$st" = failed ] && break; sleep 1
done
if [ "$st" = ready ]; then
ready=$((ready + 1))
# server-side processing time: updated_at (ready) - created_at (upload)
curl -fsS "$BASE_URL/datasets/$id" -o "$TMP5/ds$i.json"
proc="$(python3 "$HERE/lib/helpers.py" elapsed "$TMP5/ds$i.json" 2>/dev/null || echo 0)"
maxproc="$(awk -v a="$maxproc" -v b="$proc" 'BEGIN{print (b>a)?b:a}')"
fi
curl -sS -o /dev/null -X DELETE "$BASE_URL/datasets/$id"
done
cs="$(echo $codes | tr ' ' '\n' | sort | uniq -c | awk '{printf "%s%s: %s", (NR>1?", ":""), $2, $1}')"
extra_set ".uploads = {count: 20, codes: \"$cs\", min_ms: $mins, max_ms: $maxs, ready: $ready, max_processing_s: $maxproc}"
log "S5 uploads: $cs; ready $ready/20 (max processing ${maxproc}s = updated_at - created_at); uploads deleted"
rm -rf "$TMP5" ;;
S6)
baseline S6
rz="$(curl -sS -o "$OUT/readyz_after.json" -w '%{http_code}' "$BASE_URL/readyz")"
extra_set ".readyz_after = $rz | .readyz_body = $(cat "$OUT/readyz_after.json")"
log "S6: /readyz $rz $(cat "$OUT/readyz_after.json")" ;;
*) echo "unknown scenario $S" >&2; exit 2 ;;
esac
done
if [ -s "$OUT/stats.csv" ]; then # peak CPU % / memory MiB per container
peaks="$(awk -F, '{gsub("%","",$2); split($3,m," / "); v=m[1]; mult=1;
if (v ~ /GiB$/) {mult=1024} else if (v ~ /KiB$/) {mult=1/1024} else if (v ~ /kB$/) {mult=1/1024} else if (v ~ /MB$/) {mult=1};
gsub(/[A-Za-z]+$/,"",v); mem=v*mult; if ($2>cpu[$1]) cpu[$1]=$2; if (mem>mm[$1]) mm[$1]=mem }
END { printf "{"; n=0; for (k in cpu) { printf "%s\"%s\":{\"cpu\":%s,\"mem_mib\":%.1f}", (n++?",":""), k, cpu[k], mm[k] } printf "}" }' "$OUT/stats.csv")"
extra_set ".stats_peak = $peaks"
fi
python3 "$HERE/lib/load_analyze.py" "$OUT"

111
test/acceptance/lp_gen.py Executable file
View File

@@ -0,0 +1,111 @@
#!/usr/bin/env python3
"""Generates the synthetic input rasters of the Landscape Program (ЛП) test.
Writes 13 grid-aligned GeoTIFFs, one per input slot (file name = slot code),
plus one deliberately misaligned raster for the negative test (L13):
<out>/lp-z-biodiversity.tif ... <out>/lp-p-tourism.tif (13 files)
<out>/misaligned-lp-u-veg.tif (half size, 2x pixel)
Every input raster: int16, EPSG:32642 (UTM 42N), 100 m pixels, default
3000 x 2000 pixels, classes 1..5 from smooth trigonometric patterns (a
different pattern per slot, so ties and every class combination occur), and a
NoData (-9999) frame 40 pixels wide. Extra edge cases exercise the pixel rules:
* lp-u-crop: a 200 x 200 NoData block — U is computed from water, soil and
veg only (the crop/veg union rule of u_integrated / u_max);
* lp-u-veg: a 100 x 100 NoData block overlapping half of the crop block —
where both crop and veg are missing, U must be NoData;
* lp-p-snow: a 120 x 120 block of 0 and a 60 x 60 block of 6 — values that
are not NoData but outside 1..5, hence invalid (the "valid iff 1 <= v <= 5"
rule).
Output is deterministic (no random numbers), so the same command always
produces byte-identical pixel values.
python lp_gen.py --out /tmp/lp-inputs [--width 3000 --height 2000]
"""
import argparse
import os
import sys
import numpy as np
import rasterio
from rasterio.transform import from_origin
SLOTS = [
"lp-z-biodiversity", "lp-z-lcr", "lp-z-soil-protection", "lp-z-wr",
"lp-u-surf-water", "lp-u-soil", "lp-u-crop", "lp-u-veg",
"lp-p-irrigation", "lp-p-snow", "lp-p-crop", "lp-p-pasture", "lp-p-tourism",
]
NODATA = -9999
CRS = "EPSG:32642"
ORIGIN_X, ORIGIN_Y = 300000.0, 6100000.0 # UTM 42N, North Kazakhstan region
PIXEL = 100.0
FRAME = 40
def pattern(k, width, height):
"""Smooth class field 1..5 for slot index k."""
y, x = np.mgrid[0:height, 0:width].astype(np.float64)
fx = 2 * np.pi * (1.0 + 0.37 * k) / width
fy = 2 * np.pi * (0.8 + 0.29 * ((k * 5) % 7)) / height
v = (np.sin(fx * x + 0.9 * k) * np.cos(fy * y + 0.4 * k)
+ 0.5 * np.sin(fx * 2.3 * x + fy * 1.7 * y + k))
v = (v + 1.5) / 3.0 # ~0..1
cls = np.floor(np.clip(v, 0, 0.999999) * 5).astype(np.int16) + 1
return cls
def block(arr, r0, c0, h, w, value):
arr[r0:r0 + h, c0:c0 + w] = value
def write(path, arr, transform):
profile = dict(driver="GTiff", width=arr.shape[1], height=arr.shape[0], count=1,
dtype="int16", crs=CRS, transform=transform, nodata=NODATA,
tiled=True, blockxsize=256, blockysize=256, compress="DEFLATE")
with rasterio.open(path, "w", **profile) as dst:
dst.write(arr, 1)
def main():
ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
ap.add_argument("--out", required=True, help="output directory")
ap.add_argument("--width", type=int, default=3000)
ap.add_argument("--height", type=int, default=2000)
args = ap.parse_args()
w, h = args.width, args.height
if w < 400 or h < 400:
sys.exit("width/height must be >= 400")
os.makedirs(args.out, exist_ok=True)
transform = from_origin(ORIGIN_X, ORIGIN_Y, PIXEL, PIXEL)
for k, slot in enumerate(SLOTS):
arr = pattern(k, w, h)
arr[:FRAME, :] = NODATA
arr[-FRAME:, :] = NODATA
arr[:, :FRAME] = NODATA
arr[:, -FRAME:] = NODATA
cr, cc = h // 2 - 100, w // 2 - 100 # centre block anchor
if slot == "lp-u-crop":
block(arr, cr, cc, 200, 200, NODATA)
if slot == "lp-u-veg":
block(arr, cr + 50, cc + 150, 100, 100, NODATA)
if slot == "lp-p-snow":
block(arr, 300, 300, 120, 120, 0)
block(arr, 300, 500, 60, 60, 6)
write(os.path.join(args.out, slot + ".tif"), arr, transform)
print("wrote", slot + ".tif")
# Misaligned raster for L13: half the size, double pixel => same footprint,
# different grid.
mis = pattern(7, w // 2, h // 2)
write(os.path.join(args.out, "misaligned-lp-u-veg.tif"), mis,
from_origin(ORIGIN_X, ORIGIN_Y, PIXEL * 2, PIXEL * 2))
print("wrote misaligned-lp-u-veg.tif (%dx%d, %d m)" % (w // 2, h // 2, int(PIXEL * 2)))
return 0
if __name__ == "__main__":
sys.exit(main())

72
test/acceptance/lp_run.sh Executable file
View File

@@ -0,0 +1,72 @@
#!/usr/bin/env bash
# Landscape Program (ЛП) end-to-end check on the acceptance stand:
# 1. lp_gen.py — 13 synthetic input rasters (default 3000 x 2000)
# 2. upload — POST /landscape/slots/{code} for the 13 slots
# 3. run — POST /landscape/runs {"from_stage":0}, wait for it
# 4. lp_verify.py — pixel-by-pixel check of all 15 outputs (mode a)
#
# test/acceptance/lp_run.sh # BASE_URL defaults to http://localhost:18080
# LP_WIDTH=6000 LP_HEIGHT=4000 test/acceptance/lp_run.sh
#
# The uploaded inputs and the run's outputs are left in place so the run can be
# re-verified later (python lp_verify.py --run <id>); they are ordinary datasets
# of the "landscape" category. WRITES to the stand: never point it at production.
set -euo pipefail
HERE="$(cd "$(dirname "$0")" && pwd)"
BASE_URL="${BASE_URL:-http://localhost:18080}"
BASE_URL="${BASE_URL%/}"
PYTHON="${PYTHON:-python3}"
WAIT_TIMEOUT="${WAIT_TIMEOUT:-600}"
LP_WIDTH="${LP_WIDTH:-3000}"
LP_HEIGHT="${LP_HEIGHT:-2000}"
case "$BASE_URL" in
*dssgis.dwh.kz*) echo "refusing to write to $BASE_URL (production); use lp_verify.py for read-only checks" >&2; exit 2;;
esac
TMP="$(mktemp -d "${TMPDIR:-/tmp}/gis-lp.XXXXXX")"
trap 'rm -rf "$TMP"' EXIT
STAMP="$(date +%Y%m%d-%H%M%S)"
SLOTS="lp-z-biodiversity lp-z-lcr lp-z-soil-protection lp-z-wr lp-u-surf-water lp-u-soil lp-u-crop lp-u-veg lp-p-irrigation lp-p-snow lp-p-crop lp-p-pasture lp-p-tourism"
echo "== 1. generate inputs (${LP_WIDTH}x${LP_HEIGHT})"
"$PYTHON" "$HERE/lp_gen.py" --out "$TMP/in" --width "$LP_WIDTH" --height "$LP_HEIGHT"
echo "== 2. upload into 13 slots ($BASE_URL)"
ids=""
for s in $SLOTS; do
out="$(curl -sS -w ' %{http_code}' -X POST "$BASE_URL/landscape/slots/$s" -F "file=@$TMP/in/$s.tif" \
-F "name=ЛП приёмка $s $STAMP" -F start_date=2025-01-01 -F end_date=2025-12-31)"
code="${out##* }"; body="${out% *}"
[ "$code" = 201 ] || { echo "upload $s failed: HTTP $code $body" >&2; exit 1; }
id="$(jq -r .id <<< "$body")"; ids="$ids $id"
printf ' %-22s %s HTTP %s\n' "$s" "$id" "$code"
done
for id in $ids; do
deadline=$(( $(date +%s) + WAIT_TIMEOUT ))
until [ "$(curl -sS "$BASE_URL/datasets/$id" | jq -r .status)" = ready ]; do
[ "$(date +%s)" -lt "$deadline" ] || { echo "dataset $id not ready" >&2; exit 1; }
sleep 1
done
done
echo " all 13 inputs ready"
echo "== 3. run {\"from_stage\":0}"
run="$(curl -sS -X POST "$BASE_URL/landscape/runs" -H 'Content-Type: application/json' -d '{"from_stage":0}')"
RUN="$(jq -r .id <<< "$run")"
[ -n "$RUN" ] && [ "$RUN" != null ] || { echo "run not created: $run" >&2; exit 1; }
echo " run $RUN queued"
deadline=$(( $(date +%s) + WAIT_TIMEOUT ))
while :; do
st="$(curl -sS "$BASE_URL/landscape/runs/$RUN" | jq -r .status)"
case "$st" in succeeded|failed) break;; esac
[ "$(date +%s)" -lt "$deadline" ] || { echo "run $RUN still $st" >&2; exit 1; }
sleep 1
done
curl -sS "$BASE_URL/landscape/runs/$RUN" > "$TMP/run.json"
echo " run $RUN $st: $(jq -r '[.nodes[] | "\(.node_code):\(.status)"] | sort | join(" ")' "$TMP/run.json")"
"$PYTHON" "$HERE/lib/helpers.py" durations "$TMP/run.json" | sed 's/^/ /'
[ "$st" = succeeded ] || { echo "run failed: $(jq -r .error "$TMP/run.json")" >&2; exit 1; }
echo "== 4. lp_verify.py --base-url $BASE_URL --run $RUN"
"$PYTHON" "$HERE/lp_verify.py" --base-url "$BASE_URL" --run "$RUN" --cache "$TMP/cache"

436
test/acceptance/lp_verify.py Executable file
View File

@@ -0,0 +1,436 @@
#!/usr/bin/env python3
"""Independent pixel-by-pixel check of a Landscape Program (ЛП) run.
Re-implements every node of the ЛП graph in numpy — 00 z_integrated,
01 u_integrated, 02 z_max, 03 u_max, 10 e_ecological_constraint,
20 c_conflicts, 21 p_max, 30 func_zoning (V1–V3 with the run's thresholds and
tie_priority), 31 c_max — from the 13 leaf input rasters the run actually used,
and compares the result with all 15 output rasters the run published.
The program is READ-ONLY: it issues only HTTP GET requests, so it can be
pointed at any stand, production included:
python lp_verify.py --base-url http://localhost:18080 --run <run-id>
python lp_verify.py --base-url https://dssgis.dwh.kz --run <run-id>
Data flow:
GET /landscape/runs/{id} run params, per-node inputs and outputs
GET /datasets/{id}.cog every input (13) and output (15) raster
(or /datasets/{id}/download for inputs with --inputs-from download)
The leaf inputs are taken from the run's node `inputs` (group = slot code), the
outputs from the node `outputs`. Expected values are computed through the whole
chain from the leaf inputs only (E from the re-computed Z/U, C and V from the
re-computed E, ...), so a wrong intermediate output cannot hide a later error.
The run's own chain references (node 10 reading node 00's output, ...) are
checked for consistency as well.
Large rasters (production: 22219 x 14977) are processed in windows of
--block-rows x --block-cols pixels (default 512 x 4096, aligned with the
512-pixel COG tiles), so memory stays bounded (~0.5 GB).
Pixel rules (source of truth: internal/landscape/kernels.go):
valid(v) 1 <= v <= 5 (NoData and anything else is invalid); class
values are used as int16(v) (truncation)
classify(x) float32 x: < 1 -> NoData, <= 1.8 -> 1, <= 2.6 -> 2,
<= 3.4 -> 3, <= 4.2 -> 4, <= 5.0 -> 5, else NoData
z_integrated all Z valid: classify(0.25*Z1 + 0.25*Z2 + 0.25*Z3 + 0.25*Z4)
(float32, summed left to right)
u_integrated water & soil & (crop | veg): classify(num/den),
num = 0.25*water + 0.25*soil [+ 0.25*crop] [+ 0.25*veg],
den = 0.5 [+ 0.25] [+ 0.25] (float32)
z_max all Z valid: max; u_max: U domain as above, max of valid
e Z_int & U_int valid: max(Z, U)
c_conflicts P_j & E valid: min(P_j, E)
p_max, c_max max over valid inputs; NoData when none is valid
func_zoning E valid & any P_k valid; candidate k: P_k valid & rule(E, P_k)
V1: E<=2 & S>=v1_min_s_e12 | E==3 & S>=v1_min_s_e3
V2: E<=2 | E==3 & S>=v2_min_s_e3 | E==4 & S>=v2_min_s_e4
V3: E<=4
no candidate -> 1; else sector with max S (ties by
tie_priority) + 1
Exit code: 0 when every output matches pixel for pixel, 1 on any mismatch,
2 on a usage / data error.
"""
import argparse
import json
import os
import shutil
import sys
import tempfile
import time
import urllib.error
import urllib.request
import numpy as np
try:
import rasterio
from rasterio.windows import Window
except ImportError: # pragma: no cover
sys.exit("lp_verify.py needs numpy and rasterio: pip install numpy rasterio")
NODATA = np.int16(-9999)
SLOTS = [
"lp-z-biodiversity", "lp-z-lcr", "lp-z-soil-protection", "lp-z-wr",
"lp-u-surf-water", "lp-u-soil", "lp-u-crop", "lp-u-veg",
"lp-p-irrigation", "lp-p-snow", "lp-p-crop", "lp-p-pasture", "lp-p-tourism",
]
Z = SLOTS[0:4]
U = SLOTS[4:8] # water, soil, crop, veg
P = SLOTS[8:13] # sector order 1..5
C = ["lp-c-irrigation", "lp-c-snow", "lp-c-crop", "lp-c-pasture", "lp-c-tourism"]
OUTPUTS = [
"lp-z-integrated", "lp-u-integrated", "lp-z-max", "lp-u-max", "lp-e",
*C, "lp-p-max", "lp-v1-conservative", "lp-v2-balanced", "lp-v3-development", "lp-c-max",
]
OUTPUT_NODE = {
"lp-z-integrated": "00", "lp-u-integrated": "01", "lp-z-max": "02", "lp-u-max": "03", "lp-e": "10",
**{c: "20" for c in C}, "lp-p-max": "21",
"lp-v1-conservative": "30", "lp-v2-balanced": "30", "lp-v3-development": "30", "lp-c-max": "31",
}
DEFAULT_PARAMS = {"tie_priority": [1, 2, 3, 4, 5], "v1_min_s_e12": 3, "v1_min_s_e3": 4,
"v2_min_s_e3": 3, "v2_min_s_e4": 5}
F32 = np.float32
# ----------------------------------------------------------------- pixel rules
def valid(v):
return (v >= 1) & (v <= 5)
def cls(v, ok):
"""int16(v) for valid pixels (Go truncation), 0 elsewhere."""
return np.where(ok, np.trunc(np.where(ok, v, 0)), 0).astype(np.int16)
def classify(x):
x = x.astype(np.float32, copy=False)
return np.select(
[x < F32(1.0), x <= F32(1.8), x <= F32(2.6), x <= F32(3.4), x <= F32(4.2), x <= F32(5.0)],
[NODATA, 1, 2, 3, 4, 5], default=NODATA).astype(np.int16)
def quarter(v):
# float32(float32(0.25) * float32(v)), as in the Go kernels
return F32(0.25) * v.astype(np.float32)
def z_integrated(z):
ok = valid(z[0]) & valid(z[1]) & valid(z[2]) & valid(z[3])
score = ((quarter(z[0]) + quarter(z[1])) + quarter(z[2])) + quarter(z[3])
return np.where(ok, classify(score), NODATA).astype(np.int16)
def u_domain(u):
water, soil, crop, veg = u
crop_ok, veg_ok = valid(crop), valid(veg)
return valid(water) & valid(soil) & (crop_ok | veg_ok), crop_ok, veg_ok
def u_integrated(u):
water, soil, crop, veg = u
ok, crop_ok, veg_ok = u_domain(u)
num = quarter(water) + quarter(soil)
den = np.full(num.shape, F32(0.5), dtype=np.float32)
num = np.where(crop_ok, num + quarter(crop), num).astype(np.float32)
den = np.where(crop_ok, den + F32(0.25), den).astype(np.float32)
num = np.where(veg_ok, num + quarter(veg), num).astype(np.float32)
den = np.where(veg_ok, den + F32(0.25), den).astype(np.float32)
with np.errstate(invalid="ignore", divide="ignore"):
score = (num / den).astype(np.float32)
return np.where(ok, classify(score), NODATA).astype(np.int16)
def z_max(z):
ok = valid(z[0]) & valid(z[1]) & valid(z[2]) & valid(z[3])
m = np.maximum.reduce([cls(v, ok) for v in z])
return np.where(ok, m, NODATA).astype(np.int16)
def u_max(u):
water, soil, crop, veg = u
ok, crop_ok, veg_ok = u_domain(u)
m = np.maximum(cls(water, ok), cls(soil, ok))
m = np.where(crop_ok, np.maximum(m, cls(crop, crop_ok)), m)
m = np.where(veg_ok, np.maximum(m, cls(veg, veg_ok)), m)
return np.where(ok, m, NODATA).astype(np.int16)
def e_constraint(zi, ui):
ok = valid(zi) & valid(ui)
return np.where(ok, np.maximum(zi, ui), NODATA).astype(np.int16)
def c_conflict(p, e):
ok = valid(p) & valid(e)
return np.where(ok, np.minimum(cls(p, ok), e), NODATA).astype(np.int16)
def max_any_valid(layers):
oks = [valid(v) for v in layers]
m = np.maximum.reduce([cls(v, ok) for v, ok in zip(layers, oks)])
anyok = np.logical_or.reduce(oks)
return np.where(anyok, m, NODATA).astype(np.int16)
def func_zoning(e, p, params):
eok = valid(e)
ev = np.where(eok, e, 0).astype(np.int16)
base = [valid(v) for v in p]
s = [cls(v, ok) for v, ok in zip(p, base)]
any_base = np.logical_or.reduce(base)
t = params
rules = [
lambda e_, s_: ((e_ <= 2) & (s_ >= t["v1_min_s_e12"])) | ((e_ == 3) & (s_ >= t["v1_min_s_e3"])),
lambda e_, s_: (e_ <= 2) | ((e_ == 3) & (s_ >= t["v2_min_s_e3"])) | ((e_ == 4) & (s_ >= t["v2_min_s_e4"])),
lambda e_, s_: e_ <= 4,
]
outs = []
for rule in rules:
cand = [b & rule(ev, sk) for b, sk in zip(base, s)]
any_cand = np.logical_or.reduce(cand)
max_s = np.maximum.reduce([np.where(ck, sk, 0) for ck, sk in zip(cand, s)]).astype(np.int16)
sector = np.zeros(ev.shape, dtype=np.int16)
done = np.zeros(ev.shape, dtype=bool)
for code in t["tie_priority"]:
k = code - 1
hit = cand[k] & (s[k] == max_s) & ~done
sector[hit] = code
done |= hit
v = np.where(any_cand, sector + 1, 1).astype(np.int16)
outs.append(np.where(eok & any_base, v, NODATA).astype(np.int16))
return outs
def expected_block(inp, params):
"""inp: dict slot -> float64 array. Returns dict output group -> int16."""
z = [inp[g] for g in Z]
u = [inp[g] for g in U]
p = [inp[g] for g in P]
out = {
"lp-z-integrated": z_integrated(z),
"lp-u-integrated": u_integrated(u),
"lp-z-max": z_max(z),
"lp-u-max": u_max(u),
}
e = e_constraint(out["lp-z-integrated"], out["lp-u-integrated"])
out["lp-e"] = e
for cg, pk in zip(C, p):
out[cg] = c_conflict(pk, e)
out["lp-p-max"] = max_any_valid(p)
v1, v2, v3 = func_zoning(e, p, params)
out["lp-v1-conservative"], out["lp-v2-balanced"], out["lp-v3-development"] = v1, v2, v3
out["lp-c-max"] = max_any_valid([out[c] for c in C])
return out
# ------------------------------------------------------------------- HTTP (GET)
def http_get(url, timeout=120, retries=3):
req = urllib.request.Request(url, method="GET", headers={"User-Agent": "lp_verify/1.0"})
last = None
for attempt in range(retries):
try:
return urllib.request.urlopen(req, timeout=timeout)
except urllib.error.HTTPError as e:
if e.code < 500:
raise
last = e
except (urllib.error.URLError, TimeoutError, ConnectionError) as e:
last = e
time.sleep(2 * (attempt + 1))
raise last
def get_json(url):
with http_get(url) as r:
return json.loads(r.read().decode("utf-8"))
def download(url, path, wait=0):
"""GET url into path (reused when already cached). A 409 means the COG is
still being produced (conversion is asynchronous): retry for up to `wait`
seconds."""
if os.path.exists(path) and os.path.getsize(path) > 0:
return "cached"
tmp = path + ".part"
deadline = time.time() + wait
while True:
try:
r = http_get(url, timeout=600)
break
except urllib.error.HTTPError as e:
if e.code != 409 or time.time() >= deadline:
raise
time.sleep(2)
with r, open(tmp, "wb") as f:
shutil.copyfileobj(r, f, length=4 << 20)
os.replace(tmp, path)
return "%.1f MB" % (os.path.getsize(path) / 1e6)
# ----------------------------------------------------------------------- main
def parse_points(s):
pts = []
for part in filter(None, (x.strip() for x in s.split(";"))):
c, r = part.split(",")
pts.append((int(c), int(r)))
return pts
def main():
ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
ap.add_argument("--base-url", default=os.environ.get("BASE_URL", "http://localhost:18080"))
ap.add_argument("--run", required=True, help="landscape run id")
ap.add_argument("--cache", help="directory for downloaded rasters (kept; existing files are reused)")
ap.add_argument("--inputs-from", choices=["cog", "download"], default="cog",
help="read inputs from their COG (default) or the original upload (what the node read)")
ap.add_argument("--block-rows", type=int, default=512)
ap.add_argument("--block-cols", type=int, default=4096)
ap.add_argument("--wait", type=int, default=180,
help="seconds to keep retrying a COG that is not ready yet (HTTP 409), default 180")
ap.add_argument("--points", help='spot-check pixels "col,row;col,row;..." and print their values')
args = ap.parse_args()
base = args.base_url.rstrip("/")
run = get_json("%s/landscape/runs/%s" % (base, args.run))
params = dict(DEFAULT_PARAMS)
raw_params = run.get("params") or {}
for k in DEFAULT_PARAMS:
if raw_params.get(k):
params[k] = raw_params[k]
print("run %s status=%s from_stage=%s" % (run["id"], run["status"], run.get("from_stage")))
print("params %s" % json.dumps(params))
nodes = sorted(run.get("nodes") or [], key=lambda n: n["node_code"])
print("nodes %s" % " ".join("%s:%s" % (n["node_code"], n["status"]) for n in nodes))
if run["status"] != "succeeded":
print("ERROR: run is not succeeded; nothing to verify")
return 2
errors = []
slot_ids, out_ids = {}, {}
for n in nodes:
for ref in n.get("inputs") or []:
g, did = ref["group"], ref["dataset_id"]
if g in SLOTS:
if slot_ids.setdefault(g, did) != did:
errors.append("slot %s resolved to two datasets (%s, %s)" % (g, slot_ids[g], did))
for ref in n.get("outputs") or []:
out_ids[ref["group"]] = ref["dataset_id"]
if OUTPUT_NODE.get(ref["group"]) != n["node_code"]:
errors.append("output %s reported by node %s" % (ref["group"], n["node_code"]))
# chain consistency: an intermediate input must be what this run produced
for n in nodes:
for ref in n.get("inputs") or []:
g = ref["group"]
if g in OUTPUTS and out_ids.get(g) != ref["dataset_id"]:
errors.append("node %s read %s=%s but the run's output is %s" % (n["node_code"], g, ref["dataset_id"], out_ids.get(g)))
missing = [g for g in SLOTS if g not in slot_ids] + [g for g in OUTPUTS if g not in out_ids]
if missing:
errors.append("run does not reference: %s" % ", ".join(missing))
if errors:
for e in errors:
print("ERROR:", e)
return 2
cache = args.cache or tempfile.mkdtemp(prefix="lp_verify_")
os.makedirs(cache, exist_ok=True)
files = {}
t0 = time.time()
for g in SLOTS + OUTPUTS:
did = slot_ids.get(g) or out_ids[g]
if g in SLOTS and args.inputs_from == "download":
url, path = "%s/datasets/%s/download" % (base, did), os.path.join(cache, did + ".orig.tif")
else:
url, path = "%s/datasets/%s.cog" % (base, did), os.path.join(cache, did + ".cog.tif")
try:
info = download(url, path, args.wait)
except urllib.error.HTTPError as e:
print("ERROR: GET %s -> HTTP %d (output not ready or missing?)" % (url, e.code))
return 2
files[g] = path
print("fetch %-22s %s %s" % (g, did, info))
print("downloaded in %.1f s to %s" % (time.time() - t0, cache))
with rasterio.Env(GDAL_CACHEMAX=512):
ds = {g: rasterio.open(files[g]) for g in SLOTS + OUTPUTS}
ref = ds[SLOTS[0]]
for g, d in ds.items():
if (d.width, d.height) != (ref.width, ref.height) or not d.transform.almost_equals(ref.transform) or d.crs != ref.crs:
errors.append("%s grid %dx%d %s differs from %s %dx%d %s" % (
g, d.width, d.height, d.crs, SLOTS[0], ref.width, ref.height, ref.crs))
for g in OUTPUTS:
if ds[g].dtypes[0] != "int16" or ds[g].nodata != -9999:
errors.append("output %s: dtype %s nodata %s (want int16, -9999)" % (g, ds[g].dtypes[0], ds[g].nodata))
if errors:
for e in errors:
print("ERROR:", e)
return 2
W, H = ref.width, ref.height
print("grid %dx%d %s pixel %.3f" % (W, H, ref.crs, ref.transform.a))
if args.points:
return spot_check(ds, parse_points(args.points), params)
mism = {g: 0 for g in OUTPUTS}
validn = {g: 0 for g in OUTPUTS}
first = {}
t0 = time.time()
for r0 in range(0, H, args.block_rows):
h = min(args.block_rows, H - r0)
for c0 in range(0, W, args.block_cols):
w = min(args.block_cols, W - c0)
win = Window(c0, r0, w, h)
inp = {g: ds[g].read(1, window=win, masked=False).astype(np.float64) for g in SLOTS}
exp = expected_block(inp, params)
for g in OUTPUTS:
act = ds[g].read(1, window=win, masked=False)
bad = exp[g] != act
nbad = int(bad.sum())
validn[g] += int((exp[g] != NODATA).sum())
if nbad:
mism[g] += nbad
if g not in first:
rr, cc = np.argwhere(bad)[0]
first[g] = (c0 + int(cc), r0 + int(rr), int(exp[g][rr, cc]), int(act[rr, cc]))
print("compared in %.1f s" % (time.time() - t0))
for d in ds.values():
d.close()
print()
print("%-20s %-4s %-38s %12s %12s %10s" % ("output", "node", "dataset_id", "pixels", "valid(exp)", "mismatch"))
total = 0
for g in OUTPUTS:
total += mism[g]
line = "%-20s %-4s %-38s %12d %12d %10d" % (g, OUTPUT_NODE[g], out_ids[g], W * H, validn[g], mism[g])
if g in first:
line += " first at col=%d row=%d expected=%d actual=%d" % first[g]
print(line)
print()
print("TOTAL mismatching pixels: %d -> %s" % (total, "PASS" if total == 0 else "FAIL"))
return 0 if total == 0 else 1
def spot_check(ds, points, params):
bad = 0
shown = ["lp-z-integrated", "lp-u-integrated", "lp-e", *C, "lp-v1-conservative", "lp-v2-balanced", "lp-v3-development"]
for col, row in points:
win = Window(col, row, 1, 1)
inp = {g: ds[g].read(1, window=win, masked=False).astype(np.float64) for g in SLOTS}
exp = expected_block(inp, params)
vals = " ".join("%s=%d" % (g.replace("lp-", ""), int(inp[g][0, 0])) for g in SLOTS)
print("point (%d,%d) inputs: %s" % (col, row, vals))
for g in shown:
act = int(ds[g].read(1, window=win, masked=False)[0, 0])
e = int(exp[g][0, 0])
ok = act == e
bad += 0 if ok else 1
print(" %-20s expected=%6d actual=%6d %s" % (g, e, act, "OK" if ok else "MISMATCH"))
print("SPOT CHECK: %d points, %d mismatches -> %s" % (len(points), bad, "PASS" if bad == 0 else "FAIL"))
return 0 if bad == 0 else 1
if __name__ == "__main__":
sys.exit(main())

View File

@@ -0,0 +1,2 @@
numpy>=1.24
rasterio>=1.3

131
test/acceptance/scan.sh Executable file
View File

@@ -0,0 +1,131 @@
#!/usr/bin/env bash
# Security re-scan (Приложение 4 «Отчет об устранении уязвимостей»):
# 1. govulncheck, source mode — vulnerabilities reachable from the code
# 2. govulncheck -mode=binary — the binary shipped in the image
# 3. gosec — static analysis (HIGH severity counted)
# 4. trivy fs — dependencies, secrets, IaC misconfig
# 5. trivy image — the application image (built here)
# 6. golangci-lint (if installed) — code quality, informational only
#
# test/acceptance/scan.sh
#
# Environment: SCAN_IMAGE (default gis-scan:local, built from
# build/package/Dockerfile), SKIP_IMAGE=1 (no docker: skip 2 via image and 5;
# the binary for 2 is then built locally with cgo + GDAL).
# govulncheck has no severity scale: every vulnerability reachable from the
# code (source) or present in the binary symbol table is counted as High.
# Prints High/Critical per tool; exit code 1 if any High/Critical is found.
# Raw reports: test/acceptance/results/scan-<timestamp>/.
set -euo pipefail
HERE="$(cd "$(dirname "$0")" && pwd)"
ROOT="$(cd "$HERE/../.." && pwd)"
STAMP="$(date +%Y%m%d-%H%M%S)"
OUT="$HERE/results/scan-$STAMP"; mkdir -p "$OUT"
SCAN_IMAGE="${SCAN_IMAGE:-gis-scan:local}"
SKIP_IMAGE="${SKIP_IMAGE:-0}"
GOBIN_DIR="$(go env GOPATH 2>/dev/null)/bin"
PATH="$PATH:$GOBIN_DIR"
cd "$ROOT"
ROWS=""; FAIL=0
row() { # tool target high critical note
ROWS="${ROWS}$(printf '%-26s | %-40s | %5s | %8s | %s' "$1" "$2" "$3" "$4" "$5")"$'\n'
case "$3$4" in *[!0-9]*) ;; *) [ "$3" -eq 0 ] && [ "$4" -eq 0 ] || FAIL=1;; esac
}
have() { command -v "$1" >/dev/null 2>&1; }
log() { echo "[$(date +%H:%M:%S)] $*"; }
log "commit $(git rev-parse --short HEAD 2>/dev/null || echo '?'), $(go version)"
# --- govulncheck (source) -------------------------------------------------
gv_count() { # json stream -> number of distinct OSV ids with a call trace (reachable)
jq -rs '[.[] | select(.finding != null) | .finding | select((.trace // []) | length > 0)
| select(.trace[0].function != null) | .osv] | unique | length' "$1"
}
gv_ids() { jq -rs '[.[] | select(.finding != null) | .finding | select(.trace[0].function != null) | .osv] | unique | join(", ")' "$1"; }
if have govulncheck; then
log "govulncheck ./... (source)"
govulncheck -format json ./... > "$OUT/govulncheck-source.json" 2> "$OUT/govulncheck-source.err" || true
n="$(gv_count "$OUT/govulncheck-source.json")"
mods="$(jq -rs '[.[] | select(.finding != null) | .finding.osv] | unique | length' "$OUT/govulncheck-source.json")"
row "govulncheck (source)" "./..." "$n" 0 "reachable: ${n}$( [ "$n" -gt 0 ] && echo " ($(gv_ids "$OUT/govulncheck-source.json"))"); in dependencies but not called: $((mods - n))"
else row "govulncheck (source)" "./..." "n/a" "n/a" "not installed (go install golang.org/x/vuln/cmd/govulncheck@latest)"; fi
# --- application image + binary -------------------------------------------
BIN=""
if [ "$SKIP_IMAGE" != 1 ] && have docker; then
log "docker build -t $SCAN_IMAGE -f build/package/Dockerfile ."
docker build -q -t "$SCAN_IMAGE" -f build/package/Dockerfile . > "$OUT/docker-build.txt"
cid="$(docker create "$SCAN_IMAGE")"
docker cp "$cid:/usr/local/bin/gis" "$OUT/gis" >/dev/null; docker rm "$cid" >/dev/null
BIN="$OUT/gis"; BIN_SRC="image $SCAN_IMAGE:/usr/local/bin/gis"
else
log "go build (cgo + GDAL) for binary-mode scan"
CGO_ENABLED=1 go build -trimpath -o "$OUT/gis" ./cmd/gis; BIN="$OUT/gis"; BIN_SRC="local build"
fi
# --- govulncheck (binary) -------------------------------------------------
if have govulncheck; then
log "govulncheck -mode=binary ($BIN_SRC)"
govulncheck -mode=binary -format json "$BIN" > "$OUT/govulncheck-binary.json" 2> "$OUT/govulncheck-binary.err" || true
n="$(gv_count "$OUT/govulncheck-binary.json")"
gov="$(go version "$BIN" 2>/dev/null | awk '{print $2}')"
row "govulncheck (binary)" "$BIN_SRC" "$n" 0 "built with $gov; vulnerable symbols present: $n$( [ "$n" -gt 0 ] && echo " ($(gv_ids "$OUT/govulncheck-binary.json"))")"
fi
# --- gosec -----------------------------------------------------------------
if have gosec; then
log "gosec ./..."
gosec -quiet -fmt json -out "$OUT/gosec.json" ./... 2> "$OUT/gosec.err" || true
hi="$(jq '[.Issues[]? | select(.severity=="HIGH")] | length' "$OUT/gosec.json")"
me="$(jq '[.Issues[]? | select(.severity=="MEDIUM")] | length' "$OUT/gosec.json")"
lo="$(jq '[.Issues[]? | select(.severity=="LOW")] | length' "$OUT/gosec.json")"
ns="$(jq '.Stats.nosec // 0' "$OUT/gosec.json")"
row "gosec" "./... ($(jq '.Stats.files' "$OUT/gosec.json") files)" "$hi" 0 "MEDIUM $me, LOW $lo; #nosec-annotated $ns"
else row "gosec" "./..." "n/a" "n/a" "not installed"; fi
# --- trivy ------------------------------------------------------------------
trivy_counts() { # json -> "high critical"
jq -r '[.Results[]? | (.Vulnerabilities[]?, .Secrets[]?, .Misconfigurations[]?) | .Severity] as $s
| "\([$s[] | select(.=="HIGH")] | length) \([$s[] | select(.=="CRITICAL")] | length)"' "$1"
}
if have trivy; then
log "trivy fs (vuln, secret, misconfig)"
trivy fs --quiet --scanners vuln,secret,misconfig --format json --output "$OUT/trivy-fs.json" \
--skip-dirs test/acceptance/results --skip-dirs graft . 2> "$OUT/trivy-fs.err" || true
read -r h c <<< "$(trivy_counts "$OUT/trivy-fs.json")"
all="$(jq '[.Results[]? | (.Vulnerabilities[]?, .Secrets[]?, .Misconfigurations[]?)] | length' "$OUT/trivy-fs.json")"
row "trivy fs" "repository (go.mod, Dockerfile, compose)" "$h" "$c" "all severities: $all"
if [ -n "$BIN" ] && [ "$SKIP_IMAGE" != 1 ] && have docker; then
log "trivy image $SCAN_IMAGE"
trivy image --quiet --format json --output "$OUT/trivy-image.json" "$SCAN_IMAGE" 2> "$OUT/trivy-image.err" || true
read -r h c <<< "$(trivy_counts "$OUT/trivy-image.json")"
all="$(jq '[.Results[]? | .Vulnerabilities[]?] | length' "$OUT/trivy-image.json")"
os="$(jq -r '.Metadata.OS | "\(.Family) \(.Name)"' "$OUT/trivy-image.json")"
row "trivy image" "$SCAN_IMAGE ($os)" "$h" "$c" "all severities: $all"
fi
else row "trivy" "-" "n/a" "n/a" "not installed"; fi
# --- golangci-lint (informational) ------------------------------------------
if have golangci-lint; then
log "golangci-lint run ./..."
note=""
golangci-lint run --output.json.path "$OUT/golangci-lint.json" ./... > "$OUT/golangci-lint.txt" 2>&1 || true
if grep -q "unsupported version of the configuration" "$OUT/golangci-lint.txt"; then
# .golangci.yml is in the v1 format; a v2 binary refuses it — use the default linters
note=" (.golangci.yml is v1 format, $(golangci-lint version 2>/dev/null | awk '{print $4}') ran with --no-config)"
golangci-lint run --no-config --output.json.path "$OUT/golangci-lint.json" ./... > "$OUT/golangci-lint.txt" 2>&1 || true
fi
n="$(jq '[.Issues[]?] | length' "$OUT/golangci-lint.json" 2>/dev/null || echo '?')"
row "golangci-lint" "./..." "-" "-" "issues: $n$note; no severity scale, informational"
fi
echo
printf '%-26s | %-40s | %5s | %8s | %s\n' "Tool" "Target" "High" "Critical" "Notes"
printf '%s\n' "---------------------------+------------------------------------------+-------+----------+------------------------------"
printf '%s' "$ROWS"
echo
if [ "$FAIL" = 0 ]; then echo "RESULT: 0 High / 0 Critical -> PASS"; else echo "RESULT: High/Critical findings present -> FAIL"; fi
echo "raw reports: $OUT"
exit "$FAIL"

101
test/acceptance/seed.sh Executable file
View File

@@ -0,0 +1,101 @@
#!/usr/bin/env bash
# Seeds the acceptance stand with datasets for the load test (Приложение 3,
# п. 3.3), so the read endpoints work on a realistic catalogue.
#
# test/acceptance/seed.sh create (idempotent) category "acc-load":
# SEED_VECTOR (25) vector GeoJSON,
# SEED_KATO (25) vector_with_kato GeoJSON with
# KATO mapping (14 districts x 3 years each),
# SEED_RASTER (50) GeoTIFF rasters; the
# vector_with_kato datasets form the group
# "acc-load-population"
# test/acceptance/seed.sh clean delete everything seed.sh created
#
# Writes the ids load.sh needs to test/acceptance/results/seed.env.
# WRITES to the stand: never point it at production.
set -euo pipefail
HERE="$(cd "$(dirname "$0")" && pwd)"
BASE_URL="${BASE_URL:-http://localhost:18080}"
BASE_URL="${BASE_URL%/}"
PYTHON="${PYTHON:-python3}"
SEED_VECTOR="${SEED_VECTOR:-25}"
SEED_KATO="${SEED_KATO:-25}"
SEED_RASTER="${SEED_RASTER:-50}"
CAT_CODE=acc-load
GRP_CODE=acc-load-population
ENV_OUT="$HERE/results/seed.env"
mkdir -p "$HERE/results"
case "$BASE_URL" in
*dssgis.dwh.kz*) echo "refusing to write to $BASE_URL (production)" >&2; exit 2;;
esac
J=(-H 'Content-Type: application/json')
cat_id() { curl -fsS "$BASE_URL/categories" | jq -r --arg c "$CAT_CODE" '.[]|select(.code==$c)|.id' | head -n1; }
grp_id() { curl -fsS "$BASE_URL/dataset-groups?code=$GRP_CODE" | jq -r '.data[0].id // empty'; }
ids_in_category() { # all dataset ids of a category (paginated)
local cat="$1" page=1 out
while :; do
out="$(curl -fsS "$BASE_URL/datasets?category_id=$cat&page_size=100&page=$page")"
jq -r '.data[].id' <<< "$out"
[ "$page" -lt "$(jq -r .total_pages <<< "$out")" ] || break
page=$((page + 1))
done
}
if [ "${1:-}" = clean ]; then
CAT="$(cat_id || true)"; GRP="$(grp_id || true)"
if [ -n "$CAT" ]; then
n=0; for id in $(ids_in_category "$CAT"); do curl -sS -o /dev/null -X DELETE "$BASE_URL/datasets/$id"; n=$((n + 1)); done
echo "deleted $n datasets"
fi
[ -n "$GRP" ] && curl -sS -o /dev/null -w "group delete %{http_code}\n" -X DELETE "$BASE_URL/dataset-groups/$GRP"
[ -n "$CAT" ] && curl -sS -o /dev/null -w "category delete %{http_code}\n" -X DELETE "$BASE_URL/categories/$CAT"
rm -f "$ENV_OUT"
exit 0
fi
CAT="$(cat_id || true)"
if [ -z "$CAT" ]; then
CAT="$(curl -fsS -X POST "$BASE_URL/categories" "${J[@]}" -d "{\"code\":\"$CAT_CODE\",\"name\":\"Нагрузочное тестирование\",\"description\":\"Данные для test/acceptance/load.sh\"}" | jq -r .id)"
fi
GRP="$(grp_id || true)"
if [ -z "$GRP" ]; then
GRP="$(curl -fsS -X POST "$BASE_URL/dataset-groups" "${J[@]}" -d "{\"code\":\"$GRP_CODE\",\"name\":\"Население СКО (нагрузочный тест)\"}" | jq -r .id)"
fi
want=$((SEED_VECTOR + SEED_KATO + SEED_RASTER))
have="$(curl -fsS "$BASE_URL/datasets?category_id=$CAT&status=ready&page_size=1" | jq -r .total)"
if [ "$have" -lt "$want" ]; then
echo "seeding $BASE_URL: $SEED_VECTOR vector, $SEED_KATO vector_with_kato, $SEED_RASTER raster (category $CAT_CODE)"
TMP="$(mktemp -d "${TMPDIR:-/tmp}/gis-seed.XXXXXX")"; trap 'rm -rf "$TMP"' EXIT
"$PYTHON" "$HERE/lib/helpers.py" ndvi "$TMP/ndvi_seed.tif"
upload() { curl -fsS -X POST "$BASE_URL/datasets" "$@" | jq -r .id; }
for i in $(seq 1 "$SEED_VECTOR"); do
upload -F "file=@$HERE/data/plots_nko.geojson" -F file_type=vector -F "category_id=$CAT" -F "name=Участки $i" >/dev/null
done
kato_ids=""
for i in $(seq 1 "$SEED_KATO"); do
y=$((2000 + i))
kato_ids="$kato_ids $(upload -F "file=@$HERE/data/population_kato.geojson" -F file_type=vector_with_kato -F "category_id=$CAT" \
-F "name=Население $i" -F "dg_id=$GRP" -F "start_date=$y-01-01" -F "end_date=$y-12-31")"
done
for i in $(seq 1 "$SEED_RASTER"); do
upload -F "file=@$TMP/ndvi_seed.tif" -F file_type=raster -F "category_id=$CAT" -F "name=NDVI $i" >/dev/null
done
MAP='{"kato_column":"kato","year_columns":[{"column":"Y2022","start_date":"2022-01-01","end_date":"2022-12-31"},{"column":"Y2023","start_date":"2023-01-01","end_date":"2023-12-31"},{"column":"Y2024","start_date":"2024-01-01","end_date":"2024-12-31"}]}'
for id in $kato_ids; do
for _ in $(seq 1 120); do
[ "$(curl -fsS "$BASE_URL/datasets/$id" | jq -r .status)" = awaiting_mapping ] && break; sleep 1
done
curl -fsS -o /dev/null -X POST "$BASE_URL/datasets/$id/mapping" "${J[@]}" -d "$MAP"
done
for _ in $(seq 1 300); do
have="$(curl -fsS "$BASE_URL/datasets?category_id=$CAT&status=ready&page_size=1" | jq -r .total)"
[ "$have" -ge "$want" ] && break; sleep 1
done
fi
VWK="$(curl -fsS "$BASE_URL/datasets?category_id=$CAT&file_type=vector_with_kato&status=ready&page_size=1" | jq -r '.data[0].id')"
printf 'CAT_ID=%s\nGROUP_ID=%s\nVWK_ID=%s\n' "$CAT" "$GRP" "$VWK" > "$ENV_OUT"
echo "seed: category $CAT_CODE has $have ready datasets (wanted $want); ids in $ENV_OUT"
cat "$ENV_OUT"

54
test/acceptance/stand.sh Executable file
View File

@@ -0,0 +1,54 @@
#!/usr/bin/env bash
# Brings the isolated acceptance stand (docker compose project gis-acceptance)
# up or down.
#
# test/acceptance/stand.sh up build the image from this checkout, start
# all services, apply migrations, import the
# 14 districts, wait for /readyz
# test/acceptance/stand.sh down stop and remove the stand with its volumes
# test/acceptance/stand.sh ps|logs pass-through to docker compose
#
# Environment: MINIO_IMAGE, POSTGIS_PLATFORM, APP_IMAGE, HTTP_MAX_UPLOAD_BYTES,
# LP_MAX_WORKERS, LP_MAX_NODES (see compose.acceptance.yml).
set -euo pipefail
HERE="$(cd "$(dirname "$0")" && pwd)"
ROOT="$(cd "$HERE/../.." && pwd)"
PROJECT=gis-acceptance
compose() {
docker compose -p "$PROJECT" -f "$ROOT/deployments/docker-compose.yml" -f "$HERE/compose.acceptance.yml" "$@"
}
cmd="${1:-up}"
shift || true
case "$cmd" in
up)
echo "== build application image from $(git -C "$ROOT" rev-parse --short HEAD 2>/dev/null || echo 'this checkout')"
compose build app worker
echo "== start infrastructure ($PROJECT)"
compose up -d --wait postgres minio rabbitmq
echo "== migrate up"
compose run --rm --no-deps app migrate up
echo "== import districts"
compose run --rm --no-deps app import-districts /stubs/districts.geojson
compose up -d app worker
echo "== wait for http://localhost:18080/readyz"
for _ in $(seq 1 60); do
if curl -fsS http://localhost:18080/readyz >/dev/null 2>&1; then
curl -sS http://localhost:18080/readyz; echo
compose ps --format 'table {{.Service}}\t{{.Image}}\t{{.Status}}\t{{.Ports}}'
exit 0
fi
sleep 2
done
echo "stand did not become ready" >&2
compose logs --tail 50 app worker >&2
exit 1
;;
down)
compose down -v --remove-orphans
;;
*)
compose "$cmd" "$@"
;;
esac