Compare commits
26 Commits
d5c291fb9b
...
stage8/sec
| Author | SHA1 | Date | |
|---|---|---|---|
| ca87d6961b | |||
| c6229c46f7 | |||
| 5b44cbda2f | |||
| 2544d4e146 | |||
| dae9294b4d | |||
| d06d67436b | |||
| 4051506b95 | |||
| bcda76003c | |||
| 6aec00ee29 | |||
| ad3c9fbdcd | |||
| d74277552f | |||
| 5fba77b59c | |||
| ff6d5fc412 | |||
| bc524f3d98 | |||
| 32eba016b3 | |||
| 831ddd1154 | |||
| 922812b765 | |||
| 37d4f3519b | |||
| 5ca6d9e662 | |||
| 91c2cab66e | |||
| bea7e116a0 | |||
| c930f5928b | |||
| 1e0844ca82 | |||
| 00a0cd1b8e | |||
| 47b2e8653d | |||
| 017f869380 |
@@ -9,4 +9,8 @@ S3_SECRET_KEY: minioadmin
|
||||
S3_BUCKET: geofiles
|
||||
S3_USE_SSL: "false"
|
||||
|
||||
DB_PORT=5423
|
||||
DB_PORT=5423
|
||||
|
||||
# Landscape Program parallelism (bounds worker peak memory)
|
||||
LP_MAX_WORKERS=8
|
||||
LP_MAX_NODES=2
|
||||
|
||||
227
README.md
227
README.md
@@ -19,6 +19,8 @@ internal/
|
||||
transport/http/ chi router, middleware, handlers
|
||||
storage/s3/ MinIO/S3 object storage
|
||||
messaging/rabbitmq/ connection, publisher, consumer
|
||||
landscape/ Landscape Program graph, kernels, window executor
|
||||
raster/ GDAL raster I/O (COG conversion + godal engine)
|
||||
platform/logger/ slog setup
|
||||
pkg/httputil/ generic JSON/validation HTTP helpers
|
||||
migrations/ embedded goose SQL migrations
|
||||
@@ -32,6 +34,9 @@ api/openapi.yaml OpenAPI 3.1.1 spec (embedded + served at /openapi.yaml)
|
||||
|
||||
- **Category** — hierarchical (self-referencing `parent_id`). Full CRUD; cycle-safe
|
||||
on update.
|
||||
- **DatasetGroup** — a named group of "одноимённые" datasets (see *Dataset
|
||||
groups* below). Full CRUD; deleting a group still referenced by datasets is
|
||||
rejected with 409.
|
||||
- **Dataset** — a geo file uploaded to S3/MinIO (`file_type`: `vector_with_kato |
|
||||
vector | raster`), belonging to one Category. Carries `code`/`name`/`description`/
|
||||
`unit` metadata, a user-defined `meta` (JSONB) blob, an `automated` flag, a
|
||||
@@ -72,10 +77,10 @@ the KATO column and the year columns:
|
||||
`parse_error`).
|
||||
3. The client polls `GET /datasets/{id}` until `awaiting_mapping`, then submits
|
||||
`POST /datasets/{id}/mapping` with the chosen `kato_column` and a
|
||||
`year_columns` map (each `{column, date}`). Validated against the detected
|
||||
columns; `status` → `extracting`.
|
||||
`year_columns` map (each `{column, start_date}`, optionally `end_date` for
|
||||
a period). Validated against the detected columns; `status` → `extracting`.
|
||||
4. A second worker job **unpivots** the attribute table into long-format
|
||||
`dataset_observations` — one row per `(kato_code, date)` with a numeric
|
||||
`dataset_observations` — one row per `(kato_code, start_date)` with a numeric
|
||||
`value` (or `value_text` for non-numeric cells); `status` → `ready`. Read
|
||||
them via `GET /datasets/{id}/observations` (paginated, optional
|
||||
`?kato_code=`).
|
||||
@@ -84,12 +89,191 @@ the KATO column and the year columns:
|
||||
curl -X POST https://dssgis.dwh.kz/datasets/<id>/mapping -H 'Content-Type: application/json' -d '{
|
||||
"kato_column": "като",
|
||||
"year_columns": [
|
||||
{"column": "F_2023", "date": "2023-01-01"},
|
||||
{"column": "D_2025", "date": "2025-01-01"}
|
||||
{"column": "F_2023", "start_date": "2023-01-01"},
|
||||
{"column": "D_2025", "start_date": "2025-01-01"}
|
||||
]
|
||||
}'
|
||||
```
|
||||
|
||||
### Dataset periods
|
||||
|
||||
A dataset carries an optional period — `start_date` and `end_date`
|
||||
(`YYYY-MM-DD`, both nullable, `end_date` inclusive and never before
|
||||
`start_date`) — plus an optional `dg_id` linking it to a *dataset group*: the
|
||||
"одноимённые" datasets that describe the same thing for different periods. All
|
||||
three are accepted on `POST /datasets` (form fields), `PUT /datasets/{id}` and
|
||||
`PATCH /datasets/{id}` (an explicit `null`, or omission on PUT, clears them).
|
||||
|
||||
`GET /datasets` filters on them with `?dg_id=`, and with the inclusive bounds
|
||||
`?start_date_from=`, `?start_date_to=`, `?end_date_from=` and `?end_date_to=`
|
||||
(each `YYYY-MM-DD`).
|
||||
|
||||
### Dataset groups
|
||||
|
||||
A **dataset group** (`/dataset-groups`) collects the datasets that describe the
|
||||
same indicator for different periods — the "одноимённые" datasets. A group has a
|
||||
slug `code` (unique), a `name` and a `description`, and datasets join it through
|
||||
their `dg_id`.
|
||||
|
||||
**Freshness.** `GET /dataset-groups/{id}/latest` returns the group's freshest
|
||||
dataset: the greatest `end_date` wins, datasets without an `end_date` come last,
|
||||
and ties are broken by the newest `created_at`. Narrow the candidates with
|
||||
`?status=` and `?file_type=` (for example `?status=ready&file_type=raster` to
|
||||
get the latest published raster). A group with no matching dataset yields 404.
|
||||
|
||||
**Shared SLD.** A group can carry one SLD (Styled Layer Descriptor) style file
|
||||
(`POST`/`GET /dataset-groups/{id}/sld`, multipart `file`, `.sld` or `.xml`),
|
||||
which acts as the fallback style for every dataset in the group:
|
||||
`GET /datasets/{id}/sld` returns the dataset's own SLD when it has one, and
|
||||
otherwise falls back to its group's — 404 only when neither exists. Deleting a
|
||||
group also removes its stored SLD object.
|
||||
|
||||
## Landscape Program (ЛП)
|
||||
|
||||
The **Landscape Program** is a DAG of per-pixel raster computations that turns
|
||||
ecological value (Z), vulnerability (U) and sector suitability (P) class rasters
|
||||
into an ecological constraint (E), per-sector conflicts (C) and three functional
|
||||
zoning scenarios (V1–V3). Every input and output of every node is a *dataset
|
||||
group*; the leaf inputs are the **slots** users upload to, and every result is an
|
||||
ordinary `raster` dataset (COG + SLD) served by the existing dataset endpoints.
|
||||
|
||||
The graph lives in Go (`internal/landscape/graph.go`); the groups and the
|
||||
`landscape` category are seeded by migration `00011_landscape.sql`. A node code
|
||||
`XY` reads as `X` = stage, `Y` = function within the stage: a stage starts only
|
||||
after every node of all lower stages finished, and nodes of one stage run
|
||||
concurrently.
|
||||
|
||||
| Node | Name | Inputs (group codes) | Outputs (group codes) |
|
||||
|---|---|---|---|
|
||||
| 00 | z_integrated | `lp-z-biodiversity`, `lp-z-lcr`, `lp-z-soil-protection`, `lp-z-wr` | `lp-z-integrated` |
|
||||
| 01 | u_integrated | `lp-u-surf-water`, `lp-u-soil`, `lp-u-crop`, `lp-u-veg` | `lp-u-integrated` |
|
||||
| 02 | z_max | Z1..Z4 as above | `lp-z-max` |
|
||||
| 03 | u_max | U as above | `lp-u-max` |
|
||||
| 10 | e_ecological_constraint | `lp-z-integrated`, `lp-u-integrated` | `lp-e` |
|
||||
| 20 | c_conflicts | `lp-p-irrigation`, `lp-p-snow`, `lp-p-crop`, `lp-p-pasture`, `lp-p-tourism`, `lp-e` | `lp-c-irrigation`, `lp-c-snow`, `lp-c-crop`, `lp-c-pasture`, `lp-c-tourism` |
|
||||
| 21 | p_max | five P groups | `lp-p-max` |
|
||||
| 30 | func_zoning | `lp-e`, five P groups (original suitability) | `lp-v1-conservative`, `lp-v2-balanced`, `lp-v3-development` |
|
||||
| 31 | c_max | five C groups | `lp-c-max` |
|
||||
|
||||
Sector order everywhere: 1 irrigation, 2 snow, 3 crop, 4 pasture, 5 tourism.
|
||||
|
||||
### Pixel rules
|
||||
|
||||
A pixel value is **valid** iff `1 <= v <= 5`; anything else (nodata included) is
|
||||
invalid. Output rasters are `int16` with nodata `-9999`. `classify(x)` maps a
|
||||
float32 score to a class: `<= 1.8 → 1`, `<= 2.6 → 2`, `<= 3.4 → 3`, `<= 4.2 → 4`,
|
||||
`<= 5.0 → 5` (and requires `x >= 1.0`).
|
||||
|
||||
| Node | Valid domain (pixel) | Value |
|
||||
|---|---|---|
|
||||
| z_integrated | all four Z valid | `classify(0.25*Z1 + 0.25*Z2 + 0.25*Z3 + 0.25*Z4)` |
|
||||
| u_integrated | water ∧ soil ∧ (crop ∨ veg) | `classify(num/den)` with crop/veg contributing `0.25` each only when valid |
|
||||
| z_max / u_max | as above | `max` over the valid inputs |
|
||||
| e_ecological_constraint | Z ∧ U valid | `max(Z, U)` |
|
||||
| c_conflicts | `P_j` ∧ E valid | `min(P_j, E)` per sector |
|
||||
| p_max / c_max | at least one input valid | `max` over the valid ones |
|
||||
| func_zoning | E valid ∧ at least one `P_k` valid | winning sector + 1, else 1 |
|
||||
|
||||
Functional zoning takes `S_k = P_k`, the original suitability rasters (the conflict
|
||||
layers `C_k` are diagnostics and are not zoning inputs). A sector is a candidate when `S_k` and `E`
|
||||
are valid and the scenario rule holds — V1: `E <= 2 && S >= v1_min_s_e12` or
|
||||
`E == 3 && S >= v1_min_s_e3`; V2: `E <= 2`, or `E == 3 && S >= v2_min_s_e3`, or
|
||||
`E == 4 && S >= v2_min_s_e4`; V3: `E <= 4`. With no candidate the pixel is class
|
||||
1 (regulated); otherwise the candidate with the greatest `S` wins, ties broken by
|
||||
`tie_priority`, and the class is `sector + 1` (2 irrigation, 3 snow, 4 crop,
|
||||
5 pasture, 6 tourism).
|
||||
|
||||
Inputs are **not** resampled or reprojected: a node whose inputs do not share one
|
||||
grid (size, geotransform, CRS) fails with a clear message, and so does the run.
|
||||
|
||||
### Uploading inputs, running, reading results
|
||||
|
||||
```sh
|
||||
# 1. fill a slot (repeat per input group; see GET /landscape/slots for the codes)
|
||||
curl -X POST https://dssgis.dwh.kz/landscape/slots/lp-z-biodiversity \
|
||||
-F file=@z_biodiversity.tif -F name="Z Biodiversity 2025" \
|
||||
-F start_date=2025-01-01 -F end_date=2025-12-31
|
||||
|
||||
# 2. queue a run (202); the worker executes the graph asynchronously
|
||||
curl -X POST https://dssgis.dwh.kz/landscape/runs \
|
||||
-H 'Content-Type: application/json' -d '{"from_stage": 0}'
|
||||
|
||||
# 3. follow it, then read the layers
|
||||
curl https://dssgis.dwh.kz/landscape/runs/<run-id>
|
||||
curl https://dssgis.dwh.kz/landscape/results
|
||||
```
|
||||
|
||||
`GET /landscape/slots` lists every leaf slot with the dataset currently filling
|
||||
it (any status), so the UI can show what is still missing. The upload is the
|
||||
regular raster path: `.tif`/`.tiff` with a TIFF magic-byte check, stored in the
|
||||
slot's group under the `landscape` category, COG conversion enqueued.
|
||||
|
||||
**Input resolution.** For each input group a node takes, in order: the run's
|
||||
`overrides[group]`, else whatever this run already produced for that group, else
|
||||
the freshest `ready` raster of the group (greatest `end_date`, NULLs last, newest
|
||||
`created_at` first). A missing input fails the node and the run.
|
||||
|
||||
**Incremental recompute.** Each node is fingerprinted over its code, its resolved
|
||||
inputs and the params that actually affect it. Without `from_stage`, a node whose
|
||||
fingerprint matches an earlier `succeeded` node — with all of its output datasets
|
||||
still present and `ready` — is marked `skipped` and its outputs reused. With
|
||||
`from_stage: N`, every node of stage `N` and above is recomputed regardless;
|
||||
`from_stage: 0` recomputes everything. `overrides` pins a specific dataset
|
||||
(existing, `raster`, `ready`) for a group instead of resolving the freshest one.
|
||||
Run params: `window_size` (256..8192, default 2048), `tie_priority` (a
|
||||
permutation of 1..5, default `[1,2,3,4,5]`), `v1_min_s_e12`/`v1_min_s_e3`
|
||||
(defaults 3/4) and `v2_min_s_e3`/`v2_min_s_e4` (defaults 3/5).
|
||||
|
||||
**Results.** Every output is a normal dataset in its group, named
|
||||
`<Output name> <run date>`, `automated = true`, with
|
||||
`meta = {lp_run_id, lp_node, inputs}` and the min/max period of its inputs.
|
||||
`GET /landscape/results` returns one entry per output group with its freshest
|
||||
`ready` dataset, a `cog_url` (`/datasets/{id}.cog`) and, when the group carries
|
||||
one, an `sld_url` (`/dataset-groups/{id}/sld`). A node failure fails its stage;
|
||||
later stages are not started and the run ends `failed` with the message. Jobs are
|
||||
nacked without requeue, so there is no automatic retry — start a new run.
|
||||
|
||||
### Memory and parallelism
|
||||
|
||||
A run has two nested levels of parallelism: nodes of a stage run concurrently,
|
||||
and each node processes its windows with its own worker pool. Every worker holds
|
||||
one window buffer per input **and** per output, so the worker's peak resident set
|
||||
is roughly
|
||||
|
||||
```
|
||||
nodes × workers-per-node × window_size² × (inputs + outputs) × 8 bytes
|
||||
```
|
||||
|
||||
(the widest node, `20 c_conflicts`, has 6 inputs and 5 outputs). Two environment
|
||||
variables bound that product, independently of how many cores the host has:
|
||||
|
||||
| Variable | Default | Meaning |
|
||||
| --- | --- | --- |
|
||||
| `LP_MAX_WORKERS` | `8` | Total window-worker budget shared by all nodes of a run. |
|
||||
| `LP_MAX_NODES` | `2` | How many graph nodes of one stage run at the same time. |
|
||||
|
||||
Each node gets `max(1, LP_MAX_WORKERS / LP_MAX_NODES)` workers, further capped by
|
||||
`GOMAXPROCS` and by the number of windows, so the total stays at
|
||||
`LP_MAX_WORKERS`. At the defaults and `window_size: 2048` that is about
|
||||
8 × 2048² × 11 × 8 B ≈ 2.9 GiB worst case — lower `LP_MAX_WORKERS` (or
|
||||
`window_size`) on a small host, raise it on a large one.
|
||||
|
||||
### Build requirements
|
||||
|
||||
The raster math uses [godal](https://github.com/airbusgeo/godal), a **cgo**
|
||||
binding to GDAL, so the binary must be built with `CGO_ENABLED=1` against GDAL
|
||||
headers and libraries:
|
||||
|
||||
```sh
|
||||
brew install gdal # macOS; Linux: libgdal-dev / gdal-dev
|
||||
go build ./... # cgo is on by default for a native build
|
||||
```
|
||||
|
||||
`build/package/Dockerfile` already handles the image: the builder stage adds
|
||||
`gdal-dev gcc g++ musl-dev pkgconfig` and builds with `CGO_ENABLED=1`, and the
|
||||
runtime stage installs `gdal` (the shared library godal needs) plus `gdal-tools`
|
||||
(`gdal_translate`/`gdalinfo` for the COG conversion).
|
||||
|
||||
## Getting started
|
||||
|
||||
```sh
|
||||
@@ -117,15 +301,31 @@ server runs it is served at `/openapi.yaml`, with an interactive **Redoc** UI at
|
||||
| GET | `/categories/{id}` | get |
|
||||
| PUT | `/categories/{id}` | update |
|
||||
| DELETE | `/categories/{id}` | delete |
|
||||
| GET | `/datasets` | paginated list of summaries (`?page=`, `?page_size=`, `?category_id=`) |
|
||||
| POST | `/datasets` | upload (multipart: `file`, `file_type`, `category_id`, `code`, `name`, `description?`, `unit?`, `meta?` (JSON), `automated?` (bool)) |
|
||||
| GET | `/datasets` | paginated list of summaries (`?page=`, `?page_size=`, `?category_id=`, `?dg_id=`, `?start_date_from=`, `?start_date_to=`, `?end_date_from=`, `?end_date_to=`) |
|
||||
| POST | `/datasets` | upload (multipart: `file`, `file_type`, `category_id`, `code`, `name`, `description?`, `unit?`, `meta?` (JSON), `automated?` (bool), `dg_id?`, `start_date?`, `end_date?`) |
|
||||
| GET | `/datasets/{id}` | full dataset (geometry as GeoJSON, `bbox` for rasters) |
|
||||
| GET | `/datasets/{id}.geojson` | GeoJSON `FeatureCollection`; plain `vector` returns its geometry as a single feature with the extracted attribute table as top-level properties; `vector_with_kato` always ignores its own geometry and joins the `districts` table by KATO, mapping observations onto each polygon |
|
||||
| GET | `/datasets/{id}/status` | processing status; long-polls with `?current=<status>` (holds up to `?wait=` secs, default 25, max 60) |
|
||||
| GET | `/datasets/{id}/download` | download the stored file |
|
||||
| POST | `/datasets/{id}/mapping` | set KATO column + year→date map (vector_with_kato) |
|
||||
| POST | `/datasets/{id}/mapping` | set KATO column + year→period map (vector_with_kato) |
|
||||
| GET | `/datasets/{id}/observations` | paginated unpivoted values (`?kato_code=`, `?page=`, `?page_size=`) |
|
||||
| POST | `/datasets/{id}/sld` | upload/replace the dataset's SLD (multipart `file`, `.sld`/`.xml`) |
|
||||
| GET | `/datasets/{id}/sld` | download the dataset's SLD, falling back to its group's |
|
||||
| DELETE | `/datasets/{id}` | delete (row + object) |
|
||||
| GET | `/dataset-groups` | paginated list (`?code=`, `?page=`, `?page_size=`) |
|
||||
| POST | `/dataset-groups` | create (`code`, `name`, `description?`) |
|
||||
| GET | `/dataset-groups/{id}` | get |
|
||||
| PUT | `/dataset-groups/{id}` | update |
|
||||
| DELETE | `/dataset-groups/{id}` | delete (409 while datasets reference it) |
|
||||
| GET | `/dataset-groups/{id}/latest` | freshest dataset of the group (`?status=`, `?file_type=`) |
|
||||
| POST | `/dataset-groups/{id}/sld` | upload/replace the group's shared SLD |
|
||||
| GET | `/dataset-groups/{id}/sld` | download the group's shared SLD |
|
||||
| GET | `/landscape/slots` | ЛП input slots + the dataset filling each |
|
||||
| POST | `/landscape/slots/{code}` | upload a GeoTIFF into a slot (multipart: `file`, `name?`, `start_date?`, `end_date?`) |
|
||||
| GET | `/landscape/runs` | paginated runs, newest first (`?page=`, `?page_size=`) |
|
||||
| POST | `/landscape/runs` | queue a run (`from_stage?`, `overrides?`, `params?`); 202 |
|
||||
| GET | `/landscape/runs/{id}` | run with per-node status, inputs, outputs, timings |
|
||||
| GET | `/landscape/results` | per output group: latest layer, `cog_url`, `sld_url` |
|
||||
|
||||
Example upload:
|
||||
|
||||
@@ -165,6 +365,17 @@ make lint # golangci-lint (if installed)
|
||||
CI (`.github/workflows/ci.yml`) runs build, vet, `go test -race`, and golangci-lint
|
||||
on every push and pull request.
|
||||
|
||||
Repository tests in `internal/repository/postgres` run against a real database
|
||||
when `TEST_DB_URL` points at one with migrations applied, and are skipped
|
||||
otherwise. They catch Postgres-only failures (parameter type deduction, casts,
|
||||
constraints) that in-memory stubs cannot:
|
||||
|
||||
```sh
|
||||
docker run -d --name gis-test-pg -e POSTGRES_USER=gis -e POSTGRES_PASSWORD=gis -e POSTGRES_DB=gis -p 5434:5432 postgis/postgis:17-3.5
|
||||
DB_URL='postgres://gis:gis@127.0.0.1:5434/gis?sslmode=disable' S3_ENDPOINT=x S3_ACCESS_KEY=x S3_SECRET_KEY=x RABBITMQ_URL=amqp://x go run ./cmd/gis migrate up
|
||||
TEST_DB_URL='postgres://gis:gis@127.0.0.1:5434/gis?sslmode=disable' go test ./internal/repository/postgres/ -v
|
||||
```
|
||||
|
||||
## Adding a feature
|
||||
|
||||
Each new domain is one vertical slice mirroring Category/Dataset:
|
||||
|
||||
946
api/openapi.yaml
946
api/openapi.yaml
File diff suppressed because it is too large
Load Diff
@@ -1,5 +1,12 @@
|
||||
# Build context is the repo root: docker build -f build/package/Dockerfile .
|
||||
FROM golang:1.26.1-alpine AS builder
|
||||
# Go patch release pinned (not the floating 1.26 tag) so the shipped standard
|
||||
# library is reproducible and includes the security fixes up to go1.26.9;
|
||||
# keep in step with the `toolchain` line in go.mod.
|
||||
FROM golang:1.26.9-alpine3.24 AS builder
|
||||
|
||||
# godal (GDAL bindings) needs cgo, a C/C++ toolchain and the GDAL headers.
|
||||
# g++ is required: parts of the binding compile as C++ against GDAL's headers.
|
||||
RUN apk add --no-cache gdal-dev gcc g++ musl-dev pkgconfig
|
||||
|
||||
WORKDIR /src
|
||||
|
||||
@@ -7,13 +14,18 @@ COPY go.mod go.sum ./
|
||||
RUN go mod download
|
||||
|
||||
COPY . .
|
||||
RUN CGO_ENABLED=0 GOOS=linux go build -trimpath -o /out/gis ./cmd/gis
|
||||
RUN CGO_ENABLED=1 GOOS=linux go build -trimpath -o /out/gis ./cmd/gis
|
||||
|
||||
FROM alpine:3.20
|
||||
# Must stay on the same Alpine release as the builder — hence the pinned
|
||||
# `golang:1.26.9-alpine3.24` above rather than the floating `-alpine` tag. A
|
||||
# mismatch ships a different libgdal soname, and the image still builds green
|
||||
# while the binary dies at startup with "libgdal.so.NN: No such file".
|
||||
# Bump both lines together.
|
||||
FROM alpine:3.24
|
||||
|
||||
# gdal-tools provides gdal_translate / gdalinfo for raster COG conversion and
|
||||
# ogr2ogr for extracting vector feature geometry (worker).
|
||||
RUN apk add --no-cache ca-certificates tzdata gdal-tools \
|
||||
# gdal: shared library for godal (raster math in the worker).
|
||||
# gdal-tools: gdal_translate / gdalinfo (COG conversion) and ogr2ogr.
|
||||
RUN apk add --no-cache ca-certificates tzdata gdal gdal-tools \
|
||||
&& adduser -D -u 10001 app
|
||||
|
||||
WORKDIR /app
|
||||
|
||||
@@ -2,6 +2,9 @@
|
||||
|
||||
# HTTP server
|
||||
PORT=8080
|
||||
# Upper bound for any request body, uploads included (bytes; default 2 GiB).
|
||||
# Larger requests are rejected with 413. 0 disables the cap.
|
||||
HTTP_MAX_UPLOAD_BYTES=2147483648
|
||||
|
||||
# Postgres
|
||||
DB_URL=postgres://gis:gis@127.0.0.1:5432/gis?sslmode=disable
|
||||
@@ -28,3 +31,7 @@ RABBITMQ_QUEUE=gis.events.example
|
||||
|
||||
# Host port mapping for the postgres container (docker-compose)
|
||||
DB_PORT=5432
|
||||
|
||||
# Landscape Program parallelism (bounds worker peak memory)
|
||||
LP_MAX_WORKERS=8
|
||||
LP_MAX_NODES=2
|
||||
|
||||
@@ -1,16 +1,21 @@
|
||||
# Credentials below default to well-known development values so that
|
||||
# `docker compose up` works out of the box. They MUST be overridden on any
|
||||
# shared or reachable host: put strong values for POSTGRES_PASSWORD,
|
||||
# MINIO_ROOT_USER, MINIO_ROOT_PASSWORD, RABBITMQ_DEFAULT_USER and
|
||||
# RABBITMQ_DEFAULT_PASS in deployments/.env (not committed).
|
||||
x-app-common: &app-common
|
||||
build:
|
||||
context: ..
|
||||
dockerfile: build/package/Dockerfile
|
||||
environment: &app-environment
|
||||
PORT: 8080
|
||||
DB_URL: postgres://gis:gis@postgres:5432/gis?sslmode=disable
|
||||
DB_URL: postgres://gis:${POSTGRES_PASSWORD:-gis}@postgres:5432/gis?sslmode=disable
|
||||
S3_ENDPOINT: minio:9000
|
||||
S3_ACCESS_KEY: minioadmin
|
||||
S3_SECRET_KEY: minioadmin
|
||||
S3_ACCESS_KEY: ${MINIO_ROOT_USER:-minioadmin}
|
||||
S3_SECRET_KEY: ${MINIO_ROOT_PASSWORD:-minioadmin}
|
||||
S3_BUCKET: geofiles
|
||||
S3_USE_SSL: "false"
|
||||
RABBITMQ_URL: amqp://guest:guest@rabbitmq:5672/
|
||||
RABBITMQ_URL: amqp://${RABBITMQ_DEFAULT_USER:-guest}:${RABBITMQ_DEFAULT_PASS:-guest}@rabbitmq:5672/
|
||||
RABBITMQ_EXCHANGE: gis.events
|
||||
RABBITMQ_QUEUE: gis.events.example
|
||||
depends_on:
|
||||
@@ -47,7 +52,7 @@ services:
|
||||
image: postgis/postgis:17-3.5
|
||||
environment:
|
||||
POSTGRES_USER: gis
|
||||
POSTGRES_PASSWORD: gis
|
||||
POSTGRES_PASSWORD: ${POSTGRES_PASSWORD:-gis}
|
||||
POSTGRES_DB: gis
|
||||
ports:
|
||||
- "${DB_PORT:-5432}:5432"
|
||||
@@ -67,8 +72,8 @@ services:
|
||||
- "9000:9000"
|
||||
- "9001:9001"
|
||||
environment:
|
||||
MINIO_ROOT_USER: minioadmin
|
||||
MINIO_ROOT_PASSWORD: minioadmin
|
||||
MINIO_ROOT_USER: ${MINIO_ROOT_USER:-minioadmin}
|
||||
MINIO_ROOT_PASSWORD: ${MINIO_ROOT_PASSWORD:-minioadmin}
|
||||
volumes:
|
||||
- minio_data:/data
|
||||
healthcheck:
|
||||
@@ -84,8 +89,8 @@ services:
|
||||
- "5672:5672"
|
||||
- "15672:15672"
|
||||
environment:
|
||||
RABBITMQ_DEFAULT_USER: guest
|
||||
RABBITMQ_DEFAULT_PASS: guest
|
||||
RABBITMQ_DEFAULT_USER: ${RABBITMQ_DEFAULT_USER:-guest}
|
||||
RABBITMQ_DEFAULT_PASS: ${RABBITMQ_DEFAULT_PASS:-guest}
|
||||
volumes:
|
||||
- rabbitmq_data:/var/lib/rabbitmq
|
||||
healthcheck:
|
||||
|
||||
2661
docs/superpowers/plans/2026-09-05-landscape-program.md
Normal file
2661
docs/superpowers/plans/2026-09-05-landscape-program.md
Normal file
File diff suppressed because it is too large
Load Diff
320
docs/superpowers/specs/2026-09-05-landscape-program-design.md
Normal file
320
docs/superpowers/specs/2026-09-05-landscape-program-design.md
Normal file
@@ -0,0 +1,320 @@
|
||||
# Landscape Program (ЛП) + dataset groups and dates — design
|
||||
|
||||
Date: 2026-09-05. Source: `LP_20260902.excalidraw`, `TZ_functional_zoning_go.md`,
|
||||
`zoning_core.py`, `Landscape.ipynb` (reference numpy implementation).
|
||||
Decisions taken with the product owner are recorded inline as **Decision**.
|
||||
|
||||
## 1. Goals
|
||||
|
||||
1. **Dataset versioning.** Datasets can belong to a *dataset group* ("одноимённые"
|
||||
datasets), carry a `start_date` / `end_date` period, be filtered by those, and the
|
||||
"freshest" dataset of a group can be fetched.
|
||||
2. **Landscape Program pipeline.** A DAG of raster computations (Z, U, E, C, P_max,
|
||||
C_max, functional zoning V1–V3) executed by the worker, with results stored as
|
||||
ordinary raster datasets (COG + SLD) served by the existing dataset endpoints,
|
||||
incremental recompute, and a dedicated upload for input rasters.
|
||||
|
||||
Non-goals: spatial operations (resampling, reprojection), GUI, changing the
|
||||
vector_with_kato flow beyond the column rename below.
|
||||
|
||||
## 2. Part 1 — Dataset groups and dates
|
||||
|
||||
### 2.1 Schema (migration `00010_dataset_groups_and_dates.sql`)
|
||||
|
||||
```sql
|
||||
CREATE TABLE dataset_groups (
|
||||
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
|
||||
code VARCHAR(255) NOT NULL UNIQUE, -- slug, same rules as categories.code
|
||||
name VARCHAR(255) NOT NULL,
|
||||
description TEXT NOT NULL DEFAULT '',
|
||||
sld_storage_key TEXT,
|
||||
sld_filename VARCHAR(255),
|
||||
sld_content_type VARCHAR(255),
|
||||
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
|
||||
updated_at TIMESTAMPTZ NOT NULL DEFAULT now()
|
||||
);
|
||||
|
||||
ALTER TABLE datasets ADD COLUMN dg_id UUID REFERENCES dataset_groups(id) ON DELETE RESTRICT;
|
||||
ALTER TABLE datasets ADD COLUMN start_date DATE;
|
||||
ALTER TABLE datasets ADD COLUMN end_date DATE;
|
||||
CREATE INDEX idx_datasets_dg_id_end_date ON datasets (dg_id, end_date DESC NULLS LAST, created_at DESC);
|
||||
CREATE INDEX idx_datasets_start_date ON datasets (start_date);
|
||||
CREATE INDEX idx_datasets_end_date ON datasets (end_date);
|
||||
|
||||
-- Decision: rename the observation/mapping period columns (breaking API change).
|
||||
ALTER TABLE dataset_observations RENAME COLUMN date TO start_date;
|
||||
ALTER TABLE dataset_observations RENAME COLUMN date_to TO end_date;
|
||||
-- unique index and idx_dataset_observations_kato_date recreated on the new names.
|
||||
-- datasets.year_columns JSONB: every element's keys "date" -> "start_date",
|
||||
-- "date_to" -> "end_date" (UPDATE with jsonb_agg over jsonb_array_elements; NULL-safe).
|
||||
```
|
||||
|
||||
Constraint: `end_date >= start_date` when both set (CHECK constraint + service validation).
|
||||
|
||||
### 2.2 Domain changes
|
||||
|
||||
- `domain.DatasetGroup{ID, Code, Name, Description, SldStorageKey, SldFilename,
|
||||
SldContentType, CreatedAt, UpdatedAt}`.
|
||||
- `domain.Dataset` and `domain.DatasetSummary` gain `DatasetGroupID *uuid.UUID`
|
||||
(`json:"dg_id"`), `StartDate *string` (`json:"start_date"`, `YYYY-MM-DD`),
|
||||
`EndDate *string` (`json:"end_date"`).
|
||||
- `domain.Observation`: `Date`→`StartDate` (`json:"start_date"`), `DateTo`→`EndDate`
|
||||
(`json:"end_date"`). `domain.YearColumn`: same rename. Mapping request body uses
|
||||
`start_date` (required) / `end_date` (optional). Everything that previously said
|
||||
`date`/`date_to` in code, OpenAPI and README is renamed. GeoJSON output keeps its
|
||||
`data` (keyed by start date) and `periods` (start→end) shape.
|
||||
- `domain.DatasetFilter` gains `DatasetGroupID *uuid.UUID`, `StartDateFrom`,
|
||||
`StartDateTo`, `EndDateFrom`, `EndDateTo *string` (inclusive bounds).
|
||||
|
||||
### 2.3 API
|
||||
|
||||
| Method & path | Behaviour |
|
||||
|---|---|
|
||||
| `GET /dataset-groups` | paginated list, `?code=` exact filter optional |
|
||||
| `POST /dataset-groups` | `{code, name, description}`; 201 |
|
||||
| `GET /dataset-groups/{id}` | group |
|
||||
| `PUT /dataset-groups/{id}` | replace `{code, name, description}` |
|
||||
| `DELETE /dataset-groups/{id}` | 204; 409 when datasets reference it; also removes group SLD object |
|
||||
| `GET /dataset-groups/{id}/latest` | freshest dataset of the group: order by `end_date DESC NULLS LAST, created_at DESC`; optional `?status=`, `?file_type=`; 404 when none |
|
||||
| `POST /dataset-groups/{id}/sld` | multipart `file` (`.sld`/`.xml`), replaces; mirrors dataset SLD upload |
|
||||
| `GET /dataset-groups/{id}/sld` | download; 404 when none |
|
||||
| `GET /datasets` | new filters `dg_id`, `start_date_from`, `start_date_to`, `end_date_from`, `end_date_to` (all `YYYY-MM-DD`, inclusive, any combination) |
|
||||
| `POST /datasets` | new optional form fields `dg_id`, `start_date`, `end_date` |
|
||||
| `PUT /datasets/{id}` | body gains `dg_id`, `start_date`, `end_date` (nullable; omitted/null clears) |
|
||||
| `PATCH /datasets/{id}` | sparse; same three fields, `null` clears |
|
||||
| `GET /datasets/{id}/sld` | **fallback**: when the dataset has no SLD but its group has one, the group SLD is served |
|
||||
|
||||
Validation: `dg_id` must reference an existing group (404/422 as for `category_id`);
|
||||
dates parse as `2006-01-02`; `end_date` not before `start_date`.
|
||||
|
||||
`DatasetSummary` (list responses) includes `dg_id`, `start_date`, `end_date`.
|
||||
|
||||
### 2.4 Freshness rule
|
||||
|
||||
"Самый свежий" = greatest `end_date`; datasets with `NULL end_date` sort after all
|
||||
dated ones; ties broken by `created_at DESC`. The same rule is used by the pipeline
|
||||
to resolve inputs (with `status = ready`, `file_type = raster`).
|
||||
|
||||
## 3. Part 2 — Landscape Program
|
||||
|
||||
### 3.1 Computation graph
|
||||
|
||||
Node code `XY`: `X` = stage, `Y` = function in the stage. A stage starts only when
|
||||
every node of all lower stages finished. Every input and output of every node is a
|
||||
**dataset group**; leaf groups are the *slots* users upload to.
|
||||
|
||||
| Node | Name | Inputs (group codes) | Outputs (group codes) |
|
||||
|---|---|---|---|
|
||||
| 00 | z_integrated | `lp-z-biodiversity`, `lp-z-lcr`, `lp-z-soil-protection`, `lp-z-wr` | `lp-z-integrated` |
|
||||
| 01 | u_integrated | `lp-u-surf-water`, `lp-u-soil`, `lp-u-crop`, `lp-u-veg` | `lp-u-integrated` |
|
||||
| 02 | z_max | Z1..Z4 as above | `lp-z-max` |
|
||||
| 03 | u_max | U as above | `lp-u-max` |
|
||||
| 10 | e_ecological_constraint | `lp-z-integrated`, `lp-u-integrated` | `lp-e` |
|
||||
| 20 | c_conflicts | `lp-p-irrigation`, `lp-p-snow`, `lp-p-crop`, `lp-p-pasture`, `lp-p-tourism`, `lp-e` | `lp-c-irrigation`, `lp-c-snow`, `lp-c-crop`, `lp-c-pasture`, `lp-c-tourism` |
|
||||
| 21 | p_max | five P groups | `lp-p-max` |
|
||||
| 30 | func_zoning | `lp-e`, five P groups (original suitability) | `lp-v1-conservative`, `lp-v2-balanced`, `lp-v3-development` |
|
||||
| 31 | c_max | five C groups | `lp-c-max` |
|
||||
|
||||
Sector order everywhere: 1 irrigation, 2 snow, 3 crop, 4 pasture, 5 tourism.
|
||||
**Decision (revised 2026-09-11):** func_zoning takes S_k = P_k, the original suitability
|
||||
rasters, as in the reference notebook ("Three scenarios" cells) and `zoning_core.py`.
|
||||
The diagram folder `300-…-Equals_102_202` pointing at the C layers was a drawing error;
|
||||
C_k = min(P_k, E) are diagnostic conflict layers and never replace P_k in zoning.
|
||||
|
||||
Groups and the category `landscape` ("Ландшафтная программа") are seeded by
|
||||
migration `00011_landscape.sql` (`INSERT … ON CONFLICT (code) DO NOTHING`). The
|
||||
graph itself lives in Go (`internal/landscape/graph.go`), not in the DB.
|
||||
|
||||
### 3.2 Pixel rules (from the reference notebook; must match it bitwise)
|
||||
|
||||
Notation: a value is *valid* iff `1 <= v <= 5`; anything else (including nodata) is
|
||||
invalid. Output nodata is `-9999`. `classify(x)` maps a float32 score to a class:
|
||||
`x <= 1.8 → 1`, `<= 2.6 → 2`, `<= 3.4 → 3`, `<= 4.2 → 4`, `<= 5.0 → 5` (and requires
|
||||
`x >= 1.0`); comparisons are done in float32 against float32 constants, exactly as
|
||||
numpy does with a float32 array.
|
||||
|
||||
| Node | Valid domain (pixel) | Value |
|
||||
|---|---|---|
|
||||
| z_integrated | all four Z valid | `classify(0.25*Z1 + 0.25*Z2 + 0.25*Z3 + 0.25*Z4)` (float32) |
|
||||
| u_integrated | water ∧ soil ∧ (crop ∨ veg) | `classify(num/den)`, `num = 0.25*water + 0.25*soil + [0.25*crop if crop valid] + [0.25*veg if veg valid]`, `den = 0.5 + [0.25] + [0.25]` |
|
||||
| z_max | all four valid | `max(Z1..Z4)` |
|
||||
| u_max | water ∧ soil ∧ (crop ∨ veg) | `max` over the valid ones of water, soil, crop, veg |
|
||||
| e | Z ∧ U valid | `max(Z, U)` |
|
||||
| c_conflicts (5 outputs) | P_j ∧ E valid | `min(P_j, E)` |
|
||||
| p_max | at least one P valid | `max` over valid P |
|
||||
| c_max | at least one C valid | `max` over valid C |
|
||||
| func_zoning (3 outputs) | see TZ §4; S_k = P_k | see below |
|
||||
|
||||
Functional zoning (TZ `TZ_functional_zoning_go.md`, reference `zoning_core.py`):
|
||||
|
||||
```
|
||||
eValid = valid(E); base_k = eValid && valid(S_k); analysisValid = eValid && any(base_k)
|
||||
if !analysisValid → -9999 in all scenarios
|
||||
candidate_k = base_k && rule_v(E, S_k)
|
||||
V1: (E <= 2 && S >= v1_min_s_e12) || (E == 3 && S >= v1_min_s_e3) (defaults 3, 4)
|
||||
V2: (E <= 2) || (E == 3 && S >= v2_min_s_e3) || (E == 4 && S >= v2_min_s_e4) (defaults 3, 5)
|
||||
V3: (E <= 4)
|
||||
no candidates → class 1 (regulated); else winners = candidates with S == max(S over candidates);
|
||||
sector = first of tie_priority (default [1,2,3,4,5]) among winners; class = sector + 1
|
||||
```
|
||||
|
||||
Output classes: 1 regulated, 2 irrigation, 3 snow, 4 crop, 5 pasture, 6 tourism.
|
||||
|
||||
### 3.3 Raster I/O abstraction (**Decision:** godal behind an interface)
|
||||
|
||||
Package `internal/raster` (existing) gains an engine-agnostic interface so the
|
||||
implementation can be swapped (godal today; `gdal_calc`, ENVI+CLI or another binding
|
||||
later) without touching the pipeline:
|
||||
|
||||
```go
|
||||
type Grid struct { Width, Height int; GeoTransform [6]float64; ProjectionWKT string }
|
||||
type Window struct { X, Y, Width, Height int }
|
||||
type Reader interface {
|
||||
Grid() Grid
|
||||
NoData() (value float64, ok bool)
|
||||
ReadFloat64(w Window, dst []float64) error // band 1, converted by the driver
|
||||
Close() error
|
||||
}
|
||||
type WriterOptions struct { NoData int16; TileSize int; Compression string /* DEFLATE|ZSTD */ }
|
||||
type Writer interface {
|
||||
WriteInt16(w Window, src []int16) error // band 1
|
||||
Close() error
|
||||
}
|
||||
type IO interface {
|
||||
Open(ctx context.Context, path string) (Reader, error)
|
||||
Create(ctx context.Context, path string, g Grid, o WriterOptions) (Writer, error)
|
||||
SameGrid(a, b Grid) bool // size, transform (abs tol 1e-9), CRS (GDAL IsSame)
|
||||
}
|
||||
```
|
||||
|
||||
`internal/raster/godal.go` implements `IO` with `github.com/airbusgeo/godal` (cgo).
|
||||
Output GeoTIFF: `int16`, nodata `-9999`, `TILED=YES`, `BLOCKXSIZE=BLOCKYSIZE=512`,
|
||||
`COMPRESS=DEFLATE`, `PREDICTOR=2`, `BIGTIFF=IF_SAFER`; georeference copied from the
|
||||
node's first input (for func_zoning: E). Inputs are read as float64 regardless of
|
||||
their storage type; multi-band inputs use band 1.
|
||||
|
||||
The existing `RasterConverter` (gdal_translate / gdalinfo / ogr2ogr CLI) stays as is
|
||||
and keeps producing the COG for every result dataset.
|
||||
|
||||
Build: `CGO_ENABLED=1`; `go.mod` gets `toolchain go1.26.2` (the local SDK is
|
||||
incomplete) and the godal dependency. Dockerfile: builder `golang:1.26-alpine` +
|
||||
`gdal-dev gcc musl-dev pkgconfig`; runtime `alpine:3.22` + `gdal gdal-tools`.
|
||||
|
||||
### 3.4 Compute engine (`internal/landscape`)
|
||||
|
||||
- `graph.go`: slots, nodes, stages, group codes, node-level param subsets.
|
||||
- `kernels.go`: pure per-window functions over `[]float64` inputs → `[]int16`
|
||||
outputs, one per node. No I/O, fully unit-tested (TZ §7 edge cases included).
|
||||
- `compute.go`: `Compute(ctx, io raster.IO, node Node, inputs map[string]string
|
||||
/*group→local path*/, outDir string, p Params) (map[string]string /*group→path*/, error)`.
|
||||
Opens inputs, **fails if grids differ** (**Decision:** no resampling), creates
|
||||
outputs on the first input's grid, iterates windows of `p.WindowSize`
|
||||
(default 2048) with a worker pool: each worker owns its own `Reader` handles
|
||||
(GDAL datasets are not goroutine-safe); writes are serialised per output through a
|
||||
mutex. Window order does not affect results.
|
||||
- Nodes of one stage run concurrently (`errgroup`); stages run sequentially.
|
||||
|
||||
### 3.5 Runs, incremental recompute
|
||||
|
||||
Migration `00011_landscape.sql`:
|
||||
|
||||
```sql
|
||||
CREATE TABLE lp_runs (
|
||||
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
|
||||
status VARCHAR(32) NOT NULL, -- queued | running | succeeded | failed
|
||||
from_stage SMALLINT, -- NULL = incremental
|
||||
params JSONB NOT NULL,
|
||||
overrides JSONB NOT NULL DEFAULT '{}', -- {group_code: dataset_id}
|
||||
error TEXT,
|
||||
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
|
||||
started_at TIMESTAMPTZ,
|
||||
finished_at TIMESTAMPTZ
|
||||
);
|
||||
CREATE TABLE lp_run_nodes (
|
||||
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
|
||||
run_id UUID NOT NULL REFERENCES lp_runs(id) ON DELETE CASCADE,
|
||||
node_code VARCHAR(8) NOT NULL,
|
||||
stage SMALLINT NOT NULL,
|
||||
status VARCHAR(32) NOT NULL, -- pending | running | skipped | succeeded | failed
|
||||
fingerprint TEXT NOT NULL,
|
||||
input_datasets JSONB NOT NULL, -- [{group, dataset_id}]
|
||||
output_datasets JSONB, -- [{group, dataset_id}]
|
||||
error TEXT,
|
||||
started_at TIMESTAMPTZ,
|
||||
finished_at TIMESTAMPTZ,
|
||||
UNIQUE (run_id, node_code)
|
||||
);
|
||||
CREATE INDEX idx_lp_run_nodes_lookup ON lp_run_nodes (node_code, fingerprint, status);
|
||||
```
|
||||
|
||||
Run execution (worker, queue `gis.landscape.run`, routing key `landscape.run`):
|
||||
|
||||
1. Mark run `running`. For each stage in order, for each node concurrently:
|
||||
2. **Resolve inputs** per input group: `overrides[group]` if present; else the dataset
|
||||
produced for that group earlier in *this* run; else the freshest `ready` raster of
|
||||
the group (§2.4). Missing → node fails with a clear message.
|
||||
3. **Fingerprint** = sha256 of node code, sorted `(group, dataset_id)` pairs, and the
|
||||
node-relevant params. If `from_stage` is NULL or `stage < from_stage`, and a
|
||||
previous `succeeded` node row with the same fingerprint exists whose output
|
||||
datasets all still exist → mark `skipped`, reuse those outputs. Otherwise compute.
|
||||
4. Download inputs to a temp dir, `Compute`, then for each output create a raster
|
||||
dataset (§3.6), record `output_datasets`, mark `succeeded`.
|
||||
5. Any node failure fails the stage; remaining stages are not started; run `failed`
|
||||
with the error. Otherwise run `succeeded`.
|
||||
|
||||
`from_stage = 0` recomputes everything ("триггер на перерасчёт всего").
|
||||
|
||||
### 3.6 Result datasets
|
||||
|
||||
Every output is stored through a new service method
|
||||
`DatasetService.CreateRasterFromFile(ctx, CreateRasterInput{CategoryID, GroupID,
|
||||
Name, Filename, Path, StartDate, EndDate, Meta})`: uploads the file, inserts a
|
||||
`raster` dataset (`automated = true`, `status = processing`), enqueues the existing
|
||||
COG conversion. `Name` = `<Output name> <run started_at date>`, `Filename` =
|
||||
`<group>.tif`, `start_date`/`end_date` = min/max over the node's inputs (NULL if all
|
||||
NULL), `meta = {"lp_run_id", "lp_node", "inputs": [{group, dataset_id}]}`.
|
||||
Downstream nodes read the original object (`storage_key`), not the COG, so they never
|
||||
wait for conversion. SLD for a result comes from its group (dataset SLD fallback, §2.3).
|
||||
|
||||
### 3.7 Landscape API (`/landscape`)
|
||||
|
||||
| Method & path | Behaviour |
|
||||
|---|---|
|
||||
| `GET /landscape/slots` | every leaf slot: `code`, `title`, group id, latest dataset summary (any status) or null |
|
||||
| `POST /landscape/slots/{code}` | **dedicated input upload**: multipart `file` (`.tif/.tiff`, TIFF magic check), optional `name`, `start_date`, `end_date`. Creates a `raster` dataset in the slot's group under the `landscape` category via the regular upload path (COG conversion enqueued). 201 Dataset. Unknown slot → 404 |
|
||||
| `POST /landscape/runs` | body `{from_stage?: 0..3, overrides?: {group_code: dataset_id}, params?: {window_size?, tie_priority?, v1_min_s_e12?, v1_min_s_e3?, v2_min_s_e3?, v2_min_s_e4?}}`; validates overrides (existing ready raster), inserts run `queued` + pending node rows, publishes job; 202 with run |
|
||||
| `GET /landscape/runs` | paginated, newest first |
|
||||
| `GET /landscape/runs/{id}` | run with nodes (status, inputs, outputs, error, timings) |
|
||||
| `GET /landscape/results` | per output group: `group`, `title`, `node`, latest `ready` dataset (or null) with `cog_url = /datasets/{id}.cog`, `sld_url` when the group has an SLD |
|
||||
|
||||
Params defaults: `window_size 2048`, `tie_priority [1,2,3,4,5]`, `v1 3/4`, `v2 3/5`.
|
||||
Validation: tie_priority is a permutation of 1..5; thresholds in 1..5; window_size
|
||||
256..8192.
|
||||
|
||||
### 3.8 Error handling
|
||||
|
||||
- Grid mismatch, invalid CRS, missing input, GDAL error → node `failed` with message;
|
||||
run `failed`. No retry (consistent with existing consumers: nack without requeue).
|
||||
- Non-nodata values outside 1..5 are treated as invalid pixels (TZ §2), not fatal.
|
||||
- Result dataset creation failure after compute → node failed, temp files removed.
|
||||
- Temp files always cleaned with `defer`.
|
||||
|
||||
### 3.9 Testing
|
||||
|
||||
- Kernels: table tests incl. TZ §7 cases (invalid E, all S invalid, E=5, E=4 with
|
||||
several S=5, all S equal, applicability of crop/veg for U, "any valid" for max nodes).
|
||||
- `Compute`: tests with an in-memory fake `raster.IO` (10×10 grids), including grid
|
||||
mismatch failure and window tiling with sizes that do not divide the raster.
|
||||
- godal implementation: round-trip test on a temp GeoTIFF (runs when GDAL present).
|
||||
- Services: stub repositories as in `dataset_test.go`; run planning (fingerprint,
|
||||
skip/reuse, from_stage), input resolution order (override → this run → latest).
|
||||
- HTTP: handler tests for validation paths.
|
||||
- Part 1: filter clause building, latest ordering (nulls last), rename in mapping and
|
||||
observations, SLD fallback.
|
||||
|
||||
### 3.10 Out of scope / follow-ups
|
||||
|
||||
- Resampling/alignment of inputs (fails by design).
|
||||
- Continuous (float) intermediate rasters and diagnostic layers from the notebook
|
||||
(tie counts, component counts, critical flags).
|
||||
- Real-data acceptance run against the Drive test set (inputs there are not aligned).
|
||||
23
go.mod
23
go.mod
@@ -2,7 +2,10 @@ module gis
|
||||
|
||||
go 1.26.1
|
||||
|
||||
toolchain go1.26.9
|
||||
|
||||
require (
|
||||
github.com/airbusgeo/godal v0.0.18
|
||||
github.com/caarlos0/env/v11 v11.4.1
|
||||
github.com/go-chi/chi/v5 v5.3.0
|
||||
github.com/go-playground/validator/v10 v10.30.3
|
||||
@@ -11,10 +14,10 @@ require (
|
||||
github.com/joho/godotenv v1.5.1
|
||||
github.com/minio/minio-go/v7 v7.2.0
|
||||
github.com/pressly/goose/v3 v3.27.1
|
||||
github.com/rabbitmq/amqp091-go v1.12.0
|
||||
github.com/rabbitmq/amqp091-go v1.13.0
|
||||
github.com/spf13/cobra v1.10.2
|
||||
golang.org/x/sync v0.20.0
|
||||
golang.org/x/text v0.37.0
|
||||
golang.org/x/sync v0.23.0
|
||||
golang.org/x/text v0.42.0
|
||||
modernc.org/sqlite v1.53.0
|
||||
)
|
||||
|
||||
@@ -28,10 +31,9 @@ require (
|
||||
github.com/jackc/pgpassfile v1.0.0 // indirect
|
||||
github.com/jackc/pgservicefile v0.0.0-20240606120523-5a60cdf6a761 // indirect
|
||||
github.com/jackc/puddle/v2 v2.2.2 // indirect
|
||||
github.com/klauspost/compress v1.18.6 // indirect
|
||||
github.com/klauspost/compress v1.18.7 // indirect
|
||||
github.com/klauspost/cpuid/v2 v2.2.11 // indirect
|
||||
github.com/klauspost/crc32 v1.3.0 // indirect
|
||||
github.com/kr/text v0.2.0 // indirect
|
||||
github.com/leodido/go-urn v1.4.0 // indirect
|
||||
github.com/mattn/go-isatty v0.0.21 // indirect
|
||||
github.com/mfridman/interpolate v0.0.2 // indirect
|
||||
@@ -43,13 +45,16 @@ require (
|
||||
github.com/rs/xid v1.6.0 // indirect
|
||||
github.com/sethvargo/go-retry v0.3.0 // indirect
|
||||
github.com/spf13/pflag v1.0.10 // indirect
|
||||
github.com/spiffe/go-spiffe/v2 v2.9.0 // indirect
|
||||
github.com/stretchr/testify v1.12.1 // indirect
|
||||
github.com/tinylib/msgp v1.6.1 // indirect
|
||||
github.com/zeebo/xxh3 v1.1.0 // indirect
|
||||
go.uber.org/multierr v1.11.0 // indirect
|
||||
go.yaml.in/yaml/v3 v3.0.4 // indirect
|
||||
golang.org/x/crypto v0.52.0 // indirect
|
||||
golang.org/x/net v0.54.0 // indirect
|
||||
golang.org/x/sys v0.45.0 // indirect
|
||||
go.yaml.in/yaml/v3 v3.0.5 // indirect
|
||||
golang.org/x/crypto v0.57.0 // indirect
|
||||
golang.org/x/net v0.60.0 // indirect
|
||||
golang.org/x/sys v0.48.0 // indirect
|
||||
google.golang.org/protobuf v1.36.12 // indirect
|
||||
gopkg.in/ini.v1 v1.67.2 // indirect
|
||||
modernc.org/libc v1.73.4 // indirect
|
||||
modernc.org/mathutil v1.7.1 // indirect
|
||||
|
||||
142
go.sum
142
go.sum
@@ -1,18 +1,58 @@
|
||||
cel.dev/expr v0.19.2 h1:V354PbqIXr9IQdwy4SYA4xa0HXaWq1BUPAGzugBY5V4=
|
||||
cel.dev/expr v0.19.2/go.mod h1:MrpN08Q+lEBs+bGYdLxxHkZoUSsCp0nSKTs0nTymJgw=
|
||||
cloud.google.com/go v0.118.3 h1:jsypSnrE/w4mJysioGdMBg4MiW/hHx/sArFpaBWHdME=
|
||||
cloud.google.com/go v0.118.3/go.mod h1:Lhs3YLnBlwJ4KA6nuObNMZ/fCbOQBPuWKPoE0Wa/9Vc=
|
||||
cloud.google.com/go/auth v0.15.0 h1:Ly0u4aA5vG/fsSsxu98qCQBemXtAtJf+95z9HK+cxps=
|
||||
cloud.google.com/go/auth v0.15.0/go.mod h1:WJDGqZ1o9E9wKIL+IwStfyn/+s59zl4Bi+1KQNVXLZ8=
|
||||
cloud.google.com/go/auth/oauth2adapt v0.2.7 h1:/Lc7xODdqcEw8IrZ9SvwnlLX6j9FHQM74z6cBk9Rw6M=
|
||||
cloud.google.com/go/auth/oauth2adapt v0.2.7/go.mod h1:NTbTTzfvPl1Y3V1nPpOgl2w6d/FjO7NNUQaWSox6ZMc=
|
||||
cloud.google.com/go/compute/metadata v0.6.0 h1:A6hENjEsCDtC1k8byVsgwvVcioamEHvZ4j01OwKxG9I=
|
||||
cloud.google.com/go/compute/metadata v0.6.0/go.mod h1:FjyFAW1MW0C203CEOMDTu3Dk1FlqW3Rga40jzHL4hfg=
|
||||
cloud.google.com/go/iam v1.4.1 h1:cFC25Nv+u5BkTR/BT1tXdoF2daiVbZ1RLx2eqfQ9RMM=
|
||||
cloud.google.com/go/iam v1.4.1/go.mod h1:2vUEJpUG3Q9p2UdsyksaKpDzlwOrnMzS30isdReIcLM=
|
||||
cloud.google.com/go/monitoring v1.24.0 h1:csSKiCJ+WVRgNkRzzz3BPoGjFhjPY23ZTcaenToJxMM=
|
||||
cloud.google.com/go/monitoring v1.24.0/go.mod h1:Bd1PRK5bmQBQNnuGwHBfUamAV1ys9049oEPHnn4pcsc=
|
||||
cloud.google.com/go/storage v1.51.0 h1:ZVZ11zCiD7b3k+cH5lQs/qcNaoSz3U9I0jgwVzqDlCw=
|
||||
cloud.google.com/go/storage v1.51.0/go.mod h1:YEJfu/Ki3i5oHC/7jyTgsGZwdQ8P9hqMqvpi5kRKGgc=
|
||||
github.com/GoogleCloudPlatform/opentelemetry-operations-go/detectors/gcp v1.25.0 h1:3c8yed4lgqTt+oTQ+JNMDo+F4xprBf+O/il4ZC0nRLw=
|
||||
github.com/GoogleCloudPlatform/opentelemetry-operations-go/detectors/gcp v1.25.0/go.mod h1:obipzmGjfSjam60XLwGfqUkJsfiheAl+TUjG+4yzyPM=
|
||||
github.com/GoogleCloudPlatform/opentelemetry-operations-go/exporter/metric v0.51.0 h1:fYE9p3esPxA/C0rQ0AHhP0drtPXDRhaWiwg1DPqO7IU=
|
||||
github.com/GoogleCloudPlatform/opentelemetry-operations-go/exporter/metric v0.51.0/go.mod h1:BnBReJLvVYx2CS/UHOgVz2BXKXD9wsQPxZug20nZhd0=
|
||||
github.com/GoogleCloudPlatform/opentelemetry-operations-go/internal/resourcemapping v0.51.0 h1:6/0iUd0xrnX7qt+mLNRwg5c0PGv8wpE8K90ryANQwMI=
|
||||
github.com/GoogleCloudPlatform/opentelemetry-operations-go/internal/resourcemapping v0.51.0/go.mod h1:otE2jQekW/PqXk1Awf5lmfokJx4uwuqcj1ab5SpGeW0=
|
||||
github.com/airbusgeo/errs v0.0.3 h1:+lUcbfm45tPLz/DMUC38QT+y9N60aKrPv4MXqVb7Noc=
|
||||
github.com/airbusgeo/errs v0.0.3/go.mod h1:/U1fafLfbdxZXtIqdTGeGGzLqnZcnGJXsJ0S7TaP4P0=
|
||||
github.com/airbusgeo/godal v0.0.18 h1:NQJ9P7OgSs5zVtlWH/tOta7QHYbbtha7Nttf63YM01Q=
|
||||
github.com/airbusgeo/godal v0.0.18/go.mod h1:Mpn/6exBzEbIx4DaG2rIjCmAiF9Ww4SJN2ZtwVsSwCo=
|
||||
github.com/airbusgeo/osio v0.1.3 h1:IxNf1co8lxwqJCziV+5qZyZbkT6JYdmR3L0oomNffrk=
|
||||
github.com/airbusgeo/osio v0.1.3/go.mod h1:d4qlaDSgsMilFoi+/62vwFZStuYIHIDovYzIRjTK+Jw=
|
||||
github.com/caarlos0/env/v11 v11.4.1 h1:fYwH0sWEsBSMPG7t4e/PEfTFzrWrpjyygXyUnWiSwEw=
|
||||
github.com/caarlos0/env/v11 v11.4.1/go.mod h1:qupehSf/Y0TUTsxKywqRt/vJjN5nz6vauiYEUUr8P4U=
|
||||
github.com/cespare/xxhash/v2 v2.3.0 h1:UL815xU9SqsFlibzuggzjXhog7bL6oX9BbNZnL2UFvs=
|
||||
github.com/cespare/xxhash/v2 v2.3.0/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs=
|
||||
github.com/cncf/xds/go v0.0.0-20250121191232-2f005788dc42 h1:Om6kYQYDUk5wWbT0t0q6pvyM49i9XZAv9dDrkDA7gjk=
|
||||
github.com/cncf/xds/go v0.0.0-20250121191232-2f005788dc42/go.mod h1:W+zGtBO5Y1IgJhy4+A9GOqVhqLpfZi+vwmdNXUehLA8=
|
||||
github.com/cpuguy83/go-md2man/v2 v2.0.6/go.mod h1:oOW0eioCTA6cOiMLiUPZOpcVxMig6NIQQ7OS05n1F4g=
|
||||
github.com/creack/pty v1.1.9/go.mod h1:oKZEueFk5CKHvIhNR5MUki03XCEU+Q6VDXinZuGJ33E=
|
||||
github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
|
||||
github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
|
||||
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
|
||||
github.com/dustin/go-humanize v1.0.1 h1:GzkhY7T5VNhEkwH0PVJgjz+fX1rhBrR7pRT3mDkpeCY=
|
||||
github.com/dustin/go-humanize v1.0.1/go.mod h1:Mu1zIs6XwVuF/gI1OepvI0qD18qycQx+mFykh5fBlto=
|
||||
github.com/envoyproxy/go-control-plane/envoy v1.32.4 h1:jb83lalDRZSpPWW2Z7Mck/8kXZ5CQAFYVjQcdVIr83A=
|
||||
github.com/envoyproxy/go-control-plane/envoy v1.32.4/go.mod h1:Gzjc5k8JcJswLjAx1Zm+wSYE20UrLtt7JZMWiWQXQEw=
|
||||
github.com/envoyproxy/protoc-gen-validate v1.2.1 h1:DEo3O99U8j4hBFwbJfrz9VtgcDfUKS7KJ7spH3d86P8=
|
||||
github.com/envoyproxy/protoc-gen-validate v1.2.1/go.mod h1:d/C80l/jxXLdfEIhX1W2TmLfsJ31lvEjwamM4DxlWXU=
|
||||
github.com/felixge/httpsnoop v1.0.4 h1:NFTV2Zj1bL4mc9sqWACXbQFVBBg2W3GPvqp8/ESS2Wg=
|
||||
github.com/felixge/httpsnoop v1.0.4/go.mod h1:m8KPJKqk1gH5J9DgRY2ASl2lWCfGKXixSwevea8zH2U=
|
||||
github.com/gabriel-vasile/mimetype v1.4.13 h1:46nXokslUBsAJE/wMsp5gtO500a4F3Nkz9Ufpk2AcUM=
|
||||
github.com/gabriel-vasile/mimetype v1.4.13/go.mod h1:d+9Oxyo1wTzWdyVUPMmXFvp4F9tea18J8ufA774AB3s=
|
||||
github.com/go-chi/chi/v5 v5.3.0 h1:halUjDxhshgXHMrao5bB8eNBXo/rnzwr8m5m36glehM=
|
||||
github.com/go-chi/chi/v5 v5.3.0/go.mod h1:R+tYY2hNuVUUjxoPtqUdgBqevM9s9njzkTLutVsOCto=
|
||||
github.com/go-jose/go-jose/v4 v4.1.5 h1:RjgjO2LOtWOJKUC5wpwY9LR3B3vwVAz6JS2YHfYU6eA=
|
||||
github.com/go-jose/go-jose/v4 v4.1.5/go.mod h1:x4oUasVrzR7071A4TnHLGSPpNOm2a21K9Kf04k1rs08=
|
||||
github.com/go-logr/logr v1.4.3 h1:CjnDlHq8ikf6E492q6eKboGOC0T8CDaOvkHCIg8idEI=
|
||||
github.com/go-logr/logr v1.4.3/go.mod h1:9T104GzyrTigFIr8wt5mBrctHMim0Nb2HLGrmQ40KvY=
|
||||
github.com/go-logr/stdr v1.2.2 h1:hSWxHoqTgW2S2qGc0LTAI563KZ5YKYRhT3MFKZMbjag=
|
||||
github.com/go-logr/stdr v1.2.2/go.mod h1:mMo/vtBO5dYbehREoey6XUKy/eSumjCCveDpRre4VKE=
|
||||
github.com/go-playground/assert/v2 v2.2.0 h1:JvknZsQTYeFEAhQwI4qEt9cyV5ONwRHC+lYKSsYSR8s=
|
||||
github.com/go-playground/assert/v2 v2.2.0/go.mod h1:VDjEfimB/XKnb+ZQfWdccd7VUvScMdVu0Titje2rxJ4=
|
||||
github.com/go-playground/locales v0.14.1 h1:EWaQ/wswjilfKLTECiXz7Rh+3BjFhfDFKv/oXslEjJA=
|
||||
@@ -23,8 +63,16 @@ github.com/go-playground/validator/v10 v10.30.3 h1:4MU6YkEwx7GbcPJOZxrtbu+QfF3pJ
|
||||
github.com/go-playground/validator/v10 v10.30.3/go.mod h1:4Axh7oCNGcoGkqLoE4YWt6n20mcEIsPRlB7vPk3lpyc=
|
||||
github.com/google/pprof v0.0.0-20250317173921-a4b03ec1a45e h1:ijClszYn+mADRFY17kjQEVQ1XRhq2/JR1M3sGqeJoxs=
|
||||
github.com/google/pprof v0.0.0-20250317173921-a4b03ec1a45e/go.mod h1:boTsfXsheKC2y+lKOCMpSfarhxDeIzfZG1jqGcPl3cA=
|
||||
github.com/google/s2a-go v0.1.9 h1:LGD7gtMgezd8a/Xak7mEWL0PjoTQFvpRudN895yqKW0=
|
||||
github.com/google/s2a-go v0.1.9/go.mod h1:YA0Ei2ZQL3acow2O62kdp9UlnvMmU7kA6Eutn0dXayM=
|
||||
github.com/google/uuid v1.6.0 h1:NIvaJDMOsjHA8n1jAhLSgzrAzy1Hgr+hNrb57e+94F0=
|
||||
github.com/google/uuid v1.6.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
|
||||
github.com/googleapis/enterprise-certificate-proxy v0.3.5 h1:VgzTY2jogw3xt39CusEnFJWm7rlsq5yL5q9XdLOuP5g=
|
||||
github.com/googleapis/enterprise-certificate-proxy v0.3.5/go.mod h1:MkHOF77EYAE7qfSuSS9PU6g4Nt4e11cnsDUowfwewLA=
|
||||
github.com/googleapis/gax-go/v2 v2.14.1 h1:hb0FFeiPaQskmvakKu5EbCbpntQn48jyHuvrkurSS/Q=
|
||||
github.com/googleapis/gax-go/v2 v2.14.1/go.mod h1:Hb/NubMaVM88SrNkvl8X/o8XWwDJEPqouaLeN2IUxoA=
|
||||
github.com/hashicorp/golang-lru v1.0.2 h1:dV3g9Z/unq5DpblPpw+Oqcv4dU/1omnb4Ok8iPY6p1c=
|
||||
github.com/hashicorp/golang-lru v1.0.2/go.mod h1:iADmTwqILo4mZ8BN3D2Q6+9jd8WM5uGBxy+E8yxSoD4=
|
||||
github.com/hashicorp/golang-lru/v2 v2.0.7 h1:a+bsQ5rvGLjzHuww6tVxozPZFVghXaHOwFs4luLUK2k=
|
||||
github.com/hashicorp/golang-lru/v2 v2.0.7/go.mod h1:QeFd9opnmA6QUJc5vARoKUSoFhyfM2/ZepoAG6RGpeM=
|
||||
github.com/inconshreveable/mousetrap v1.1.0 h1:wN+x4NVGpMsO7ErUn/mUI3vEoE6Jt13X2s0bqwp9tc8=
|
||||
@@ -39,17 +87,13 @@ github.com/jackc/puddle/v2 v2.2.2 h1:PR8nw+E/1w0GLuRFSmiioY6UooMp6KJv0/61nB7icHo
|
||||
github.com/jackc/puddle/v2 v2.2.2/go.mod h1:vriiEXHvEE654aYKXXjOvZM39qJ0q+azkZFrfEOc3H4=
|
||||
github.com/joho/godotenv v1.5.1 h1:7eLL/+HRGLY0ldzfGMeQkb7vMd0as4CfYvUVzLqw0N0=
|
||||
github.com/joho/godotenv v1.5.1/go.mod h1:f4LDr5Voq0i2e/R5DDNOoa2zzDfwtkZa6DnEwAbqwq4=
|
||||
github.com/klauspost/compress v1.18.6 h1:2jupLlAwFm95+YDR+NwD2MEfFO9d4z4Prjl1XXDjuao=
|
||||
github.com/klauspost/compress v1.18.6/go.mod h1:cwPg85FWrGar70rWktvGQj8/hthj3wpl0PGDogxkrSQ=
|
||||
github.com/klauspost/compress v1.18.7 h1:aUyZsS4kH3QTKurYhAOwAHxllVPnOthb3vPfnF1Ehjw=
|
||||
github.com/klauspost/compress v1.18.7/go.mod h1:cwPg85FWrGar70rWktvGQj8/hthj3wpl0PGDogxkrSQ=
|
||||
github.com/klauspost/cpuid/v2 v2.0.1/go.mod h1:FInQzS24/EEf25PyTYn52gqo7WaD8xa0213Md/qVLRg=
|
||||
github.com/klauspost/cpuid/v2 v2.2.11 h1:0OwqZRYI2rFrjS4kvkDnqJkKHdHaRnCm68/DY4OxRzU=
|
||||
github.com/klauspost/cpuid/v2 v2.2.11/go.mod h1:hqwkgyIinND0mEev00jJYCxPNVRVXFQeu1XKlok6oO0=
|
||||
github.com/klauspost/crc32 v1.3.0 h1:sSmTt3gUt81RP655XGZPElI0PelVTZ6YwCRnPSupoFM=
|
||||
github.com/klauspost/crc32 v1.3.0/go.mod h1:D7kQaZhnkX/Y0tstFGf8VUzv2UofNGqCjnC3zdHB0Hw=
|
||||
github.com/kr/pretty v0.3.0 h1:WgNl7dwNpEZ6jJ9k1snq4pZsg7DOEN8hP9Xw0Tsjwk0=
|
||||
github.com/kr/pretty v0.3.0/go.mod h1:640gp4NfQd8pI5XOwp5fnNeVWj67G7CFk/SaSQn7NBk=
|
||||
github.com/kr/text v0.2.0 h1:5Nx0Ya0ZqY2ygV366QzturHI13Jq95ApcVaJBhpS+AY=
|
||||
github.com/kr/text v0.2.0/go.mod h1:eLer722TekiGuMkidMxC/pM04lWEeraHUUmBw8l2grE=
|
||||
github.com/leodido/go-urn v1.4.0 h1:WT9HwE9SGECu3lg4d/dIA+jxlljEa1/ffXKmRjqdmIQ=
|
||||
github.com/leodido/go-urn v1.4.0/go.mod h1:bvxc+MVxLKB4z00jd1z+Dvzr47oO32F/QSNjSBOlFxI=
|
||||
github.com/mattn/go-isatty v0.0.21 h1:xYae+lCNBP7QuW4PUnNG61ffM4hVIfm+zUzDuSzYLGs=
|
||||
@@ -66,16 +110,15 @@ github.com/ncruces/go-strftime v1.0.0 h1:HMFp8mLCTPp341M/ZnA4qaf7ZlsbTc+miZjCLOF
|
||||
github.com/ncruces/go-strftime v1.0.0/go.mod h1:Fwc5htZGVVkseilnfgOVb9mKy6w1naJmn9CehxcKcls=
|
||||
github.com/philhofer/fwd v1.2.0 h1:e6DnBTl7vGY+Gz322/ASL4Gyp1FspeMvx1RNDoToZuM=
|
||||
github.com/philhofer/fwd v1.2.0/go.mod h1:RqIHx9QI14HlwKwm98g9Re5prTQ6LdeRQn+gXJFxsJM=
|
||||
github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
|
||||
github.com/planetscale/vtprotobuf v0.6.1-0.20240319094008-0393e58bdf10 h1:GFCKgmp0tecUJ0sJuv4pzYCqS9+RGSn52M3FUwPs+uo=
|
||||
github.com/planetscale/vtprotobuf v0.6.1-0.20240319094008-0393e58bdf10/go.mod h1:t/avpk3KcrXxUnYOhZhMXJlSEyie6gQbtLq5NM3loB8=
|
||||
github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
|
||||
github.com/pressly/goose/v3 v3.27.1 h1:6uEvcprBybDmW4hcz3gYujhARhye+GoWKhEWyzD5sh4=
|
||||
github.com/pressly/goose/v3 v3.27.1/go.mod h1:maruOxsPnIG2yHHyo8UqKWXYKFcH7Q76csUV7+7KYoM=
|
||||
github.com/rabbitmq/amqp091-go v1.12.0 h1:V0v14Iqfs+MwHWihJt/nGS5Ulu0vw572b2Co3mwunkI=
|
||||
github.com/rabbitmq/amqp091-go v1.12.0/go.mod h1:Hy4jKW5kQART1u+JkDTF9YYOQUHXqMuhrgxOEeS7G4o=
|
||||
github.com/rabbitmq/amqp091-go v1.13.0 h1:L8NA1WtF76C6KA3LAoufjfLgbist/If1UQYcsOjtxXA=
|
||||
github.com/rabbitmq/amqp091-go v1.13.0/go.mod h1:Hy4jKW5kQART1u+JkDTF9YYOQUHXqMuhrgxOEeS7G4o=
|
||||
github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec h1:W09IVJc94icq4NjY3clb7Lk8O1qJ8BdBEF8z0ibU0rE=
|
||||
github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec/go.mod h1:qqbHyh8v60DhA7CoWK5oRCqLrMHRGoxYCSS9EjAz6Eo=
|
||||
github.com/rogpeppe/go-internal v1.14.1 h1:UQB4HGPB6osV0SQTLymcB4TgvyWu6ZyliaW0tI/otEQ=
|
||||
github.com/rogpeppe/go-internal v1.14.1/go.mod h1:MaRKkUm5W0goXpeCfT7UZI6fk/L7L7so1lCWt35ZSgc=
|
||||
github.com/rs/xid v1.6.0 h1:fV591PaemRlL6JfRxGDEPl69wICngIQ3shQtzfy2gxU=
|
||||
github.com/rs/xid v1.6.0/go.mod h1:7XoLgs4eV+QndskICGsho+ADou8ySMSjJKDIan90Nz0=
|
||||
github.com/russross/blackfriday/v2 v2.1.0/go.mod h1:+Rmxgy9KzJVeS9/2gXHxylqXiyQDYRxCVz55jmeOWTM=
|
||||
@@ -86,6 +129,8 @@ github.com/spf13/cobra v1.10.2/go.mod h1:7C1pvHqHw5A4vrJfjNwvOdzYu0Gml16OCs2GRiT
|
||||
github.com/spf13/pflag v1.0.9/go.mod h1:McXfInJRrz4CZXVZOBLb0bTZqETkiAhM9Iw0y3An2Bg=
|
||||
github.com/spf13/pflag v1.0.10 h1:4EBh2KAYBwaONj6b2Ye1GiHfwjqyROoF4RwYO+vPwFk=
|
||||
github.com/spf13/pflag v1.0.10/go.mod h1:McXfInJRrz4CZXVZOBLb0bTZqETkiAhM9Iw0y3An2Bg=
|
||||
github.com/spiffe/go-spiffe/v2 v2.9.0 h1:g5qTg3qIP1HXV+GzuyEfSJFVxUCpEP5Ui1OINVJHPB4=
|
||||
github.com/spiffe/go-spiffe/v2 v2.9.0/go.mod h1:w2CLWKLMTX/PPYUEUPv3ltH0RXsw5S8suwNF46w9/Aw=
|
||||
github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
|
||||
github.com/stretchr/objx v0.4.0/go.mod h1:YvHI0jy2hoMjB+UWwv71VJQ9isScKT/TqJzVSSt89Yw=
|
||||
github.com/stretchr/objx v0.5.0/go.mod h1:Yh+to48EsGEfYuaHDzXPcE3xhTkx73EhmCGUpEOglKo=
|
||||
@@ -95,41 +140,74 @@ github.com/stretchr/testify v1.7.0/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/
|
||||
github.com/stretchr/testify v1.7.1/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
|
||||
github.com/stretchr/testify v1.8.0/go.mod h1:yNjHg4UonilssWZ8iaSj1OCr/vHnekPRkoO+kdMU+MU=
|
||||
github.com/stretchr/testify v1.8.4/go.mod h1:sz/lmYIOXD/1dqDmKjjqLyZ2RngseejIcXlSw2iwfAo=
|
||||
github.com/stretchr/testify v1.11.1 h1:7s2iGBzp5EwR7/aIZr8ao5+dra3wiQyKjjFuvgVKu7U=
|
||||
github.com/stretchr/testify v1.11.1/go.mod h1:wZwfW3scLgRK+23gO65QZefKpKQRnfz6sD981Nm4B6U=
|
||||
github.com/stretchr/testify v1.12.1 h1:EuwCh5fleGS7H32xRwO3wRGT7DxrDhLAT6FF8MpWDWE=
|
||||
github.com/stretchr/testify v1.12.1/go.mod h1:MDEgiDPPsNp5cuIrHPPCyornHKgEVbtFUmoNlxoYthg=
|
||||
github.com/tinylib/msgp v1.6.1 h1:ESRv8eL3u+DNHUoSAAQRE50Hm162zqAnBoGv9PzScPY=
|
||||
github.com/tinylib/msgp v1.6.1/go.mod h1:RSp0LW9oSxFut3KzESt5Voq4GVWyS+PSulT77roAqEA=
|
||||
github.com/zeebo/assert v1.3.0 h1:g7C04CbJuIDKNPFHmsk4hwZDO5O+kntRxzaUoNXj+IQ=
|
||||
github.com/zeebo/assert v1.3.0/go.mod h1:Pq9JiuJQpG8JLJdtkwrJESF0Foym2/D9XMU5ciN/wJ0=
|
||||
github.com/zeebo/xxh3 v1.1.0 h1:s7DLGDK45Dyfg7++yxI0khrfwq9661w9EN78eP/UZVs=
|
||||
github.com/zeebo/xxh3 v1.1.0/go.mod h1:IisAie1LELR4xhVinxWS5+zf1lA4p0MW4T+w+W07F5s=
|
||||
go.opentelemetry.io/auto/sdk v1.2.1 h1:jXsnJ4Lmnqd11kwkBV2LgLoFMZKizbCi5fNZ/ipaZ64=
|
||||
go.opentelemetry.io/auto/sdk v1.2.1/go.mod h1:KRTj+aOaElaLi+wW1kO/DZRXwkF4C5xPbEe3ZiIhN7Y=
|
||||
go.opentelemetry.io/contrib/detectors/gcp v1.34.0 h1:JRxssobiPg23otYU5SbWtQC//snGVIM3Tx6QRzlQBao=
|
||||
go.opentelemetry.io/contrib/detectors/gcp v1.34.0/go.mod h1:cV4BMFcscUR/ckqLkbfQmF0PRsq8w/lMGzdbCSveBHo=
|
||||
go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc v0.59.0 h1:rgMkmiGfix9vFJDcDi1PK8WEQP4FLQwLDfhp5ZLpFeE=
|
||||
go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc v0.59.0/go.mod h1:ijPqXp5P6IRRByFVVg9DY8P5HkxkHE5ARIa+86aXPf4=
|
||||
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.68.0 h1:CqXxU8VOmDefoh0+ztfGaymYbhdB/tT3zs79QaZTNGY=
|
||||
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.68.0/go.mod h1:BuhAPThV8PBHBvg8ZzZ/Ok3idOdhWIodywz2xEcRbJo=
|
||||
go.opentelemetry.io/otel v1.43.0 h1:mYIM03dnh5zfN7HautFE4ieIig9amkNANT+xcVxAj9I=
|
||||
go.opentelemetry.io/otel v1.43.0/go.mod h1:JuG+u74mvjvcm8vj8pI5XiHy1zDeoCS2LB1spIq7Ay0=
|
||||
go.opentelemetry.io/otel/metric v1.43.0 h1:d7638QeInOnuwOONPp4JAOGfbCEpYb+K6DVWvdxGzgM=
|
||||
go.opentelemetry.io/otel/metric v1.43.0/go.mod h1:RDnPtIxvqlgO8GRW18W6Z/4P462ldprJtfxHxyKd2PY=
|
||||
go.opentelemetry.io/otel/sdk v1.34.0 h1:95zS4k/2GOy069d321O8jWgYsW3MzVV+KuSPKp7Wr1A=
|
||||
go.opentelemetry.io/otel/sdk v1.34.0/go.mod h1:0e/pNiaMAqaykJGKbi+tSjWfNNHMTxoC9qANsCzbyxU=
|
||||
go.opentelemetry.io/otel/sdk/metric v1.34.0 h1:5CeK9ujjbFVL5c1PhLuStg1wxA7vQv7ce1EK0Gyvahk=
|
||||
go.opentelemetry.io/otel/sdk/metric v1.34.0/go.mod h1:jQ/r8Ze28zRKoNRdkjCZxfs6YvBTG1+YIqyFVFYec5w=
|
||||
go.opentelemetry.io/otel/trace v1.43.0 h1:BkNrHpup+4k4w+ZZ86CZoHHEkohws8AY+WTX09nk+3A=
|
||||
go.opentelemetry.io/otel/trace v1.43.0/go.mod h1:/QJhyVBUUswCphDVxq+8mld+AvhXZLhe+8WVFxiFff0=
|
||||
go.uber.org/goleak v1.3.0 h1:2K3zAYmnTNqV73imy9J1T3WC+gmCePx2hEGkimedGto=
|
||||
go.uber.org/goleak v1.3.0/go.mod h1:CoHD4mav9JJNrW/WLlf7HGZPjdw8EucARQHekz1X6bE=
|
||||
go.uber.org/multierr v1.11.0 h1:blXXJkSxSSfBVBlC76pxqeO+LN3aDfLQo+309xJstO0=
|
||||
go.uber.org/multierr v1.11.0/go.mod h1:20+QtiLqy0Nd6FdQB9TLXag12DsQkrbs3htMFfDN80Y=
|
||||
go.yaml.in/yaml/v3 v3.0.4 h1:tfq32ie2Jv2UxXFdLJdh3jXuOzWiL1fo0bu/FbuKpbc=
|
||||
go.yaml.in/yaml/v3 v3.0.4/go.mod h1:DhzuOOF2ATzADvBadXxruRBLzYTpT36CKvDb3+aBEFg=
|
||||
golang.org/x/crypto v0.52.0 h1:RMs7fP2rXdep0CftQlK8Uf+kibLm7qkCcradZWYz988=
|
||||
golang.org/x/crypto v0.52.0/go.mod h1:1QgfPxDqh0T2M/elOJtp9RvuR95kVjir0e6/BvEmGbc=
|
||||
golang.org/x/mod v0.36.0 h1:JJjpVx6myfUsUdAzZuOSTTmRE0PfZeNWzzvKrP7amb4=
|
||||
golang.org/x/mod v0.36.0/go.mod h1:moc6ELqsWcOw5Ef3xVprK5ul/MvtVvkIXLziUOICjUQ=
|
||||
golang.org/x/net v0.54.0 h1:2zJIZAxAHV/OHCDTCOHAYehQzLfSXuf/5SoL/Dv6w/w=
|
||||
golang.org/x/net v0.54.0/go.mod h1:Sj4oj8jK6XmHpBZU/zWHw3BV3abl4Kvi+Ut7cQcY+cQ=
|
||||
golang.org/x/sync v0.20.0 h1:e0PTpb7pjO8GAtTs2dQ6jYa5BWYlMuX047Dco/pItO4=
|
||||
golang.org/x/sync v0.20.0/go.mod h1:9xrNwdLfx4jkKbNva9FpL6vEN7evnE43NNNJQ2LF3+0=
|
||||
golang.org/x/sys v0.45.0 h1:dO4czNzziLiiXplLQgBCEpCvXQ3dnkn0SdaZSYdQ+FY=
|
||||
golang.org/x/sys v0.45.0/go.mod h1:4GL1E5IUh+htKOUEOaiffhrAeqysfVGipDYzABqnCmw=
|
||||
golang.org/x/text v0.37.0 h1:Cqjiwd9eSg8e0QAkyCaQTNHFIIzWtidPahFWR83rTrc=
|
||||
golang.org/x/text v0.37.0/go.mod h1:a5sjxXGs9hsn/AJVwuElvCAo9v8QYLzvavO5z2PiM38=
|
||||
golang.org/x/tools v0.45.0 h1:18qN3FAooORvApf5XjCXgsuayZOEtXf6JK18I3+ONa8=
|
||||
golang.org/x/tools v0.45.0/go.mod h1:LuUGqqaXcXMEFEruIVJVm5mgDD8vww/z/SR1gQ4uE/0=
|
||||
go.yaml.in/yaml/v3 v3.0.5 h1:N6y/pJk8buWs9NY5ERU2HSMfm+IuD/OtfdAnq6kESPw=
|
||||
go.yaml.in/yaml/v3 v3.0.5/go.mod h1:HVTZu1O7/Vkt2N+BFy8Zza+lnLsABggaTM2ZpNIGuKg=
|
||||
golang.org/x/crypto v0.57.0 h1:3ZVCjf8Ggz7zneR/EHRVx68Ctf+2pmIMP2UFhh9cC6M=
|
||||
golang.org/x/crypto v0.57.0/go.mod h1:Fdz0i5U6CoizGwLda9DttjSk6qlZo25zYNtR+ycvuZA=
|
||||
golang.org/x/mod v0.41.0 h1:qJmnOUb4YB+FsEuM3HcWucdZASCPGhsX6uljO6pog0c=
|
||||
golang.org/x/mod v0.41.0/go.mod h1:Ek9pY8RKWXwsWvd3rQiHYtMqkjSUV+s1Rj7j4H5Ur6o=
|
||||
golang.org/x/net v0.60.0 h1:79p50tfZlm0J9YfoDsSi639qSXNGVwEzOPLCxM2FsYU=
|
||||
golang.org/x/net v0.60.0/go.mod h1:2DA/G1UfVbCpQPeWTmMPGY7Cs2PkBkwu743bVX5PIVg=
|
||||
golang.org/x/oauth2 v0.28.0 h1:CrgCKl8PPAVtLnU3c+EDw6x11699EWlsDeWNWKdIOkc=
|
||||
golang.org/x/oauth2 v0.28.0/go.mod h1:onh5ek6nERTohokkhCD/y2cV4Do3fxFHFuAejCkRWT8=
|
||||
golang.org/x/sync v0.23.0 h1:KameEIfc1IkluZyXWLn39Wd4tURc6GbCiISGiZm2bQk=
|
||||
golang.org/x/sync v0.23.0/go.mod h1:sUUOizhqBxiL6pEWpqNLUiaJn1ShEbZ6BBqskPbjZm0=
|
||||
golang.org/x/sys v0.48.0 h1:bbX/i/6MgT9BVLM9RT1thmxL04yeTAhbEz4SyadbXoo=
|
||||
golang.org/x/sys v0.48.0/go.mod h1:hNLxWAXmnKAxqDtdwIYC4bM9oQPEecfsnNMuSxOs3og=
|
||||
golang.org/x/text v0.42.0 h1:JbOZXgfeCPU9gacVtYliJqOhD+zhrEqK4LfdpmlUZqI=
|
||||
golang.org/x/text v0.42.0/go.mod h1:ojzP1Z+2QtioaF8DTtO8K5q7JWVVYwZKenzujK0Zd0E=
|
||||
golang.org/x/time v0.10.0 h1:3usCWA8tQn0L8+hFJQNgzpWbd89begxN66o1Ojdn5L4=
|
||||
golang.org/x/time v0.10.0/go.mod h1:3BpzKBy/shNhVucY/MWOyx10tF3SFh9QdLuxbVysPQM=
|
||||
golang.org/x/tools v0.49.0 h1:3NI7VXzL9+1WZD52Dx2ttoPwD5DWrFGpl9mFZDlmisI=
|
||||
golang.org/x/tools v0.49.0/go.mod h1:SJNXV9DBKT0UbdttsQjbfJlAE/q+y36++zo3uL3N0Oo=
|
||||
google.golang.org/api v0.224.0 h1:Ir4UPtDsNiwIOHdExr3fAj4xZ42QjK7uQte3lORLJwU=
|
||||
google.golang.org/api v0.224.0/go.mod h1:3V39my2xAGkodXy0vEqcEtkqgw2GtrFL5WuBZlCTCOQ=
|
||||
google.golang.org/genproto v0.0.0-20250303144028-a0af3efb3deb h1:ITgPrl429bc6+2ZraNSzMDk3I95nmQln2fuPstKwFDE=
|
||||
google.golang.org/genproto v0.0.0-20250303144028-a0af3efb3deb/go.mod h1:sAo5UzpjUwgFBCzupwhcLcxHVDK7vG5IqI30YnwX2eE=
|
||||
google.golang.org/genproto/googleapis/api v0.0.0-20250303144028-a0af3efb3deb h1:p31xT4yrYrSM/G4Sn2+TNUkVhFCbG9y8itM2S6Th950=
|
||||
google.golang.org/genproto/googleapis/api v0.0.0-20250303144028-a0af3efb3deb/go.mod h1:jbe3Bkdp+Dh2IrslsFCklNhweNTBgSYanP1UXhJDhKg=
|
||||
google.golang.org/genproto/googleapis/rpc v0.0.0-20260420184626-e10c466a9529 h1:XF8+t6QQiS0o9ArVan/HW8Q7cycNPGsJf6GA2nXxYAg=
|
||||
google.golang.org/genproto/googleapis/rpc v0.0.0-20260420184626-e10c466a9529/go.mod h1:4Hqkh8ycfw05ld/3BWL7rJOSfebL2Q+DVDeRgYgxUU8=
|
||||
google.golang.org/grpc v1.80.0 h1:Xr6m2WmWZLETvUNvIUmeD5OAagMw3FiKmMlTdViWsHM=
|
||||
google.golang.org/grpc v1.80.0/go.mod h1:ho/dLnxwi3EDJA4Zghp7k2Ec1+c2jqup0bFkw07bwF4=
|
||||
google.golang.org/protobuf v1.36.12 h1:pJOKDDOyeXErUroCihFAd5LQuwXBSpVnKGrj5o/fwxc=
|
||||
google.golang.org/protobuf v1.36.12/go.mod h1:HTf+CrKn2C3g5S8VImy6tdcUvCska2kB7j23XfzDpco=
|
||||
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
|
||||
gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c h1:Hei/4ADfdWqJk1ZMxUNpqntNwaWcugrBjAiHlqqRiVk=
|
||||
gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c/go.mod h1:JHkPIbrfpd72SG/EVd6muEfDQjcINNoR0C8j2r3qZ4Q=
|
||||
gopkg.in/ini.v1 v1.67.2 h1:JtOSMb9OuaCZKr7h5D/h6iii14sK0hLbplTc6frx4Ss=
|
||||
gopkg.in/ini.v1 v1.67.2/go.mod h1:x/cyOwCgZqOkJoDIJ3c1KNHMo10+nLGAhh+kn3Zizss=
|
||||
gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
|
||||
gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
|
||||
gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
|
||||
modernc.org/cc/v4 v4.28.4 h1:Hd/4Es+MBj+/7hSdZaisNyu6bv3V0Dp2MdllyfqaH+c=
|
||||
modernc.org/cc/v4 v4.28.4/go.mod h1:OnovgIhbbMXMu1aISnJ0wvVD1KnW+cAUJkIrAWh+kVI=
|
||||
|
||||
@@ -38,6 +38,8 @@ type App struct {
|
||||
publisher *rabbitmq.Publisher
|
||||
categories *service.CategoryService
|
||||
datasets *service.DatasetService
|
||||
groups *service.DatasetGroupService
|
||||
landscape *service.LandscapeService
|
||||
eventRepo *postgres.EventRepository
|
||||
}
|
||||
|
||||
@@ -68,10 +70,16 @@ func New(ctx context.Context) (*App, error) {
|
||||
|
||||
categoryRepo := postgres.NewCategoryRepository(pool)
|
||||
datasetRepo := postgres.NewDatasetRepository(pool)
|
||||
groupRepo := postgres.NewDatasetGroupRepository(pool)
|
||||
eventRepo := postgres.NewEventRepository(pool)
|
||||
|
||||
lpRepo := postgres.NewLandscapeRepository(pool)
|
||||
|
||||
publisher := rabbitmq.NewPublisher(rabbit)
|
||||
jobPublisher := rabbitmq.NewDatasetJobPublisher(publisher)
|
||||
lpJobs := rabbitmq.NewLandscapeJobPublisher(publisher)
|
||||
|
||||
datasetsSvc := service.NewDatasetService(datasetRepo, store, categoryRepo, groupRepo, jobPublisher, parser.Columns, parser.Rows, raster.NewGDALConverter())
|
||||
|
||||
return &App{
|
||||
Cfg: cfg,
|
||||
@@ -81,30 +89,47 @@ func New(ctx context.Context) (*App, error) {
|
||||
rabbit: rabbit,
|
||||
publisher: publisher,
|
||||
categories: service.NewCategoryService(categoryRepo),
|
||||
datasets: service.NewDatasetService(datasetRepo, store, categoryRepo, jobPublisher, parser.Columns, parser.Rows, raster.NewGDALConverter()),
|
||||
eventRepo: eventRepo,
|
||||
datasets: datasetsSvc,
|
||||
groups: service.NewDatasetGroupService(groupRepo, datasetRepo, store),
|
||||
// A nil ComputeFunc selects landscape.ComputeWithWorkers, so each node
|
||||
// gets the per-node share of the configured worker budget.
|
||||
landscape: service.NewLandscapeService(
|
||||
lpRepo, datasetRepo, groupRepo, categoryRepo, datasetsSvc, store,
|
||||
lpJobs, raster.NewGodalIO(), nil, log,
|
||||
service.LandscapeOptions{
|
||||
MaxWorkers: cfg.Landscape.MaxWorkers,
|
||||
MaxConcurrentNodes: cfg.Landscape.MaxConcurrentNodes,
|
||||
},
|
||||
),
|
||||
eventRepo: eventRepo,
|
||||
}, nil
|
||||
}
|
||||
|
||||
// Handler builds the HTTP handler with all routes and readiness checks wired.
|
||||
func (a *App) Handler() stdhttp.Handler {
|
||||
validate := validator.New(validator.WithRequiredStructEnabled())
|
||||
validate.RegisterValidation("slug", func(fl validator.FieldLevel) bool {
|
||||
if err := validate.RegisterValidation("slug", func(fl validator.FieldLevel) bool {
|
||||
return domain.ValidSlug(fl.Field().String())
|
||||
})
|
||||
}); err != nil {
|
||||
// Only fails for an empty tag or nil func: a programming error.
|
||||
panic(fmt.Sprintf("register slug validation: %v", err))
|
||||
}
|
||||
|
||||
health := transporthttp.NewHealthHandler(map[string]transporthttp.ReadinessCheck{
|
||||
"postgres": func(ctx context.Context) error { return a.pool.Ping(ctx) },
|
||||
"s3": func(ctx context.Context) error { return a.store.Ping(ctx) },
|
||||
"rabbitmq": func(_ context.Context) error { return a.rabbit.Ping() },
|
||||
})
|
||||
}, a.Log)
|
||||
|
||||
return transporthttp.NewRouter(transporthttp.RouterDeps{
|
||||
Logger: a.Log,
|
||||
Health: health,
|
||||
Categories: transporthttp.NewCategoryHandler(a.categories, validate),
|
||||
Datasets: transporthttp.NewDatasetHandler(a.datasets, validate),
|
||||
OpenAPISpec: api.Spec,
|
||||
Logger: a.Log,
|
||||
Health: health,
|
||||
Categories: transporthttp.NewCategoryHandler(a.categories, validate),
|
||||
Datasets: transporthttp.NewDatasetHandler(a.datasets, validate),
|
||||
DatasetGroups: transporthttp.NewDatasetGroupHandler(a.groups, validate),
|
||||
Landscape: transporthttp.NewLandscapeHandler(a.landscape, validate),
|
||||
OpenAPISpec: api.Spec,
|
||||
MaxBodyBytes: a.Cfg.HTTP.MaxUploadBytes,
|
||||
})
|
||||
}
|
||||
|
||||
@@ -120,10 +145,20 @@ func (a *App) Consumers() []*rabbitmq.Consumer {
|
||||
a.PropertiesConsumer(),
|
||||
a.ExtractConsumer(),
|
||||
a.ConvertConsumer(),
|
||||
a.LandscapeConsumer(),
|
||||
a.ExampleConsumer(),
|
||||
}
|
||||
}
|
||||
|
||||
// LandscapeConsumer builds the Landscape Program run consumer.
|
||||
func (a *App) LandscapeConsumer() *rabbitmq.Consumer {
|
||||
handler := rabbitmq.NewLandscapeRunHandler(a.landscape, a.Log)
|
||||
return rabbitmq.NewConsumer(
|
||||
a.rabbit, rabbitmq.LandscapeRunQueue, rabbitmq.LandscapeRunRoutingKey,
|
||||
"gis-landscape-runner", handler, a.Log,
|
||||
)
|
||||
}
|
||||
|
||||
// PropertiesConsumer builds the plain-vector properties-extraction consumer.
|
||||
func (a *App) PropertiesConsumer() *rabbitmq.Consumer {
|
||||
handler := rabbitmq.NewPropertiesHandler(a.datasets, a.Log)
|
||||
@@ -178,11 +213,12 @@ func (a *App) ReprocessDataset(ctx context.Context, id uuid.UUID) (domain.Datase
|
||||
return a.datasets.Reprocess(ctx, id)
|
||||
}
|
||||
|
||||
// ReprocessAllDatasets re-enqueues the processing job for every dataset,
|
||||
// restarting each one's asynchronous pipeline. It returns how many jobs were
|
||||
// enqueued and a per-dataset map of any failures.
|
||||
func (a *App) ReprocessAllDatasets(ctx context.Context) (int, map[uuid.UUID]error, error) {
|
||||
return a.datasets.ReprocessAll(ctx)
|
||||
// ReprocessAllDatasets re-enqueues the processing job for every dataset
|
||||
// matching fileType (nil matches every file type), restarting each one's
|
||||
// asynchronous pipeline. It returns how many jobs were enqueued and a
|
||||
// per-dataset map of any failures.
|
||||
func (a *App) ReprocessAllDatasets(ctx context.Context, fileType *domain.FileType) (int, map[uuid.UUID]error, error) {
|
||||
return a.datasets.ReprocessAll(ctx, fileType)
|
||||
}
|
||||
|
||||
// Close releases all resources in reverse order of acquisition.
|
||||
|
||||
@@ -2,31 +2,53 @@ package cli
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"strings"
|
||||
|
||||
"gis/internal/app"
|
||||
"gis/internal/domain"
|
||||
|
||||
"github.com/google/uuid"
|
||||
"github.com/spf13/cobra"
|
||||
)
|
||||
|
||||
var reprocessAll bool
|
||||
var (
|
||||
reprocessAll bool
|
||||
reprocessType string
|
||||
)
|
||||
|
||||
var reprocessCmd = &cobra.Command{
|
||||
Use: "reprocess [dataset-id]",
|
||||
Short: "Re-enqueue the processing job for uploaded datasets",
|
||||
Long: "Re-publish the RabbitMQ message that drives an uploaded dataset's\n" +
|
||||
"asynchronous processing, selecting the right step from its file type\n" +
|
||||
"(vector_with_kato -> parse, vector -> properties, raster -> cog).\n\n" +
|
||||
"(vector_with_kato -> parse, vector -> properties, raster -> cog).\n" +
|
||||
"Before re-enqueuing, each dataset's stored geometry and GeoJSON are\n" +
|
||||
"cleared so a stale (e.g. previously miscomputed) value is never served\n" +
|
||||
"while the job is pending.\n\n" +
|
||||
"Pass a dataset id to reprocess a single dataset, or --all to reprocess\n" +
|
||||
"every dataset.\n\n" +
|
||||
"every dataset. --type restricts --all to one file type\n" +
|
||||
"(vector_with_kato, vector, or raster).\n\n" +
|
||||
"Examples:\n" +
|
||||
" gis reprocess 06818b2b-1fc5-47d9-a764-db2d4cb3df75\n" +
|
||||
" gis reprocess --all",
|
||||
" gis reprocess --all\n" +
|
||||
" gis reprocess --all --type=vector",
|
||||
Args: cobra.MaximumNArgs(1),
|
||||
RunE: func(cmd *cobra.Command, args []string) error {
|
||||
if reprocessAll == (len(args) == 1) {
|
||||
return fmt.Errorf("provide exactly one of <dataset-id> or --all")
|
||||
}
|
||||
if reprocessType != "" && !reprocessAll {
|
||||
return fmt.Errorf("--type can only be used together with --all")
|
||||
}
|
||||
|
||||
var fileType *domain.FileType
|
||||
if reprocessType != "" {
|
||||
ft := domain.FileType(strings.TrimSpace(reprocessType))
|
||||
if !ft.Valid() {
|
||||
return fmt.Errorf("invalid --type %q: must be one of vector_with_kato, vector, raster", reprocessType)
|
||||
}
|
||||
fileType = &ft
|
||||
}
|
||||
|
||||
ctx, cancel := signalContext()
|
||||
defer cancel()
|
||||
@@ -38,7 +60,7 @@ var reprocessCmd = &cobra.Command{
|
||||
defer application.Close()
|
||||
|
||||
if reprocessAll {
|
||||
enqueued, failures, err := application.ReprocessAllDatasets(ctx)
|
||||
enqueued, failures, err := application.ReprocessAllDatasets(ctx, fileType)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
@@ -47,7 +69,7 @@ var reprocessCmd = &cobra.Command{
|
||||
"dataset_id", id, "error", ferr)
|
||||
}
|
||||
application.Log.Info("re-enqueued all dataset processing",
|
||||
"enqueued", enqueued, "failed", len(failures))
|
||||
"file_type", reprocessType, "enqueued", enqueued, "failed", len(failures))
|
||||
if len(failures) > 0 {
|
||||
return fmt.Errorf("%d of %d datasets failed to re-enqueue", len(failures), enqueued+len(failures))
|
||||
}
|
||||
@@ -71,4 +93,5 @@ var reprocessCmd = &cobra.Command{
|
||||
|
||||
func init() {
|
||||
reprocessCmd.Flags().BoolVar(&reprocessAll, "all", false, "reprocess every dataset")
|
||||
reprocessCmd.Flags().StringVar(&reprocessType, "type", "", "restrict --all to one file type (vector_with_kato, vector, raster)")
|
||||
}
|
||||
|
||||
@@ -15,7 +15,7 @@ var publishExample bool
|
||||
|
||||
var workerCmd = &cobra.Command{
|
||||
Use: "worker",
|
||||
Short: "Run the RabbitMQ consumers (dataset parsing + example)",
|
||||
Short: "Run the RabbitMQ consumers (dataset processing, landscape runs + example)",
|
||||
RunE: func(cmd *cobra.Command, args []string) error {
|
||||
ctx, cancel := signalContext()
|
||||
defer cancel()
|
||||
|
||||
@@ -12,10 +12,11 @@ import (
|
||||
// Config holds all configuration for the application. Values are read from
|
||||
// environment variables; a local .env file (if present) is loaded first.
|
||||
type Config struct {
|
||||
HTTP HTTPConfig
|
||||
DB DBConfig
|
||||
S3 S3Config
|
||||
RabbitMQ RabbitMQConfig
|
||||
HTTP HTTPConfig
|
||||
DB DBConfig
|
||||
S3 S3Config
|
||||
RabbitMQ RabbitMQConfig
|
||||
Landscape LandscapeConfig
|
||||
}
|
||||
|
||||
// HTTPConfig configures the HTTP server.
|
||||
@@ -26,6 +27,9 @@ type HTTPConfig struct {
|
||||
WriteTimeout time.Duration `env:"HTTP_WRITE_TIMEOUT" envDefault:"120s"`
|
||||
IdleTimeout time.Duration `env:"HTTP_IDLE_TIMEOUT" envDefault:"60s"`
|
||||
ShutdownTimeout time.Duration `env:"HTTP_SHUTDOWN_TIMEOUT" envDefault:"10s"`
|
||||
// MaxUploadBytes caps the size of any request body, multipart uploads
|
||||
// included (default 2 GiB); larger requests get 413. 0 disables the cap.
|
||||
MaxUploadBytes int64 `env:"HTTP_MAX_UPLOAD_BYTES" envDefault:"2147483648"`
|
||||
}
|
||||
|
||||
// Addr returns the listen address for the HTTP server.
|
||||
@@ -58,6 +62,19 @@ type RabbitMQConfig struct {
|
||||
Queue string `env:"RABBITMQ_QUEUE" envDefault:"gis.events.example"`
|
||||
}
|
||||
|
||||
// LandscapeConfig bounds the Landscape Program's parallelism, and with it the
|
||||
// worker's peak memory: a run holds up to MaxConcurrentNodes nodes in flight,
|
||||
// each running a pool of MaxWorkers/MaxConcurrentNodes goroutines that own one
|
||||
// window buffer per input and output. The defaults (8 total workers spread
|
||||
// over 2 nodes) keep that product bounded regardless of how many cores the
|
||||
// host has.
|
||||
type LandscapeConfig struct {
|
||||
// MaxWorkers is the total window-worker budget shared by all nodes of a run.
|
||||
MaxWorkers int `env:"LP_MAX_WORKERS" envDefault:"8"`
|
||||
// MaxConcurrentNodes is how many graph nodes of one stage run at a time.
|
||||
MaxConcurrentNodes int `env:"LP_MAX_NODES" envDefault:"2"`
|
||||
}
|
||||
|
||||
// Load reads configuration from the environment, loading an optional .env file
|
||||
// from the current working directory first.
|
||||
func Load() (*Config, error) {
|
||||
|
||||
@@ -71,18 +71,30 @@ type DatasetFilter struct {
|
||||
FileType *FileType
|
||||
Automated *bool
|
||||
Status *string
|
||||
// DatasetGroupID restricts the listing to one dataset group.
|
||||
DatasetGroupID *uuid.UUID
|
||||
// StartDateFrom / StartDateTo and EndDateFrom / EndDateTo are inclusive
|
||||
// YYYY-MM-DD bounds on the dataset's period columns.
|
||||
StartDateFrom *string
|
||||
StartDateTo *string
|
||||
EndDateFrom *string
|
||||
EndDateTo *string
|
||||
}
|
||||
|
||||
// Observation is a single unpivoted value from a dataset's attribute table,
|
||||
// keyed by KATO code and date. Exactly one of Value / ValueText is typically
|
||||
// set (numeric vs non-numeric cell); both may be nil for an empty cell.
|
||||
// keyed by KATO code and date (optionally a StartDate..EndDate period). Exactly
|
||||
// one of Value / ValueText is typically set (numeric vs non-numeric cell); both
|
||||
// may be nil for an empty cell.
|
||||
type Observation struct {
|
||||
ID uuid.UUID `json:"id"`
|
||||
DatasetID uuid.UUID `json:"dataset_id"`
|
||||
KatoCode string `json:"kato_code"`
|
||||
Date string `json:"date"`
|
||||
Value *float64 `json:"value"`
|
||||
ValueText *string `json:"value_text"`
|
||||
StartDate string `json:"start_date"`
|
||||
// EndDate is the period's end date. Nil for a point-in-time observation
|
||||
// (pre-existing behavior).
|
||||
EndDate *string `json:"end_date,omitempty"`
|
||||
Value *float64 `json:"value"`
|
||||
ValueText *string `json:"value_text"`
|
||||
}
|
||||
|
||||
// allowedExtensions lists the accepted lowercase file extensions (including the
|
||||
@@ -115,9 +127,23 @@ func ExtensionAllowedFor(ft FileType, ext string) bool {
|
||||
func ValidateFileContent(ext string, head []byte) error {
|
||||
switch ext {
|
||||
case ".tif", ".tiff":
|
||||
// TIFF: little-endian "II*\0" or big-endian "MM\0*".
|
||||
if !bytes.HasPrefix(head, []byte("II*\x00")) && !bytes.HasPrefix(head, []byte("MM\x00*")) {
|
||||
return fmt.Errorf("file is not a valid TIFF/GeoTIFF")
|
||||
// Classic TIFF: little-endian "II*\0" or big-endian "MM\0*".
|
||||
// BigTIFF (version 43) uses "II+\0" / "MM\0+"; GDAL writes it for
|
||||
// large rasters (BIGTIFF=IF_SAFER), so the pipeline's own COG output
|
||||
// must pass this check too.
|
||||
tiffMagic := [][]byte{
|
||||
[]byte("II*\x00"), []byte("MM\x00*"),
|
||||
[]byte("II+\x00"), []byte("MM\x00+"),
|
||||
}
|
||||
ok := false
|
||||
for _, m := range tiffMagic {
|
||||
if bytes.HasPrefix(head, m) {
|
||||
ok = true
|
||||
break
|
||||
}
|
||||
}
|
||||
if !ok {
|
||||
return fmt.Errorf("file is not a valid TIFF/BigTIFF GeoTIFF")
|
||||
}
|
||||
case ".zip":
|
||||
// ZIP local-file or empty-archive signature.
|
||||
@@ -134,10 +160,30 @@ func ValidateFileContent(ext string, head []byte) error {
|
||||
if b, ok := firstMeaningfulByte(head); !ok || b != '{' {
|
||||
return fmt.Errorf("file is not valid GeoJSON")
|
||||
}
|
||||
case ".sld", ".xml":
|
||||
// SLD is XML; a lightweight sniff for the opening tag.
|
||||
if b, ok := firstMeaningfulByte(head); !ok || b != '<' {
|
||||
return fmt.Errorf("file is not valid XML/SLD")
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// AllowedSLDExtensions lists the accepted lowercase extensions (including the
|
||||
// dot) for a dataset's SLD (Styled Layer Descriptor) style file.
|
||||
var AllowedSLDExtensions = []string{".sld", ".xml"}
|
||||
|
||||
// ExtensionAllowedForSLD reports whether ext (lowercase, with dot) is a valid
|
||||
// SLD file extension.
|
||||
func ExtensionAllowedForSLD(ext string) bool {
|
||||
for _, e := range AllowedSLDExtensions {
|
||||
if e == ext {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
// firstMeaningfulByte returns the first non-whitespace byte after an optional
|
||||
// UTF-8 BOM.
|
||||
func firstMeaningfulByte(head []byte) (byte, bool) {
|
||||
@@ -160,18 +206,30 @@ type AttributeColumn struct {
|
||||
Samples []string `json:"samples,omitempty"`
|
||||
}
|
||||
|
||||
// YearColumn maps an attribute column to the date it represents,
|
||||
// e.g. {"column": "F_2023", "date": "2023-01-01"}.
|
||||
// YearColumn maps an attribute column to the date (or date period) it
|
||||
// represents, e.g. {"column": "F_2023", "start_date": "2023-01-01"} for a
|
||||
// single point in time, or {"column": "F_2020_2023", "start_date":
|
||||
// "2020-01-01", "end_date": "2023-12-31"} for a period. EndDate is optional;
|
||||
// when nil the mapping is a single point in time, preserving the original
|
||||
// (pre-period) behavior.
|
||||
type YearColumn struct {
|
||||
Column string `json:"column"`
|
||||
Date string `json:"date"`
|
||||
Column string `json:"column"`
|
||||
StartDate string `json:"start_date"`
|
||||
EndDate *string `json:"end_date,omitempty"`
|
||||
}
|
||||
|
||||
// DatasetSummary is the lightweight view of a dataset used in list responses.
|
||||
// It omits the heavy geometry/attribute/JSONB fields.
|
||||
type DatasetSummary struct {
|
||||
ID uuid.UUID `json:"id"`
|
||||
CategoryID uuid.UUID `json:"category_id"`
|
||||
ID uuid.UUID `json:"id"`
|
||||
CategoryID uuid.UUID `json:"category_id"`
|
||||
// DatasetGroupID links the dataset to its dataset group ("одноимённые"
|
||||
// datasets versioned by period). Nullable.
|
||||
DatasetGroupID *uuid.UUID `json:"dg_id"`
|
||||
// StartDate / EndDate is the period the dataset describes (YYYY-MM-DD).
|
||||
// Both nullable; EndDate drives the "freshest in group" rule.
|
||||
StartDate *string `json:"start_date"`
|
||||
EndDate *string `json:"end_date"`
|
||||
Name string `json:"name"`
|
||||
Description *string `json:"description"`
|
||||
Unit *string `json:"unit"`
|
||||
@@ -184,21 +242,33 @@ type DatasetSummary struct {
|
||||
|
||||
// Dataset is a geo file stored in the object store and grouped under a category.
|
||||
type Dataset struct {
|
||||
ID uuid.UUID `json:"id"`
|
||||
CategoryID uuid.UUID `json:"category_id"`
|
||||
Name string `json:"name"`
|
||||
Description *string `json:"description"`
|
||||
Unit *string `json:"unit"`
|
||||
Filename string `json:"filename"`
|
||||
StorageKey string `json:"storage_key"`
|
||||
ID uuid.UUID `json:"id"`
|
||||
CategoryID uuid.UUID `json:"category_id"`
|
||||
// DatasetGroupID links the dataset to its dataset group ("одноимённые"
|
||||
// datasets versioned by period). Nullable.
|
||||
DatasetGroupID *uuid.UUID `json:"dg_id"`
|
||||
// StartDate / EndDate is the period the dataset describes (YYYY-MM-DD).
|
||||
// Both nullable; EndDate drives the "freshest in group" rule.
|
||||
StartDate *string `json:"start_date"`
|
||||
EndDate *string `json:"end_date"`
|
||||
Name string `json:"name"`
|
||||
Description *string `json:"description"`
|
||||
Unit *string `json:"unit"`
|
||||
Filename string `json:"filename"`
|
||||
StorageKey string `json:"storage_key"`
|
||||
// CogStorageKey points to the Cloud-Optimized GeoTIFF for rasters. Nullable.
|
||||
CogStorageKey *string `json:"cog_storage_key"`
|
||||
FileType FileType `json:"file_type"`
|
||||
SizeBytes int64 `json:"size_bytes"`
|
||||
ContentType string `json:"content_type"`
|
||||
// SldStorageKey points to the dataset's SLD (Styled Layer Descriptor) style
|
||||
// file, if one has been uploaded. Nullable; applies to every file type.
|
||||
SldStorageKey *string `json:"sld_storage_key"`
|
||||
SldFilename *string `json:"sld_filename"`
|
||||
SldContentType *string `json:"sld_content_type"`
|
||||
// Properties holds tabular data extracted from the file (e.g. a shapefile's
|
||||
// attribute table). Nullable.
|
||||
Properties json.RawMessage `json:"properties"`
|
||||
// attribute table). Nullable; not exposed in dataset responses.
|
||||
Properties json.RawMessage `json:"-"`
|
||||
// Meta holds arbitrary user-defined data. Nullable.
|
||||
Meta json.RawMessage `json:"meta"`
|
||||
// Automated is a user-defined flag.
|
||||
@@ -215,8 +285,9 @@ type Dataset struct {
|
||||
// ParseError holds the failure reason when Status is failed. Nullable.
|
||||
ParseError *string `json:"parse_error"`
|
||||
// Geometry is the dataset's spatial geometry, serialized as GeoJSON.
|
||||
// Nullable; populated from the file's spatial data.
|
||||
Geometry json.RawMessage `json:"geometry"`
|
||||
// Nullable; populated from the file's spatial data. Not exposed in dataset
|
||||
// responses.
|
||||
Geometry json.RawMessage `json:"-"`
|
||||
// GeoJSON is the pre-assembled GeoJSON FeatureCollection served by the
|
||||
// .geojson endpoint. It is generated and persisted at processing time for
|
||||
// vector and vector_with_kato datasets. Nullable; not exposed on the dataset
|
||||
@@ -228,3 +299,37 @@ type Dataset struct {
|
||||
CreatedAt time.Time `json:"created_at"`
|
||||
UpdatedAt time.Time `json:"updated_at"`
|
||||
}
|
||||
|
||||
// DatasetFields is the set of editable metadata columns written by
|
||||
// UpdateFields. Nil pointers clear the corresponding nullable column.
|
||||
type DatasetFields struct {
|
||||
CategoryID uuid.UUID
|
||||
Name string
|
||||
Description *string
|
||||
Unit *string
|
||||
DatasetGroupID *uuid.UUID
|
||||
StartDate *string
|
||||
EndDate *string
|
||||
}
|
||||
|
||||
// ValidateDatePeriod checks that start/end (YYYY-MM-DD, either may be nil)
|
||||
// parse and that end is not before start. Returns an ErrValidation-wrapped
|
||||
// error naming the offending field.
|
||||
func ValidateDatePeriod(start, end *string) error {
|
||||
var from, to time.Time
|
||||
var err error
|
||||
if start != nil {
|
||||
if from, err = time.Parse("2006-01-02", *start); err != nil {
|
||||
return fmt.Errorf("%w: start_date %q is not YYYY-MM-DD", ErrValidation, *start)
|
||||
}
|
||||
}
|
||||
if end != nil {
|
||||
if to, err = time.Parse("2006-01-02", *end); err != nil {
|
||||
return fmt.Errorf("%w: end_date %q is not YYYY-MM-DD", ErrValidation, *end)
|
||||
}
|
||||
}
|
||||
if start != nil && end != nil && to.Before(from) {
|
||||
return fmt.Errorf("%w: end_date %q is before start_date %q", ErrValidation, *end, *start)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
23
internal/domain/dataset_group.go
Normal file
23
internal/domain/dataset_group.go
Normal file
@@ -0,0 +1,23 @@
|
||||
package domain
|
||||
|
||||
import (
|
||||
"time"
|
||||
|
||||
"github.com/google/uuid"
|
||||
)
|
||||
|
||||
// DatasetGroup groups "одноимённые" datasets (the same indicator delivered for
|
||||
// different periods). The freshest dataset of a group is the one with the
|
||||
// greatest end_date. A group may carry a shared SLD style used as a fallback by
|
||||
// datasets that have none of their own.
|
||||
type DatasetGroup struct {
|
||||
ID uuid.UUID `json:"id"`
|
||||
Code string `json:"code"`
|
||||
Name string `json:"name"`
|
||||
Description string `json:"description"`
|
||||
SldStorageKey *string `json:"sld_storage_key"`
|
||||
SldFilename *string `json:"sld_filename"`
|
||||
SldContentType *string `json:"sld_content_type"`
|
||||
CreatedAt time.Time `json:"created_at"`
|
||||
UpdatedAt time.Time `json:"updated_at"`
|
||||
}
|
||||
@@ -1,6 +1,9 @@
|
||||
package domain
|
||||
|
||||
import "testing"
|
||||
import (
|
||||
"errors"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestFileTypeValid(t *testing.T) {
|
||||
valid := []FileType{FileTypeVectorWithKato, FileTypeVector, FileTypeRaster}
|
||||
@@ -27,7 +30,10 @@ func TestValidateFileContent(t *testing.T) {
|
||||
}{
|
||||
{"valid tiff LE", ".tif", []byte("II*\x00rest"), true},
|
||||
{"valid tiff BE", ".tiff", []byte("MM\x00*rest"), true},
|
||||
{"valid bigtiff LE", ".tif", []byte("II+\x00rest"), true},
|
||||
{"valid bigtiff BE", ".tiff", []byte("MM\x00+rest"), true},
|
||||
{"bad tiff", ".tif", []byte("not a tiff"), false},
|
||||
{"bad tiff near-miss prefix", ".tif", []byte("II-\x00rest"), false},
|
||||
{"valid zip", ".zip", []byte("PK\x03\x04rest"), true},
|
||||
{"bad zip", ".zip", []byte("RARrest"), false},
|
||||
{"valid gpkg", ".gpkg", []byte("SQLite format 3\x00rest"), true},
|
||||
@@ -72,3 +78,31 @@ func TestExtensionAllowedFor(t *testing.T) {
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestValidateDatePeriod(t *testing.T) {
|
||||
s := func(v string) *string { return &v }
|
||||
cases := []struct {
|
||||
name string
|
||||
start, end *string
|
||||
wantErr bool
|
||||
}{
|
||||
{"both nil", nil, nil, false},
|
||||
{"start only", s("2020-01-01"), nil, false},
|
||||
{"ordered", s("2020-01-01"), s("2020-12-31"), false},
|
||||
{"same day", s("2020-01-01"), s("2020-01-01"), false},
|
||||
{"reversed", s("2020-12-31"), s("2020-01-01"), true},
|
||||
{"bad start", s("01.01.2020"), nil, true},
|
||||
{"bad end", nil, s("2020-13-01"), true},
|
||||
}
|
||||
for _, tc := range cases {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
err := ValidateDatePeriod(tc.start, tc.end)
|
||||
if (err != nil) != tc.wantErr {
|
||||
t.Fatalf("err=%v wantErr=%v", err, tc.wantErr)
|
||||
}
|
||||
if err != nil && !errors.Is(err, ErrValidation) {
|
||||
t.Fatalf("expected ErrValidation, got %v", err)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
68
internal/domain/landscape.go
Normal file
68
internal/domain/landscape.go
Normal file
@@ -0,0 +1,68 @@
|
||||
package domain
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"time"
|
||||
|
||||
"github.com/google/uuid"
|
||||
)
|
||||
|
||||
// LPRunStatus is the lifecycle status of a Landscape Program run.
|
||||
type LPRunStatus string
|
||||
|
||||
// Landscape Program run statuses.
|
||||
const (
|
||||
LPRunQueued LPRunStatus = "queued"
|
||||
LPRunRunning LPRunStatus = "running"
|
||||
LPRunSucceeded LPRunStatus = "succeeded"
|
||||
LPRunFailed LPRunStatus = "failed"
|
||||
)
|
||||
|
||||
// LPNodeStatus is the lifecycle status of a single node within a Landscape
|
||||
// Program run.
|
||||
type LPNodeStatus string
|
||||
|
||||
// Landscape Program run node statuses.
|
||||
const (
|
||||
LPNodePending LPNodeStatus = "pending"
|
||||
LPNodeRunning LPNodeStatus = "running"
|
||||
LPNodeSkipped LPNodeStatus = "skipped"
|
||||
LPNodeSucceeded LPNodeStatus = "succeeded"
|
||||
LPNodeFailed LPNodeStatus = "failed"
|
||||
)
|
||||
|
||||
// LPDatasetRef references a dataset produced for, or consumed from, a
|
||||
// dataset group slot in the Landscape Program graph.
|
||||
type LPDatasetRef struct {
|
||||
Group string `json:"group"`
|
||||
DatasetID uuid.UUID `json:"dataset_id"`
|
||||
}
|
||||
|
||||
// LPRun is a single execution of the Landscape Program graph.
|
||||
type LPRun struct {
|
||||
ID uuid.UUID `json:"id"`
|
||||
Status LPRunStatus `json:"status"`
|
||||
FromStage *int `json:"from_stage"`
|
||||
Params json.RawMessage `json:"params"`
|
||||
Overrides map[string]uuid.UUID `json:"overrides"`
|
||||
Error *string `json:"error"`
|
||||
CreatedAt time.Time `json:"created_at"`
|
||||
StartedAt *time.Time `json:"started_at"`
|
||||
FinishedAt *time.Time `json:"finished_at"`
|
||||
Nodes []LPRunNode `json:"nodes"`
|
||||
}
|
||||
|
||||
// LPRunNode is the execution state of a single graph node within an LPRun.
|
||||
type LPRunNode struct {
|
||||
ID uuid.UUID `json:"id"`
|
||||
RunID uuid.UUID `json:"run_id"`
|
||||
NodeCode string `json:"node_code"`
|
||||
Stage int `json:"stage"`
|
||||
Status LPNodeStatus `json:"status"`
|
||||
Fingerprint string `json:"fingerprint"`
|
||||
Inputs []LPDatasetRef `json:"inputs"`
|
||||
Outputs []LPDatasetRef `json:"outputs"`
|
||||
Error *string `json:"error"`
|
||||
StartedAt *time.Time `json:"started_at"`
|
||||
FinishedAt *time.Time `json:"finished_at"`
|
||||
}
|
||||
225
internal/landscape/compute.go
Normal file
225
internal/landscape/compute.go
Normal file
@@ -0,0 +1,225 @@
|
||||
package landscape
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"runtime"
|
||||
"sync"
|
||||
|
||||
"gis/internal/raster"
|
||||
|
||||
"golang.org/x/sync/errgroup"
|
||||
)
|
||||
|
||||
// MaxWorkersPerNode is the per-node worker cap Compute uses. Peak memory is
|
||||
// roughly nodes x workers x window^2 x (inputs+outputs), so callers that run
|
||||
// several nodes at once should divide their total worker budget between them
|
||||
// and call ComputeWithWorkers instead.
|
||||
const MaxWorkersPerNode = 8
|
||||
|
||||
// Compute runs one node over its input rasters with the default per-node
|
||||
// worker cap (MaxWorkersPerNode). See ComputeWithWorkers.
|
||||
func Compute(ctx context.Context, rio raster.IO, node Node, inputs map[string]string, outDir string, p Params) (map[string]string, error) {
|
||||
return ComputeWithWorkers(ctx, rio, node, inputs, outDir, p, MaxWorkersPerNode)
|
||||
}
|
||||
|
||||
// ComputeWithWorkers runs one node over its input rasters with an explicit
|
||||
// worker cap. See the package doc for the concurrency model: windows are
|
||||
// processed by a worker pool of at most workers goroutines; each worker owns
|
||||
// its own Reader handles (GDAL datasets are not goroutine-safe) and writes are
|
||||
// serialised per output. Window order does not affect the result. On any
|
||||
// error, no output files are left behind: ComputeWithWorkers best-effort
|
||||
// removes every output it created before returning. A workers value below 1 is
|
||||
// treated as 1.
|
||||
func ComputeWithWorkers(ctx context.Context, rio raster.IO, node Node, inputs map[string]string, outDir string, p Params, workers int) (map[string]string, error) {
|
||||
p = p.WithDefaults()
|
||||
if p.WindowSize <= 0 {
|
||||
// WithDefaults only fills in a zero value; guard against a negative
|
||||
// one too, since raster.Windows would otherwise loop forever.
|
||||
p.WindowSize = DefaultParams().WindowSize
|
||||
}
|
||||
paths := make([]string, len(node.Inputs))
|
||||
for i, g := range node.Inputs {
|
||||
path, ok := inputs[g]
|
||||
if !ok || path == "" {
|
||||
return nil, fmt.Errorf("node %s (%s): missing input %q", node.Code, node.Name, g)
|
||||
}
|
||||
paths[i] = path
|
||||
}
|
||||
|
||||
// Validate grids with one set of handles before creating anything.
|
||||
grid, err := checkGrids(ctx, rio, node, paths)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
outPaths := make(map[string]string, len(node.Outputs))
|
||||
writers := make([]raster.Writer, len(node.Outputs))
|
||||
createdPaths := make([]string, 0, len(node.Outputs))
|
||||
for j, g := range node.Outputs {
|
||||
path := filepath.Join(outDir, g+".tif")
|
||||
w, err := rio.Create(ctx, path, grid, raster.DefaultWriterOptions())
|
||||
if err != nil {
|
||||
closeWriters(writers[:j])
|
||||
removeOutputs(createdPaths)
|
||||
return nil, fmt.Errorf("node %s: create output %s: %w", node.Code, g, err)
|
||||
}
|
||||
writers[j] = w
|
||||
outPaths[g] = path
|
||||
createdPaths = append(createdPaths, path)
|
||||
}
|
||||
|
||||
windows := raster.Windows(grid.Width, grid.Height, p.WindowSize)
|
||||
// The first window raster.Windows produces is always the largest: it
|
||||
// starts at (0,0) so both its width and height are the un-clipped
|
||||
// min(WindowSize, dimension). Size per-worker buffers off it instead of
|
||||
// WindowSize*WindowSize, which would over-allocate whenever the raster is
|
||||
// smaller than one window (the common case for small/test rasters, and
|
||||
// for the last row/column of any raster).
|
||||
var size int
|
||||
if len(windows) > 0 {
|
||||
size = windows[0].Width * windows[0].Height
|
||||
}
|
||||
jobs := make(chan raster.Window)
|
||||
var writeMu sync.Mutex
|
||||
eg, gctx := errgroup.WithContext(ctx)
|
||||
// Cap workers well below GOMAXPROCS too: each worker holds a full set of
|
||||
// per-window input/output buffers, and nodes can have many inputs and
|
||||
// outputs (e.g. node 20: 6 inputs, 5 outputs), so an unbounded worker
|
||||
// count multiplies memory with core count and with concurrently running
|
||||
// nodes. The caller's budget is the third bound.
|
||||
workers = min(runtime.GOMAXPROCS(0), len(windows), max(1, workers))
|
||||
for w := 0; w < workers; w++ {
|
||||
eg.Go(func() error {
|
||||
readers, err := openAll(gctx, rio, node, paths)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
defer closeReaders(readers)
|
||||
in := make([][]float64, len(paths))
|
||||
for i := range in {
|
||||
in[i] = make([]float64, size)
|
||||
}
|
||||
out := make([][]int16, len(writers))
|
||||
for j := range out {
|
||||
out[j] = make([]int16, size)
|
||||
}
|
||||
for win := range jobs {
|
||||
n := win.Width * win.Height
|
||||
inView := make([][]float64, len(in))
|
||||
for i, r := range readers {
|
||||
if err := r.ReadFloat64(win, in[i][:n]); err != nil {
|
||||
return fmt.Errorf("node %s: read %s window %+v: %w", node.Code, node.Inputs[i], win, err)
|
||||
}
|
||||
inView[i] = in[i][:n]
|
||||
}
|
||||
outView := make([][]int16, len(out))
|
||||
for j := range out {
|
||||
outView[j] = out[j][:n]
|
||||
}
|
||||
node.Kernel(inView, outView, p)
|
||||
writeMu.Lock()
|
||||
for j, wr := range writers {
|
||||
if err := wr.WriteInt16(win, outView[j]); err != nil {
|
||||
writeMu.Unlock()
|
||||
return fmt.Errorf("node %s: write %s window %+v: %w", node.Code, node.Outputs[j], win, err)
|
||||
}
|
||||
}
|
||||
writeMu.Unlock()
|
||||
}
|
||||
return nil
|
||||
})
|
||||
}
|
||||
eg.Go(func() error {
|
||||
defer close(jobs)
|
||||
for _, win := range windows {
|
||||
select {
|
||||
case jobs <- win:
|
||||
case <-gctx.Done():
|
||||
return gctx.Err()
|
||||
}
|
||||
}
|
||||
return nil
|
||||
})
|
||||
runErr := eg.Wait()
|
||||
if err := closeWriters(writers); err != nil && runErr == nil {
|
||||
runErr = err
|
||||
}
|
||||
if runErr != nil {
|
||||
removeOutputs(createdPaths)
|
||||
return nil, runErr
|
||||
}
|
||||
return outPaths, nil
|
||||
}
|
||||
|
||||
// checkGrids opens every input once, verifies all grids match the first, and
|
||||
// returns that grid.
|
||||
func checkGrids(ctx context.Context, rio raster.IO, node Node, paths []string) (raster.Grid, error) {
|
||||
readers, err := openAll(ctx, rio, node, paths)
|
||||
if err != nil {
|
||||
return raster.Grid{}, err
|
||||
}
|
||||
defer closeReaders(readers)
|
||||
if len(readers) == 0 {
|
||||
return raster.Grid{}, fmt.Errorf("node %s (%s): has no inputs", node.Code, node.Name)
|
||||
}
|
||||
ref := readers[0].Grid()
|
||||
for i := 1; i < len(readers); i++ {
|
||||
if g := readers[i].Grid(); !rio.SameGrid(ref, g) {
|
||||
return raster.Grid{}, fmt.Errorf("node %s: input %s grid (%dx%d, %v) differs from %s (%dx%d, %v); inputs must be aligned",
|
||||
node.Code, node.Inputs[i], g.Width, g.Height, g.GeoTransform, node.Inputs[0], ref.Width, ref.Height, ref.GeoTransform)
|
||||
}
|
||||
}
|
||||
return ref, nil
|
||||
}
|
||||
|
||||
// openAll opens paths in order, wrapping any failure with the node code and
|
||||
// the input group name (paths is always node.Inputs resolved to files, in
|
||||
// the same order) for context.
|
||||
func openAll(ctx context.Context, rio raster.IO, node Node, paths []string) ([]raster.Reader, error) {
|
||||
readers := make([]raster.Reader, 0, len(paths))
|
||||
for i, p := range paths {
|
||||
r, err := rio.Open(ctx, p)
|
||||
if err != nil {
|
||||
closeReaders(readers)
|
||||
name := p
|
||||
if i < len(node.Inputs) {
|
||||
name = node.Inputs[i]
|
||||
}
|
||||
return nil, fmt.Errorf("node %s: input %s: %w", node.Code, name, err)
|
||||
}
|
||||
readers = append(readers, r)
|
||||
}
|
||||
return readers, nil
|
||||
}
|
||||
|
||||
func closeReaders(rs []raster.Reader) {
|
||||
for _, r := range rs {
|
||||
if r != nil {
|
||||
_ = r.Close()
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func closeWriters(ws []raster.Writer) error {
|
||||
var first error
|
||||
for _, w := range ws {
|
||||
if w == nil {
|
||||
continue
|
||||
}
|
||||
if err := w.Close(); err != nil && first == nil {
|
||||
first = err
|
||||
}
|
||||
}
|
||||
return first
|
||||
}
|
||||
|
||||
// removeOutputs best-effort deletes every path in paths, ignoring errors. It
|
||||
// is used to leave no partially-written output behind after a failed run.
|
||||
func removeOutputs(paths []string) {
|
||||
for _, p := range paths {
|
||||
_ = os.Remove(p)
|
||||
}
|
||||
}
|
||||
252
internal/landscape/compute_test.go
Normal file
252
internal/landscape/compute_test.go
Normal file
@@ -0,0 +1,252 @@
|
||||
package landscape
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"testing"
|
||||
|
||||
"gis/internal/raster"
|
||||
)
|
||||
|
||||
func grid(w, h int) raster.Grid {
|
||||
return raster.Grid{Width: w, Height: h, GeoTransform: [6]float64{0, 30, 0, 0, 0, -30}, ProjectionWKT: "EPSG:32642"}
|
||||
}
|
||||
|
||||
func constRaster(w, h int, v float64) []float64 {
|
||||
d := make([]float64, w*h)
|
||||
for i := range d {
|
||||
d[i] = v
|
||||
}
|
||||
return d
|
||||
}
|
||||
|
||||
func TestCompute_E_TilesWindowsAndWritesEveryPixel(t *testing.T) {
|
||||
fio := newFakeIO()
|
||||
g := grid(10, 7)
|
||||
z := constRaster(10, 7, 2)
|
||||
u := constRaster(10, 7, 4)
|
||||
z[0] = -9999 // pixel 0 invalid
|
||||
fio.put("z.tif", g, z)
|
||||
fio.put("u.tif", g, u)
|
||||
node, _ := NodeByCode("10")
|
||||
p := DefaultParams()
|
||||
p.WindowSize = 4 // forces partial windows (10x7 with 4x4 tiles)
|
||||
out, err := Compute(context.Background(), fio, node, map[string]string{OutZIntegrated: "z.tif", OutUIntegrated: "u.tif"}, "/out", p)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
path := out[OutE]
|
||||
if path != filepath.Join("/out", OutE+".tif") {
|
||||
t.Fatalf("unexpected output path %q", path)
|
||||
}
|
||||
res := fio.files[path]
|
||||
if res.grid != g {
|
||||
t.Fatal("output grid must copy the first input's grid")
|
||||
}
|
||||
if res.opts != raster.DefaultWriterOptions() {
|
||||
t.Fatalf("output created with %+v, want raster.DefaultWriterOptions() = %+v", res.opts, raster.DefaultWriterOptions())
|
||||
}
|
||||
for i, v := range res.data {
|
||||
want := 4.0
|
||||
if i == 0 {
|
||||
want = -9999
|
||||
}
|
||||
if v != want {
|
||||
t.Fatalf("pixel %d = %v want %v", i, v, want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestCompute_FailsOnGridMismatch(t *testing.T) {
|
||||
fio := newFakeIO()
|
||||
fio.put("z.tif", grid(10, 7), constRaster(10, 7, 2))
|
||||
fio.put("u.tif", grid(9, 7), constRaster(9, 7, 4))
|
||||
node, _ := NodeByCode("10")
|
||||
_, err := Compute(context.Background(), fio, node, map[string]string{OutZIntegrated: "z.tif", OutUIntegrated: "u.tif"}, "/out", DefaultParams())
|
||||
if err == nil {
|
||||
t.Fatal("expected grid mismatch error")
|
||||
}
|
||||
if _, created := fio.files[filepath.Join("/out", OutE+".tif")]; created {
|
||||
t.Fatal("no output must be created on mismatch")
|
||||
}
|
||||
}
|
||||
|
||||
func TestCompute_FailsOnMissingInput(t *testing.T) {
|
||||
fio := newFakeIO()
|
||||
fio.put("z.tif", grid(2, 2), constRaster(2, 2, 2))
|
||||
node, _ := NodeByCode("10")
|
||||
if _, err := Compute(context.Background(), fio, node, map[string]string{OutZIntegrated: "z.tif"}, "/out", DefaultParams()); err == nil {
|
||||
t.Fatal("expected error for missing input group")
|
||||
}
|
||||
}
|
||||
|
||||
func TestCompute_MultiOutputNode(t *testing.T) {
|
||||
fio := newFakeIO()
|
||||
g := grid(3, 1)
|
||||
paths := map[string]string{}
|
||||
for i, s := range []string{SlotPIrrigation, SlotPSnow, SlotPCrop, SlotPPasture, SlotPTourism} {
|
||||
paths[s] = s + ".tif"
|
||||
fio.put(paths[s], g, constRaster(3, 1, float64(i+1)))
|
||||
}
|
||||
paths[OutE] = "e.tif"
|
||||
fio.put("e.tif", g, constRaster(3, 1, 3))
|
||||
node, _ := NodeByCode("20")
|
||||
out, err := Compute(context.Background(), fio, node, paths, "/o", DefaultParams())
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if len(out) != 5 {
|
||||
t.Fatalf("want 5 outputs, got %d", len(out))
|
||||
}
|
||||
if fio.files[out[OutCTourism]].data[0] != 3 || fio.files[out[OutCIrrigation]].data[0] != 1 {
|
||||
t.Fatal("min(P, E) not applied per output")
|
||||
}
|
||||
}
|
||||
|
||||
// TestCompute_MultiOutputNode_TiledWindows runs the 6-input/5-output node 20
|
||||
// with a window size much smaller than the raster, so per-worker output
|
||||
// buffers are reused across windows of differing pixel counts (partial
|
||||
// windows at the raster's right/bottom edges).
|
||||
func TestCompute_MultiOutputNode_TiledWindows(t *testing.T) {
|
||||
fio := newFakeIO()
|
||||
w, h := 5, 3
|
||||
g := grid(w, h)
|
||||
paths := map[string]string{}
|
||||
for i, s := range []string{SlotPIrrigation, SlotPSnow, SlotPCrop, SlotPPasture, SlotPTourism} {
|
||||
paths[s] = s + ".tif"
|
||||
fio.put(paths[s], g, constRaster(w, h, float64(i+1)))
|
||||
}
|
||||
paths[OutE] = "e.tif"
|
||||
fio.put("e.tif", g, constRaster(w, h, 3))
|
||||
node, _ := NodeByCode("20")
|
||||
p := DefaultParams()
|
||||
p.WindowSize = 2 // 5x3 with 2x2 tiles: several differently-sized windows
|
||||
out, err := Compute(context.Background(), fio, node, paths, "/o2", p)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
codes := []string{OutCIrrigation, OutCSnow, OutCCrop, OutCPasture, OutCTourism}
|
||||
for i, code := range codes {
|
||||
want := min(i+1, 3) // min(P_sector, E=3)
|
||||
res := fio.files[out[code]]
|
||||
for px, v := range res.data {
|
||||
if v != float64(want) {
|
||||
t.Fatalf("%s pixel %d = %v want %v", code, px, v, want)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// TestCompute_ZeroParamsUsesDefaults exercises Compute with a completely
|
||||
// zero-valued Params, relying on WithDefaults to fill in WindowSize and the
|
||||
// other node-independent knobs.
|
||||
func TestCompute_ZeroParamsUsesDefaults(t *testing.T) {
|
||||
fio := newFakeIO()
|
||||
g := grid(4, 4)
|
||||
fio.put("z.tif", g, constRaster(4, 4, 2))
|
||||
fio.put("u.tif", g, constRaster(4, 4, 3))
|
||||
node, _ := NodeByCode("10")
|
||||
out, err := Compute(context.Background(), fio, node, map[string]string{OutZIntegrated: "z.tif", OutUIntegrated: "u.tif"}, "/z", Params{})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
for _, v := range fio.files[out[OutE]].data {
|
||||
if v != 3 {
|
||||
t.Fatalf("pixel = %v want 3", v)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// TestCompute_NegativeWindowSizeClampedToDefault guards against
|
||||
// raster.Windows looping forever on a non-positive window size: WithDefaults
|
||||
// only fills in a zero value, so Compute must clamp negatives itself.
|
||||
func TestCompute_NegativeWindowSizeClampedToDefault(t *testing.T) {
|
||||
fio := newFakeIO()
|
||||
g := grid(2, 2)
|
||||
fio.put("z.tif", g, constRaster(2, 2, 2))
|
||||
fio.put("u.tif", g, constRaster(2, 2, 3))
|
||||
node, _ := NodeByCode("10")
|
||||
p := DefaultParams()
|
||||
p.WindowSize = -5
|
||||
out, err := Compute(context.Background(), fio, node, map[string]string{OutZIntegrated: "z.tif", OutUIntegrated: "u.tif"}, "/neg", p)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if fio.files[out[OutE]].data[0] != 3 {
|
||||
t.Fatal("unexpected result")
|
||||
}
|
||||
}
|
||||
|
||||
// TestCompute_FailureMidRunReturnsError simulates a read failure partway
|
||||
// through a run (the fake fails every ReadFloat64 against u.tif from the
|
||||
// first call on) and checks Compute surfaces the error instead of returning
|
||||
// a partial result.
|
||||
func TestCompute_FailureMidRunReturnsError(t *testing.T) {
|
||||
fio := newFakeIO()
|
||||
g := grid(10, 7)
|
||||
fio.put("z.tif", g, constRaster(10, 7, 2))
|
||||
fio.put("u.tif", g, constRaster(10, 7, 4))
|
||||
fio.failReadAfterN("u.tif", 0) // fail starting from the very first read
|
||||
node, _ := NodeByCode("10")
|
||||
p := DefaultParams()
|
||||
p.WindowSize = 4
|
||||
if _, err := Compute(context.Background(), fio, node, map[string]string{OutZIntegrated: "z.tif", OutUIntegrated: "u.tif"}, "/out-fail", p); err == nil {
|
||||
t.Fatal("expected error from simulated read failure")
|
||||
}
|
||||
}
|
||||
|
||||
// TestRemoveOutputs_DeletesFiles unit-tests the cleanup helper Compute uses
|
||||
// after a failed run, against the real filesystem (the fake IO above has no
|
||||
// on-disk files to assert against).
|
||||
func TestRemoveOutputs_DeletesFiles(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
var paths []string
|
||||
for i := 0; i < 3; i++ {
|
||||
p := filepath.Join(dir, fmt.Sprintf("out%d.tif", i))
|
||||
if err := os.WriteFile(p, []byte("data"), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
paths = append(paths, p)
|
||||
}
|
||||
removeOutputs(paths)
|
||||
for _, p := range paths {
|
||||
if _, err := os.Stat(p); !os.IsNotExist(err) {
|
||||
t.Fatalf("expected %s to be removed, stat err = %v", p, err)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// TestComputeWithWorkers_RespectsCap checks the caller's worker budget bounds
|
||||
// the pool: each pool worker opens one reader per input and holds a buffer per
|
||||
// input and output, so the open count bounds peak memory. checkGrids opens one
|
||||
// extra set of handles before the pool starts, hence the (workers+1) bound.
|
||||
func TestComputeWithWorkers_RespectsCap(t *testing.T) {
|
||||
node, _ := NodeByCode("10")
|
||||
inputs := len(node.Inputs)
|
||||
|
||||
for _, workers := range []int{1, 2, 4, 0, -3} {
|
||||
t.Run(fmt.Sprintf("workers=%d", workers), func(t *testing.T) {
|
||||
fio := newFakeIO()
|
||||
g := grid(64, 64)
|
||||
fio.put("z.tif", g, constRaster(64, 64, 2))
|
||||
fio.put("u.tif", g, constRaster(64, 64, 4))
|
||||
p := DefaultParams()
|
||||
p.WindowSize = 8 // 64 windows, so the pool is never window-bound
|
||||
|
||||
if _, err := ComputeWithWorkers(context.Background(), fio, node,
|
||||
map[string]string{OutZIntegrated: "z.tif", OutUIntegrated: "u.tif"}, "/out", p, workers); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
effective := max(1, workers)
|
||||
if got, limit := fio.openCount(), (effective+1)*inputs; got > limit {
|
||||
t.Fatalf("opened %d readers with workers=%d, want at most %d", got, workers, limit)
|
||||
}
|
||||
if fio.openCount() < inputs {
|
||||
t.Fatalf("opened %d readers, want at least one set of %d", fio.openCount(), inputs)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
122
internal/landscape/fakeio_test.go
Normal file
122
internal/landscape/fakeio_test.go
Normal file
@@ -0,0 +1,122 @@
|
||||
package landscape
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"sync"
|
||||
|
||||
"gis/internal/raster"
|
||||
)
|
||||
|
||||
// fakeIO keeps rasters in memory keyed by path.
|
||||
type fakeIO struct {
|
||||
mu sync.Mutex
|
||||
files map[string]*fakeRaster
|
||||
|
||||
// failReadAfter, when set for a path, makes every ReadFloat64 call
|
||||
// against that path fail once the cumulative read count (across all
|
||||
// readers/workers) exceeds the configured threshold. Used to simulate a
|
||||
// mid-run I/O failure.
|
||||
failReadAfter map[string]int
|
||||
readCounts map[string]int
|
||||
|
||||
// opens counts every successful Open. Each pool worker opens one reader
|
||||
// per input, so the count bounds the number of concurrent handles.
|
||||
opens int
|
||||
}
|
||||
|
||||
// openCount returns how many readers have been opened so far.
|
||||
func (f *fakeIO) openCount() int {
|
||||
f.mu.Lock()
|
||||
defer f.mu.Unlock()
|
||||
return f.opens
|
||||
}
|
||||
|
||||
type fakeRaster struct {
|
||||
grid raster.Grid
|
||||
data []float64 // row-major
|
||||
opts raster.WriterOptions
|
||||
}
|
||||
|
||||
func newFakeIO() *fakeIO { return &fakeIO{files: map[string]*fakeRaster{}} }
|
||||
|
||||
func (f *fakeIO) put(path string, g raster.Grid, data []float64) {
|
||||
f.files[path] = &fakeRaster{grid: g, data: data}
|
||||
}
|
||||
|
||||
// failReadAfterN makes ReadFloat64 against path fail starting from the
|
||||
// (n+1)-th call to it.
|
||||
func (f *fakeIO) failReadAfterN(path string, n int) {
|
||||
f.mu.Lock()
|
||||
defer f.mu.Unlock()
|
||||
if f.failReadAfter == nil {
|
||||
f.failReadAfter = map[string]int{}
|
||||
f.readCounts = map[string]int{}
|
||||
}
|
||||
f.failReadAfter[path] = n
|
||||
}
|
||||
|
||||
type fakeReader struct {
|
||||
f *fakeIO
|
||||
path string
|
||||
r *fakeRaster
|
||||
}
|
||||
|
||||
func (r *fakeReader) Grid() raster.Grid { return r.r.grid }
|
||||
func (r *fakeReader) NoData() (float64, bool) { return -9999, true }
|
||||
func (r *fakeReader) Close() error { return nil }
|
||||
func (r *fakeReader) ReadFloat64(w raster.Window, dst []float64) error {
|
||||
if r.f != nil {
|
||||
r.f.mu.Lock()
|
||||
if n, ok := r.f.failReadAfter[r.path]; ok {
|
||||
r.f.readCounts[r.path]++
|
||||
if r.f.readCounts[r.path] > n {
|
||||
r.f.mu.Unlock()
|
||||
return fmt.Errorf("fake: simulated read failure for %s", r.path)
|
||||
}
|
||||
}
|
||||
r.f.mu.Unlock()
|
||||
}
|
||||
for y := 0; y < w.Height; y++ {
|
||||
copy(dst[y*w.Width:(y+1)*w.Width], r.r.data[(w.Y+y)*r.r.grid.Width+w.X:(w.Y+y)*r.r.grid.Width+w.X+w.Width])
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
type fakeWriter struct {
|
||||
f *fakeIO
|
||||
r *fakeRaster
|
||||
}
|
||||
|
||||
func (w *fakeWriter) WriteInt16(win raster.Window, src []int16) error {
|
||||
w.f.mu.Lock()
|
||||
defer w.f.mu.Unlock()
|
||||
for y := 0; y < win.Height; y++ {
|
||||
for x := 0; x < win.Width; x++ {
|
||||
w.r.data[(win.Y+y)*w.r.grid.Width+win.X+x] = float64(src[y*win.Width+x])
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
func (w *fakeWriter) Close() error { return nil }
|
||||
|
||||
func (f *fakeIO) Open(_ context.Context, path string) (raster.Reader, error) {
|
||||
f.mu.Lock()
|
||||
defer f.mu.Unlock()
|
||||
r, ok := f.files[path]
|
||||
if !ok {
|
||||
return nil, fmt.Errorf("fake: %s not found", path)
|
||||
}
|
||||
f.opens++
|
||||
return &fakeReader{f: f, path: path, r: r}, nil
|
||||
}
|
||||
|
||||
func (f *fakeIO) Create(_ context.Context, path string, g raster.Grid, o raster.WriterOptions) (raster.Writer, error) {
|
||||
f.mu.Lock()
|
||||
defer f.mu.Unlock()
|
||||
r := &fakeRaster{grid: g, data: make([]float64, g.Width*g.Height), opts: o}
|
||||
f.files[path] = r
|
||||
return &fakeWriter{f: f, r: r}, nil
|
||||
}
|
||||
|
||||
func (f *fakeIO) SameGrid(a, b raster.Grid) bool { return a == b }
|
||||
157
internal/landscape/graph.go
Normal file
157
internal/landscape/graph.go
Normal file
@@ -0,0 +1,157 @@
|
||||
// Package landscape defines the Landscape Program (ЛП) computation graph — the
|
||||
// per-pixel formulas that turn ecological (Z), vulnerability (U) and
|
||||
// suitability (P) class rasters into constraints (E), conflicts (C) and
|
||||
// functional zoning (V1–V3) — together with a window-based executor.
|
||||
package landscape
|
||||
|
||||
import "sort"
|
||||
|
||||
const (
|
||||
// NoData is the output nodata value of every LP raster.
|
||||
NoData int16 = -9999
|
||||
// CategoryCode is the dataset category holding every LP dataset.
|
||||
CategoryCode = "landscape"
|
||||
)
|
||||
|
||||
// Group codes. Slots are leaf inputs users upload; the rest are node outputs.
|
||||
const (
|
||||
SlotZBiodiversity = "lp-z-biodiversity"
|
||||
SlotZLCR = "lp-z-lcr"
|
||||
SlotZSoilProtection = "lp-z-soil-protection"
|
||||
SlotZWR = "lp-z-wr"
|
||||
SlotUSurfWater = "lp-u-surf-water"
|
||||
SlotUSoil = "lp-u-soil"
|
||||
SlotUCrop = "lp-u-crop"
|
||||
SlotUVeg = "lp-u-veg"
|
||||
SlotPIrrigation = "lp-p-irrigation"
|
||||
SlotPSnow = "lp-p-snow"
|
||||
SlotPCrop = "lp-p-crop"
|
||||
SlotPPasture = "lp-p-pasture"
|
||||
SlotPTourism = "lp-p-tourism"
|
||||
|
||||
OutZIntegrated = "lp-z-integrated"
|
||||
OutUIntegrated = "lp-u-integrated"
|
||||
OutZMax = "lp-z-max"
|
||||
OutUMax = "lp-u-max"
|
||||
OutE = "lp-e"
|
||||
OutCIrrigation = "lp-c-irrigation"
|
||||
OutCSnow = "lp-c-snow"
|
||||
OutCCrop = "lp-c-crop"
|
||||
OutCPasture = "lp-c-pasture"
|
||||
OutCTourism = "lp-c-tourism"
|
||||
OutPMax = "lp-p-max"
|
||||
OutV1 = "lp-v1-conservative"
|
||||
OutV2 = "lp-v2-balanced"
|
||||
OutV3 = "lp-v3-development"
|
||||
OutCMax = "lp-c-max"
|
||||
)
|
||||
|
||||
type Slot struct{ Code, Title string }
|
||||
type Output struct{ Code, Title string }
|
||||
|
||||
// Kernel computes one window. in[i] follows Node.Inputs order and out[j]
|
||||
// follows Node.Outputs order; every slice has exactly window-pixel length.
|
||||
type Kernel func(in [][]float64, out [][]int16, p Params)
|
||||
|
||||
// Node is one function of the graph. Stage = first digit of Code.
|
||||
type Node struct {
|
||||
Code, Name string
|
||||
Stage int
|
||||
Inputs []string
|
||||
Outputs []string
|
||||
Kernel Kernel
|
||||
}
|
||||
|
||||
var Slots = []Slot{
|
||||
{SlotZBiodiversity, "Z Biodiversity"}, {SlotZLCR, "Z LCR"}, {SlotZSoilProtection, "Z Soil Protection"}, {SlotZWR, "Z Water Resources"},
|
||||
{SlotUSurfWater, "U Surface Water"}, {SlotUSoil, "U Soil"}, {SlotUCrop, "U Crop"}, {SlotUVeg, "U Vegetation"},
|
||||
{SlotPIrrigation, "P Irrigation"}, {SlotPSnow, "P Snow"}, {SlotPCrop, "P Crop"}, {SlotPPasture, "P Pasture"}, {SlotPTourism, "P Tourism"},
|
||||
}
|
||||
|
||||
var Outputs = []Output{
|
||||
{OutZIntegrated, "Z Integrated"}, {OutUIntegrated, "U Integrated"}, {OutZMax, "Z Max"}, {OutUMax, "U Max"},
|
||||
{OutE, "E Ecological Constraint"},
|
||||
{OutCIrrigation, "C Irrigation"}, {OutCSnow, "C Snow"}, {OutCCrop, "C Crop"}, {OutCPasture, "C Pasture"}, {OutCTourism, "C Tourism"},
|
||||
{OutPMax, "P Max"},
|
||||
{OutV1, "V1 Conservative"}, {OutV2, "V2 Balanced"}, {OutV3, "V3 Development"}, {OutCMax, "C Max"},
|
||||
}
|
||||
|
||||
var zInputs = []string{SlotZBiodiversity, SlotZLCR, SlotZSoilProtection, SlotZWR}
|
||||
var uInputs = []string{SlotUSurfWater, SlotUSoil, SlotUCrop, SlotUVeg} // order matters: water, soil, crop, veg
|
||||
var pInputs = []string{SlotPIrrigation, SlotPSnow, SlotPCrop, SlotPPasture, SlotPTourism} // sector order 1..5
|
||||
var cOutputs = []string{OutCIrrigation, OutCSnow, OutCCrop, OutCPasture, OutCTourism}
|
||||
|
||||
var Nodes = []Node{
|
||||
{Code: "00", Name: "z_integrated", Stage: 0, Inputs: zInputs, Outputs: []string{OutZIntegrated}, Kernel: kernelZIntegrated},
|
||||
{Code: "01", Name: "u_integrated", Stage: 0, Inputs: uInputs, Outputs: []string{OutUIntegrated}, Kernel: kernelUIntegrated},
|
||||
{Code: "02", Name: "z_max", Stage: 0, Inputs: zInputs, Outputs: []string{OutZMax}, Kernel: kernelZMax},
|
||||
{Code: "03", Name: "u_max", Stage: 0, Inputs: uInputs, Outputs: []string{OutUMax}, Kernel: kernelUMax},
|
||||
{Code: "10", Name: "e_ecological_constraint", Stage: 1, Inputs: []string{OutZIntegrated, OutUIntegrated}, Outputs: []string{OutE}, Kernel: kernelE},
|
||||
{Code: "20", Name: "c_conflicts", Stage: 2, Inputs: append(append([]string{}, pInputs...), OutE), Outputs: cOutputs, Kernel: kernelCConflicts},
|
||||
{Code: "21", Name: "p_max", Stage: 2, Inputs: pInputs, Outputs: []string{OutPMax}, Kernel: kernelMaxAnyValid},
|
||||
// func_zoning ranks the original suitability rasters P1..P5 against E (the
|
||||
// reference notebook and zoning_core.py); the conflict layers C_k are
|
||||
// diagnostics and are not zoning inputs. It stays in stage 3 to keep the
|
||||
// diagram's folder numbering (30x) even though it only depends on stage 1.
|
||||
{Code: "30", Name: "func_zoning", Stage: 3, Inputs: append([]string{OutE}, pInputs...), Outputs: []string{OutV1, OutV2, OutV3}, Kernel: kernelFuncZoning},
|
||||
{Code: "31", Name: "c_max", Stage: 3, Inputs: cOutputs, Outputs: []string{OutCMax}, Kernel: kernelMaxAnyValid},
|
||||
}
|
||||
|
||||
func NodeByCode(code string) (Node, bool) {
|
||||
for _, n := range Nodes {
|
||||
if n.Code == code {
|
||||
return n, true
|
||||
}
|
||||
}
|
||||
return Node{}, false
|
||||
}
|
||||
|
||||
func SlotByCode(code string) (Slot, bool) {
|
||||
for _, s := range Slots {
|
||||
if s.Code == code {
|
||||
return s, true
|
||||
}
|
||||
}
|
||||
return Slot{}, false
|
||||
}
|
||||
|
||||
func OutputByCode(code string) (Output, bool) {
|
||||
for _, o := range Outputs {
|
||||
if o.Code == code {
|
||||
return o, true
|
||||
}
|
||||
}
|
||||
return Output{}, false
|
||||
}
|
||||
|
||||
// NodeForOutput returns the node producing the given output group.
|
||||
func NodeForOutput(group string) (Node, bool) {
|
||||
for _, n := range Nodes {
|
||||
for _, o := range n.Outputs {
|
||||
if o == group {
|
||||
return n, true
|
||||
}
|
||||
}
|
||||
}
|
||||
return Node{}, false
|
||||
}
|
||||
|
||||
// Stages groups nodes by stage, ascending; within a stage nodes are sorted by code.
|
||||
func Stages() [][]Node {
|
||||
byStage := map[int][]Node{}
|
||||
for _, n := range Nodes {
|
||||
byStage[n.Stage] = append(byStage[n.Stage], n)
|
||||
}
|
||||
keys := make([]int, 0, len(byStage))
|
||||
for k := range byStage {
|
||||
keys = append(keys, k)
|
||||
}
|
||||
sort.Ints(keys)
|
||||
out := make([][]Node, 0, len(keys))
|
||||
for _, k := range keys {
|
||||
ns := byStage[k]
|
||||
sort.Slice(ns, func(i, j int) bool { return ns[i].Code < ns[j].Code })
|
||||
out = append(out, ns)
|
||||
}
|
||||
return out
|
||||
}
|
||||
153
internal/landscape/graph_test.go
Normal file
153
internal/landscape/graph_test.go
Normal file
@@ -0,0 +1,153 @@
|
||||
package landscape
|
||||
|
||||
import (
|
||||
"regexp"
|
||||
"testing"
|
||||
)
|
||||
|
||||
var slugRe = regexp.MustCompile(`^[a-z0-9]+(?:-[a-z0-9]+)*$`)
|
||||
|
||||
// isSlot reports whether code names one of the leaf input slots.
|
||||
func isSlot(code string) bool {
|
||||
_, ok := SlotByCode(code)
|
||||
return ok
|
||||
}
|
||||
|
||||
func TestNodeInputsComeFromSlotsOrLowerStage(t *testing.T) {
|
||||
for _, n := range Nodes {
|
||||
for _, in := range n.Inputs {
|
||||
if isSlot(in) {
|
||||
continue
|
||||
}
|
||||
producer, ok := NodeForOutput(in)
|
||||
if !ok {
|
||||
t.Errorf("node %s: input %q is neither a slot nor a known node output", n.Code, in)
|
||||
continue
|
||||
}
|
||||
if producer.Stage >= n.Stage {
|
||||
t.Errorf("node %s (stage %d): input %q is produced by node %s at stage %d, want stage < %d",
|
||||
n.Code, n.Stage, in, producer.Code, producer.Stage, n.Stage)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestEveryOutputHasExactlyOneProducer(t *testing.T) {
|
||||
count := map[string]int{}
|
||||
for _, n := range Nodes {
|
||||
for _, o := range n.Outputs {
|
||||
count[o]++
|
||||
}
|
||||
}
|
||||
for _, o := range Outputs {
|
||||
if c := count[o.Code]; c != 1 {
|
||||
t.Errorf("output %q has %d producers, want exactly 1", o.Code, c)
|
||||
}
|
||||
}
|
||||
// No node should produce something that isn't in the Outputs list.
|
||||
for code, c := range count {
|
||||
if _, ok := OutputByCode(code); !ok {
|
||||
t.Errorf("node output %q (%d producers) is not declared in Outputs", code, c)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestStagesGroupingAndOrder(t *testing.T) {
|
||||
stages := Stages()
|
||||
if len(stages) != 4 {
|
||||
t.Fatalf("got %d stages, want 4", len(stages))
|
||||
}
|
||||
want := [][]string{
|
||||
{"00", "01", "02", "03"},
|
||||
{"10"},
|
||||
{"20", "21"},
|
||||
{"30", "31"},
|
||||
}
|
||||
for i, group := range stages {
|
||||
var codes []string
|
||||
for _, n := range group {
|
||||
codes = append(codes, n.Code)
|
||||
}
|
||||
if len(codes) != len(want[i]) {
|
||||
t.Fatalf("stage group %d: got codes %v, want %v", i, codes, want[i])
|
||||
}
|
||||
for j, c := range codes {
|
||||
if c != want[i][j] {
|
||||
t.Errorf("stage group %d: got codes %v, want %v", i, codes, want[i])
|
||||
break
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestAllGroupCodesAreValidSlugs(t *testing.T) {
|
||||
for _, s := range Slots {
|
||||
if !slugRe.MatchString(s.Code) {
|
||||
t.Errorf("slot code %q is not a valid slug", s.Code)
|
||||
}
|
||||
}
|
||||
for _, o := range Outputs {
|
||||
if !slugRe.MatchString(o.Code) {
|
||||
t.Errorf("output code %q is not a valid slug", o.Code)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestNodeByCode(t *testing.T) {
|
||||
n, ok := NodeByCode("00")
|
||||
if !ok || n.Name != "z_integrated" {
|
||||
t.Fatalf("NodeByCode(00) = %+v, %v", n, ok)
|
||||
}
|
||||
if _, ok := NodeByCode("99"); ok {
|
||||
t.Fatal("NodeByCode(99) should not be found")
|
||||
}
|
||||
}
|
||||
|
||||
func TestSlotByCode(t *testing.T) {
|
||||
s, ok := SlotByCode(SlotZBiodiversity)
|
||||
if !ok || s.Title != "Z Biodiversity" {
|
||||
t.Fatalf("SlotByCode(%s) = %+v, %v", SlotZBiodiversity, s, ok)
|
||||
}
|
||||
if _, ok := SlotByCode("nope"); ok {
|
||||
t.Fatal("SlotByCode(nope) should not be found")
|
||||
}
|
||||
}
|
||||
|
||||
func TestOutputByCode(t *testing.T) {
|
||||
o, ok := OutputByCode(OutE)
|
||||
if !ok || o.Title != "E Ecological Constraint" {
|
||||
t.Fatalf("OutputByCode(%s) = %+v, %v", OutE, o, ok)
|
||||
}
|
||||
if _, ok := OutputByCode("nope"); ok {
|
||||
t.Fatal("OutputByCode(nope) should not be found")
|
||||
}
|
||||
}
|
||||
|
||||
func TestNodeForOutput(t *testing.T) {
|
||||
n, ok := NodeForOutput(OutV1)
|
||||
if !ok || n.Code != "30" {
|
||||
t.Fatalf("NodeForOutput(%s) = %+v, %v", OutV1, n, ok)
|
||||
}
|
||||
if _, ok := NodeForOutput("nope"); ok {
|
||||
t.Fatal("NodeForOutput(nope) should not be found")
|
||||
}
|
||||
}
|
||||
|
||||
// Functional zoning ranks the ORIGINAL suitability rasters P1..P5 against E
|
||||
// (reference notebook cells "Three scenarios" and zoning_core.py); the
|
||||
// conflict layers C_k are diagnostics and must not replace P_k here.
|
||||
func TestFuncZoningTakesEAndOriginalP(t *testing.T) {
|
||||
n, ok := NodeByCode("30")
|
||||
if !ok {
|
||||
t.Fatal("node 30 missing")
|
||||
}
|
||||
want := []string{OutE, SlotPIrrigation, SlotPSnow, SlotPCrop, SlotPPasture, SlotPTourism}
|
||||
if len(n.Inputs) != len(want) {
|
||||
t.Fatalf("node 30 inputs = %v, want %v", n.Inputs, want)
|
||||
}
|
||||
for i := range want {
|
||||
if n.Inputs[i] != want[i] {
|
||||
t.Fatalf("node 30 input[%d] = %s, want %s (order is E, then sectors 1..5)", i, n.Inputs[i], want[i])
|
||||
}
|
||||
}
|
||||
}
|
||||
218
internal/landscape/kernels.go
Normal file
218
internal/landscape/kernels.go
Normal file
@@ -0,0 +1,218 @@
|
||||
package landscape
|
||||
|
||||
// valid reports whether a class value is in the analysis domain (1..5). Any
|
||||
// other value, including nodata and NaN, is invalid.
|
||||
func valid(v float64) bool { return v >= 1 && v <= 5 }
|
||||
|
||||
// classify maps a float32 score onto classes 1..5 with the fixed intervals
|
||||
// used by the reference notebook (comparisons in float32, like numpy on a
|
||||
// float32 array). Scores outside [1, 5] yield NoData.
|
||||
func classify(x float32) int16 {
|
||||
switch {
|
||||
case x < 1.0:
|
||||
return NoData
|
||||
case x <= 1.8:
|
||||
return 1
|
||||
case x <= 2.6:
|
||||
return 2
|
||||
case x <= 3.4:
|
||||
return 3
|
||||
case x <= 4.2:
|
||||
return 4
|
||||
case x <= 5.0:
|
||||
return 5
|
||||
default:
|
||||
return NoData
|
||||
}
|
||||
}
|
||||
|
||||
// kernelZIntegrated: all four Z valid -> classify(0.25*Z1 + 0.25*Z2 + 0.25*Z3 + 0.25*Z4).
|
||||
func kernelZIntegrated(in [][]float64, out [][]int16, _ Params) {
|
||||
for i := range out[0] {
|
||||
z1, z2, z3, z4 := in[0][i], in[1][i], in[2][i], in[3][i]
|
||||
if !(valid(z1) && valid(z2) && valid(z3) && valid(z4)) {
|
||||
out[0][i] = NoData
|
||||
continue
|
||||
}
|
||||
// explicit conversions forbid FMA fusion so results match numpy's per-op rounding
|
||||
score := float32(float32(0.25)*float32(z1)) + float32(float32(0.25)*float32(z2)) +
|
||||
float32(float32(0.25)*float32(z3)) + float32(float32(0.25)*float32(z4))
|
||||
out[0][i] = classify(score)
|
||||
}
|
||||
}
|
||||
|
||||
// kernelUIntegrated: water and soil required, at least one of crop/veg;
|
||||
// U = (0.25*water + 0.25*soil + [0.25*crop] + [0.25*veg]) / (0.5 + [0.25] + [0.25]).
|
||||
func kernelUIntegrated(in [][]float64, out [][]int16, _ Params) {
|
||||
for i := range out[0] {
|
||||
water, soil, crop, veg := in[0][i], in[1][i], in[2][i], in[3][i]
|
||||
cropOK, vegOK := valid(crop), valid(veg)
|
||||
if !(valid(water) && valid(soil) && (cropOK || vegOK)) {
|
||||
out[0][i] = NoData
|
||||
continue
|
||||
}
|
||||
// explicit conversions forbid FMA fusion so results match numpy's per-op rounding
|
||||
num := float32(float32(0.25)*float32(water)) + float32(float32(0.25)*float32(soil))
|
||||
den := float32(0.5)
|
||||
if cropOK {
|
||||
num += float32(float32(0.25) * float32(crop))
|
||||
den += 0.25
|
||||
}
|
||||
if vegOK {
|
||||
num += float32(float32(0.25) * float32(veg))
|
||||
den += 0.25
|
||||
}
|
||||
out[0][i] = classify(num / den)
|
||||
}
|
||||
}
|
||||
|
||||
// kernelZMax: all four valid -> max.
|
||||
func kernelZMax(in [][]float64, out [][]int16, _ Params) {
|
||||
for i := range out[0] {
|
||||
m, ok := int16(0), true
|
||||
for _, layer := range in {
|
||||
v := layer[i]
|
||||
if !valid(v) {
|
||||
ok = false
|
||||
break
|
||||
}
|
||||
if int16(v) > m {
|
||||
m = int16(v)
|
||||
}
|
||||
}
|
||||
if !ok {
|
||||
m = NoData
|
||||
}
|
||||
out[0][i] = m
|
||||
}
|
||||
}
|
||||
|
||||
// kernelUMax: water and soil required plus one of crop/veg; max over valid layers.
|
||||
func kernelUMax(in [][]float64, out [][]int16, _ Params) {
|
||||
for i := range out[0] {
|
||||
water, soil, crop, veg := in[0][i], in[1][i], in[2][i], in[3][i]
|
||||
cropOK, vegOK := valid(crop), valid(veg)
|
||||
if !(valid(water) && valid(soil) && (cropOK || vegOK)) {
|
||||
out[0][i] = NoData
|
||||
continue
|
||||
}
|
||||
m := max(int16(water), int16(soil))
|
||||
if cropOK {
|
||||
m = max(m, int16(crop))
|
||||
}
|
||||
if vegOK {
|
||||
m = max(m, int16(veg))
|
||||
}
|
||||
out[0][i] = m
|
||||
}
|
||||
}
|
||||
|
||||
// kernelE: Z and U valid -> max(Z, U).
|
||||
func kernelE(in [][]float64, out [][]int16, _ Params) {
|
||||
for i := range out[0] {
|
||||
z, u := in[0][i], in[1][i]
|
||||
if !(valid(z) && valid(u)) {
|
||||
out[0][i] = NoData
|
||||
continue
|
||||
}
|
||||
out[0][i] = max(int16(z), int16(u))
|
||||
}
|
||||
}
|
||||
|
||||
// kernelCConflicts: inputs P1..P5, E; output j = min(Pj, E) where both valid.
|
||||
func kernelCConflicts(in [][]float64, out [][]int16, _ Params) {
|
||||
e := in[5]
|
||||
for j := 0; j < 5; j++ {
|
||||
p := in[j]
|
||||
for i := range out[j] {
|
||||
if !(valid(p[i]) && valid(e[i])) {
|
||||
out[j][i] = NoData
|
||||
continue
|
||||
}
|
||||
out[j][i] = min(int16(p[i]), int16(e[i]))
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// kernelMaxAnyValid: max over the valid inputs; NoData when none is valid.
|
||||
// Used for P_max and C_max.
|
||||
func kernelMaxAnyValid(in [][]float64, out [][]int16, _ Params) {
|
||||
for i := range out[0] {
|
||||
m, any := int16(0), false
|
||||
for _, layer := range in {
|
||||
if v := layer[i]; valid(v) {
|
||||
any = true
|
||||
m = max(m, int16(v))
|
||||
}
|
||||
}
|
||||
if !any {
|
||||
m = NoData
|
||||
}
|
||||
out[0][i] = m
|
||||
}
|
||||
}
|
||||
|
||||
// kernelFuncZoning implements TZ_functional_zoning_go.md §4. Inputs: E, then
|
||||
// S1..S5 (irrigation, snow, crop, pasture, tourism). Outputs: V1, V2, V3.
|
||||
func kernelFuncZoning(in [][]float64, out [][]int16, p Params) {
|
||||
e := in[0]
|
||||
// The int16 conversions below cannot overflow (gosec G115): thresholds and
|
||||
// tie_priority codes are limited to 1..5 by Params.Validate, and pixel
|
||||
// values are converted only after valid() has bounded them to 1..5.
|
||||
rules := [3]func(e, s int16) bool{
|
||||
func(e, s int16) bool {
|
||||
return (e <= 2 && s >= int16(p.V1MinSE12)) || (e == 3 && s >= int16(p.V1MinSE3)) // #nosec G115 -- 1..5, see above
|
||||
},
|
||||
func(e, s int16) bool {
|
||||
return e <= 2 || (e == 3 && s >= int16(p.V2MinSE3)) || (e == 4 && s >= int16(p.V2MinSE4)) // #nosec G115 -- 1..5, see above
|
||||
},
|
||||
func(e, s int16) bool { return e <= 4 },
|
||||
}
|
||||
var s [5]int16
|
||||
var base [5]bool
|
||||
for i := range out[0] {
|
||||
if !valid(e[i]) {
|
||||
out[0][i], out[1][i], out[2][i] = NoData, NoData, NoData
|
||||
continue
|
||||
}
|
||||
ev := int16(e[i])
|
||||
anyBase := false
|
||||
for k := 0; k < 5; k++ {
|
||||
base[k] = valid(in[k+1][i])
|
||||
if base[k] {
|
||||
s[k] = int16(in[k+1][i])
|
||||
anyBase = true
|
||||
}
|
||||
}
|
||||
if !anyBase {
|
||||
out[0][i], out[1][i], out[2][i] = NoData, NoData, NoData
|
||||
continue
|
||||
}
|
||||
for v := 0; v < 3; v++ {
|
||||
var cand [5]bool
|
||||
maxS, anyCand := int16(0), false
|
||||
for k := 0; k < 5; k++ {
|
||||
cand[k] = base[k] && rules[v](ev, s[k])
|
||||
if cand[k] {
|
||||
anyCand = true
|
||||
if s[k] > maxS {
|
||||
maxS = s[k]
|
||||
}
|
||||
}
|
||||
}
|
||||
if !anyCand {
|
||||
out[v][i] = 1 // regulated
|
||||
continue
|
||||
}
|
||||
sector := int16(0)
|
||||
for _, code := range p.TiePriority {
|
||||
k := code - 1
|
||||
if cand[k] && s[k] == maxS {
|
||||
sector = int16(code) // #nosec G115 -- tie_priority code is 1..5, see above
|
||||
break
|
||||
}
|
||||
}
|
||||
out[v][i] = sector + 1
|
||||
}
|
||||
}
|
||||
}
|
||||
123
internal/landscape/kernels_test.go
Normal file
123
internal/landscape/kernels_test.go
Normal file
@@ -0,0 +1,123 @@
|
||||
package landscape
|
||||
|
||||
import (
|
||||
"reflect"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func run(k Kernel, nOut int, in ...[]float64) [][]int16 {
|
||||
out := make([][]int16, nOut)
|
||||
for j := range out {
|
||||
out[j] = make([]int16, len(in[0]))
|
||||
}
|
||||
k(in, out, DefaultParams())
|
||||
return out
|
||||
}
|
||||
|
||||
func TestClassify(t *testing.T) {
|
||||
cases := map[float32]int16{1.0: 1, 1.8: 1, 1.81: 2, 2.6: 2, 2.75: 3, 3.4: 3, 3.5: 4, 4.2: 4, 4.25: 5, 5.0: 5}
|
||||
for x, want := range cases {
|
||||
if got := classify(x); got != want {
|
||||
t.Errorf("classify(%v)=%d want %d", x, got, want)
|
||||
}
|
||||
}
|
||||
if classify(0.9) != NoData || classify(5.1) != NoData {
|
||||
t.Error("out of range must be NoData")
|
||||
}
|
||||
}
|
||||
|
||||
func TestKernelZIntegrated(t *testing.T) {
|
||||
// pixel0: all valid (1,2,3,4) -> 2.5 -> class 2 ; pixel1: one nodata -> NoData ; pixel2: all 5 -> 5 -> class 5
|
||||
got := run(kernelZIntegrated, 1,
|
||||
[]float64{1, -9999, 5}, []float64{2, 2, 5}, []float64{3, 3, 5}, []float64{4, 4, 5})
|
||||
if want := []int16{2, NoData, 5}; !reflect.DeepEqual(got[0], want) {
|
||||
t.Fatalf("got %v want %v", got[0], want)
|
||||
}
|
||||
}
|
||||
|
||||
func TestKernelUIntegrated_Applicability(t *testing.T) {
|
||||
// order: water, soil, crop, veg
|
||||
// p0: all valid 1,1,5,5 -> (0.25+0.25+1.25+1.25)/1.0 = 3.0 -> class 3
|
||||
// p1: crop invalid: 1,1,-,5 -> (0.25+0.25+1.25)/0.75 = 2.333 -> class 2
|
||||
// p2: crop and veg invalid -> NoData
|
||||
// p3: water invalid -> NoData
|
||||
got := run(kernelUIntegrated, 1,
|
||||
[]float64{1, 1, 1, -9999}, []float64{1, 1, 1, 1}, []float64{5, 0, 0, 5}, []float64{5, 5, 0, 5})
|
||||
if want := []int16{3, 2, NoData, NoData}; !reflect.DeepEqual(got[0], want) {
|
||||
t.Fatalf("got %v want %v", got[0], want)
|
||||
}
|
||||
}
|
||||
|
||||
func TestKernelZMax_RequiresAllValid(t *testing.T) {
|
||||
got := run(kernelZMax, 1, []float64{1, 1}, []float64{4, 4}, []float64{2, -9999}, []float64{3, 3})
|
||||
if want := []int16{4, NoData}; !reflect.DeepEqual(got[0], want) {
|
||||
t.Fatalf("got %v want %v", got[0], want)
|
||||
}
|
||||
}
|
||||
|
||||
func TestKernelUMax_IgnoresInvalidCropVeg(t *testing.T) {
|
||||
// water, soil, crop, veg ; p0: 1,2,-,5 -> 5 ; p1: 3,2,0,0 -> NoData (needs crop or veg) ; p2: 3,2,1,0 -> 3
|
||||
got := run(kernelUMax, 1, []float64{1, 3, 3}, []float64{2, 2, 2}, []float64{0, 0, 1}, []float64{5, 0, 0})
|
||||
if want := []int16{5, NoData, 3}; !reflect.DeepEqual(got[0], want) {
|
||||
t.Fatalf("got %v want %v", got[0], want)
|
||||
}
|
||||
}
|
||||
|
||||
func TestKernelE(t *testing.T) {
|
||||
got := run(kernelE, 1, []float64{2, 5, -9999}, []float64{4, 1, 3})
|
||||
if want := []int16{4, 5, NoData}; !reflect.DeepEqual(got[0], want) {
|
||||
t.Fatalf("got %v want %v", got[0], want)
|
||||
}
|
||||
}
|
||||
|
||||
func TestKernelCConflicts(t *testing.T) {
|
||||
// 5 P then E; p0: E=3 -> min(P,3); p1: E invalid -> all NoData; p2: P2 invalid
|
||||
e := []float64{3, 0, 4}
|
||||
got := run(kernelCConflicts, 5,
|
||||
[]float64{5, 5, 5}, []float64{1, 1, 7}, []float64{3, 3, 3}, []float64{4, 4, 4}, []float64{2, 2, 2}, e)
|
||||
want := [][]int16{{3, NoData, 4}, {1, NoData, NoData}, {3, NoData, 3}, {3, NoData, 4}, {2, NoData, 2}}
|
||||
if !reflect.DeepEqual(got, want) {
|
||||
t.Fatalf("got %v want %v", got, want)
|
||||
}
|
||||
}
|
||||
|
||||
func TestKernelMaxAnyValid(t *testing.T) {
|
||||
got := run(kernelMaxAnyValid, 1, []float64{1, 0, 0}, []float64{4, 0, 2}, []float64{-9999, 0, 0})
|
||||
if want := []int16{4, NoData, 2}; !reflect.DeepEqual(got[0], want) {
|
||||
t.Fatalf("got %v want %v", got[0], want)
|
||||
}
|
||||
}
|
||||
|
||||
// TZ §7 edge cases. Inputs: E, then C1..C5 (irrigation, snow, crop, pasture, tourism).
|
||||
func TestKernelFuncZoning_TZEdgeCases(t *testing.T) {
|
||||
e := []float64{0, 3, 5, 4, 2, 2, 3}
|
||||
s1 := []float64{3, 0, 3, 5, 3, 2, 4}
|
||||
s2 := []float64{3, 0, 3, 5, 3, 2, 4}
|
||||
s3 := []float64{3, 0, 3, 2, 3, 2, 5}
|
||||
s4 := []float64{3, 0, 3, 2, 3, 2, 1}
|
||||
s5 := []float64{3, 0, 3, 2, 3, 2, 1}
|
||||
got := run(kernelFuncZoning, 3, e, s1, s2, s3, s4, s5)
|
||||
// pixel: 0 E invalid -> NoData ; 1 all S invalid -> NoData ; 2 E=5 -> regulated(1) everywhere ;
|
||||
// 3 E=4, S=5 for sectors 1,2 -> V1 regulated, V2/V3 sector 1 -> class 2 ;
|
||||
// 4 all S equal (3), E=2 -> irrigation (2) in all ; 5 E=2, S=2 -> V1 regulated (S<3), V2 best=irrigation (2), V3 irrigation (2)
|
||||
// 6 E=3: S=[4,4,5,1,1] -> V1 (S>=4): winners 1,2,3 max 5 -> crop (4); V2 (S>=3) -> crop (4); V3 -> crop (4)
|
||||
want := [][]int16{
|
||||
{NoData, NoData, 1, 1, 2, 1, 4}, // V1
|
||||
{NoData, NoData, 1, 2, 2, 2, 4}, // V2
|
||||
{NoData, NoData, 1, 2, 2, 2, 4}, // V3
|
||||
}
|
||||
if !reflect.DeepEqual(got, want) {
|
||||
t.Fatalf("got %v want %v", got, want)
|
||||
}
|
||||
}
|
||||
|
||||
func TestKernelFuncZoning_TiePriorityIsConfigurable(t *testing.T) {
|
||||
in := [][]float64{{2}, {3}, {3}, {3}, {3}, {3}}
|
||||
out := [][]int16{{0}, {0}, {0}}
|
||||
p := DefaultParams()
|
||||
p.TiePriority = []int{5, 4, 3, 2, 1}
|
||||
kernelFuncZoning(in, out, p)
|
||||
if out[0][0] != 6 || out[1][0] != 6 || out[2][0] != 6 {
|
||||
t.Fatalf("expected tourism (6) everywhere, got %v", out)
|
||||
}
|
||||
}
|
||||
81
internal/landscape/params.go
Normal file
81
internal/landscape/params.go
Normal file
@@ -0,0 +1,81 @@
|
||||
package landscape
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"fmt"
|
||||
)
|
||||
|
||||
// Params are the run-time knobs of the graph (TZ §6): window size, tie
|
||||
// priority for functional zoning, and the V1/V2 suitability thresholds.
|
||||
type Params struct {
|
||||
WindowSize int `json:"window_size"`
|
||||
TiePriority []int `json:"tie_priority"`
|
||||
V1MinSE12 int `json:"v1_min_s_e12"`
|
||||
V1MinSE3 int `json:"v1_min_s_e3"`
|
||||
V2MinSE3 int `json:"v2_min_s_e3"`
|
||||
V2MinSE4 int `json:"v2_min_s_e4"`
|
||||
}
|
||||
|
||||
// DefaultParams returns the TZ defaults.
|
||||
func DefaultParams() Params {
|
||||
return Params{WindowSize: 2048, TiePriority: []int{1, 2, 3, 4, 5}, V1MinSE12: 3, V1MinSE3: 4, V2MinSE3: 3, V2MinSE4: 5}
|
||||
}
|
||||
|
||||
// WithDefaults fills zero-valued fields from DefaultParams.
|
||||
func (p Params) WithDefaults() Params {
|
||||
d := DefaultParams()
|
||||
if p.WindowSize == 0 {
|
||||
p.WindowSize = d.WindowSize
|
||||
}
|
||||
if len(p.TiePriority) == 0 {
|
||||
p.TiePriority = d.TiePriority
|
||||
}
|
||||
if p.V1MinSE12 == 0 {
|
||||
p.V1MinSE12 = d.V1MinSE12
|
||||
}
|
||||
if p.V1MinSE3 == 0 {
|
||||
p.V1MinSE3 = d.V1MinSE3
|
||||
}
|
||||
if p.V2MinSE3 == 0 {
|
||||
p.V2MinSE3 = d.V2MinSE3
|
||||
}
|
||||
if p.V2MinSE4 == 0 {
|
||||
p.V2MinSE4 = d.V2MinSE4
|
||||
}
|
||||
return p
|
||||
}
|
||||
|
||||
// Validate checks ranges: window 256..8192, thresholds 1..5, tie_priority a
|
||||
// permutation of 1..5.
|
||||
func (p Params) Validate() error {
|
||||
var errs []error
|
||||
if p.WindowSize < 256 || p.WindowSize > 8192 {
|
||||
errs = append(errs, fmt.Errorf("window_size must be in 256..8192"))
|
||||
}
|
||||
for name, v := range map[string]int{"v1_min_s_e12": p.V1MinSE12, "v1_min_s_e3": p.V1MinSE3, "v2_min_s_e3": p.V2MinSE3, "v2_min_s_e4": p.V2MinSE4} {
|
||||
if v < 1 || v > 5 {
|
||||
errs = append(errs, fmt.Errorf("%s must be in 1..5", name))
|
||||
}
|
||||
}
|
||||
seen := map[int]bool{}
|
||||
for _, k := range p.TiePriority {
|
||||
if k < 1 || k > 5 || seen[k] {
|
||||
errs = append(errs, fmt.Errorf("tie_priority must be a permutation of 1..5"))
|
||||
break
|
||||
}
|
||||
seen[k] = true
|
||||
}
|
||||
if len(p.TiePriority) != 5 && len(seen) == len(p.TiePriority) {
|
||||
errs = append(errs, fmt.Errorf("tie_priority must list all of 1..5"))
|
||||
}
|
||||
return errors.Join(errs...)
|
||||
}
|
||||
|
||||
// ForNode keeps only the fields that influence the node's result (window size
|
||||
// never does), so fingerprints do not change when irrelevant knobs change.
|
||||
func (p Params) ForNode(code string) Params {
|
||||
if code == "30" {
|
||||
return Params{TiePriority: p.TiePriority, V1MinSE12: p.V1MinSE12, V1MinSE3: p.V1MinSE3, V2MinSE3: p.V2MinSE3, V2MinSE4: p.V2MinSE4}
|
||||
}
|
||||
return Params{}
|
||||
}
|
||||
58
internal/landscape/params_test.go
Normal file
58
internal/landscape/params_test.go
Normal file
@@ -0,0 +1,58 @@
|
||||
package landscape
|
||||
|
||||
import (
|
||||
"reflect"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestDefaultParamsValidate(t *testing.T) {
|
||||
if err := DefaultParams().Validate(); err != nil {
|
||||
t.Fatalf("DefaultParams().Validate() = %v, want nil", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestWithDefaultsFillsZeroFields(t *testing.T) {
|
||||
got := Params{}.WithDefaults()
|
||||
want := DefaultParams()
|
||||
if !reflect.DeepEqual(got, want) {
|
||||
t.Fatalf("Params{}.WithDefaults() = %+v, want %+v", got, want)
|
||||
}
|
||||
}
|
||||
|
||||
func TestValidate_TiePriorityDuplicateInvalid(t *testing.T) {
|
||||
p := DefaultParams()
|
||||
p.TiePriority = []int{1, 1, 2, 3, 4}
|
||||
if err := p.Validate(); err == nil {
|
||||
t.Fatal("expected error for tie_priority with duplicate values")
|
||||
}
|
||||
}
|
||||
|
||||
func TestValidate_WindowSizeOutOfRangeInvalid(t *testing.T) {
|
||||
p := DefaultParams()
|
||||
p.WindowSize = 100
|
||||
if err := p.Validate(); err == nil {
|
||||
t.Fatal("expected error for window_size below 256")
|
||||
}
|
||||
}
|
||||
|
||||
func TestForNode_NonFuncZoningNodeIsZero(t *testing.T) {
|
||||
p := DefaultParams()
|
||||
got := p.ForNode("00")
|
||||
if !reflect.DeepEqual(got, Params{}) {
|
||||
t.Fatalf("ForNode(00) = %+v, want zero value", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestForNode_FuncZoningHasNoWindowSize(t *testing.T) {
|
||||
p := DefaultParams()
|
||||
got := p.ForNode("30")
|
||||
if got.WindowSize != 0 {
|
||||
t.Fatalf("ForNode(30).WindowSize = %d, want 0", got.WindowSize)
|
||||
}
|
||||
if !reflect.DeepEqual(got.TiePriority, p.TiePriority) {
|
||||
t.Fatalf("ForNode(30).TiePriority = %v, want %v", got.TiePriority, p.TiePriority)
|
||||
}
|
||||
if got.V1MinSE12 != p.V1MinSE12 || got.V1MinSE3 != p.V1MinSE3 || got.V2MinSE3 != p.V2MinSE3 || got.V2MinSE4 != p.V2MinSE4 {
|
||||
t.Fatalf("ForNode(30) thresholds = %+v, want matching %+v", got, p)
|
||||
}
|
||||
}
|
||||
94
internal/messaging/rabbitmq/landscape_consumer.go
Normal file
94
internal/messaging/rabbitmq/landscape_consumer.go
Normal file
@@ -0,0 +1,94 @@
|
||||
package rabbitmq
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"log/slog"
|
||||
"time"
|
||||
|
||||
"github.com/google/uuid"
|
||||
amqp "github.com/rabbitmq/amqp091-go"
|
||||
)
|
||||
|
||||
const (
|
||||
// LandscapeRunRoutingKey routes Landscape Program run jobs.
|
||||
LandscapeRunRoutingKey = "landscape.run"
|
||||
// LandscapeRunQueue receives Landscape Program run jobs.
|
||||
LandscapeRunQueue = "gis.landscape.run"
|
||||
// markRunFailedTimeout bounds the best-effort terminal status write made
|
||||
// after Execute failed, on a context detached from the delivery's.
|
||||
markRunFailedTimeout = 10 * time.Second
|
||||
)
|
||||
|
||||
// LandscapeJob is the message body for a Landscape Program run job.
|
||||
type LandscapeJob struct {
|
||||
RunID uuid.UUID `json:"run_id"`
|
||||
}
|
||||
|
||||
// LandscapeJobPublisher publishes Landscape Program run jobs; it implements
|
||||
// service.LandscapeJobEnqueuer.
|
||||
type LandscapeJobPublisher struct {
|
||||
pub *Publisher
|
||||
}
|
||||
|
||||
// NewLandscapeJobPublisher returns a LandscapeJobPublisher.
|
||||
func NewLandscapeJobPublisher(pub *Publisher) *LandscapeJobPublisher {
|
||||
return &LandscapeJobPublisher{pub: pub}
|
||||
}
|
||||
|
||||
// EnqueueRun publishes an execution job for the given run.
|
||||
func (p *LandscapeJobPublisher) EnqueueRun(ctx context.Context, runID uuid.UUID) error {
|
||||
body, err := json.Marshal(LandscapeJob{RunID: runID})
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
return p.pub.Publish(ctx, LandscapeRunRoutingKey, body)
|
||||
}
|
||||
|
||||
// LandscapeProcessor executes Landscape Program runs; implemented by the service.
|
||||
type LandscapeProcessor interface {
|
||||
Execute(ctx context.Context, runID uuid.UUID) error
|
||||
// MarkRunFailed records a terminal failure on the run so it does not remain
|
||||
// stuck in an in-progress status.
|
||||
MarkRunFailed(ctx context.Context, runID uuid.UUID, reason string) error
|
||||
}
|
||||
|
||||
// landscapeHandler executes one Landscape Program run per delivery.
|
||||
type landscapeHandler struct {
|
||||
proc LandscapeProcessor
|
||||
log *slog.Logger
|
||||
}
|
||||
|
||||
func (h landscapeHandler) Handle(ctx context.Context, d amqp.Delivery) error {
|
||||
var job LandscapeJob
|
||||
if err := json.Unmarshal(d.Body, &job); err != nil {
|
||||
return fmt.Errorf("decode landscape run job: %w", err)
|
||||
}
|
||||
h.log.Info("processing landscape run", "run_id", job.RunID)
|
||||
|
||||
err := h.proc.Execute(ctx, job.RunID)
|
||||
if err != nil {
|
||||
// The delivery is nacked without requeue (see Consumer.dispatch), so the
|
||||
// run will not be retried. Record the failure on the run itself;
|
||||
// otherwise it stays "running" forever. This covers the errors Execute
|
||||
// returns before it can mark the run itself (a missing run, a failed
|
||||
// status write), since Execute already marks the run failed for every
|
||||
// node failure. Best-effort: log if marking fails, but still return the
|
||||
// original error so the delivery is nacked.
|
||||
// Detached from ctx: Execute may well have failed *because* ctx was
|
||||
// cancelled (shutdown), and the write must still land or the run stays
|
||||
// "running" forever.
|
||||
mctx, cancel := context.WithTimeout(context.WithoutCancel(ctx), markRunFailedTimeout)
|
||||
if markErr := h.proc.MarkRunFailed(mctx, job.RunID, err.Error()); markErr != nil {
|
||||
h.log.Error("mark landscape run failed", "run_id", job.RunID, "error", markErr)
|
||||
}
|
||||
cancel()
|
||||
}
|
||||
return err
|
||||
}
|
||||
|
||||
// NewLandscapeRunHandler returns a handler that executes Landscape Program runs.
|
||||
func NewLandscapeRunHandler(p LandscapeProcessor, log *slog.Logger) Handler {
|
||||
return landscapeHandler{proc: p, log: log}
|
||||
}
|
||||
90
internal/messaging/rabbitmq/landscape_consumer_test.go
Normal file
90
internal/messaging/rabbitmq/landscape_consumer_test.go
Normal file
@@ -0,0 +1,90 @@
|
||||
package rabbitmq
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"testing"
|
||||
|
||||
"github.com/google/uuid"
|
||||
amqp "github.com/rabbitmq/amqp091-go"
|
||||
)
|
||||
|
||||
// fakeLandscapeProcessor records MarkRunFailed calls and returns a preset error
|
||||
// from Execute.
|
||||
type fakeLandscapeProcessor struct {
|
||||
runErr error
|
||||
executed []uuid.UUID
|
||||
failedID uuid.UUID
|
||||
failedWith string
|
||||
failedCnt int
|
||||
}
|
||||
|
||||
func (f *fakeLandscapeProcessor) Execute(_ context.Context, runID uuid.UUID) error {
|
||||
f.executed = append(f.executed, runID)
|
||||
return f.runErr
|
||||
}
|
||||
|
||||
func (f *fakeLandscapeProcessor) MarkRunFailed(_ context.Context, runID uuid.UUID, reason string) error {
|
||||
f.failedCnt++
|
||||
f.failedID = runID
|
||||
f.failedWith = reason
|
||||
return nil
|
||||
}
|
||||
|
||||
func landscapeDelivery(t *testing.T, id uuid.UUID) amqp.Delivery {
|
||||
t.Helper()
|
||||
body, err := json.Marshal(LandscapeJob{RunID: id})
|
||||
if err != nil {
|
||||
t.Fatalf("marshal job: %v", err)
|
||||
}
|
||||
return amqp.Delivery{Body: body}
|
||||
}
|
||||
|
||||
func TestLandscapeHandler_MarksRunFailedOnExecuteError(t *testing.T) {
|
||||
id := uuid.New()
|
||||
fp := &fakeLandscapeProcessor{runErr: errors.New("node 00: grid mismatch")}
|
||||
h := NewLandscapeRunHandler(fp, discardLogger())
|
||||
|
||||
err := h.Handle(context.Background(), landscapeDelivery(t, id))
|
||||
if err == nil {
|
||||
t.Fatal("expected the run error to be returned so the delivery is nacked")
|
||||
}
|
||||
if len(fp.executed) != 1 || fp.executed[0] != id {
|
||||
t.Fatalf("Execute called with %v, want [%s]", fp.executed, id)
|
||||
}
|
||||
if fp.failedCnt != 1 {
|
||||
t.Fatalf("expected MarkRunFailed to be called once, got %d", fp.failedCnt)
|
||||
}
|
||||
if fp.failedID != id {
|
||||
t.Fatalf("MarkRunFailed called with wrong id: got %s want %s", fp.failedID, id)
|
||||
}
|
||||
if fp.failedWith != "node 00: grid mismatch" {
|
||||
t.Fatalf("MarkRunFailed called with wrong reason: %q", fp.failedWith)
|
||||
}
|
||||
}
|
||||
|
||||
func TestLandscapeHandler_DoesNotMarkFailedOnSuccess(t *testing.T) {
|
||||
fp := &fakeLandscapeProcessor{runErr: nil}
|
||||
h := NewLandscapeRunHandler(fp, discardLogger())
|
||||
|
||||
if err := h.Handle(context.Background(), landscapeDelivery(t, uuid.New())); err != nil {
|
||||
t.Fatalf("unexpected error: %v", err)
|
||||
}
|
||||
if fp.failedCnt != 0 {
|
||||
t.Fatalf("expected MarkRunFailed not to be called on success, got %d", fp.failedCnt)
|
||||
}
|
||||
}
|
||||
|
||||
func TestLandscapeHandler_UndecodableBodyIsNacked(t *testing.T) {
|
||||
fp := &fakeLandscapeProcessor{}
|
||||
h := NewLandscapeRunHandler(fp, discardLogger())
|
||||
|
||||
err := h.Handle(context.Background(), amqp.Delivery{Body: []byte("not json")})
|
||||
if err == nil {
|
||||
t.Fatal("expected a decode error so the delivery is nacked")
|
||||
}
|
||||
if len(fp.executed) != 0 {
|
||||
t.Fatalf("Execute must not run for an undecodable body, got %v", fp.executed)
|
||||
}
|
||||
}
|
||||
@@ -112,7 +112,9 @@ func gpkgScan(db *sql.DB, table string, names []string, limit int) []map[string]
|
||||
for i, n := range names {
|
||||
quoted[i] = quoteIdent(n)
|
||||
}
|
||||
query := fmt.Sprintf("SELECT %s FROM %s", strings.Join(quoted, ", "), quoteIdent(table))
|
||||
// Identifiers cannot be bound as parameters; every one is quoted with
|
||||
// quoteIdent (embedded quotes doubled) and the LIMIT is an int.
|
||||
query := fmt.Sprintf("SELECT %s FROM %s", strings.Join(quoted, ", "), quoteIdent(table)) // #nosec G201 -- identifiers quoted by quoteIdent
|
||||
if limit >= 0 {
|
||||
query += fmt.Sprintf(" LIMIT %d", limit)
|
||||
}
|
||||
|
||||
@@ -25,7 +25,8 @@ func NewGDALConverter() *GDALConverter {
|
||||
// ToCOG converts the source raster to a Cloud-Optimized GeoTIFF at dst. The COG
|
||||
// driver builds internal tiling and overviews.
|
||||
func (c *GDALConverter) ToCOG(ctx context.Context, src, dst string) error {
|
||||
cmd := exec.CommandContext(ctx, "gdal_translate",
|
||||
// Fixed binary, no shell; src/dst are service-generated temp-file paths.
|
||||
cmd := exec.CommandContext(ctx, "gdal_translate", // #nosec G204 -- see above
|
||||
"-of", "COG",
|
||||
"-co", "COMPRESS="+c.compression,
|
||||
src, dst,
|
||||
@@ -41,7 +42,7 @@ func (c *GDALConverter) ToCOG(ctx context.Context, src, dst string) error {
|
||||
// Footprint returns the raster's footprint as a GeoJSON polygon in EPSG:4326, or
|
||||
// nil if the raster has no spatial reference.
|
||||
func (c *GDALConverter) Footprint(ctx context.Context, src string) ([]byte, error) {
|
||||
out, err := exec.CommandContext(ctx, "gdalinfo", "-json", src).Output()
|
||||
out, err := exec.CommandContext(ctx, "gdalinfo", "-json", src).Output() // #nosec G204 -- fixed binary, no shell; src is a service temp file
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("gdalinfo: %w", err)
|
||||
}
|
||||
@@ -58,18 +59,22 @@ func (c *GDALConverter) Footprint(ctx context.Context, src string) ([]byte, erro
|
||||
return info.Wgs84Extent, nil
|
||||
}
|
||||
|
||||
// VectorGeometry reads every feature of a vector file and returns their combined
|
||||
// geometry as a GeoJSON GeometryCollection reprojected to EPSG:4326, or nil if
|
||||
// the file has no features. The caller (PostGIS) dissolves the collection into
|
||||
// the union of all features. Zipped ESRI shapefiles are read in place via GDAL's
|
||||
// /vsizip/ virtual filesystem; GeoJSON and GeoPackage are read directly.
|
||||
func (c *GDALConverter) VectorGeometry(ctx context.Context, src string) ([]byte, error) {
|
||||
// VectorFeatures reads every feature of a vector file and returns their
|
||||
// geometries, reprojected to EPSG:4326, in file order (nil if the file has no
|
||||
// features). A feature with no geometry yields a nil entry at its index, so
|
||||
// the result stays aligned with attribute rows read from the same file (the
|
||||
// shapefile format itself has no other way to pair a .shp geometry record with
|
||||
// its .dbf attribute row than matching by position). Zipped ESRI shapefiles
|
||||
// are read in place via GDAL's /vsizip/ virtual filesystem; GeoJSON and
|
||||
// GeoPackage are read directly.
|
||||
func (c *GDALConverter) VectorFeatures(ctx context.Context, src string) ([]json.RawMessage, error) {
|
||||
input := src
|
||||
if strings.EqualFold(filepath.Ext(src), ".zip") {
|
||||
input = "/vsizip/" + src
|
||||
}
|
||||
|
||||
cmd := exec.CommandContext(ctx, "ogr2ogr",
|
||||
// Fixed binary, no shell; input is a service-generated temp-file path.
|
||||
cmd := exec.CommandContext(ctx, "ogr2ogr", // #nosec G204 -- see above
|
||||
"-f", "GeoJSON",
|
||||
"-t_srs", "EPSG:4326",
|
||||
"/vsistdout/", input,
|
||||
@@ -89,20 +94,16 @@ func (c *GDALConverter) VectorGeometry(ctx context.Context, src string) ([]byte,
|
||||
if err := json.Unmarshal(out, &fc); err != nil {
|
||||
return nil, fmt.Errorf("parse ogr2ogr output: %w", err)
|
||||
}
|
||||
|
||||
geoms := make([]json.RawMessage, 0, len(fc.Features))
|
||||
for _, f := range fc.Features {
|
||||
if len(f.Geometry) == 0 || string(f.Geometry) == "null" {
|
||||
continue
|
||||
}
|
||||
geoms = append(geoms, f.Geometry)
|
||||
}
|
||||
if len(geoms) == 0 {
|
||||
if len(fc.Features) == 0 {
|
||||
return nil, nil
|
||||
}
|
||||
|
||||
return json.Marshal(struct {
|
||||
Type string `json:"type"`
|
||||
Geometries []json.RawMessage `json:"geometries"`
|
||||
}{Type: "GeometryCollection", Geometries: geoms})
|
||||
geoms := make([]json.RawMessage, len(fc.Features))
|
||||
for i, f := range fc.Features {
|
||||
if len(f.Geometry) == 0 || string(f.Geometry) == "null" {
|
||||
continue
|
||||
}
|
||||
geoms[i] = f.Geometry
|
||||
}
|
||||
return geoms, nil
|
||||
}
|
||||
|
||||
140
internal/raster/godal.go
Normal file
140
internal/raster/godal.go
Normal file
@@ -0,0 +1,140 @@
|
||||
package raster
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"math"
|
||||
"sync"
|
||||
|
||||
"github.com/airbusgeo/godal"
|
||||
)
|
||||
|
||||
var registerOnce sync.Once
|
||||
|
||||
// GodalIO implements IO with the GDAL bindings (github.com/airbusgeo/godal).
|
||||
// GDAL datasets are not goroutine-safe: callers must not share a Reader or
|
||||
// Writer between goroutines without external synchronisation.
|
||||
type GodalIO struct{}
|
||||
|
||||
// NewGodalIO registers GDAL drivers (once) and returns a GodalIO.
|
||||
func NewGodalIO() *GodalIO {
|
||||
registerOnce.Do(godal.RegisterAll)
|
||||
return &GodalIO{}
|
||||
}
|
||||
|
||||
// DefaultWriterOptions returns the Landscape Program output profile.
|
||||
func DefaultWriterOptions() WriterOptions {
|
||||
return WriterOptions{NoData: -9999, TileSize: 512, Compression: "DEFLATE"}
|
||||
}
|
||||
|
||||
type godalReader struct {
|
||||
ds *godal.Dataset
|
||||
grid Grid
|
||||
}
|
||||
|
||||
func (g *GodalIO) Open(_ context.Context, path string) (Reader, error) {
|
||||
ds, err := godal.Open(path)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("open raster %s: %w", path, err)
|
||||
}
|
||||
st := ds.Structure()
|
||||
if st.NBands < 1 {
|
||||
ds.Close()
|
||||
return nil, fmt.Errorf("open raster %s: no bands", path)
|
||||
}
|
||||
gt, err := ds.GeoTransform()
|
||||
if err != nil {
|
||||
ds.Close()
|
||||
return nil, fmt.Errorf("open raster %s: geotransform: %w", path, err)
|
||||
}
|
||||
return &godalReader{ds: ds, grid: Grid{Width: st.SizeX, Height: st.SizeY, GeoTransform: gt, ProjectionWKT: ds.Projection()}}, nil
|
||||
}
|
||||
|
||||
func (r *godalReader) Grid() Grid { return r.grid }
|
||||
|
||||
func (r *godalReader) NoData() (float64, bool) { return r.ds.Bands()[0].NoData() }
|
||||
|
||||
func (r *godalReader) ReadFloat64(w Window, dst []float64) error {
|
||||
if len(dst) < w.Width*w.Height {
|
||||
return fmt.Errorf("read window: dst has %d elements, need %d", len(dst), w.Width*w.Height)
|
||||
}
|
||||
return r.ds.Bands()[0].Read(w.X, w.Y, dst[:w.Width*w.Height], w.Width, w.Height)
|
||||
}
|
||||
|
||||
func (r *godalReader) Close() error { return r.ds.Close() }
|
||||
|
||||
type godalWriter struct{ ds *godal.Dataset }
|
||||
|
||||
func (g *GodalIO) Create(_ context.Context, path string, grid Grid, o WriterOptions) (Writer, error) {
|
||||
return createGTiff(path, grid, o, "IF_SAFER")
|
||||
}
|
||||
|
||||
// createGTiff writes a tiled Int16 GeoTIFF. bigTIFF is the value of the
|
||||
// BIGTIFF creation option ("IF_SAFER" in production); it is a parameter so
|
||||
// tests can force a BigTIFF header.
|
||||
func createGTiff(path string, grid Grid, o WriterOptions, bigTIFF string) (Writer, error) {
|
||||
tile := fmt.Sprint(o.TileSize)
|
||||
ds, err := godal.Create(godal.GTiff, path, 1, godal.Int16, grid.Width, grid.Height,
|
||||
godal.CreationOption("TILED=YES", "BLOCKXSIZE="+tile, "BLOCKYSIZE="+tile,
|
||||
"COMPRESS="+o.Compression, "PREDICTOR=2", "BIGTIFF="+bigTIFF))
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("create raster %s: %w", path, err)
|
||||
}
|
||||
if err := ds.SetGeoTransform(grid.GeoTransform); err != nil {
|
||||
ds.Close()
|
||||
return nil, fmt.Errorf("create raster %s: geotransform: %w", path, err)
|
||||
}
|
||||
if grid.ProjectionWKT != "" {
|
||||
if err := ds.SetProjection(grid.ProjectionWKT); err != nil {
|
||||
ds.Close()
|
||||
return nil, fmt.Errorf("create raster %s: projection: %w", path, err)
|
||||
}
|
||||
}
|
||||
if err := ds.Bands()[0].SetNoData(float64(o.NoData)); err != nil {
|
||||
ds.Close()
|
||||
return nil, fmt.Errorf("create raster %s: nodata: %w", path, err)
|
||||
}
|
||||
return &godalWriter{ds: ds}, nil
|
||||
}
|
||||
|
||||
func (w *godalWriter) WriteInt16(win Window, src []int16) error {
|
||||
if len(src) < win.Width*win.Height {
|
||||
return fmt.Errorf("write window: src has %d elements, need %d", len(src), win.Width*win.Height)
|
||||
}
|
||||
return w.ds.Bands()[0].Write(win.X, win.Y, src[:win.Width*win.Height], win.Width, win.Height)
|
||||
}
|
||||
|
||||
func (w *godalWriter) Close() error { return w.ds.Close() }
|
||||
|
||||
// SameGrid reports whether two grids describe the same pixels: equal size,
|
||||
// geotransform within 1e-9 (absolute), and the same CRS (GDAL IsSame; an
|
||||
// empty projection only matches an empty projection).
|
||||
//
|
||||
// ProjectionWKT is parsed with godal.NewSpatialRef rather than
|
||||
// NewSpatialRefFromWKT: despite the field name, callers may pass any user
|
||||
// projection string GDAL's SetProjection accepts (e.g. "EPSG:32642"), not
|
||||
// only WKT, and NewSpatialRefFromWKT rejects non-WKT input.
|
||||
func (g *GodalIO) SameGrid(a, b Grid) bool {
|
||||
if a.Width != b.Width || a.Height != b.Height {
|
||||
return false
|
||||
}
|
||||
for i := range a.GeoTransform {
|
||||
if math.Abs(a.GeoTransform[i]-b.GeoTransform[i]) > 1e-9 {
|
||||
return false
|
||||
}
|
||||
}
|
||||
if a.ProjectionWKT == "" || b.ProjectionWKT == "" {
|
||||
return a.ProjectionWKT == b.ProjectionWKT
|
||||
}
|
||||
sa, err := godal.NewSpatialRef(a.ProjectionWKT)
|
||||
if err != nil {
|
||||
return false
|
||||
}
|
||||
defer sa.Close()
|
||||
sb, err := godal.NewSpatialRef(b.ProjectionWKT)
|
||||
if err != nil {
|
||||
return false
|
||||
}
|
||||
defer sb.Close()
|
||||
return sa.IsSame(sb)
|
||||
}
|
||||
103
internal/raster/godal_test.go
Normal file
103
internal/raster/godal_test.go
Normal file
@@ -0,0 +1,103 @@
|
||||
package raster
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"io"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"testing"
|
||||
|
||||
"gis/internal/domain"
|
||||
)
|
||||
|
||||
func TestGodalIO_RoundTrip(t *testing.T) {
|
||||
rio := NewGodalIO()
|
||||
ctx := context.Background()
|
||||
path := filepath.Join(t.TempDir(), "out.tif")
|
||||
grid := Grid{Width: 10, Height: 7, GeoTransform: [6]float64{100, 30, 0, 200, 0, -30}, ProjectionWKT: "EPSG:32642"}
|
||||
|
||||
w, err := rio.Create(ctx, path, grid, DefaultWriterOptions())
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
full := make([]int16, 10*7)
|
||||
for i := range full {
|
||||
full[i] = int16(i%6) - 1 // -1..4 (-1 stands in for garbage, not nodata)
|
||||
}
|
||||
full[0] = -9999
|
||||
if err := w.WriteInt16(Window{0, 0, 10, 7}, full); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := w.Close(); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
r, err := rio.Open(ctx, path)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer r.Close()
|
||||
g := r.Grid()
|
||||
if g.Width != 10 || g.Height != 7 || g.GeoTransform != grid.GeoTransform || g.ProjectionWKT == "" {
|
||||
t.Fatalf("grid mismatch: %+v", g)
|
||||
}
|
||||
if nd, ok := r.NoData(); !ok || nd != -9999 {
|
||||
t.Fatalf("nodata = %v %v", nd, ok)
|
||||
}
|
||||
dst := make([]float64, 3*2)
|
||||
if err := r.ReadFloat64(Window{X: 8, Y: 5, Width: 2, Height: 2}, dst[:4]); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
// pixel (8,5) has index 58 -> 58%6-1 = 3 ; (9,5)=59 -> 4 ; (8,6)=68 -> 1 ; (9,6)=69 -> 2
|
||||
want := []float64{3, 4, 1, 2}
|
||||
for i, v := range want {
|
||||
if dst[i] != v {
|
||||
t.Fatalf("pixel %d = %v want %v", i, dst[i], v)
|
||||
}
|
||||
}
|
||||
if !rio.SameGrid(g, grid) {
|
||||
t.Fatal("SameGrid should be true for identical grids")
|
||||
}
|
||||
other := grid
|
||||
other.GeoTransform[0] += 1
|
||||
if rio.SameGrid(g, other) {
|
||||
t.Fatal("SameGrid should be false for shifted origin")
|
||||
}
|
||||
}
|
||||
|
||||
// TestGodalIO_BigTIFFPassesContentValidation guards the upload magic-byte
|
||||
// check against the pipeline's own output: GDAL writes a BigTIFF header
|
||||
// (version 43) whenever BIGTIFF is YES or IF_SAFER kicks in for a large
|
||||
// raster, and domain.ValidateFileContent must accept it.
|
||||
func TestGodalIO_BigTIFFPassesContentValidation(t *testing.T) {
|
||||
path := filepath.Join(t.TempDir(), "big.tif")
|
||||
grid := Grid{Width: 8, Height: 8, GeoTransform: [6]float64{100, 30, 0, 200, 0, -30}, ProjectionWKT: "EPSG:32642"}
|
||||
|
||||
w, err := createGTiff(path, grid, DefaultWriterOptions(), "YES")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := w.WriteInt16(Window{0, 0, 8, 8}, make([]int16, 8*8)); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := w.Close(); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
f, err := os.Open(path)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer f.Close()
|
||||
head := make([]byte, 4)
|
||||
if _, err := io.ReadFull(f, head); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if !bytes.HasPrefix(head, []byte("II+\x00")) && !bytes.HasPrefix(head, []byte("MM\x00+")) {
|
||||
t.Fatalf("BIGTIFF=YES did not produce a BigTIFF header: % x", head)
|
||||
}
|
||||
if err := domain.ValidateFileContent(".tif", head); err != nil {
|
||||
t.Fatalf("ValidateFileContent rejected our own BigTIFF output: %v", err)
|
||||
}
|
||||
}
|
||||
77
internal/raster/io.go
Normal file
77
internal/raster/io.go
Normal file
@@ -0,0 +1,77 @@
|
||||
package raster
|
||||
|
||||
import "context"
|
||||
|
||||
// Grid describes a raster's pixel grid: its size, geotransform, and
|
||||
// projection. Two grids describe the same pixels when IO.SameGrid reports
|
||||
// true for them.
|
||||
type Grid struct {
|
||||
Width, Height int
|
||||
GeoTransform [6]float64
|
||||
ProjectionWKT string
|
||||
}
|
||||
|
||||
// Window is a rectangular block of pixels within a raster, in pixel/line
|
||||
// coordinates with (0,0) at the top-left corner.
|
||||
type Window struct {
|
||||
X, Y, Width, Height int
|
||||
}
|
||||
|
||||
// Reader reads pixel data and metadata from an existing raster.
|
||||
type Reader interface {
|
||||
// Grid returns the raster's pixel grid.
|
||||
Grid() Grid
|
||||
// NoData returns the raster's nodata value and whether one is set.
|
||||
NoData() (float64, bool)
|
||||
// ReadFloat64 reads the pixels in w into dst, row-major, converting the
|
||||
// underlying pixel type to float64. len(dst) must be at least
|
||||
// w.Width*w.Height.
|
||||
ReadFloat64(w Window, dst []float64) error
|
||||
// Close releases the underlying dataset handle.
|
||||
Close() error
|
||||
}
|
||||
|
||||
// WriterOptions configures a raster created by IO.Create.
|
||||
type WriterOptions struct {
|
||||
// NoData is the nodata value written to the output band.
|
||||
NoData int16
|
||||
// TileSize is the internal tile size (both block dimensions) in pixels.
|
||||
TileSize int
|
||||
// Compression is the GDAL COMPRESS creation option value.
|
||||
Compression string
|
||||
}
|
||||
|
||||
// Writer writes pixel data to a raster created by IO.Create.
|
||||
type Writer interface {
|
||||
// WriteInt16 writes src, row-major, into the pixels described by w.
|
||||
// len(src) must be at least w.Width*w.Height.
|
||||
WriteInt16(w Window, src []int16) error
|
||||
// Close flushes and releases the underlying dataset handle.
|
||||
Close() error
|
||||
}
|
||||
|
||||
// IO is an engine-agnostic block I/O abstraction over raster storage. It lets
|
||||
// callers open and create rasters without depending on a particular GDAL
|
||||
// binding.
|
||||
type IO interface {
|
||||
// Open opens the raster at path for reading.
|
||||
Open(ctx context.Context, path string) (Reader, error)
|
||||
// Create creates a new single-band int16 raster at path with the given
|
||||
// grid and writer options, ready for WriteInt16 calls.
|
||||
Create(ctx context.Context, path string, g Grid, o WriterOptions) (Writer, error)
|
||||
// SameGrid reports whether a and b describe the same pixel grid: equal
|
||||
// size, geotransform, and coordinate reference system.
|
||||
SameGrid(a, b Grid) bool
|
||||
}
|
||||
|
||||
// Windows tiles a w×h raster into row-major windows of at most size×size.
|
||||
func Windows(w, h, size int) []Window {
|
||||
var out []Window
|
||||
for y := 0; y < h; y += size {
|
||||
wh := min(size, h-y)
|
||||
for x := 0; x < w; x += size {
|
||||
out = append(out, Window{X: x, Y: y, Width: min(size, w-x), Height: wh})
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
61
internal/raster/io_test.go
Normal file
61
internal/raster/io_test.go
Normal file
@@ -0,0 +1,61 @@
|
||||
package raster
|
||||
|
||||
import (
|
||||
"reflect"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestWindows(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
w, h, size int
|
||||
want []Window
|
||||
}{
|
||||
{
|
||||
name: "10x10 size4 tiles into 9 windows",
|
||||
w: 10, h: 10, size: 4,
|
||||
want: []Window{
|
||||
{X: 0, Y: 0, Width: 4, Height: 4},
|
||||
{X: 4, Y: 0, Width: 4, Height: 4},
|
||||
{X: 8, Y: 0, Width: 2, Height: 4},
|
||||
{X: 0, Y: 4, Width: 4, Height: 4},
|
||||
{X: 4, Y: 4, Width: 4, Height: 4},
|
||||
{X: 8, Y: 4, Width: 2, Height: 4},
|
||||
{X: 0, Y: 8, Width: 4, Height: 2},
|
||||
{X: 4, Y: 8, Width: 4, Height: 2},
|
||||
{X: 8, Y: 8, Width: 2, Height: 2},
|
||||
},
|
||||
},
|
||||
{
|
||||
name: "5x3 size8 fits in one window",
|
||||
w: 5, h: 3, size: 8,
|
||||
want: []Window{
|
||||
{X: 0, Y: 0, Width: 5, Height: 3},
|
||||
},
|
||||
},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
got := Windows(tt.w, tt.h, tt.size)
|
||||
if !reflect.DeepEqual(got, tt.want) {
|
||||
t.Fatalf("Windows(%d, %d, %d) = %+v, want %+v", tt.w, tt.h, tt.size, got, tt.want)
|
||||
}
|
||||
if len(got) > 0 {
|
||||
last := got[len(got)-1]
|
||||
wantLast := tt.want[len(tt.want)-1]
|
||||
if last != wantLast {
|
||||
t.Fatalf("last window = %+v, want %+v", last, wantLast)
|
||||
}
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestDefaultWriterOptions(t *testing.T) {
|
||||
got := DefaultWriterOptions()
|
||||
want := WriterOptions{NoData: -9999, TileSize: 512, Compression: "DEFLATE"}
|
||||
if got != want {
|
||||
t.Fatalf("DefaultWriterOptions() = %+v, want %+v", got, want)
|
||||
}
|
||||
}
|
||||
@@ -27,21 +27,22 @@ func NewDatasetRepository(pool *pgxpool.Pool) *DatasetRepository {
|
||||
// datasetColumns lists the dataset columns for SELECT and RETURNING. The
|
||||
// geometry is exposed as GeoJSON (jsonb) rather than its raw EWKB form, and a
|
||||
// bounding box array is derived for raster datasets only.
|
||||
const datasetColumns = `id, category_id, name, description, unit, filename, storage_key, cog_storage_key, file_type, size_bytes, content_type, properties, meta, automated, status, attribute_columns, kato_column, year_columns, parse_error, ST_AsGeoJSON(geometry)::jsonb AS geometry,
|
||||
const datasetColumns = `id, category_id, dg_id, to_char(start_date, 'YYYY-MM-DD') AS start_date, to_char(end_date, 'YYYY-MM-DD') AS end_date, name, description, unit, filename, storage_key, cog_storage_key, file_type, size_bytes, content_type, properties, meta, automated, status, attribute_columns, kato_column, year_columns, parse_error, ST_AsGeoJSON(geometry)::jsonb AS geometry,
|
||||
CASE WHEN file_type = 'raster' AND geometry IS NOT NULL
|
||||
THEN ARRAY[ST_XMin(geometry), ST_YMin(geometry), ST_XMax(geometry), ST_YMax(geometry)]
|
||||
ELSE NULL END AS bbox,
|
||||
geojson,
|
||||
geojson, sld_storage_key, sld_filename, sld_content_type,
|
||||
created_at, updated_at`
|
||||
|
||||
func scanDataset(row pgx.Row) (domain.Dataset, error) {
|
||||
var d domain.Dataset
|
||||
err := row.Scan(
|
||||
&d.ID, &d.CategoryID, &d.Name, &d.Description, &d.Unit,
|
||||
&d.ID, &d.CategoryID, &d.DatasetGroupID, &d.StartDate, &d.EndDate, &d.Name, &d.Description, &d.Unit,
|
||||
&d.Filename, &d.StorageKey, &d.CogStorageKey, &d.FileType, &d.SizeBytes, &d.ContentType,
|
||||
&d.Properties, &d.Meta, &d.Automated, &d.Status,
|
||||
&d.AttributeColumns, &d.KatoColumn, &d.YearColumns, &d.ParseError,
|
||||
&d.Geometry, &d.BBox, &d.GeoJSON, &d.CreatedAt, &d.UpdatedAt,
|
||||
&d.Geometry, &d.BBox, &d.GeoJSON, &d.SldStorageKey, &d.SldFilename, &d.SldContentType,
|
||||
&d.CreatedAt, &d.UpdatedAt,
|
||||
)
|
||||
return d, err
|
||||
}
|
||||
@@ -58,11 +59,28 @@ func nullableJSON(raw json.RawMessage) any {
|
||||
// Create inserts a new dataset and returns the stored row.
|
||||
func (r *DatasetRepository) Create(ctx context.Context, d domain.Dataset) (domain.Dataset, error) {
|
||||
row := r.pool.QueryRow(ctx,
|
||||
`INSERT INTO datasets (category_id, name, description, unit, filename, storage_key, file_type, size_bytes, content_type, properties, meta, automated, status)
|
||||
VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13)
|
||||
`INSERT INTO datasets (category_id, name, description, unit, filename, storage_key, file_type, size_bytes, content_type, properties, meta, automated, status, dg_id, start_date, end_date)
|
||||
VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13, $14, $15::date, $16::date)
|
||||
RETURNING `+datasetColumns,
|
||||
d.CategoryID, d.Name, d.Description, d.Unit, d.Filename, d.StorageKey, d.FileType, d.SizeBytes, d.ContentType,
|
||||
nullableJSON(d.Properties), nullableJSON(d.Meta), d.Automated, d.Status,
|
||||
d.DatasetGroupID, d.StartDate, d.EndDate,
|
||||
)
|
||||
out, err := scanDataset(row)
|
||||
return out, mapError(err)
|
||||
}
|
||||
|
||||
// UpdateFields overwrites a dataset's editable metadata columns. Nil pointers
|
||||
// clear the nullable columns. Returns domain.ErrNotFound if the dataset does
|
||||
// not exist, or domain.ErrConflict if the category/group does not exist.
|
||||
func (r *DatasetRepository) UpdateFields(ctx context.Context, id uuid.UUID, f domain.DatasetFields) (domain.Dataset, error) {
|
||||
row := r.pool.QueryRow(ctx,
|
||||
`UPDATE datasets
|
||||
SET category_id = $2, name = $3, description = $4, unit = $5,
|
||||
dg_id = $6, start_date = $7::date, end_date = $8::date, updated_at = now()
|
||||
WHERE id = $1
|
||||
RETURNING `+datasetColumns,
|
||||
id, f.CategoryID, f.Name, f.Description, f.Unit, f.DatasetGroupID, f.StartDate, f.EndDate,
|
||||
)
|
||||
out, err := scanDataset(row)
|
||||
return out, mapError(err)
|
||||
@@ -211,6 +229,38 @@ func (r *DatasetRepository) SetGeoJSON(ctx context.Context, id uuid.UUID, geojso
|
||||
return nil
|
||||
}
|
||||
|
||||
// SetSLD stores the dataset's SLD (Styled Layer Descriptor) style file
|
||||
// metadata, replacing any previous one, and returns the updated row.
|
||||
func (r *DatasetRepository) SetSLD(ctx context.Context, id uuid.UUID, key, filename, contentType string) (domain.Dataset, error) {
|
||||
row := r.pool.QueryRow(ctx,
|
||||
`UPDATE datasets
|
||||
SET sld_storage_key = $2, sld_filename = $3, sld_content_type = $4, updated_at = now()
|
||||
WHERE id = $1
|
||||
RETURNING `+datasetColumns,
|
||||
id, key, filename, contentType,
|
||||
)
|
||||
out, err := scanDataset(row)
|
||||
return out, mapError(err)
|
||||
}
|
||||
|
||||
// ClearGeometryAndGeoJSON nulls out a dataset's stored geometry and persisted
|
||||
// GeoJSON. Used before re-enqueuing a dataset's processing job so a stale
|
||||
// (possibly incorrect) geometry/GeoJSON is never served while the job is
|
||||
// pending; the async worker repopulates both once it completes.
|
||||
func (r *DatasetRepository) ClearGeometryAndGeoJSON(ctx context.Context, id uuid.UUID) error {
|
||||
tag, err := r.pool.Exec(ctx,
|
||||
`UPDATE datasets SET geometry = NULL, geojson = NULL, updated_at = now() WHERE id = $1`,
|
||||
id,
|
||||
)
|
||||
if err != nil {
|
||||
return mapError(err)
|
||||
}
|
||||
if tag.RowsAffected() == 0 {
|
||||
return domain.ErrNotFound
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// ReplaceObservations atomically replaces all observations for a dataset.
|
||||
func (r *DatasetRepository) ReplaceObservations(ctx context.Context, datasetID uuid.UUID, obs []domain.Observation) error {
|
||||
tx, err := r.pool.Begin(ctx)
|
||||
@@ -226,15 +276,23 @@ func (r *DatasetRepository) ReplaceObservations(ctx context.Context, datasetID u
|
||||
if len(obs) > 0 {
|
||||
rows := make([][]any, len(obs))
|
||||
for i, o := range obs {
|
||||
d, err := time.Parse("2006-01-02", o.Date)
|
||||
d, err := time.Parse("2006-01-02", o.StartDate)
|
||||
if err != nil {
|
||||
return fmt.Errorf("invalid observation date %q: %w", o.Date, err)
|
||||
return fmt.Errorf("invalid observation start_date %q: %w", o.StartDate, err)
|
||||
}
|
||||
rows[i] = []any{datasetID, o.KatoCode, d, o.Value, o.ValueText}
|
||||
var endDate any
|
||||
if o.EndDate != nil {
|
||||
dt, err := time.Parse("2006-01-02", *o.EndDate)
|
||||
if err != nil {
|
||||
return fmt.Errorf("invalid observation end_date %q: %w", *o.EndDate, err)
|
||||
}
|
||||
endDate = dt
|
||||
}
|
||||
rows[i] = []any{datasetID, o.KatoCode, d, endDate, o.Value, o.ValueText}
|
||||
}
|
||||
_, err := tx.CopyFrom(ctx,
|
||||
pgx.Identifier{"dataset_observations"},
|
||||
[]string{"dataset_id", "kato_code", "date", "value", "value_text"},
|
||||
[]string{"dataset_id", "kato_code", "start_date", "end_date", "value", "value_text"},
|
||||
pgx.CopyFromRows(rows),
|
||||
)
|
||||
if err != nil {
|
||||
@@ -245,16 +303,16 @@ func (r *DatasetRepository) ReplaceObservations(ctx context.Context, datasetID u
|
||||
return mapError(tx.Commit(ctx))
|
||||
}
|
||||
|
||||
const observationColumns = `id, dataset_id, kato_code, to_char(date, 'YYYY-MM-DD') AS date, value, value_text`
|
||||
const observationColumns = `id, dataset_id, kato_code, to_char(start_date, 'YYYY-MM-DD') AS start_date, to_char(end_date, 'YYYY-MM-DD') AS end_date, value, value_text`
|
||||
|
||||
func scanObservation(row pgx.Row) (domain.Observation, error) {
|
||||
var o domain.Observation
|
||||
err := row.Scan(&o.ID, &o.DatasetID, &o.KatoCode, &o.Date, &o.Value, &o.ValueText)
|
||||
err := row.Scan(&o.ID, &o.DatasetID, &o.KatoCode, &o.StartDate, &o.EndDate, &o.Value, &o.ValueText)
|
||||
return o, err
|
||||
}
|
||||
|
||||
// ListObservations returns a page of observations for a dataset, optionally
|
||||
// filtered by KATO code, ordered by (kato_code, date).
|
||||
// filtered by KATO code, ordered by (kato_code, start_date).
|
||||
func (r *DatasetRepository) ListObservations(ctx context.Context, datasetID uuid.UUID, katoCode *string, limit, offset int) ([]domain.Observation, error) {
|
||||
base := `SELECT ` + observationColumns + ` FROM dataset_observations WHERE dataset_id = $1`
|
||||
|
||||
@@ -264,11 +322,11 @@ func (r *DatasetRepository) ListObservations(ctx context.Context, datasetID uuid
|
||||
)
|
||||
if katoCode != nil {
|
||||
rows, err = r.pool.Query(ctx,
|
||||
base+` AND kato_code = $2 ORDER BY kato_code, date LIMIT $3 OFFSET $4`,
|
||||
base+` AND kato_code = $2 ORDER BY kato_code, start_date LIMIT $3 OFFSET $4`,
|
||||
datasetID, *katoCode, limit, offset)
|
||||
} else {
|
||||
rows, err = r.pool.Query(ctx,
|
||||
base+` ORDER BY kato_code, date LIMIT $2 OFFSET $3`,
|
||||
base+` ORDER BY kato_code, start_date LIMIT $2 OFFSET $3`,
|
||||
datasetID, limit, offset)
|
||||
}
|
||||
if err != nil {
|
||||
@@ -310,12 +368,25 @@ func (r *DatasetRepository) GetByID(ctx context.Context, id uuid.UUID) (domain.D
|
||||
return out, mapError(err)
|
||||
}
|
||||
|
||||
const datasetSummaryColumns = `id, category_id, name, description, unit, file_type, size_bytes, status, created_at, updated_at`
|
||||
// LatestInGroup returns the freshest dataset of a group: greatest end_date
|
||||
// (NULL end_date sorts last), ties broken by created_at DESC. status and
|
||||
// fileType optionally restrict candidates. Returns domain.ErrNotFound when no
|
||||
// dataset matches.
|
||||
func (r *DatasetRepository) LatestInGroup(ctx context.Context, groupID uuid.UUID, status *string, fileType *domain.FileType) (domain.Dataset, error) {
|
||||
where, args := datasetFilterClause(domain.DatasetFilter{DatasetGroupID: &groupID, Status: status, FileType: fileType}, nil)
|
||||
row := r.pool.QueryRow(ctx,
|
||||
`SELECT `+datasetColumns+` FROM datasets`+where+
|
||||
` ORDER BY end_date DESC NULLS LAST, created_at DESC LIMIT 1`, args...)
|
||||
out, err := scanDataset(row)
|
||||
return out, mapError(err)
|
||||
}
|
||||
|
||||
const datasetSummaryColumns = `id, category_id, dg_id, to_char(start_date, 'YYYY-MM-DD') AS start_date, to_char(end_date, 'YYYY-MM-DD') AS end_date, name, description, unit, file_type, size_bytes, status, created_at, updated_at`
|
||||
|
||||
func scanDatasetSummary(row pgx.Row) (domain.DatasetSummary, error) {
|
||||
var d domain.DatasetSummary
|
||||
err := row.Scan(
|
||||
&d.ID, &d.CategoryID, &d.Name, &d.Description, &d.Unit,
|
||||
&d.ID, &d.CategoryID, &d.DatasetGroupID, &d.StartDate, &d.EndDate, &d.Name, &d.Description, &d.Unit,
|
||||
&d.FileType, &d.SizeBytes, &d.Status, &d.CreatedAt, &d.UpdatedAt,
|
||||
)
|
||||
return d, err
|
||||
@@ -342,6 +413,22 @@ func datasetFilterClause(f domain.DatasetFilter, args []any) (string, []any) {
|
||||
args = append(args, *f.Status)
|
||||
conds = append(conds, fmt.Sprintf("status = $%d", len(args)))
|
||||
}
|
||||
if f.DatasetGroupID != nil {
|
||||
args = append(args, *f.DatasetGroupID)
|
||||
conds = append(conds, fmt.Sprintf("dg_id = $%d", len(args)))
|
||||
}
|
||||
for _, b := range []struct {
|
||||
col, op string
|
||||
val *string
|
||||
}{
|
||||
{"start_date", ">=", f.StartDateFrom}, {"start_date", "<=", f.StartDateTo},
|
||||
{"end_date", ">=", f.EndDateFrom}, {"end_date", "<=", f.EndDateTo},
|
||||
} {
|
||||
if b.val != nil {
|
||||
args = append(args, *b.val)
|
||||
conds = append(conds, fmt.Sprintf("%s %s $%d::date", b.col, b.op, len(args)))
|
||||
}
|
||||
}
|
||||
if len(conds) == 0 {
|
||||
return "", args
|
||||
}
|
||||
|
||||
115
internal/repository/postgres/dataset_db_test.go
Normal file
115
internal/repository/postgres/dataset_db_test.go
Normal file
@@ -0,0 +1,115 @@
|
||||
package postgres
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"gis/internal/domain"
|
||||
|
||||
"github.com/google/uuid"
|
||||
)
|
||||
|
||||
// TestDatasetRepository_GroupsPeriodsAndLatest exercises the Postgres-only
|
||||
// parts of the dataset/group repositories added with dataset periods: the
|
||||
// ::date casts on *string parameters, the date filter clause, and the
|
||||
// "freshest in group" ordering (end_date DESC NULLS LAST, created_at DESC).
|
||||
func TestDatasetRepository_GroupsPeriodsAndLatest(t *testing.T) {
|
||||
pool := testPool(t)
|
||||
ctx := context.Background()
|
||||
groups := NewDatasetGroupRepository(pool)
|
||||
datasets := NewDatasetRepository(pool)
|
||||
categories := NewCategoryRepository(pool)
|
||||
|
||||
suffix := uuid.NewString()[:8]
|
||||
cat, err := categories.Create(ctx, domain.Category{Code: "dbtest-" + suffix, Name: "db test"})
|
||||
if err != nil {
|
||||
t.Fatalf("create category: %v", err)
|
||||
}
|
||||
t.Cleanup(func() { _, _ = pool.Exec(ctx, `DELETE FROM categories WHERE id = $1`, cat.ID) })
|
||||
|
||||
group, err := groups.Create(ctx, domain.DatasetGroup{Code: "dbtest-group-" + suffix, Name: "db test group"})
|
||||
if err != nil {
|
||||
t.Fatalf("create group: %v", err)
|
||||
}
|
||||
t.Cleanup(func() { _, _ = pool.Exec(ctx, `DELETE FROM dataset_groups WHERE id = $1`, group.ID) })
|
||||
|
||||
s := func(v string) *string { return &v }
|
||||
mk := func(name string, start, end *string) domain.Dataset {
|
||||
d, err := datasets.Create(ctx, domain.Dataset{
|
||||
CategoryID: cat.ID, DatasetGroupID: &group.ID, Name: name, Filename: name + ".tif",
|
||||
StorageKey: "dbtest/" + suffix + "/" + name, FileType: domain.FileTypeRaster,
|
||||
ContentType: "image/tiff", Status: domain.DatasetStatusReady,
|
||||
StartDate: start, EndDate: end,
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("create dataset %s: %v", name, err)
|
||||
}
|
||||
t.Cleanup(func() { _, _ = pool.Exec(ctx, `DELETE FROM datasets WHERE id = $1`, d.ID) })
|
||||
time.Sleep(5 * time.Millisecond) // distinct created_at for the tie-break
|
||||
return d
|
||||
}
|
||||
|
||||
old := mk("old", s("2019-01-01"), s("2019-12-31"))
|
||||
fresh := mk("fresh", s("2020-01-01"), s("2020-12-31"))
|
||||
undated := mk("undated", nil, nil)
|
||||
|
||||
if old.StartDate == nil || *old.StartDate != "2019-01-01" || *old.EndDate != "2019-12-31" || old.DatasetGroupID == nil {
|
||||
t.Fatalf("dates/group not round-tripped: %+v", old)
|
||||
}
|
||||
|
||||
latest, err := datasets.LatestInGroup(ctx, group.ID, nil, nil)
|
||||
if err != nil {
|
||||
t.Fatalf("LatestInGroup: %v", err)
|
||||
}
|
||||
if latest.ID != fresh.ID {
|
||||
t.Fatalf("latest = %s, want fresh %s (undated %s must sort last)", latest.Name, fresh.ID, undated.ID)
|
||||
}
|
||||
ft := domain.FileTypeVector
|
||||
if _, err := datasets.LatestInGroup(ctx, group.ID, nil, &ft); !errors.Is(err, domain.ErrNotFound) {
|
||||
t.Fatalf("no vector in group: err = %v, want ErrNotFound", err)
|
||||
}
|
||||
|
||||
page, err := datasets.ListSummaries(ctx, domain.DatasetFilter{
|
||||
DatasetGroupID: &group.ID, EndDateFrom: s("2020-01-01"), EndDateTo: s("2020-12-31"),
|
||||
}, 10, 0)
|
||||
if err != nil {
|
||||
t.Fatalf("ListSummaries with date bounds: %v", err)
|
||||
}
|
||||
if len(page) != 1 || page[0].ID != fresh.ID {
|
||||
t.Fatalf("date filter returned %+v, want only fresh", page)
|
||||
}
|
||||
n, err := datasets.Count(ctx, domain.DatasetFilter{DatasetGroupID: &group.ID, StartDateFrom: s("2019-06-01")})
|
||||
if err != nil || n != 1 {
|
||||
t.Fatalf("Count(start_date_from) = %d, %v; want 1", n, err)
|
||||
}
|
||||
|
||||
updated, err := datasets.UpdateFields(ctx, undated.ID, domain.DatasetFields{
|
||||
CategoryID: cat.ID, Name: "undated-now-dated", DatasetGroupID: &group.ID,
|
||||
StartDate: s("2021-01-01"), EndDate: s("2021-12-31"),
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("UpdateFields: %v", err)
|
||||
}
|
||||
if updated.EndDate == nil || *updated.EndDate != "2021-12-31" {
|
||||
t.Fatalf("UpdateFields did not store the period: %+v", updated)
|
||||
}
|
||||
if _, err := datasets.UpdateFields(ctx, undated.ID, domain.DatasetFields{
|
||||
CategoryID: cat.ID, Name: "bad", StartDate: s("2021-12-31"), EndDate: s("2021-01-01"),
|
||||
}); !errors.Is(err, domain.ErrValidation) {
|
||||
t.Fatalf("reversed period must hit the CHECK constraint: err = %v", err)
|
||||
}
|
||||
|
||||
if err := groups.Delete(ctx, group.ID); !errors.Is(err, domain.ErrConflict) {
|
||||
t.Fatalf("deleting a referenced group: err = %v, want ErrConflict", err)
|
||||
}
|
||||
g2, err := groups.SetSLD(ctx, group.ID, "k", "style.sld", "application/xml")
|
||||
if err != nil || g2.SldStorageKey == nil || *g2.SldStorageKey != "k" {
|
||||
t.Fatalf("SetSLD: %+v, %v", g2, err)
|
||||
}
|
||||
byCode, err := groups.GetByCode(ctx, group.Code)
|
||||
if err != nil || byCode.ID != group.ID {
|
||||
t.Fatalf("GetByCode: %+v, %v", byCode, err)
|
||||
}
|
||||
}
|
||||
117
internal/repository/postgres/dataset_group.go
Normal file
117
internal/repository/postgres/dataset_group.go
Normal file
@@ -0,0 +1,117 @@
|
||||
package postgres
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
|
||||
"gis/internal/domain"
|
||||
|
||||
"github.com/google/uuid"
|
||||
"github.com/jackc/pgx/v5"
|
||||
"github.com/jackc/pgx/v5/pgxpool"
|
||||
)
|
||||
|
||||
// DatasetGroupRepository persists dataset groups in Postgres.
|
||||
type DatasetGroupRepository struct{ pool *pgxpool.Pool }
|
||||
|
||||
// NewDatasetGroupRepository returns a DatasetGroupRepository backed by pool.
|
||||
func NewDatasetGroupRepository(pool *pgxpool.Pool) *DatasetGroupRepository {
|
||||
return &DatasetGroupRepository{pool: pool}
|
||||
}
|
||||
|
||||
const datasetGroupColumns = `id, code, name, description, sld_storage_key, sld_filename, sld_content_type, created_at, updated_at`
|
||||
|
||||
func scanDatasetGroup(row pgx.Row) (domain.DatasetGroup, error) {
|
||||
var g domain.DatasetGroup
|
||||
err := row.Scan(&g.ID, &g.Code, &g.Name, &g.Description, &g.SldStorageKey, &g.SldFilename, &g.SldContentType, &g.CreatedAt, &g.UpdatedAt)
|
||||
return g, err
|
||||
}
|
||||
|
||||
// Create inserts a new dataset group and returns the stored row.
|
||||
func (r *DatasetGroupRepository) Create(ctx context.Context, g domain.DatasetGroup) (domain.DatasetGroup, error) {
|
||||
row := r.pool.QueryRow(ctx,
|
||||
`INSERT INTO dataset_groups (code, name, description) VALUES ($1, $2, $3) RETURNING `+datasetGroupColumns,
|
||||
g.Code, g.Name, g.Description)
|
||||
out, err := scanDatasetGroup(row)
|
||||
return out, mapError(err)
|
||||
}
|
||||
|
||||
// GetByID returns the group with the given id.
|
||||
func (r *DatasetGroupRepository) GetByID(ctx context.Context, id uuid.UUID) (domain.DatasetGroup, error) {
|
||||
out, err := scanDatasetGroup(r.pool.QueryRow(ctx, `SELECT `+datasetGroupColumns+` FROM dataset_groups WHERE id = $1`, id))
|
||||
return out, mapError(err)
|
||||
}
|
||||
|
||||
// GetByCode returns the group with the given (unique) code.
|
||||
func (r *DatasetGroupRepository) GetByCode(ctx context.Context, code string) (domain.DatasetGroup, error) {
|
||||
out, err := scanDatasetGroup(r.pool.QueryRow(ctx, `SELECT `+datasetGroupColumns+` FROM dataset_groups WHERE code = $1`, code))
|
||||
return out, mapError(err)
|
||||
}
|
||||
|
||||
// List returns a page of groups ordered by code; code (exact) is optional.
|
||||
func (r *DatasetGroupRepository) List(ctx context.Context, code *string, limit, offset int) ([]domain.DatasetGroup, error) {
|
||||
where, args := groupWhere(code)
|
||||
args = append(args, limit, offset)
|
||||
rows, err := r.pool.Query(ctx, fmt.Sprintf(`SELECT %s FROM dataset_groups%s ORDER BY code LIMIT $%d OFFSET $%d`,
|
||||
datasetGroupColumns, where, len(args)-1, len(args)), args...)
|
||||
if err != nil {
|
||||
return nil, mapError(err)
|
||||
}
|
||||
defer rows.Close()
|
||||
out := make([]domain.DatasetGroup, 0)
|
||||
for rows.Next() {
|
||||
g, err := scanDatasetGroup(rows)
|
||||
if err != nil {
|
||||
return nil, mapError(err)
|
||||
}
|
||||
out = append(out, g)
|
||||
}
|
||||
return out, mapError(rows.Err())
|
||||
}
|
||||
|
||||
// Count returns how many groups match the optional code filter.
|
||||
func (r *DatasetGroupRepository) Count(ctx context.Context, code *string) (int, error) {
|
||||
where, args := groupWhere(code)
|
||||
var n int
|
||||
err := r.pool.QueryRow(ctx, `SELECT count(*) FROM dataset_groups`+where, args...).Scan(&n)
|
||||
return n, mapError(err)
|
||||
}
|
||||
|
||||
// groupWhere builds the optional exact-code WHERE clause and its arguments.
|
||||
func groupWhere(code *string) (string, []any) {
|
||||
if code == nil {
|
||||
return "", nil
|
||||
}
|
||||
return " WHERE code = $1", []any{*code}
|
||||
}
|
||||
|
||||
// Update replaces a group's editable fields and returns the stored row.
|
||||
func (r *DatasetGroupRepository) Update(ctx context.Context, g domain.DatasetGroup) (domain.DatasetGroup, error) {
|
||||
row := r.pool.QueryRow(ctx,
|
||||
`UPDATE dataset_groups SET code = $2, name = $3, description = $4, updated_at = now() WHERE id = $1 RETURNING `+datasetGroupColumns,
|
||||
g.ID, g.Code, g.Name, g.Description)
|
||||
out, err := scanDatasetGroup(row)
|
||||
return out, mapError(err)
|
||||
}
|
||||
|
||||
// Delete removes a group. Returns domain.ErrNotFound if absent and
|
||||
// domain.ErrConflict (via the FK) if datasets still reference it.
|
||||
func (r *DatasetGroupRepository) Delete(ctx context.Context, id uuid.UUID) error {
|
||||
tag, err := r.pool.Exec(ctx, `DELETE FROM dataset_groups WHERE id = $1`, id)
|
||||
if err != nil {
|
||||
return mapError(err)
|
||||
}
|
||||
if tag.RowsAffected() == 0 {
|
||||
return domain.ErrNotFound
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// SetSLD points the group at a stored SLD style object.
|
||||
func (r *DatasetGroupRepository) SetSLD(ctx context.Context, id uuid.UUID, key, filename, contentType string) (domain.DatasetGroup, error) {
|
||||
row := r.pool.QueryRow(ctx,
|
||||
`UPDATE dataset_groups SET sld_storage_key = $2, sld_filename = $3, sld_content_type = $4, updated_at = now() WHERE id = $1 RETURNING `+datasetGroupColumns,
|
||||
id, key, filename, contentType)
|
||||
out, err := scanDatasetGroup(row)
|
||||
return out, mapError(err)
|
||||
}
|
||||
@@ -9,14 +9,14 @@ import (
|
||||
)
|
||||
|
||||
// ListAllObservations returns every observation for a dataset ordered by
|
||||
// (kato_code, date). Unlike ListObservations it is unpaginated: it backs GeoJSON
|
||||
// (kato_code, start_date). Unlike ListObservations it is unpaginated: it backs GeoJSON
|
||||
// assembly, where all of a dataset's observations are needed at once.
|
||||
func (r *DatasetRepository) ListAllObservations(ctx context.Context, datasetID uuid.UUID) ([]domain.Observation, error) {
|
||||
rows, err := r.pool.Query(ctx,
|
||||
`SELECT `+observationColumns+`
|
||||
FROM dataset_observations
|
||||
WHERE dataset_id = $1
|
||||
ORDER BY kato_code, date`,
|
||||
ORDER BY kato_code, start_date`,
|
||||
datasetID)
|
||||
if err != nil {
|
||||
return nil, mapError(err)
|
||||
|
||||
239
internal/repository/postgres/landscape.go
Normal file
239
internal/repository/postgres/landscape.go
Normal file
@@ -0,0 +1,239 @@
|
||||
package postgres
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
|
||||
"gis/internal/domain"
|
||||
|
||||
"github.com/google/uuid"
|
||||
"github.com/jackc/pgx/v5"
|
||||
"github.com/jackc/pgx/v5/pgxpool"
|
||||
)
|
||||
|
||||
// LandscapeRepository persists Landscape Program runs in Postgres.
|
||||
type LandscapeRepository struct {
|
||||
pool *pgxpool.Pool
|
||||
}
|
||||
|
||||
// NewLandscapeRepository returns a LandscapeRepository backed by the given pool.
|
||||
func NewLandscapeRepository(pool *pgxpool.Pool) *LandscapeRepository {
|
||||
return &LandscapeRepository{pool: pool}
|
||||
}
|
||||
|
||||
const runColumns = `id, status, from_stage, params, overrides, error, created_at, started_at, finished_at`
|
||||
|
||||
// scanRun scans a run row. overrides is stored as jsonb but scanned into
|
||||
// []byte and unmarshalled manually so an empty/NULL value maps to an empty
|
||||
// map rather than nil.
|
||||
func scanRun(row pgx.Row) (domain.LPRun, error) {
|
||||
var r domain.LPRun
|
||||
var overrides []byte
|
||||
err := row.Scan(&r.ID, &r.Status, &r.FromStage, &r.Params, &overrides, &r.Error, &r.CreatedAt, &r.StartedAt, &r.FinishedAt)
|
||||
if err != nil {
|
||||
return r, err
|
||||
}
|
||||
r.Overrides = map[string]uuid.UUID{}
|
||||
if len(overrides) > 0 {
|
||||
if err := json.Unmarshal(overrides, &r.Overrides); err != nil {
|
||||
return r, err
|
||||
}
|
||||
}
|
||||
return r, nil
|
||||
}
|
||||
|
||||
const nodeColumns = `id, run_id, node_code, stage, status, fingerprint, input_datasets, output_datasets, error, started_at, finished_at`
|
||||
|
||||
func scanNode(row pgx.Row) (domain.LPRunNode, error) {
|
||||
var n domain.LPRunNode
|
||||
err := row.Scan(&n.ID, &n.RunID, &n.NodeCode, &n.Stage, &n.Status, &n.Fingerprint, &n.Inputs, &n.Outputs, &n.Error, &n.StartedAt, &n.FinishedAt)
|
||||
return n, err
|
||||
}
|
||||
|
||||
// CreateRun inserts a new run and its nodes (status pending) in one
|
||||
// transaction, and returns the run with its nodes populated (in the order
|
||||
// they were given).
|
||||
func (r *LandscapeRepository) CreateRun(ctx context.Context, run domain.LPRun) (domain.LPRun, error) {
|
||||
tx, err := r.pool.Begin(ctx)
|
||||
if err != nil {
|
||||
return domain.LPRun{}, mapError(err)
|
||||
}
|
||||
defer tx.Rollback(ctx)
|
||||
|
||||
overrides := run.Overrides
|
||||
if overrides == nil {
|
||||
overrides = map[string]uuid.UUID{}
|
||||
}
|
||||
overridesJSON, err := json.Marshal(overrides)
|
||||
if err != nil {
|
||||
return domain.LPRun{}, err
|
||||
}
|
||||
|
||||
row := tx.QueryRow(ctx,
|
||||
`INSERT INTO lp_runs (status, from_stage, params, overrides)
|
||||
VALUES ($1, $2, $3, $4)
|
||||
RETURNING `+runColumns,
|
||||
run.Status, run.FromStage, run.Params, overridesJSON,
|
||||
)
|
||||
out, err := scanRun(row)
|
||||
if err != nil {
|
||||
return domain.LPRun{}, mapError(err)
|
||||
}
|
||||
|
||||
nodes := make([]domain.LPRunNode, 0, len(run.Nodes))
|
||||
for _, n := range run.Nodes {
|
||||
row := tx.QueryRow(ctx,
|
||||
`INSERT INTO lp_run_nodes (run_id, node_code, stage, status)
|
||||
VALUES ($1, $2, $3, $4)
|
||||
RETURNING `+nodeColumns,
|
||||
out.ID, n.NodeCode, n.Stage, domain.LPNodePending,
|
||||
)
|
||||
node, err := scanNode(row)
|
||||
if err != nil {
|
||||
return domain.LPRun{}, mapError(err)
|
||||
}
|
||||
nodes = append(nodes, node)
|
||||
}
|
||||
out.Nodes = nodes
|
||||
|
||||
if err := tx.Commit(ctx); err != nil {
|
||||
return domain.LPRun{}, mapError(err)
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
|
||||
// GetRun returns the run with the given id, with its nodes ordered by
|
||||
// node_code. Returns domain.ErrNotFound if the run does not exist.
|
||||
func (r *LandscapeRepository) GetRun(ctx context.Context, id uuid.UUID) (domain.LPRun, error) {
|
||||
row := r.pool.QueryRow(ctx, `SELECT `+runColumns+` FROM lp_runs WHERE id = $1`, id)
|
||||
out, err := scanRun(row)
|
||||
if err != nil {
|
||||
return domain.LPRun{}, mapError(err)
|
||||
}
|
||||
|
||||
rows, err := r.pool.Query(ctx,
|
||||
`SELECT `+nodeColumns+` FROM lp_run_nodes WHERE run_id = $1 ORDER BY node_code`, id)
|
||||
if err != nil {
|
||||
return domain.LPRun{}, mapError(err)
|
||||
}
|
||||
defer rows.Close()
|
||||
|
||||
nodes := make([]domain.LPRunNode, 0)
|
||||
for rows.Next() {
|
||||
n, err := scanNode(rows)
|
||||
if err != nil {
|
||||
return domain.LPRun{}, mapError(err)
|
||||
}
|
||||
nodes = append(nodes, n)
|
||||
}
|
||||
if err := rows.Err(); err != nil {
|
||||
return domain.LPRun{}, mapError(err)
|
||||
}
|
||||
out.Nodes = nodes
|
||||
return out, nil
|
||||
}
|
||||
|
||||
// ListRuns returns a page of runs (without nodes), newest first.
|
||||
func (r *LandscapeRepository) ListRuns(ctx context.Context, limit, offset int) ([]domain.LPRun, error) {
|
||||
rows, err := r.pool.Query(ctx,
|
||||
`SELECT `+runColumns+` FROM lp_runs ORDER BY created_at DESC LIMIT $1 OFFSET $2`, limit, offset)
|
||||
if err != nil {
|
||||
return nil, mapError(err)
|
||||
}
|
||||
defer rows.Close()
|
||||
|
||||
runs := make([]domain.LPRun, 0)
|
||||
for rows.Next() {
|
||||
run, err := scanRun(rows)
|
||||
if err != nil {
|
||||
return nil, mapError(err)
|
||||
}
|
||||
runs = append(runs, run)
|
||||
}
|
||||
return runs, mapError(rows.Err())
|
||||
}
|
||||
|
||||
// CountRuns returns the total number of runs.
|
||||
func (r *LandscapeRepository) CountRuns(ctx context.Context) (int, error) {
|
||||
var n int
|
||||
err := r.pool.QueryRow(ctx, `SELECT count(*) FROM lp_runs`).Scan(&n)
|
||||
return n, mapError(err)
|
||||
}
|
||||
|
||||
// SetRunStatus updates a run's status and error. Transitioning to running
|
||||
// sets started_at; transitioning to succeeded or failed sets finished_at.
|
||||
// Returns domain.ErrNotFound if the run does not exist.
|
||||
func (r *LandscapeRepository) SetRunStatus(ctx context.Context, id uuid.UUID, status domain.LPRunStatus, errMsg *string) error {
|
||||
tag, err := r.pool.Exec(ctx,
|
||||
// $2 is used both as the assigned value (varchar column) and in the CASE
|
||||
// comparisons (text literals); the explicit ::text casts keep Postgres
|
||||
// from deducing two different types for one parameter (SQLSTATE 42P08).
|
||||
`UPDATE lp_runs SET status = $2::text, error = $3,
|
||||
started_at = CASE WHEN $2::text = 'running' THEN now() ELSE started_at END,
|
||||
finished_at = CASE WHEN $2::text IN ('succeeded','failed') THEN now() ELSE finished_at END
|
||||
WHERE id = $1`,
|
||||
id, status, errMsg,
|
||||
)
|
||||
if err != nil {
|
||||
return mapError(err)
|
||||
}
|
||||
if tag.RowsAffected() == 0 {
|
||||
return domain.ErrNotFound
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// SetNodeStarted marks a run's node as running, storing its fingerprint and
|
||||
// resolved inputs, and clears any previous error. Returns domain.ErrNotFound
|
||||
// if no such node exists.
|
||||
func (r *LandscapeRepository) SetNodeStarted(ctx context.Context, runID uuid.UUID, code string, fingerprint string, inputs []domain.LPDatasetRef) error {
|
||||
tag, err := r.pool.Exec(ctx,
|
||||
`UPDATE lp_run_nodes
|
||||
SET status = 'running', fingerprint = $3, input_datasets = $4, started_at = now(), error = NULL
|
||||
WHERE run_id = $1 AND node_code = $2`,
|
||||
runID, code, fingerprint, inputs,
|
||||
)
|
||||
if err != nil {
|
||||
return mapError(err)
|
||||
}
|
||||
if tag.RowsAffected() == 0 {
|
||||
return domain.ErrNotFound
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// SetNodeFinished marks a run's node as finished with the given status,
|
||||
// storing its outputs and error. Returns domain.ErrNotFound if no such node
|
||||
// exists.
|
||||
func (r *LandscapeRepository) SetNodeFinished(ctx context.Context, runID uuid.UUID, code string, status domain.LPNodeStatus, outputs []domain.LPDatasetRef, errMsg *string) error {
|
||||
tag, err := r.pool.Exec(ctx,
|
||||
`UPDATE lp_run_nodes
|
||||
SET status = $3, output_datasets = $4, error = $5, finished_at = now()
|
||||
WHERE run_id = $1 AND node_code = $2`,
|
||||
runID, code, status, outputs, errMsg,
|
||||
)
|
||||
if err != nil {
|
||||
return mapError(err)
|
||||
}
|
||||
if tag.RowsAffected() == 0 {
|
||||
return domain.ErrNotFound
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// FindReusableOutputs returns the outputs of the most recently succeeded node
|
||||
// with the same code and fingerprint, excluding excludeRun, so its outputs
|
||||
// may be reused instead of recomputed. Returns domain.ErrNotFound if none.
|
||||
func (r *LandscapeRepository) FindReusableOutputs(ctx context.Context, code, fingerprint string, excludeRun uuid.UUID) ([]domain.LPDatasetRef, error) {
|
||||
var outputs []domain.LPDatasetRef
|
||||
err := r.pool.QueryRow(ctx,
|
||||
`SELECT output_datasets FROM lp_run_nodes
|
||||
WHERE node_code = $1 AND fingerprint = $2 AND status = 'succeeded' AND run_id <> $3 AND output_datasets IS NOT NULL
|
||||
ORDER BY finished_at DESC LIMIT 1`,
|
||||
code, fingerprint, excludeRun,
|
||||
).Scan(&outputs)
|
||||
if err != nil {
|
||||
return nil, mapError(err)
|
||||
}
|
||||
return outputs, nil
|
||||
}
|
||||
151
internal/repository/postgres/landscape_db_test.go
Normal file
151
internal/repository/postgres/landscape_db_test.go
Normal file
@@ -0,0 +1,151 @@
|
||||
package postgres
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"os"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"gis/internal/domain"
|
||||
|
||||
"github.com/google/uuid"
|
||||
"github.com/jackc/pgx/v5/pgxpool"
|
||||
)
|
||||
|
||||
// testPool connects to the database named by TEST_DB_URL (migrations already
|
||||
// applied) or skips the test. Repository SQL is otherwise only exercised by
|
||||
// hand, so these tests are the guard against Postgres-only failures such as
|
||||
// parameter type deduction (SQLSTATE 42P08) that no stub can reproduce.
|
||||
func testPool(t *testing.T) *pgxpool.Pool {
|
||||
t.Helper()
|
||||
url := os.Getenv("TEST_DB_URL")
|
||||
if url == "" {
|
||||
t.Skip("TEST_DB_URL not set; skipping database test")
|
||||
}
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
|
||||
defer cancel()
|
||||
pool, err := Connect(ctx, url)
|
||||
if err != nil {
|
||||
t.Fatalf("connect: %v", err)
|
||||
}
|
||||
t.Cleanup(pool.Close)
|
||||
return pool
|
||||
}
|
||||
|
||||
func TestLandscapeRepository_SetRunStatus_AllTransitions(t *testing.T) {
|
||||
pool := testPool(t)
|
||||
repo := NewLandscapeRepository(pool)
|
||||
ctx := context.Background()
|
||||
|
||||
run, err := repo.CreateRun(ctx, domain.LPRun{
|
||||
Status: domain.LPRunQueued,
|
||||
Params: []byte(`{}`),
|
||||
Nodes: []domain.LPRunNode{{NodeCode: "00", Stage: 0, Status: domain.LPNodePending}},
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("CreateRun: %v", err)
|
||||
}
|
||||
t.Cleanup(func() { _, _ = pool.Exec(ctx, `DELETE FROM lp_runs WHERE id = $1`, run.ID) })
|
||||
|
||||
if err := repo.SetRunStatus(ctx, run.ID, domain.LPRunRunning, nil); err != nil {
|
||||
t.Fatalf("SetRunStatus(running): %v", err)
|
||||
}
|
||||
got, err := repo.GetRun(ctx, run.ID)
|
||||
if err != nil {
|
||||
t.Fatalf("GetRun: %v", err)
|
||||
}
|
||||
if got.Status != domain.LPRunRunning || got.StartedAt == nil || got.FinishedAt != nil {
|
||||
t.Fatalf("after running: status=%s started=%v finished=%v", got.Status, got.StartedAt, got.FinishedAt)
|
||||
}
|
||||
|
||||
msg := "boom"
|
||||
if err := repo.SetRunStatus(ctx, run.ID, domain.LPRunFailed, &msg); err != nil {
|
||||
t.Fatalf("SetRunStatus(failed): %v", err)
|
||||
}
|
||||
got, err = repo.GetRun(ctx, run.ID)
|
||||
if err != nil {
|
||||
t.Fatalf("GetRun: %v", err)
|
||||
}
|
||||
if got.Status != domain.LPRunFailed || got.FinishedAt == nil || got.Error == nil || *got.Error != msg {
|
||||
t.Fatalf("after failed: status=%s finished=%v error=%v", got.Status, got.FinishedAt, got.Error)
|
||||
}
|
||||
|
||||
if err := repo.SetRunStatus(ctx, run.ID, domain.LPRunSucceeded, nil); err != nil {
|
||||
t.Fatalf("SetRunStatus(succeeded): %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestLandscapeRepository_NodeLifecycleAndReuse(t *testing.T) {
|
||||
pool := testPool(t)
|
||||
repo := NewLandscapeRepository(pool)
|
||||
ctx := context.Background()
|
||||
|
||||
newRun := func() domain.LPRun {
|
||||
run, err := repo.CreateRun(ctx, domain.LPRun{
|
||||
Status: domain.LPRunQueued,
|
||||
Params: []byte(`{"window_size":2048}`),
|
||||
Overrides: map[string]uuid.UUID{"lp-e": uuid.New()},
|
||||
Nodes: []domain.LPRunNode{{NodeCode: "00", Stage: 0, Status: domain.LPNodePending}},
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("CreateRun: %v", err)
|
||||
}
|
||||
t.Cleanup(func() { _, _ = pool.Exec(ctx, `DELETE FROM lp_runs WHERE id = $1`, run.ID) })
|
||||
return run
|
||||
}
|
||||
|
||||
inputs := []domain.LPDatasetRef{{Group: "lp-z-wr", DatasetID: uuid.New()}}
|
||||
outputs := []domain.LPDatasetRef{{Group: "lp-z-integrated", DatasetID: uuid.New()}}
|
||||
|
||||
first := newRun()
|
||||
if err := repo.SetNodeStarted(ctx, first.ID, "00", "fp-1", inputs); err != nil {
|
||||
t.Fatalf("SetNodeStarted: %v", err)
|
||||
}
|
||||
if err := repo.SetNodeFinished(ctx, first.ID, "00", domain.LPNodeSucceeded, outputs, nil); err != nil {
|
||||
t.Fatalf("SetNodeFinished: %v", err)
|
||||
}
|
||||
got, err := repo.GetRun(ctx, first.ID)
|
||||
if err != nil {
|
||||
t.Fatalf("GetRun: %v", err)
|
||||
}
|
||||
if len(got.Nodes) != 1 || got.Nodes[0].Status != domain.LPNodeSucceeded ||
|
||||
len(got.Nodes[0].Inputs) != 1 || got.Nodes[0].Inputs[0] != inputs[0] ||
|
||||
len(got.Nodes[0].Outputs) != 1 || got.Nodes[0].Outputs[0] != outputs[0] ||
|
||||
got.Nodes[0].StartedAt == nil || got.Nodes[0].FinishedAt == nil {
|
||||
t.Fatalf("node row not round-tripped: %+v", got.Nodes)
|
||||
}
|
||||
if len(got.Overrides) != 1 {
|
||||
t.Fatalf("overrides not round-tripped: %+v", got.Overrides)
|
||||
}
|
||||
|
||||
second := newRun()
|
||||
reused, err := repo.FindReusableOutputs(ctx, "00", "fp-1", second.ID)
|
||||
if err != nil {
|
||||
t.Fatalf("FindReusableOutputs: %v", err)
|
||||
}
|
||||
if len(reused) != 1 || reused[0] != outputs[0] {
|
||||
t.Fatalf("reused = %+v, want %+v", reused, outputs)
|
||||
}
|
||||
if _, err := repo.FindReusableOutputs(ctx, "00", "fp-other", second.ID); !errors.Is(err, domain.ErrNotFound) {
|
||||
t.Fatalf("unknown fingerprint: err = %v, want ErrNotFound", err)
|
||||
}
|
||||
if _, err := repo.FindReusableOutputs(ctx, "00", "fp-1", first.ID); !errors.Is(err, domain.ErrNotFound) {
|
||||
t.Fatalf("own run must be excluded: err = %v, want ErrNotFound", err)
|
||||
}
|
||||
|
||||
failMsg := "grid mismatch"
|
||||
if err := repo.SetNodeFinished(ctx, second.ID, "00", domain.LPNodeFailed, nil, &failMsg); err != nil {
|
||||
t.Fatalf("SetNodeFinished(failed): %v", err)
|
||||
}
|
||||
runs, err := repo.ListRuns(ctx, 10, 0)
|
||||
if err != nil {
|
||||
t.Fatalf("ListRuns: %v", err)
|
||||
}
|
||||
if len(runs) < 2 || runs[0].ID != second.ID {
|
||||
t.Fatalf("ListRuns must be newest first: %+v", runs)
|
||||
}
|
||||
if n, err := repo.CountRuns(ctx); err != nil || n < 2 {
|
||||
t.Fatalf("CountRuns = %d, %v", n, err)
|
||||
}
|
||||
}
|
||||
@@ -22,10 +22,23 @@ import (
|
||||
// maxParseBytes caps how much of a file is read into memory for parsing.
|
||||
const maxParseBytes = 256 << 20 // 256 MiB
|
||||
|
||||
var (
|
||||
// ErrNotRaster is returned when a raster-only operation is requested for a
|
||||
// non-raster dataset.
|
||||
ErrNotRaster = errors.New("dataset is not a raster")
|
||||
// ErrCOGNotReady is returned when a raster's Cloud-Optimized GeoTIFF has not
|
||||
// been produced yet.
|
||||
ErrCOGNotReady = errors.New("cog is not ready")
|
||||
// ErrSLDNotFound is returned when a dataset has no SLD style file uploaded.
|
||||
ErrSLDNotFound = fmt.Errorf("%w: dataset has no sld", domain.ErrNotFound)
|
||||
)
|
||||
|
||||
// DatasetRepository is the persistence behaviour DatasetService needs.
|
||||
type DatasetRepository interface {
|
||||
Create(ctx context.Context, d domain.Dataset) (domain.Dataset, error)
|
||||
GetByID(ctx context.Context, id uuid.UUID) (domain.Dataset, error)
|
||||
UpdateFields(ctx context.Context, id uuid.UUID, f domain.DatasetFields) (domain.Dataset, error)
|
||||
LatestInGroup(ctx context.Context, groupID uuid.UUID, status *string, fileType *domain.FileType) (domain.Dataset, error)
|
||||
ListSummaries(ctx context.Context, filter domain.DatasetFilter, limit, offset int) ([]domain.DatasetSummary, error)
|
||||
Count(ctx context.Context, filter domain.DatasetFilter) (int, error)
|
||||
Delete(ctx context.Context, id uuid.UUID) error
|
||||
@@ -35,7 +48,9 @@ type DatasetRepository interface {
|
||||
MarkConverted(ctx context.Context, id uuid.UUID, cogKey string, footprint []byte) error
|
||||
SetProperties(ctx context.Context, id uuid.UUID, properties, geometry []byte) error
|
||||
SetGeoJSON(ctx context.Context, id uuid.UUID, geojson []byte) error
|
||||
ClearGeometryAndGeoJSON(ctx context.Context, id uuid.UUID) error
|
||||
SaveMapping(ctx context.Context, id uuid.UUID, katoColumn string, years []domain.YearColumn) (domain.Dataset, error)
|
||||
SetSLD(ctx context.Context, id uuid.UUID, key, filename, contentType string) (domain.Dataset, error)
|
||||
ReplaceObservations(ctx context.Context, datasetID uuid.UUID, obs []domain.Observation) error
|
||||
ListObservations(ctx context.Context, datasetID uuid.UUID, katoCode *string, limit, offset int) ([]domain.Observation, error)
|
||||
CountObservations(ctx context.Context, datasetID uuid.UUID, katoCode *string) (int, error)
|
||||
@@ -71,6 +86,12 @@ type categoryReader interface {
|
||||
GetByCode(ctx context.Context, code string) (domain.Category, error)
|
||||
}
|
||||
|
||||
// groupReader lets the dataset service verify a dataset group exists before a
|
||||
// dataset is filed under it, and resolve its shared SLD as a fallback.
|
||||
type groupReader interface {
|
||||
GetByID(ctx context.Context, id uuid.UUID) (domain.DatasetGroup, error)
|
||||
}
|
||||
|
||||
// JobEnqueuer schedules asynchronous dataset jobs.
|
||||
type JobEnqueuer interface {
|
||||
EnqueueParse(ctx context.Context, datasetID uuid.UUID) error
|
||||
@@ -86,14 +107,16 @@ type ColumnParser func(filename string, data []byte) ([]domain.AttributeColumn,
|
||||
type RowParser func(filename string, data []byte) ([]map[string]string, error)
|
||||
|
||||
// RasterConverter converts a raster file to a Cloud-Optimized GeoTIFF, reads its
|
||||
// footprint, and dissolves a vector file's features into a single geometry. It
|
||||
// operates on local file paths.
|
||||
// footprint, and reads a vector file's per-feature geometry. It operates on
|
||||
// local file paths.
|
||||
type RasterConverter interface {
|
||||
ToCOG(ctx context.Context, srcPath, dstPath string) error
|
||||
Footprint(ctx context.Context, srcPath string) ([]byte, error)
|
||||
// VectorGeometry returns the combined geometry of a vector file's features as
|
||||
// GeoJSON in EPSG:4326 (nil if the file has no features).
|
||||
VectorGeometry(ctx context.Context, srcPath string) ([]byte, error)
|
||||
// VectorFeatures returns each feature's geometry as GeoJSON in EPSG:4326, in
|
||||
// file order (nil if the file has no features). A feature with no geometry
|
||||
// yields a nil entry at its index, keeping the result aligned by position
|
||||
// with attribute rows read from the same file via RowParser.
|
||||
VectorFeatures(ctx context.Context, srcPath string) ([]json.RawMessage, error)
|
||||
}
|
||||
|
||||
// UploadInput carries everything needed to store a new dataset.
|
||||
@@ -109,6 +132,11 @@ type UploadInput struct {
|
||||
ContentType string
|
||||
Size int64
|
||||
Reader io.Reader
|
||||
// DatasetGroupID optionally files the dataset under a dataset group.
|
||||
DatasetGroupID *uuid.UUID
|
||||
// StartDate / EndDate is the period the dataset describes (YYYY-MM-DD).
|
||||
StartDate *string
|
||||
EndDate *string
|
||||
}
|
||||
|
||||
// DatasetService implements dataset business rules and object storage handling.
|
||||
@@ -116,6 +144,7 @@ type DatasetService struct {
|
||||
repo DatasetRepository
|
||||
store ObjectStore
|
||||
categories categoryReader
|
||||
groups groupReader
|
||||
jobs JobEnqueuer
|
||||
parseColumns ColumnParser
|
||||
parseRows RowParser
|
||||
@@ -123,12 +152,14 @@ type DatasetService struct {
|
||||
}
|
||||
|
||||
// NewDatasetService wires the dataset repository, object store, category reader
|
||||
// (for parent validation), the job enqueuer, the column/row parsers, and the
|
||||
// raster converter.
|
||||
// (for parent validation), the dataset-group reader (for group validation and
|
||||
// the SLD fallback), the job enqueuer, the column/row parsers, and the raster
|
||||
// converter.
|
||||
func NewDatasetService(
|
||||
repo DatasetRepository,
|
||||
store ObjectStore,
|
||||
categories categoryReader,
|
||||
groups groupReader,
|
||||
jobs JobEnqueuer,
|
||||
parseColumns ColumnParser,
|
||||
parseRows RowParser,
|
||||
@@ -138,6 +169,7 @@ func NewDatasetService(
|
||||
repo: repo,
|
||||
store: store,
|
||||
categories: categories,
|
||||
groups: groups,
|
||||
jobs: jobs,
|
||||
parseColumns: parseColumns,
|
||||
parseRows: parseRows,
|
||||
@@ -171,10 +203,13 @@ func (s *DatasetService) Upload(ctx context.Context, in UploadInput) (domain.Dat
|
||||
}
|
||||
content := io.MultiReader(bytes.NewReader(head), in.Reader)
|
||||
|
||||
if _, err := s.categories.GetByID(ctx, in.CategoryID); err != nil {
|
||||
if errors.Is(err, domain.ErrNotFound) {
|
||||
return domain.Dataset{}, fmt.Errorf("%w: category does not exist", domain.ErrValidation)
|
||||
}
|
||||
if err := s.ensureCategoryExists(ctx, in.CategoryID); err != nil {
|
||||
return domain.Dataset{}, err
|
||||
}
|
||||
if err := s.ensureGroupExists(ctx, in.DatasetGroupID); err != nil {
|
||||
return domain.Dataset{}, err
|
||||
}
|
||||
if err := domain.ValidateDatePeriod(in.StartDate, in.EndDate); err != nil {
|
||||
return domain.Dataset{}, err
|
||||
}
|
||||
|
||||
@@ -191,24 +226,31 @@ func (s *DatasetService) Upload(ctx context.Context, in UploadInput) (domain.Dat
|
||||
status = domain.DatasetStatusParsing
|
||||
}
|
||||
|
||||
storageKey := fmt.Sprintf("%s/%s", uuid.New().String(), in.Filename)
|
||||
objectName, err := safeObjectName(in.Filename)
|
||||
if err != nil {
|
||||
return domain.Dataset{}, err
|
||||
}
|
||||
storageKey := fmt.Sprintf("%s/%s", uuid.New().String(), objectName)
|
||||
if err := s.store.Put(ctx, storageKey, content, in.Size, in.ContentType); err != nil {
|
||||
return domain.Dataset{}, err
|
||||
}
|
||||
|
||||
dataset, err := s.repo.Create(ctx, domain.Dataset{
|
||||
CategoryID: in.CategoryID,
|
||||
Name: name,
|
||||
Description: in.Description,
|
||||
Unit: in.Unit,
|
||||
Meta: in.Meta,
|
||||
Automated: in.Automated,
|
||||
Status: status,
|
||||
Filename: in.Filename,
|
||||
StorageKey: storageKey,
|
||||
FileType: in.FileType,
|
||||
SizeBytes: in.Size,
|
||||
ContentType: in.ContentType,
|
||||
CategoryID: in.CategoryID,
|
||||
DatasetGroupID: in.DatasetGroupID,
|
||||
StartDate: in.StartDate,
|
||||
EndDate: in.EndDate,
|
||||
Name: name,
|
||||
Description: in.Description,
|
||||
Unit: in.Unit,
|
||||
Meta: in.Meta,
|
||||
Automated: in.Automated,
|
||||
Status: status,
|
||||
Filename: in.Filename,
|
||||
StorageKey: storageKey,
|
||||
FileType: in.FileType,
|
||||
SizeBytes: in.Size,
|
||||
ContentType: in.ContentType,
|
||||
})
|
||||
if err != nil {
|
||||
// Compensate: the row was not written, so the object would be orphaned.
|
||||
@@ -225,6 +267,160 @@ func (s *DatasetService) Upload(ctx context.Context, in UploadInput) (domain.Dat
|
||||
return dataset, nil
|
||||
}
|
||||
|
||||
// UploadSLDInput carries an SLD (Styled Layer Descriptor) style file to attach
|
||||
// to an existing dataset.
|
||||
type UploadSLDInput struct {
|
||||
Filename string
|
||||
ContentType string
|
||||
Size int64
|
||||
Reader io.Reader
|
||||
}
|
||||
|
||||
// UploadSLD validates and stores an SLD style file for a dataset, replacing
|
||||
// any previously stored one. Applies to every file_type.
|
||||
func (s *DatasetService) UploadSLD(ctx context.Context, id uuid.UUID, in UploadSLDInput) (domain.Dataset, error) {
|
||||
dataset, err := s.repo.GetByID(ctx, id)
|
||||
if err != nil {
|
||||
return domain.Dataset{}, err
|
||||
}
|
||||
|
||||
ext := strings.ToLower(filepath.Ext(in.Filename))
|
||||
if !domain.ExtensionAllowedForSLD(ext) {
|
||||
return domain.Dataset{}, fmt.Errorf("%w: sld file must have one of these extensions: %s",
|
||||
domain.ErrValidation, strings.Join(domain.AllowedSLDExtensions, ", "))
|
||||
}
|
||||
|
||||
head := make([]byte, 512)
|
||||
n, err := io.ReadFull(in.Reader, head)
|
||||
if err != nil && err != io.EOF && err != io.ErrUnexpectedEOF {
|
||||
return domain.Dataset{}, fmt.Errorf("read sld upload: %w", err)
|
||||
}
|
||||
head = head[:n]
|
||||
if err := domain.ValidateFileContent(ext, head); err != nil {
|
||||
return domain.Dataset{}, fmt.Errorf("%w: %s", domain.ErrValidation, err)
|
||||
}
|
||||
content := io.MultiReader(bytes.NewReader(head), in.Reader)
|
||||
|
||||
key, err := deriveSLDKey(dataset.StorageKey, in.Filename)
|
||||
if err != nil {
|
||||
return domain.Dataset{}, err
|
||||
}
|
||||
if err := s.store.Put(ctx, key, content, in.Size, in.ContentType); err != nil {
|
||||
return domain.Dataset{}, err
|
||||
}
|
||||
|
||||
updated, err := s.repo.SetSLD(ctx, id, key, in.Filename, in.ContentType)
|
||||
if err != nil {
|
||||
// Compensate: the row was not updated, so the object would be orphaned.
|
||||
_ = s.store.Remove(ctx, key)
|
||||
return domain.Dataset{}, err
|
||||
}
|
||||
|
||||
if dataset.SldStorageKey != nil && *dataset.SldStorageKey != key {
|
||||
_ = s.store.Remove(ctx, *dataset.SldStorageKey) // best-effort: drop the replaced file
|
||||
}
|
||||
return updated, nil
|
||||
}
|
||||
|
||||
// DownloadSLD returns the dataset metadata and a reader for its stored SLD
|
||||
// style file. Fails with ErrSLDNotFound when none has been uploaded. The
|
||||
// caller must close the reader.
|
||||
func (s *DatasetService) DownloadSLD(ctx context.Context, id uuid.UUID) (domain.Dataset, io.ReadCloser, error) {
|
||||
dataset, err := s.repo.GetByID(ctx, id)
|
||||
if err != nil {
|
||||
return domain.Dataset{}, nil, err
|
||||
}
|
||||
if dataset.SldStorageKey == nil {
|
||||
// Fall back to the dataset group's shared style, if the group has one.
|
||||
if dataset.DatasetGroupID == nil {
|
||||
return domain.Dataset{}, nil, ErrSLDNotFound
|
||||
}
|
||||
group, err := s.groups.GetByID(ctx, *dataset.DatasetGroupID)
|
||||
if err != nil {
|
||||
if errors.Is(err, domain.ErrNotFound) {
|
||||
return domain.Dataset{}, nil, ErrSLDNotFound
|
||||
}
|
||||
return domain.Dataset{}, nil, err
|
||||
}
|
||||
if group.SldStorageKey == nil {
|
||||
return domain.Dataset{}, nil, ErrSLDNotFound
|
||||
}
|
||||
dataset.SldStorageKey = group.SldStorageKey
|
||||
dataset.SldFilename = group.SldFilename
|
||||
dataset.SldContentType = group.SldContentType
|
||||
}
|
||||
obj, err := s.store.Get(ctx, *dataset.SldStorageKey)
|
||||
if err != nil {
|
||||
return domain.Dataset{}, nil, err
|
||||
}
|
||||
return dataset, obj, nil
|
||||
}
|
||||
|
||||
// CreateRasterInput describes a raster file already on local disk that should
|
||||
// become a dataset (used by the Landscape Program to publish results).
|
||||
type CreateRasterInput struct {
|
||||
CategoryID uuid.UUID
|
||||
DatasetGroupID *uuid.UUID
|
||||
Name string
|
||||
Filename string // stored filename, e.g. "lp-z-integrated.tif"
|
||||
Path string // local path to read from
|
||||
StartDate *string
|
||||
EndDate *string
|
||||
Meta json.RawMessage
|
||||
Automated bool
|
||||
}
|
||||
|
||||
// CreateRasterFromFile stores a local GeoTIFF as a new raster dataset and
|
||||
// enqueues the standard COG conversion, exactly like an uploaded raster.
|
||||
func (s *DatasetService) CreateRasterFromFile(ctx context.Context, in CreateRasterInput) (domain.Dataset, error) {
|
||||
f, err := os.Open(in.Path)
|
||||
if err != nil {
|
||||
return domain.Dataset{}, err
|
||||
}
|
||||
defer f.Close()
|
||||
info, err := f.Stat()
|
||||
if err != nil {
|
||||
return domain.Dataset{}, err
|
||||
}
|
||||
return s.Upload(ctx, UploadInput{
|
||||
CategoryID: in.CategoryID,
|
||||
DatasetGroupID: in.DatasetGroupID,
|
||||
Name: in.Name,
|
||||
Meta: in.Meta,
|
||||
Automated: in.Automated,
|
||||
Filename: in.Filename,
|
||||
FileType: domain.FileTypeRaster,
|
||||
ContentType: "image/tiff",
|
||||
Size: info.Size(),
|
||||
Reader: f,
|
||||
StartDate: in.StartDate,
|
||||
EndDate: in.EndDate,
|
||||
})
|
||||
}
|
||||
|
||||
// safeObjectName reduces a user-supplied filename to a single path element so
|
||||
// it can never escape the object-key prefix it is appended to. Rejects names
|
||||
// that carry no usable base ("", ".", "..", "/").
|
||||
func safeObjectName(name string) (string, error) {
|
||||
base := path.Base(filepath.ToSlash(strings.TrimSpace(name)))
|
||||
switch base {
|
||||
case "", ".", "..", "/":
|
||||
return "", fmt.Errorf("%w: invalid filename", domain.ErrValidation)
|
||||
}
|
||||
return base, nil
|
||||
}
|
||||
|
||||
// deriveSLDKey places a dataset's SLD style file alongside the original under
|
||||
// an sld/ prefix, named after the uploaded SLD file. The filename is
|
||||
// sanitised, so the key always stays under the dataset's own prefix.
|
||||
func deriveSLDKey(storageKey, filename string) (string, error) {
|
||||
name, err := safeObjectName(filename)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
return path.Join(path.Dir(storageKey), "sld", name), nil
|
||||
}
|
||||
|
||||
// enqueueProcessing schedules the appropriate async job for a dataset based on
|
||||
// its file type: vector_with_kato is parsed for column selection, plain vector
|
||||
// has its attribute table extracted into properties, and raster is converted to
|
||||
@@ -244,26 +440,35 @@ func (s *DatasetService) enqueueProcessing(ctx context.Context, d domain.Dataset
|
||||
|
||||
// Reprocess re-enqueues the processing job for an existing dataset, restarting
|
||||
// its asynchronous pipeline from the step appropriate to its file type. Useful
|
||||
// for retrying after a transient failure or a worker that was behind the schema.
|
||||
// for retrying after a transient failure or a worker that was behind the
|
||||
// schema. Its stored geometry and GeoJSON are cleared first so a stale (e.g.
|
||||
// previously miscomputed) value is never served while the job is pending; the
|
||||
// worker repopulates both once the job completes.
|
||||
func (s *DatasetService) Reprocess(ctx context.Context, id uuid.UUID) (domain.Dataset, error) {
|
||||
dataset, err := s.repo.GetByID(ctx, id)
|
||||
if err != nil {
|
||||
return domain.Dataset{}, err
|
||||
}
|
||||
if err := s.repo.ClearGeometryAndGeoJSON(ctx, id); err != nil {
|
||||
return domain.Dataset{}, fmt.Errorf("clear stale geometry/geojson: %w", err)
|
||||
}
|
||||
if err := s.enqueueProcessing(ctx, dataset); err != nil {
|
||||
return domain.Dataset{}, fmt.Errorf("enqueue processing: %w", err)
|
||||
}
|
||||
return dataset, nil
|
||||
}
|
||||
|
||||
// ReprocessAll re-enqueues the processing job for every dataset, restarting each
|
||||
// one's asynchronous pipeline from the step appropriate to its file type. It
|
||||
// paginates through all datasets and continues past individual failures,
|
||||
// returning how many jobs were enqueued and a per-dataset map of any failures.
|
||||
func (s *DatasetService) ReprocessAll(ctx context.Context) (enqueued int, failures map[uuid.UUID]error, err error) {
|
||||
// ReprocessAll re-enqueues the processing job for every dataset matching
|
||||
// fileType (nil matches every file type), restarting each one's asynchronous
|
||||
// pipeline from the step appropriate to its file type. Each dataset's stored
|
||||
// geometry and GeoJSON are cleared first (see Reprocess). It paginates through
|
||||
// all matching datasets and continues past individual failures, returning how
|
||||
// many jobs were enqueued and a per-dataset map of any failures.
|
||||
func (s *DatasetService) ReprocessAll(ctx context.Context, fileType *domain.FileType) (enqueued int, failures map[uuid.UUID]error, err error) {
|
||||
failures = make(map[uuid.UUID]error)
|
||||
filter := domain.DatasetFilter{FileType: fileType}
|
||||
for offset := 0; ; {
|
||||
summaries, err := s.repo.ListSummaries(ctx, domain.DatasetFilter{}, MaxPageSize, offset)
|
||||
summaries, err := s.repo.ListSummaries(ctx, filter, MaxPageSize, offset)
|
||||
if err != nil {
|
||||
return enqueued, failures, err
|
||||
}
|
||||
@@ -271,6 +476,10 @@ func (s *DatasetService) ReprocessAll(ctx context.Context) (enqueued int, failur
|
||||
break
|
||||
}
|
||||
for _, sum := range summaries {
|
||||
if e := s.repo.ClearGeometryAndGeoJSON(ctx, sum.ID); e != nil {
|
||||
failures[sum.ID] = e
|
||||
continue
|
||||
}
|
||||
if e := s.enqueueProcessing(ctx, domain.Dataset{ID: sum.ID, FileType: sum.FileType}); e != nil {
|
||||
failures[sum.ID] = e
|
||||
continue
|
||||
@@ -285,12 +494,15 @@ func (s *DatasetService) ReprocessAll(ctx context.Context) (enqueued int, failur
|
||||
return enqueued, failures, nil
|
||||
}
|
||||
|
||||
// ExtractProperties reads a plain vector dataset's attribute table and spatial
|
||||
// geometry and stores them (the attribute table as a JSON array of row objects
|
||||
// in the properties column, the dissolved feature geometry in the geometry
|
||||
// column), then marks the dataset ready. Invoked by the worker. Parse failures
|
||||
// are recorded; storage failures are returned for retry. Geometry extraction is
|
||||
// best-effort: a failure leaves geometry unset rather than failing the job.
|
||||
// ExtractProperties reads a plain vector dataset's attribute table and
|
||||
// per-feature geometry and stores them: the dissolved (unioned) geometry in
|
||||
// the geometry column, and a GeoJSON FeatureCollection with one Feature per
|
||||
// original polygon (its own geometry paired with its own attribute row, by
|
||||
// position - the same pairing a shapefile's .shp/.dbf records have) as the
|
||||
// persisted GeoJSON. Then marks the dataset ready. Invoked by the worker.
|
||||
// Parse failures are recorded; storage failures are returned for retry.
|
||||
// Geometry extraction is best-effort: a failure leaves geometry unset rather
|
||||
// than failing the job.
|
||||
func (s *DatasetService) ExtractProperties(ctx context.Context, id uuid.UUID) error {
|
||||
dataset, err := s.repo.GetByID(ctx, id)
|
||||
if err != nil {
|
||||
@@ -317,20 +529,27 @@ func (s *DatasetService) ExtractProperties(ctx context.Context, id uuid.UUID) er
|
||||
}
|
||||
}
|
||||
|
||||
geometry := s.vectorGeometry(ctx, dataset.Filename, data)
|
||||
if err := s.repo.SetProperties(ctx, id, properties, geometry); err != nil {
|
||||
geoms := s.vectorFeatureGeometries(ctx, dataset.Filename, data)
|
||||
if err := s.repo.SetProperties(ctx, id, properties, dissolvedGeometryJSON(geoms)); err != nil {
|
||||
return err
|
||||
}
|
||||
// Assemble and persist the GeoJSON served by the .geojson endpoint from the
|
||||
// now-canonical (unioned) geometry and attribute table.
|
||||
return s.generateGeoJSON(ctx, id)
|
||||
|
||||
fc := domain.FeatureCollection{
|
||||
Type: domain.GeoJSONFeatureCollection,
|
||||
Features: buildVectorFeatures(rows, geoms),
|
||||
}
|
||||
raw, err := json.Marshal(fc)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
return s.repo.SetGeoJSON(ctx, id, raw)
|
||||
}
|
||||
|
||||
// vectorGeometry dissolves a vector file's features into a single GeoJSON
|
||||
// geometry via the converter. It writes the in-memory bytes to a temp file
|
||||
// (preserving the extension so the converter detects the format) because the
|
||||
// converter operates on file paths. Best-effort: any failure yields nil.
|
||||
func (s *DatasetService) vectorGeometry(ctx context.Context, filename string, data []byte) []byte {
|
||||
// vectorFeatureGeometries reads a vector file's per-feature geometries via the
|
||||
// converter. It writes the in-memory bytes to a temp file (preserving the
|
||||
// extension so the converter detects the format) because the converter
|
||||
// operates on file paths. Best-effort: any failure yields nil.
|
||||
func (s *DatasetService) vectorFeatureGeometries(ctx context.Context, filename string, data []byte) []json.RawMessage {
|
||||
ext := strings.ToLower(filepath.Ext(filename))
|
||||
f, err := os.CreateTemp("", "gis-vec-*"+ext)
|
||||
if err != nil {
|
||||
@@ -344,8 +563,61 @@ func (s *DatasetService) vectorGeometry(ctx context.Context, filename string, da
|
||||
if err := f.Close(); err != nil {
|
||||
return nil
|
||||
}
|
||||
geom, _ := s.converter.VectorGeometry(ctx, f.Name())
|
||||
return geom
|
||||
geoms, _ := s.converter.VectorFeatures(ctx, f.Name())
|
||||
return geoms
|
||||
}
|
||||
|
||||
// dissolvedGeometryJSON wraps a vector file's per-feature geometries into a
|
||||
// single GeoJSON GeometryCollection for storage in the geometry column, where
|
||||
// PostGIS dissolves it into the union of all features (nil entries and an
|
||||
// empty/nil input are dropped, yielding nil when no geometry survives).
|
||||
func dissolvedGeometryJSON(geoms []json.RawMessage) []byte {
|
||||
present := make([]json.RawMessage, 0, len(geoms))
|
||||
for _, g := range geoms {
|
||||
if len(g) == 0 || string(g) == "null" {
|
||||
continue
|
||||
}
|
||||
present = append(present, g)
|
||||
}
|
||||
if len(present) == 0 {
|
||||
return nil
|
||||
}
|
||||
b, err := json.Marshal(struct {
|
||||
Type string `json:"type"`
|
||||
Geometries []json.RawMessage `json:"geometries"`
|
||||
}{Type: "GeometryCollection", Geometries: present})
|
||||
if err != nil {
|
||||
return nil
|
||||
}
|
||||
return b
|
||||
}
|
||||
|
||||
// buildVectorFeatures pairs each row with the geometry at the same position
|
||||
// (rows and geoms both come from reading the same file in file order, which
|
||||
// is how a shapefile's .dbf attribute records correspond to its .shp geometry
|
||||
// records) into one Feature per polygon. A row or geometry beyond the other
|
||||
// slice's length, or a feature with no geometry, is skipped.
|
||||
func buildVectorFeatures(rows []map[string]string, geoms []json.RawMessage) []domain.Feature {
|
||||
n := len(geoms)
|
||||
if len(rows) < n {
|
||||
n = len(rows)
|
||||
}
|
||||
features := make([]domain.Feature, 0, n)
|
||||
for i := 0; i < n; i++ {
|
||||
if len(geoms[i]) == 0 || string(geoms[i]) == "null" {
|
||||
continue
|
||||
}
|
||||
props := make(map[string]any, len(rows[i]))
|
||||
for k, v := range rows[i] {
|
||||
props[k] = v
|
||||
}
|
||||
features = append(features, domain.Feature{
|
||||
Type: domain.GeoJSONFeature,
|
||||
Geometry: geoms[i],
|
||||
Properties: props,
|
||||
})
|
||||
}
|
||||
return features
|
||||
}
|
||||
|
||||
// hasAttributeData reports whether any row carries at least one attribute.
|
||||
@@ -420,7 +692,7 @@ func (s *DatasetService) downloadToTemp(ctx context.Context, key, pattern string
|
||||
|
||||
// uploadFile streams a local file to the object store.
|
||||
func (s *DatasetService) uploadFile(ctx context.Context, key, filePath, contentType string) error {
|
||||
f, err := os.Open(filePath)
|
||||
f, err := os.Open(filePath) // #nosec G304 -- filePath is a temp file this service created
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
@@ -517,8 +789,18 @@ func (s *DatasetService) SaveMapping(ctx context.Context, id uuid.UUID, in Mappi
|
||||
if _, ok := known[yc.Column]; !ok {
|
||||
return domain.Dataset{}, fmt.Errorf("%w: year column %q is not among the detected columns", domain.ErrValidation, yc.Column)
|
||||
}
|
||||
if _, err := time.Parse("2006-01-02", yc.Date); err != nil {
|
||||
return domain.Dataset{}, fmt.Errorf("%w: invalid date %q for column %q (want YYYY-MM-DD)", domain.ErrValidation, yc.Date, yc.Column)
|
||||
from, err := time.Parse("2006-01-02", yc.StartDate)
|
||||
if err != nil {
|
||||
return domain.Dataset{}, fmt.Errorf("%w: invalid start_date %q for column %q (want YYYY-MM-DD)", domain.ErrValidation, yc.StartDate, yc.Column)
|
||||
}
|
||||
if yc.EndDate != nil {
|
||||
to, err := time.Parse("2006-01-02", *yc.EndDate)
|
||||
if err != nil {
|
||||
return domain.Dataset{}, fmt.Errorf("%w: invalid end_date %q for column %q (want YYYY-MM-DD)", domain.ErrValidation, *yc.EndDate, yc.Column)
|
||||
}
|
||||
if to.Before(from) {
|
||||
return domain.Dataset{}, fmt.Errorf("%w: end_date %q is before start_date %q for column %q", domain.ErrValidation, *yc.EndDate, yc.StartDate, yc.Column)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -563,8 +845,8 @@ func (s *DatasetService) Extract(ctx context.Context, id uuid.UUID) error {
|
||||
return err // transient
|
||||
}
|
||||
|
||||
geometry := s.vectorGeometry(ctx, dataset.Filename, data)
|
||||
if err := s.repo.MarkReady(ctx, id, geometry); err != nil {
|
||||
geoms := s.vectorFeatureGeometries(ctx, dataset.Filename, data)
|
||||
if err := s.repo.MarkReady(ctx, id, dissolvedGeometryJSON(geoms)); err != nil {
|
||||
return err
|
||||
}
|
||||
// Assemble and persist the GeoJSON served by the .geojson endpoint by joining
|
||||
@@ -573,8 +855,10 @@ func (s *DatasetService) Extract(ctx context.Context, id uuid.UUID) error {
|
||||
}
|
||||
|
||||
// buildObservations unpivots rows into observations. Rows without a KATO code
|
||||
// are skipped; duplicate (kato, date) pairs keep the last value. Numeric cells
|
||||
// populate Value, others ValueText.
|
||||
// are skipped; duplicate (kato, start_date, end_date) tuples keep the last
|
||||
// value. Numeric cells populate Value, others ValueText. A year column's
|
||||
// EndDate (nil for a point-in-time mapping) carries through to the observation
|
||||
// unchanged.
|
||||
func buildObservations(datasetID uuid.UUID, katoColumn string, years []domain.YearColumn, rows []map[string]string) []domain.Observation {
|
||||
obs := make([]domain.Observation, 0, len(rows)*len(years))
|
||||
index := make(map[string]int)
|
||||
@@ -585,7 +869,7 @@ func buildObservations(datasetID uuid.UUID, katoColumn string, years []domain.Ye
|
||||
continue
|
||||
}
|
||||
for _, yc := range years {
|
||||
o := domain.Observation{DatasetID: datasetID, KatoCode: kato, Date: yc.Date}
|
||||
o := domain.Observation{DatasetID: datasetID, KatoCode: kato, StartDate: yc.StartDate, EndDate: yc.EndDate}
|
||||
if raw := strings.TrimSpace(row[yc.Column]); raw != "" {
|
||||
if f, err := strconv.ParseFloat(raw, 64); err == nil {
|
||||
o.Value = &f
|
||||
@@ -593,7 +877,7 @@ func buildObservations(datasetID uuid.UUID, katoColumn string, years []domain.Ye
|
||||
o.ValueText = &raw
|
||||
}
|
||||
}
|
||||
key := kato + "\x00" + yc.Date
|
||||
key := kato + "\x00" + yc.StartDate + "\x00" + endDateKey(yc.EndDate)
|
||||
if i, ok := index[key]; ok {
|
||||
obs[i] = o
|
||||
} else {
|
||||
@@ -605,6 +889,14 @@ func buildObservations(datasetID uuid.UUID, katoColumn string, years []domain.Ye
|
||||
return obs
|
||||
}
|
||||
|
||||
// endDateKey normalizes an optional period end date for use in a dedup key.
|
||||
func endDateKey(endDate *string) string {
|
||||
if endDate == nil {
|
||||
return ""
|
||||
}
|
||||
return *endDate
|
||||
}
|
||||
|
||||
// ObservationPage is a page of observations with pagination metadata.
|
||||
type ObservationPage struct {
|
||||
Items []domain.Observation
|
||||
@@ -703,7 +995,11 @@ func (s *DatasetService) generateGeoJSON(ctx context.Context, id uuid.UUID) erro
|
||||
// joins the districts table on KATO code: one Feature is emitted per KATO, its
|
||||
// boundary taken from the districts table and its observation values nested
|
||||
// under a `data` object (keyed by date) alongside `kato` and `name`. KATO codes
|
||||
// with no matching district are skipped.
|
||||
// with no matching district are skipped. When any observation carries a period
|
||||
// (EndDate set), its end date is additionally exposed under a `periods` object
|
||||
// (keyed by the same start date used in `data`); `periods` is omitted entirely
|
||||
// when every observation is point-in-time, so a dataset mapped without periods
|
||||
// produces byte-identical output to before periods existed.
|
||||
func (s *DatasetService) buildGeoJSON(ctx context.Context, id uuid.UUID) (json.RawMessage, error) {
|
||||
dataset, err := s.repo.GetByID(ctx, id)
|
||||
if err != nil {
|
||||
@@ -730,8 +1026,8 @@ func (s *DatasetService) buildGeoJSON(ctx context.Context, id uuid.UUID) (json.R
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
grouped, order := groupObservationsByKato(obs)
|
||||
features, err := s.districtFeatures(ctx, grouped, order)
|
||||
grouped, periods, order := groupObservationsByKato(obs)
|
||||
features, err := s.districtFeatures(ctx, grouped, periods, order)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
@@ -764,9 +1060,12 @@ func (s *DatasetService) loadGeoJSONDataset(ctx context.Context, id uuid.UUID) (
|
||||
|
||||
// districtFeatures builds one Feature per KATO from the districts table, with
|
||||
// the grouped observation values (keyed by date) nested under a `data` object
|
||||
// alongside `kato` and `name` in each Feature's properties. KATO codes with no
|
||||
// alongside `kato` and `name` in each Feature's properties. When periods holds
|
||||
// an entry for the KATO (at least one observation had a period), its date->
|
||||
// end_date map is additionally nested under a `periods` object; a KATO with no
|
||||
// period observations gets no `periods` key at all. KATO codes with no
|
||||
// matching district are skipped. order drives the deterministic feature order.
|
||||
func (s *DatasetService) districtFeatures(ctx context.Context, grouped map[string]map[string]any, order []string) ([]domain.Feature, error) {
|
||||
func (s *DatasetService) districtFeatures(ctx context.Context, grouped map[string]map[string]any, periods map[string]map[string]string, order []string) ([]domain.Feature, error) {
|
||||
districts, err := s.repo.DistrictGeometriesByKato(ctx, order)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
@@ -777,10 +1076,14 @@ func (s *DatasetService) districtFeatures(ctx context.Context, grouped map[strin
|
||||
if !ok {
|
||||
continue // skip KATO codes with no district boundary
|
||||
}
|
||||
props := map[string]any{"kato": kato, "name": dist.Name, "data": grouped[kato]}
|
||||
if p, ok := periods[kato]; ok {
|
||||
props["periods"] = p
|
||||
}
|
||||
features = append(features, domain.Feature{
|
||||
Type: domain.GeoJSONFeature,
|
||||
Geometry: dist.Geometry,
|
||||
Properties: map[string]any{"kato": kato, "name": dist.Name, "data": grouped[kato]},
|
||||
Properties: props,
|
||||
})
|
||||
}
|
||||
return features, nil
|
||||
@@ -823,11 +1126,15 @@ func vectorFeatureProperties(raw json.RawMessage) map[string]any {
|
||||
|
||||
// groupObservationsByKato groups observations by KATO code into date->value
|
||||
// maps. The value is the numeric Value when present, otherwise ValueText,
|
||||
// otherwise nil (an empty cell). order lists KATO codes in first-seen order so
|
||||
// the resulting feature order is deterministic.
|
||||
func groupObservationsByKato(obs []domain.Observation) (map[string]map[string]any, []string) {
|
||||
grouped := make(map[string]map[string]any)
|
||||
order := make([]string, 0)
|
||||
// otherwise nil (an empty cell). It also groups, per KATO, a start_date->end_date map
|
||||
// for observations that carry a period (EndDate set); a KATO with no period
|
||||
// observations gets no entry in periods at all, so callers can tell "no
|
||||
// periods" apart from "empty periods". order lists KATO codes in first-seen
|
||||
// order so the resulting feature order is deterministic.
|
||||
func groupObservationsByKato(obs []domain.Observation) (grouped map[string]map[string]any, periods map[string]map[string]string, order []string) {
|
||||
grouped = make(map[string]map[string]any)
|
||||
periods = make(map[string]map[string]string)
|
||||
order = make([]string, 0)
|
||||
for _, o := range obs {
|
||||
values, ok := grouped[o.KatoCode]
|
||||
if !ok {
|
||||
@@ -835,9 +1142,18 @@ func groupObservationsByKato(obs []domain.Observation) (map[string]map[string]an
|
||||
grouped[o.KatoCode] = values
|
||||
order = append(order, o.KatoCode)
|
||||
}
|
||||
values[o.Date] = observationValue(o)
|
||||
values[o.StartDate] = observationValue(o)
|
||||
|
||||
if o.EndDate != nil {
|
||||
p, ok := periods[o.KatoCode]
|
||||
if !ok {
|
||||
p = make(map[string]string)
|
||||
periods[o.KatoCode] = p
|
||||
}
|
||||
p[o.StartDate] = *o.EndDate
|
||||
}
|
||||
}
|
||||
return grouped, order
|
||||
return grouped, periods, order
|
||||
}
|
||||
|
||||
// observationValue returns the typed cell value: numeric Value, else ValueText,
|
||||
@@ -858,6 +1174,147 @@ func (s *DatasetService) Get(ctx context.Context, id uuid.UUID) (domain.Dataset,
|
||||
return s.repo.GetByID(ctx, id)
|
||||
}
|
||||
|
||||
// DatasetUpdateInput carries the full set of editable metadata fields for a
|
||||
// PUT replace. Description and Unit being nil clears those (nullable)
|
||||
// columns.
|
||||
type DatasetUpdateInput struct {
|
||||
CategoryID uuid.UUID
|
||||
Name string
|
||||
Description *string
|
||||
Unit *string
|
||||
// DatasetGroupID, StartDate and EndDate are nullable: nil clears them.
|
||||
DatasetGroupID *uuid.UUID
|
||||
StartDate *string
|
||||
EndDate *string
|
||||
}
|
||||
|
||||
// Update replaces a dataset's editable metadata (category, name, description,
|
||||
// unit). File-related and lifecycle fields are untouched.
|
||||
func (s *DatasetService) Update(ctx context.Context, id uuid.UUID, in DatasetUpdateInput) (domain.Dataset, error) {
|
||||
if err := s.ensureCategoryExists(ctx, in.CategoryID); err != nil {
|
||||
return domain.Dataset{}, err
|
||||
}
|
||||
if err := s.ensureGroupExists(ctx, in.DatasetGroupID); err != nil {
|
||||
return domain.Dataset{}, err
|
||||
}
|
||||
if err := domain.ValidateDatePeriod(in.StartDate, in.EndDate); err != nil {
|
||||
return domain.Dataset{}, err
|
||||
}
|
||||
return s.repo.UpdateFields(ctx, id, domain.DatasetFields{
|
||||
CategoryID: in.CategoryID,
|
||||
Name: in.Name,
|
||||
Description: in.Description,
|
||||
Unit: in.Unit,
|
||||
DatasetGroupID: in.DatasetGroupID,
|
||||
StartDate: in.StartDate,
|
||||
EndDate: in.EndDate,
|
||||
})
|
||||
}
|
||||
|
||||
// DatasetPatchInput carries a sparse set of metadata field changes. A nil
|
||||
// CategoryID/Name leaves that column unchanged. Description/Unit are only
|
||||
// applied when their paired Set flag is true; a nil value with Set true
|
||||
// clears the (nullable) column.
|
||||
type DatasetPatchInput struct {
|
||||
CategoryID *uuid.UUID
|
||||
Name *string
|
||||
Description *string
|
||||
DescriptionSet bool
|
||||
Unit *string
|
||||
UnitSet bool
|
||||
|
||||
DatasetGroupID *uuid.UUID
|
||||
DatasetGroupIDSet bool
|
||||
StartDate *string
|
||||
StartDateSet bool
|
||||
EndDate *string
|
||||
EndDateSet bool
|
||||
}
|
||||
|
||||
// Patch applies a sparse update to a dataset's editable metadata (category,
|
||||
// name, description, unit). Fields absent from the input are left unchanged.
|
||||
func (s *DatasetService) Patch(ctx context.Context, id uuid.UUID, in DatasetPatchInput) (domain.Dataset, error) {
|
||||
current, err := s.repo.GetByID(ctx, id)
|
||||
if err != nil {
|
||||
return domain.Dataset{}, err
|
||||
}
|
||||
|
||||
categoryID := current.CategoryID
|
||||
if in.CategoryID != nil {
|
||||
categoryID = *in.CategoryID
|
||||
if err := s.ensureCategoryExists(ctx, categoryID); err != nil {
|
||||
return domain.Dataset{}, err
|
||||
}
|
||||
}
|
||||
name := current.Name
|
||||
if in.Name != nil {
|
||||
name = *in.Name
|
||||
}
|
||||
description := current.Description
|
||||
if in.DescriptionSet {
|
||||
description = in.Description
|
||||
}
|
||||
unit := current.Unit
|
||||
if in.UnitSet {
|
||||
unit = in.Unit
|
||||
}
|
||||
groupID := current.DatasetGroupID
|
||||
if in.DatasetGroupIDSet {
|
||||
groupID = in.DatasetGroupID
|
||||
if err := s.ensureGroupExists(ctx, groupID); err != nil {
|
||||
return domain.Dataset{}, err
|
||||
}
|
||||
}
|
||||
startDate := current.StartDate
|
||||
if in.StartDateSet {
|
||||
startDate = in.StartDate
|
||||
}
|
||||
endDate := current.EndDate
|
||||
if in.EndDateSet {
|
||||
endDate = in.EndDate
|
||||
}
|
||||
if err := domain.ValidateDatePeriod(startDate, endDate); err != nil {
|
||||
return domain.Dataset{}, err
|
||||
}
|
||||
|
||||
return s.repo.UpdateFields(ctx, id, domain.DatasetFields{
|
||||
CategoryID: categoryID,
|
||||
Name: name,
|
||||
Description: description,
|
||||
Unit: unit,
|
||||
DatasetGroupID: groupID,
|
||||
StartDate: startDate,
|
||||
EndDate: endDate,
|
||||
})
|
||||
}
|
||||
|
||||
// ensureCategoryExists validates that categoryID references an existing
|
||||
// category, mirroring the check performed on upload.
|
||||
func (s *DatasetService) ensureCategoryExists(ctx context.Context, categoryID uuid.UUID) error {
|
||||
if _, err := s.categories.GetByID(ctx, categoryID); err != nil {
|
||||
if errors.Is(err, domain.ErrNotFound) {
|
||||
return fmt.Errorf("%w: category does not exist", domain.ErrValidation)
|
||||
}
|
||||
return err
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// ensureGroupExists validates that a non-nil dataset group id references an
|
||||
// existing group. A nil id (no group) is always valid.
|
||||
func (s *DatasetService) ensureGroupExists(ctx context.Context, id *uuid.UUID) error {
|
||||
if id == nil {
|
||||
return nil
|
||||
}
|
||||
if _, err := s.groups.GetByID(ctx, *id); err != nil {
|
||||
if errors.Is(err, domain.ErrNotFound) {
|
||||
return fmt.Errorf("%w: dataset group %s not found", domain.ErrValidation, id)
|
||||
}
|
||||
return err
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// Status-wait bounds and polling cadence for long polling.
|
||||
const (
|
||||
DefaultStatusWait = 25 * time.Second
|
||||
@@ -958,6 +1415,27 @@ func (s *DatasetService) Download(ctx context.Context, id uuid.UUID) (domain.Dat
|
||||
return dataset, obj, nil
|
||||
}
|
||||
|
||||
// COG returns the dataset metadata and a reader for its Cloud-Optimized GeoTIFF.
|
||||
// It fails with ErrNotRaster for non-raster datasets and ErrCOGNotReady when the
|
||||
// raster has not been converted yet. The caller must close the reader.
|
||||
func (s *DatasetService) COG(ctx context.Context, id uuid.UUID) (domain.Dataset, io.ReadCloser, error) {
|
||||
dataset, err := s.repo.GetByID(ctx, id)
|
||||
if err != nil {
|
||||
return domain.Dataset{}, nil, err
|
||||
}
|
||||
if dataset.FileType != domain.FileTypeRaster {
|
||||
return domain.Dataset{}, nil, ErrNotRaster
|
||||
}
|
||||
if dataset.CogStorageKey == nil {
|
||||
return domain.Dataset{}, nil, ErrCOGNotReady
|
||||
}
|
||||
obj, err := s.store.Get(ctx, *dataset.CogStorageKey)
|
||||
if err != nil {
|
||||
return domain.Dataset{}, nil, err
|
||||
}
|
||||
return dataset, obj, nil
|
||||
}
|
||||
|
||||
// Delete removes the dataset row and its stored object.
|
||||
func (s *DatasetService) Delete(ctx context.Context, id uuid.UUID) error {
|
||||
dataset, err := s.repo.GetByID(ctx, id)
|
||||
|
||||
189
internal/service/dataset_group.go
Normal file
189
internal/service/dataset_group.go
Normal file
@@ -0,0 +1,189 @@
|
||||
package service
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"fmt"
|
||||
"io"
|
||||
"path"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
|
||||
"gis/internal/domain"
|
||||
|
||||
"github.com/google/uuid"
|
||||
)
|
||||
|
||||
// DatasetGroupRepository is the persistence behaviour DatasetGroupService needs.
|
||||
type DatasetGroupRepository interface {
|
||||
Create(ctx context.Context, g domain.DatasetGroup) (domain.DatasetGroup, error)
|
||||
GetByID(ctx context.Context, id uuid.UUID) (domain.DatasetGroup, error)
|
||||
GetByCode(ctx context.Context, code string) (domain.DatasetGroup, error)
|
||||
List(ctx context.Context, code *string, limit, offset int) ([]domain.DatasetGroup, error)
|
||||
Count(ctx context.Context, code *string) (int, error)
|
||||
Update(ctx context.Context, g domain.DatasetGroup) (domain.DatasetGroup, error)
|
||||
Delete(ctx context.Context, id uuid.UUID) error
|
||||
SetSLD(ctx context.Context, id uuid.UUID, key, filename, contentType string) (domain.DatasetGroup, error)
|
||||
}
|
||||
|
||||
// DatasetGroupInput carries the mutable fields of a group.
|
||||
type DatasetGroupInput struct {
|
||||
Code string
|
||||
Name string
|
||||
Description string
|
||||
}
|
||||
|
||||
// DatasetGroupPage is a page of groups.
|
||||
type DatasetGroupPage struct {
|
||||
Items []domain.DatasetGroup
|
||||
Page int
|
||||
PageSize int
|
||||
Total int
|
||||
}
|
||||
|
||||
// DatasetGroupService implements dataset-group business rules.
|
||||
type DatasetGroupService struct {
|
||||
repo DatasetGroupRepository
|
||||
datasets DatasetRepository
|
||||
store ObjectStore
|
||||
}
|
||||
|
||||
// NewDatasetGroupService wires the group repository, the dataset repository
|
||||
// (for the "latest" lookup) and the object store (for the group SLD).
|
||||
func NewDatasetGroupService(repo DatasetGroupRepository, datasets DatasetRepository, store ObjectStore) *DatasetGroupService {
|
||||
return &DatasetGroupService{repo: repo, datasets: datasets, store: store}
|
||||
}
|
||||
|
||||
// errInvalidGroupCode is the validation error returned for a non-slug code.
|
||||
func errInvalidGroupCode() error {
|
||||
return fmt.Errorf("%w: code must be a slug (lowercase latin letters, digits, and dashes)", domain.ErrValidation)
|
||||
}
|
||||
|
||||
// Create persists a new group after validating its code.
|
||||
func (s *DatasetGroupService) Create(ctx context.Context, in DatasetGroupInput) (domain.DatasetGroup, error) {
|
||||
if !domain.ValidSlug(in.Code) {
|
||||
return domain.DatasetGroup{}, errInvalidGroupCode()
|
||||
}
|
||||
return s.repo.Create(ctx, domain.DatasetGroup{Code: in.Code, Name: in.Name, Description: in.Description})
|
||||
}
|
||||
|
||||
// Get returns one group by id.
|
||||
func (s *DatasetGroupService) Get(ctx context.Context, id uuid.UUID) (domain.DatasetGroup, error) {
|
||||
return s.repo.GetByID(ctx, id)
|
||||
}
|
||||
|
||||
// List returns a page of groups; page/pageSize are clamped like datasets.
|
||||
func (s *DatasetGroupService) List(ctx context.Context, code *string, page, pageSize int) (DatasetGroupPage, error) {
|
||||
if page < 1 {
|
||||
page = 1
|
||||
}
|
||||
if pageSize < 1 {
|
||||
pageSize = DefaultPageSize
|
||||
}
|
||||
if pageSize > MaxPageSize {
|
||||
pageSize = MaxPageSize
|
||||
}
|
||||
items, err := s.repo.List(ctx, code, pageSize, (page-1)*pageSize)
|
||||
if err != nil {
|
||||
return DatasetGroupPage{}, err
|
||||
}
|
||||
total, err := s.repo.Count(ctx, code)
|
||||
if err != nil {
|
||||
return DatasetGroupPage{}, err
|
||||
}
|
||||
return DatasetGroupPage{Items: items, Page: page, PageSize: pageSize, Total: total}, nil
|
||||
}
|
||||
|
||||
// Update replaces a group's editable fields.
|
||||
func (s *DatasetGroupService) Update(ctx context.Context, id uuid.UUID, in DatasetGroupInput) (domain.DatasetGroup, error) {
|
||||
if !domain.ValidSlug(in.Code) {
|
||||
return domain.DatasetGroup{}, errInvalidGroupCode()
|
||||
}
|
||||
return s.repo.Update(ctx, domain.DatasetGroup{ID: id, Code: in.Code, Name: in.Name, Description: in.Description})
|
||||
}
|
||||
|
||||
// Delete removes a group (409 via ErrConflict while datasets reference it) and
|
||||
// then best-effort removes its SLD object.
|
||||
func (s *DatasetGroupService) Delete(ctx context.Context, id uuid.UUID) error {
|
||||
g, err := s.repo.GetByID(ctx, id)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if err := s.repo.Delete(ctx, id); err != nil {
|
||||
return err
|
||||
}
|
||||
if g.SldStorageKey != nil {
|
||||
_ = s.store.Remove(ctx, *g.SldStorageKey) // best-effort: the row is gone
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// Latest returns the freshest dataset of the group (greatest end_date, NULLs
|
||||
// last, then newest), optionally restricted by status and file type.
|
||||
func (s *DatasetGroupService) Latest(ctx context.Context, id uuid.UUID, status *string, fileType *domain.FileType) (domain.Dataset, error) {
|
||||
if _, err := s.repo.GetByID(ctx, id); err != nil {
|
||||
return domain.Dataset{}, err
|
||||
}
|
||||
return s.datasets.LatestInGroup(ctx, id, status, fileType)
|
||||
}
|
||||
|
||||
// UploadSLD validates and stores the group's shared SLD, replacing any previous one.
|
||||
func (s *DatasetGroupService) UploadSLD(ctx context.Context, id uuid.UUID, in UploadSLDInput) (domain.DatasetGroup, error) {
|
||||
g, err := s.repo.GetByID(ctx, id)
|
||||
if err != nil {
|
||||
return domain.DatasetGroup{}, err
|
||||
}
|
||||
|
||||
ext := strings.ToLower(filepath.Ext(in.Filename))
|
||||
if !domain.ExtensionAllowedForSLD(ext) {
|
||||
return domain.DatasetGroup{}, fmt.Errorf("%w: sld file must have one of these extensions: %s",
|
||||
domain.ErrValidation, strings.Join(domain.AllowedSLDExtensions, ", "))
|
||||
}
|
||||
|
||||
head := make([]byte, 512)
|
||||
n, err := io.ReadFull(in.Reader, head)
|
||||
if err != nil && err != io.EOF && err != io.ErrUnexpectedEOF {
|
||||
return domain.DatasetGroup{}, fmt.Errorf("read sld upload: %w", err)
|
||||
}
|
||||
head = head[:n]
|
||||
if err := domain.ValidateFileContent(ext, head); err != nil {
|
||||
return domain.DatasetGroup{}, fmt.Errorf("%w: %s", domain.ErrValidation, err)
|
||||
}
|
||||
content := io.MultiReader(bytes.NewReader(head), in.Reader)
|
||||
|
||||
name, err := safeObjectName(in.Filename)
|
||||
if err != nil {
|
||||
return domain.DatasetGroup{}, err
|
||||
}
|
||||
key := path.Join("groups", g.ID.String(), "sld", name)
|
||||
if err := s.store.Put(ctx, key, content, in.Size, in.ContentType); err != nil {
|
||||
return domain.DatasetGroup{}, err
|
||||
}
|
||||
updated, err := s.repo.SetSLD(ctx, id, key, in.Filename, in.ContentType)
|
||||
if err != nil {
|
||||
// Compensate: the row was not updated, so the object would be orphaned.
|
||||
_ = s.store.Remove(ctx, key)
|
||||
return domain.DatasetGroup{}, err
|
||||
}
|
||||
if g.SldStorageKey != nil && *g.SldStorageKey != key {
|
||||
_ = s.store.Remove(ctx, *g.SldStorageKey) // best-effort: drop the replaced file
|
||||
}
|
||||
return updated, nil
|
||||
}
|
||||
|
||||
// DownloadSLD returns the group and a reader for its SLD; ErrSLDNotFound when
|
||||
// none has been uploaded. The caller must close the reader.
|
||||
func (s *DatasetGroupService) DownloadSLD(ctx context.Context, id uuid.UUID) (domain.DatasetGroup, io.ReadCloser, error) {
|
||||
g, err := s.repo.GetByID(ctx, id)
|
||||
if err != nil {
|
||||
return domain.DatasetGroup{}, nil, err
|
||||
}
|
||||
if g.SldStorageKey == nil {
|
||||
return domain.DatasetGroup{}, nil, ErrSLDNotFound
|
||||
}
|
||||
obj, err := s.store.Get(ctx, *g.SldStorageKey)
|
||||
if err != nil {
|
||||
return domain.DatasetGroup{}, nil, err
|
||||
}
|
||||
return g, obj, nil
|
||||
}
|
||||
283
internal/service/dataset_group_test.go
Normal file
283
internal/service/dataset_group_test.go
Normal file
@@ -0,0 +1,283 @@
|
||||
package service
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"gis/internal/domain"
|
||||
|
||||
"github.com/google/uuid"
|
||||
)
|
||||
|
||||
// stubGroupRepo is an in-memory DatasetGroupRepository for tests. It also
|
||||
// satisfies the dataset service's groupReader dependency.
|
||||
type stubGroupRepo struct {
|
||||
store map[uuid.UUID]domain.DatasetGroup
|
||||
}
|
||||
|
||||
func newStubGroupRepo() *stubGroupRepo {
|
||||
return &stubGroupRepo{store: map[uuid.UUID]domain.DatasetGroup{}}
|
||||
}
|
||||
|
||||
func (r *stubGroupRepo) Create(_ context.Context, g domain.DatasetGroup) (domain.DatasetGroup, error) {
|
||||
for _, e := range r.store {
|
||||
if e.Code == g.Code {
|
||||
return domain.DatasetGroup{}, domain.ErrConflict
|
||||
}
|
||||
}
|
||||
g.ID = uuid.New()
|
||||
r.store[g.ID] = g
|
||||
return g, nil
|
||||
}
|
||||
|
||||
func (r *stubGroupRepo) GetByID(_ context.Context, id uuid.UUID) (domain.DatasetGroup, error) {
|
||||
g, ok := r.store[id]
|
||||
if !ok {
|
||||
return domain.DatasetGroup{}, domain.ErrNotFound
|
||||
}
|
||||
return g, nil
|
||||
}
|
||||
|
||||
func (r *stubGroupRepo) GetByCode(_ context.Context, code string) (domain.DatasetGroup, error) {
|
||||
for _, g := range r.store {
|
||||
if g.Code == code {
|
||||
return g, nil
|
||||
}
|
||||
}
|
||||
return domain.DatasetGroup{}, domain.ErrNotFound
|
||||
}
|
||||
|
||||
func (r *stubGroupRepo) List(_ context.Context, code *string, _, _ int) ([]domain.DatasetGroup, error) {
|
||||
out := []domain.DatasetGroup{}
|
||||
for _, g := range r.store {
|
||||
if code == nil || g.Code == *code {
|
||||
out = append(out, g)
|
||||
}
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
|
||||
func (r *stubGroupRepo) Count(_ context.Context, code *string) (int, error) {
|
||||
l, _ := r.List(context.Background(), code, 0, 0)
|
||||
return len(l), nil
|
||||
}
|
||||
|
||||
func (r *stubGroupRepo) Update(_ context.Context, g domain.DatasetGroup) (domain.DatasetGroup, error) {
|
||||
if _, ok := r.store[g.ID]; !ok {
|
||||
return domain.DatasetGroup{}, domain.ErrNotFound
|
||||
}
|
||||
r.store[g.ID] = g
|
||||
return g, nil
|
||||
}
|
||||
|
||||
func (r *stubGroupRepo) Delete(_ context.Context, id uuid.UUID) error {
|
||||
if _, ok := r.store[id]; !ok {
|
||||
return domain.ErrNotFound
|
||||
}
|
||||
delete(r.store, id)
|
||||
return nil
|
||||
}
|
||||
|
||||
func (r *stubGroupRepo) SetSLD(_ context.Context, id uuid.UUID, key, filename, contentType string) (domain.DatasetGroup, error) {
|
||||
g, ok := r.store[id]
|
||||
if !ok {
|
||||
return domain.DatasetGroup{}, domain.ErrNotFound
|
||||
}
|
||||
g.SldStorageKey, g.SldFilename, g.SldContentType = &key, &filename, &contentType
|
||||
r.store[id] = g
|
||||
return g, nil
|
||||
}
|
||||
|
||||
func TestDatasetGroupService_Create_RequiresSlug(t *testing.T) {
|
||||
svc := NewDatasetGroupService(newStubGroupRepo(), newStubDatasetRepo(), newStubStore())
|
||||
_, err := svc.Create(context.Background(), DatasetGroupInput{Code: "Bad Code", Name: "x"})
|
||||
if !errors.Is(err, domain.ErrValidation) {
|
||||
t.Fatalf("want ErrValidation, got %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestDatasetGroupService_Latest_PrefersGreatestEndDateThenCreatedAt(t *testing.T) {
|
||||
groups := newStubGroupRepo()
|
||||
datasets := newStubDatasetRepo()
|
||||
svc := NewDatasetGroupService(groups, datasets, newStubStore())
|
||||
g, err := svc.Create(context.Background(), DatasetGroupInput{Code: "pop", Name: "Population"})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
s := func(v string) *string { return &v }
|
||||
base := time.Date(2024, 1, 1, 0, 0, 0, 0, time.UTC)
|
||||
old := domain.Dataset{ID: uuid.New(), DatasetGroupID: &g.ID, EndDate: s("2019-12-31"), Status: domain.DatasetStatusReady, CreatedAt: base}
|
||||
fresh := domain.Dataset{ID: uuid.New(), DatasetGroupID: &g.ID, EndDate: s("2020-12-31"), Status: domain.DatasetStatusReady, CreatedAt: base.Add(time.Hour)}
|
||||
undated := domain.Dataset{ID: uuid.New(), DatasetGroupID: &g.ID, Status: domain.DatasetStatusReady, CreatedAt: base.Add(2 * time.Hour)}
|
||||
for _, d := range []domain.Dataset{old, fresh, undated} {
|
||||
datasets.store[d.ID] = d
|
||||
}
|
||||
got, err := svc.Latest(context.Background(), g.ID, nil, nil)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if got.ID != fresh.ID {
|
||||
t.Fatalf("want %s, got %s", fresh.ID, got.ID)
|
||||
}
|
||||
}
|
||||
|
||||
func TestDatasetGroupService_Latest_NotFoundWhenEmpty(t *testing.T) {
|
||||
groups := newStubGroupRepo()
|
||||
svc := NewDatasetGroupService(groups, newStubDatasetRepo(), newStubStore())
|
||||
g, err := svc.Create(context.Background(), DatasetGroupInput{Code: "pop", Name: "Population"})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err := svc.Latest(context.Background(), g.ID, nil, nil); !errors.Is(err, domain.ErrNotFound) {
|
||||
t.Fatalf("want ErrNotFound, got %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestDatasetGroupService_Latest_UnknownGroupIsNotFound(t *testing.T) {
|
||||
svc := NewDatasetGroupService(newStubGroupRepo(), newStubDatasetRepo(), newStubStore())
|
||||
if _, err := svc.Latest(context.Background(), uuid.New(), nil, nil); !errors.Is(err, domain.ErrNotFound) {
|
||||
t.Fatalf("want ErrNotFound, got %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestDatasetGroupService_UploadSLD_RejectsBadExtension(t *testing.T) {
|
||||
groups := newStubGroupRepo()
|
||||
store := newStubStore()
|
||||
svc := NewDatasetGroupService(groups, newStubDatasetRepo(), store)
|
||||
g, err := svc.Create(context.Background(), DatasetGroupInput{Code: "pop", Name: "Population"})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
_, err = svc.UploadSLD(context.Background(), g.ID, UploadSLDInput{Filename: "style.txt", ContentType: "text/plain", Size: 3, Reader: strings.NewReader("<x>")})
|
||||
if !errors.Is(err, domain.ErrValidation) {
|
||||
t.Fatalf("want ErrValidation, got %v", err)
|
||||
}
|
||||
if len(store.put) != 0 {
|
||||
t.Fatalf("nothing should be stored on validation failure, got %v", store.put)
|
||||
}
|
||||
}
|
||||
|
||||
func TestDatasetGroupService_UploadSLD_StoresAndReplaces(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
groups := newStubGroupRepo()
|
||||
store := newStubStore()
|
||||
svc := NewDatasetGroupService(groups, newStubDatasetRepo(), store)
|
||||
g, err := svc.Create(ctx, DatasetGroupInput{Code: "pop", Name: "Population"})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
updated, err := svc.UploadSLD(ctx, g.ID, UploadSLDInput{
|
||||
Filename: "style.sld", ContentType: "application/xml", Size: 3, Reader: strings.NewReader("<x>"),
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
wantKey := "groups/" + g.ID.String() + "/sld/style.sld"
|
||||
if updated.SldStorageKey == nil || *updated.SldStorageKey != wantKey {
|
||||
t.Fatalf("want key %s, got %v", wantKey, updated.SldStorageKey)
|
||||
}
|
||||
|
||||
// A second upload under a different name replaces the previous object.
|
||||
if _, err := svc.UploadSLD(ctx, g.ID, UploadSLDInput{
|
||||
Filename: "other.sld", ContentType: "application/xml", Size: 3, Reader: strings.NewReader("<y>"),
|
||||
}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if len(store.removed) != 1 || store.removed[0] != wantKey {
|
||||
t.Fatalf("want previous sld removed, got %v", store.removed)
|
||||
}
|
||||
}
|
||||
|
||||
func TestDatasetGroupService_DownloadSLD_NotFoundWhenAbsent(t *testing.T) {
|
||||
svc := NewDatasetGroupService(newStubGroupRepo(), newStubDatasetRepo(), newStubStore())
|
||||
g, err := svc.Create(context.Background(), DatasetGroupInput{Code: "pop", Name: "Population"})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, _, err := svc.DownloadSLD(context.Background(), g.ID); !errors.Is(err, ErrSLDNotFound) {
|
||||
t.Fatalf("want ErrSLDNotFound, got %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestDatasetGroupService_Delete_RemovesSLDObject(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
store := newStubStore()
|
||||
svc := NewDatasetGroupService(newStubGroupRepo(), newStubDatasetRepo(), store)
|
||||
g, err := svc.Create(ctx, DatasetGroupInput{Code: "pop", Name: "Population"})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err := svc.UploadSLD(ctx, g.ID, UploadSLDInput{
|
||||
Filename: "style.sld", ContentType: "application/xml", Size: 3, Reader: strings.NewReader("<x>"),
|
||||
}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := svc.Delete(ctx, g.ID); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
wantKey := "groups/" + g.ID.String() + "/sld/style.sld"
|
||||
if len(store.removed) != 1 || store.removed[0] != wantKey {
|
||||
t.Fatalf("want sld object removed, got %v", store.removed)
|
||||
}
|
||||
if _, err := svc.Get(ctx, g.ID); !errors.Is(err, domain.ErrNotFound) {
|
||||
t.Fatalf("want ErrNotFound after delete, got %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestDatasetGroupService_List_ClampsPageSize(t *testing.T) {
|
||||
svc := NewDatasetGroupService(newStubGroupRepo(), newStubDatasetRepo(), newStubStore())
|
||||
if _, err := svc.Create(context.Background(), DatasetGroupInput{Code: "pop", Name: "Population"}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
page, err := svc.List(context.Background(), nil, 0, 10_000)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if page.Page != 1 || page.PageSize != MaxPageSize {
|
||||
t.Fatalf("want page 1 and page size %d, got %d/%d", MaxPageSize, page.Page, page.PageSize)
|
||||
}
|
||||
if page.Total != 1 {
|
||||
t.Fatalf("want total 1, got %d", page.Total)
|
||||
}
|
||||
}
|
||||
|
||||
func TestDatasetGroupService_Update_RequiresSlug(t *testing.T) {
|
||||
svc := NewDatasetGroupService(newStubGroupRepo(), newStubDatasetRepo(), newStubStore())
|
||||
g, err := svc.Create(context.Background(), DatasetGroupInput{Code: "pop", Name: "Population"})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err := svc.Update(context.Background(), g.ID, DatasetGroupInput{Code: "Bad Code", Name: "x"}); !errors.Is(err, domain.ErrValidation) {
|
||||
t.Fatalf("want ErrValidation, got %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestDatasetGroupService_UploadSLD_RejectsPathTraversal(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
groups := newStubGroupRepo()
|
||||
store := newStubStore()
|
||||
svc := NewDatasetGroupService(groups, newStubDatasetRepo(), store)
|
||||
g, err := svc.Create(ctx, DatasetGroupInput{Code: "pop", Name: "Population"})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
updated, err := svc.UploadSLD(ctx, g.ID, UploadSLDInput{
|
||||
Filename: "../../other/sld/evil.sld", ContentType: "application/xml", Size: 3, Reader: strings.NewReader("<x>"),
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
wantPrefix := "groups/" + g.ID.String() + "/sld/"
|
||||
if updated.SldStorageKey == nil || !strings.HasPrefix(*updated.SldStorageKey, wantPrefix) ||
|
||||
!strings.HasSuffix(*updated.SldStorageKey, "evil.sld") {
|
||||
t.Fatalf("sld key escaped the group prefix: %v", updated.SldStorageKey)
|
||||
}
|
||||
if len(store.put) != 1 || !strings.HasPrefix(store.put[0], wantPrefix) {
|
||||
t.Fatalf("object stored outside the group sld prefix: %v", store.put)
|
||||
}
|
||||
}
|
||||
File diff suppressed because it is too large
Load Diff
808
internal/service/landscape.go
Normal file
808
internal/service/landscape.go
Normal file
@@ -0,0 +1,808 @@
|
||||
package service
|
||||
|
||||
import (
|
||||
"context"
|
||||
"crypto/sha256"
|
||||
"encoding/hex"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"log/slog"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"sort"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"gis/internal/domain"
|
||||
"gis/internal/landscape"
|
||||
"gis/internal/raster"
|
||||
|
||||
"github.com/google/uuid"
|
||||
"golang.org/x/sync/errgroup"
|
||||
)
|
||||
|
||||
// ComputeFunc runs one graph node over its input rasters, writing one raster
|
||||
// per output group into outDir and returning the produced paths by group. It
|
||||
// mirrors landscape.Compute so tests can substitute a stub.
|
||||
type ComputeFunc func(ctx context.Context, rio raster.IO, node landscape.Node, inputs map[string]string, outDir string, p landscape.Params) (map[string]string, error)
|
||||
|
||||
// ComputeWithWorkersFunc is ComputeFunc plus an explicit per-node worker cap.
|
||||
// It mirrors landscape.ComputeWithWorkers.
|
||||
type ComputeWithWorkersFunc func(ctx context.Context, rio raster.IO, node landscape.Node, inputs map[string]string, outDir string, p landscape.Params, workers int) (map[string]string, error)
|
||||
|
||||
// Default bounds on a run's parallelism, mirroring config.LandscapeConfig.
|
||||
const (
|
||||
// DefaultMaxWorkers is the total window-worker budget of one run.
|
||||
DefaultMaxWorkers = 8
|
||||
// DefaultMaxConcurrentNodes is how many nodes of a stage run at a time.
|
||||
DefaultMaxConcurrentNodes = 2
|
||||
)
|
||||
|
||||
// terminalWriteTimeout bounds a terminal status write made on a context that
|
||||
// has already been cancelled (see terminalCtx).
|
||||
const terminalWriteTimeout = 10 * time.Second
|
||||
|
||||
// LandscapeOptions bounds a run's parallelism, and with it the worker's peak
|
||||
// memory: MaxConcurrentNodes nodes run at a time, and they share the
|
||||
// MaxWorkers window-worker budget. Zero or negative values fall back to the
|
||||
// defaults.
|
||||
type LandscapeOptions struct {
|
||||
MaxWorkers int
|
||||
MaxConcurrentNodes int
|
||||
}
|
||||
|
||||
func (o LandscapeOptions) withDefaults() LandscapeOptions {
|
||||
if o.MaxWorkers < 1 {
|
||||
o.MaxWorkers = DefaultMaxWorkers
|
||||
}
|
||||
if o.MaxConcurrentNodes < 1 {
|
||||
o.MaxConcurrentNodes = DefaultMaxConcurrentNodes
|
||||
}
|
||||
return o
|
||||
}
|
||||
|
||||
// workersPerNode splits the total worker budget over the concurrently running
|
||||
// nodes, so nodes x workers stays at MaxWorkers. Always at least 1.
|
||||
func (o LandscapeOptions) workersPerNode() int {
|
||||
return max(1, o.MaxWorkers/o.MaxConcurrentNodes)
|
||||
}
|
||||
|
||||
// LandscapeRepository is the persistence behaviour LandscapeService needs.
|
||||
type LandscapeRepository interface {
|
||||
CreateRun(ctx context.Context, run domain.LPRun) (domain.LPRun, error)
|
||||
GetRun(ctx context.Context, id uuid.UUID) (domain.LPRun, error)
|
||||
ListRuns(ctx context.Context, limit, offset int) ([]domain.LPRun, error)
|
||||
CountRuns(ctx context.Context) (int, error)
|
||||
SetRunStatus(ctx context.Context, id uuid.UUID, status domain.LPRunStatus, errMsg *string) error
|
||||
SetNodeStarted(ctx context.Context, runID uuid.UUID, code string, fingerprint string, inputs []domain.LPDatasetRef) error
|
||||
SetNodeFinished(ctx context.Context, runID uuid.UUID, code string, status domain.LPNodeStatus, outputs []domain.LPDatasetRef, errMsg *string) error
|
||||
FindReusableOutputs(ctx context.Context, code, fingerprint string, excludeRun uuid.UUID) ([]domain.LPDatasetRef, error)
|
||||
}
|
||||
|
||||
// LandscapeJobEnqueuer schedules asynchronous Landscape Program runs.
|
||||
type LandscapeJobEnqueuer interface {
|
||||
EnqueueRun(ctx context.Context, runID uuid.UUID) error
|
||||
}
|
||||
|
||||
// datasetPublisher is the slice of DatasetService the Landscape Program needs:
|
||||
// uploading a slot input and publishing a computed result.
|
||||
type datasetPublisher interface {
|
||||
Upload(ctx context.Context, in UploadInput) (domain.Dataset, error)
|
||||
CreateRasterFromFile(ctx context.Context, in CreateRasterInput) (domain.Dataset, error)
|
||||
}
|
||||
|
||||
// DatasetService is what the landscape service publishes through in production.
|
||||
var _ datasetPublisher = (*DatasetService)(nil)
|
||||
|
||||
// SlotInfo describes one leaf input slot and the dataset currently filling it.
|
||||
type SlotInfo struct {
|
||||
Code string `json:"code"`
|
||||
Title string `json:"title"`
|
||||
GroupID uuid.UUID `json:"group_id"`
|
||||
Latest *domain.DatasetSummary `json:"latest"`
|
||||
}
|
||||
|
||||
// ResultInfo describes one computed output group and its freshest ready dataset.
|
||||
type ResultInfo struct {
|
||||
Group string `json:"group"`
|
||||
Title string `json:"title"`
|
||||
Node string `json:"node"`
|
||||
GroupID uuid.UUID `json:"group_id"`
|
||||
Latest *domain.DatasetSummary `json:"latest"`
|
||||
CogURL *string `json:"cog_url"`
|
||||
SldURL *string `json:"sld_url"`
|
||||
}
|
||||
|
||||
// SlotUploadInput carries a raster upload destined for one input slot.
|
||||
type SlotUploadInput struct {
|
||||
Filename string
|
||||
ContentType string
|
||||
Size int64
|
||||
Reader io.Reader
|
||||
Name string
|
||||
StartDate *string
|
||||
EndDate *string
|
||||
}
|
||||
|
||||
// RunInput carries the options of a new Landscape Program run.
|
||||
type RunInput struct {
|
||||
// FromStage forces recomputation from this stage on (0..3); nil recomputes
|
||||
// only what changed.
|
||||
FromStage *int
|
||||
// Overrides pins a specific dataset for a group code instead of resolving
|
||||
// the freshest ready one.
|
||||
Overrides map[string]uuid.UUID
|
||||
Params landscape.Params
|
||||
}
|
||||
|
||||
// LPRunPage is a page of runs with pagination metadata.
|
||||
type LPRunPage struct {
|
||||
Items []domain.LPRun
|
||||
Page int
|
||||
PageSize int
|
||||
Total int
|
||||
}
|
||||
|
||||
// LandscapeService plans and executes Landscape Program runs.
|
||||
type LandscapeService struct {
|
||||
runs LandscapeRepository
|
||||
datasets DatasetRepository
|
||||
groups DatasetGroupRepository
|
||||
categories categoryReader
|
||||
publisher datasetPublisher
|
||||
store ObjectStore
|
||||
jobs LandscapeJobEnqueuer
|
||||
rio raster.IO
|
||||
// compute is the legacy worker-count-agnostic entry point, kept so an
|
||||
// existing ComputeFunc still wires up; runNode always calls
|
||||
// computeWithWorkers, which wraps compute when one was provided.
|
||||
compute ComputeFunc
|
||||
computeWithWorkers ComputeWithWorkersFunc
|
||||
opts LandscapeOptions
|
||||
log *slog.Logger
|
||||
}
|
||||
|
||||
// NewLandscapeService wires the run repository, the dataset and group
|
||||
// repositories (input resolution), the category reader (the "landscape"
|
||||
// category every result is filed under), the dataset publisher (result
|
||||
// datasets and slot uploads), the object store (input downloads), the job
|
||||
// enqueuer, and the raster IO / compute function the graph runs on. Pass a nil
|
||||
// compute (the production wiring) to run landscape.ComputeWithWorkers with the
|
||||
// per-node share of opts.MaxWorkers; a non-nil ComputeFunc is honoured for
|
||||
// backward compatibility but never sees the worker cap. opts bounds the run's
|
||||
// parallelism; zero fields fall back to the package defaults.
|
||||
func NewLandscapeService(
|
||||
runs LandscapeRepository,
|
||||
datasets DatasetRepository,
|
||||
groups DatasetGroupRepository,
|
||||
categories categoryReader,
|
||||
publisher datasetPublisher,
|
||||
store ObjectStore,
|
||||
jobs LandscapeJobEnqueuer,
|
||||
rio raster.IO,
|
||||
compute ComputeFunc,
|
||||
log *slog.Logger,
|
||||
opts LandscapeOptions,
|
||||
) *LandscapeService {
|
||||
// Default to the workers-aware entry point; wrap a supplied legacy
|
||||
// ComputeFunc (which has no worker parameter) when one is given.
|
||||
withWorkers := ComputeWithWorkersFunc(landscape.ComputeWithWorkers)
|
||||
if compute != nil {
|
||||
withWorkers = func(ctx context.Context, rio raster.IO, node landscape.Node, inputs map[string]string, outDir string, p landscape.Params, _ int) (map[string]string, error) {
|
||||
return compute(ctx, rio, node, inputs, outDir, p)
|
||||
}
|
||||
}
|
||||
return &LandscapeService{
|
||||
runs: runs,
|
||||
datasets: datasets,
|
||||
groups: groups,
|
||||
categories: categories,
|
||||
publisher: publisher,
|
||||
store: store,
|
||||
jobs: jobs,
|
||||
rio: rio,
|
||||
compute: compute,
|
||||
computeWithWorkers: withWorkers,
|
||||
opts: opts.withDefaults(),
|
||||
log: log,
|
||||
}
|
||||
}
|
||||
|
||||
// Slots lists every leaf input slot with its group and latest dataset (any
|
||||
// status), so the UI can show what has been uploaded.
|
||||
func (s *LandscapeService) Slots(ctx context.Context) ([]SlotInfo, error) {
|
||||
out := make([]SlotInfo, 0, len(landscape.Slots))
|
||||
for _, slot := range landscape.Slots {
|
||||
g, err := s.groups.GetByCode(ctx, slot.Code)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("dataset group %q: %w", slot.Code, err)
|
||||
}
|
||||
info := SlotInfo{Code: slot.Code, Title: slot.Title, GroupID: g.ID}
|
||||
latest, err := s.datasets.LatestInGroup(ctx, g.ID, nil, nil)
|
||||
switch {
|
||||
case err == nil:
|
||||
summary := summaryOf(latest)
|
||||
info.Latest = &summary
|
||||
case errors.Is(err, domain.ErrNotFound):
|
||||
// An empty slot is expected before the first upload.
|
||||
default:
|
||||
return nil, err
|
||||
}
|
||||
out = append(out, info)
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
|
||||
// UploadToSlot stores a raster under the slot's dataset group in the landscape
|
||||
// category, through the regular upload path (COG conversion is enqueued).
|
||||
// Returns domain.ErrNotFound when the code is not a known slot.
|
||||
func (s *LandscapeService) UploadToSlot(ctx context.Context, code string, in SlotUploadInput) (domain.Dataset, error) {
|
||||
slot, ok := landscape.SlotByCode(code)
|
||||
if !ok {
|
||||
return domain.Dataset{}, fmt.Errorf("%w: unknown landscape slot %q", domain.ErrNotFound, code)
|
||||
}
|
||||
g, err := s.groups.GetByCode(ctx, slot.Code)
|
||||
if err != nil {
|
||||
return domain.Dataset{}, fmt.Errorf("dataset group %q: %w", slot.Code, err)
|
||||
}
|
||||
cat, err := s.categories.GetByCode(ctx, landscape.CategoryCode)
|
||||
if err != nil {
|
||||
return domain.Dataset{}, fmt.Errorf("category %q: %w", landscape.CategoryCode, err)
|
||||
}
|
||||
name := in.Name
|
||||
if name == "" {
|
||||
name = in.Filename
|
||||
}
|
||||
return s.publisher.Upload(ctx, UploadInput{
|
||||
CategoryID: cat.ID,
|
||||
DatasetGroupID: &g.ID,
|
||||
Name: name,
|
||||
Filename: in.Filename,
|
||||
FileType: domain.FileTypeRaster,
|
||||
ContentType: in.ContentType,
|
||||
Size: in.Size,
|
||||
Reader: in.Reader,
|
||||
StartDate: in.StartDate,
|
||||
EndDate: in.EndDate,
|
||||
})
|
||||
}
|
||||
|
||||
// CreateRun validates the params and overrides, persists a queued run with one
|
||||
// pending node per graph node, and publishes the worker job.
|
||||
func (s *LandscapeService) CreateRun(ctx context.Context, in RunInput) (domain.LPRun, error) {
|
||||
params := in.Params.WithDefaults()
|
||||
if err := params.Validate(); err != nil {
|
||||
return domain.LPRun{}, fmt.Errorf("%w: %s", domain.ErrValidation, err)
|
||||
}
|
||||
if in.FromStage != nil {
|
||||
if *in.FromStage < 0 || *in.FromStage > maxStage() {
|
||||
return domain.LPRun{}, fmt.Errorf("%w: from_stage must be in 0..%d", domain.ErrValidation, maxStage())
|
||||
}
|
||||
}
|
||||
|
||||
overrides := map[string]uuid.UUID{}
|
||||
for group, id := range in.Overrides {
|
||||
if !knownGroup(group) {
|
||||
return domain.LPRun{}, fmt.Errorf("%w: unknown landscape group %q in overrides", domain.ErrValidation, group)
|
||||
}
|
||||
d, err := s.datasets.GetByID(ctx, id)
|
||||
if err != nil {
|
||||
if errors.Is(err, domain.ErrNotFound) {
|
||||
return domain.LPRun{}, fmt.Errorf("%w: override for %q: dataset %s does not exist", domain.ErrValidation, group, id)
|
||||
}
|
||||
return domain.LPRun{}, err
|
||||
}
|
||||
if d.FileType != domain.FileTypeRaster {
|
||||
return domain.LPRun{}, fmt.Errorf("%w: override for %q: dataset %s is not a raster", domain.ErrValidation, group, id)
|
||||
}
|
||||
if d.Status != domain.DatasetStatusReady {
|
||||
return domain.LPRun{}, fmt.Errorf("%w: override for %q: dataset %s is not ready", domain.ErrValidation, group, id)
|
||||
}
|
||||
overrides[group] = id
|
||||
}
|
||||
|
||||
// lp_runs.params is NOT NULL, and the kernels misbehave on a partially
|
||||
// filled Params, so the defaulted-and-validated value is what we persist.
|
||||
paramsJSON, err := json.Marshal(params)
|
||||
if err != nil {
|
||||
return domain.LPRun{}, err
|
||||
}
|
||||
|
||||
nodes := make([]domain.LPRunNode, 0, len(landscape.Nodes))
|
||||
for _, n := range landscape.Nodes {
|
||||
nodes = append(nodes, domain.LPRunNode{NodeCode: n.Code, Stage: n.Stage, Status: domain.LPNodePending})
|
||||
}
|
||||
|
||||
run, err := s.runs.CreateRun(ctx, domain.LPRun{
|
||||
Status: domain.LPRunQueued,
|
||||
FromStage: in.FromStage,
|
||||
Params: paramsJSON,
|
||||
Overrides: overrides,
|
||||
Nodes: nodes,
|
||||
})
|
||||
if err != nil {
|
||||
return domain.LPRun{}, err
|
||||
}
|
||||
|
||||
if err := s.jobs.EnqueueRun(ctx, run.ID); err != nil {
|
||||
msg := fmt.Sprintf("failed to enqueue: %s", err)
|
||||
if setErr := s.runs.SetRunStatus(ctx, run.ID, domain.LPRunFailed, &msg); setErr != nil {
|
||||
s.log.Error("mark landscape run failed", "run_id", run.ID, "error", setErr)
|
||||
}
|
||||
return domain.LPRun{}, fmt.Errorf("enqueue landscape run: %w", err)
|
||||
}
|
||||
return run, nil
|
||||
}
|
||||
|
||||
// GetRun returns one run with its nodes.
|
||||
func (s *LandscapeService) GetRun(ctx context.Context, id uuid.UUID) (domain.LPRun, error) {
|
||||
return s.runs.GetRun(ctx, id)
|
||||
}
|
||||
|
||||
// ListRuns returns a page of runs, newest first.
|
||||
func (s *LandscapeService) ListRuns(ctx context.Context, page, pageSize int) (LPRunPage, error) {
|
||||
if page < 1 {
|
||||
page = 1
|
||||
}
|
||||
if pageSize < 1 {
|
||||
pageSize = DefaultPageSize
|
||||
}
|
||||
if pageSize > MaxPageSize {
|
||||
pageSize = MaxPageSize
|
||||
}
|
||||
items, err := s.runs.ListRuns(ctx, pageSize, (page-1)*pageSize)
|
||||
if err != nil {
|
||||
return LPRunPage{}, err
|
||||
}
|
||||
total, err := s.runs.CountRuns(ctx)
|
||||
if err != nil {
|
||||
return LPRunPage{}, err
|
||||
}
|
||||
return LPRunPage{Items: items, Page: page, PageSize: pageSize, Total: total}, nil
|
||||
}
|
||||
|
||||
// Results lists every output group with its freshest ready dataset and the URLs
|
||||
// to render it.
|
||||
func (s *LandscapeService) Results(ctx context.Context) ([]ResultInfo, error) {
|
||||
ready := domain.DatasetStatusReady
|
||||
fileType := domain.FileTypeRaster
|
||||
|
||||
out := make([]ResultInfo, 0, len(landscape.Outputs))
|
||||
for _, o := range landscape.Outputs {
|
||||
g, err := s.groups.GetByCode(ctx, o.Code)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("dataset group %q: %w", o.Code, err)
|
||||
}
|
||||
info := ResultInfo{Group: o.Code, Title: o.Title, GroupID: g.ID}
|
||||
if node, ok := landscape.NodeForOutput(o.Code); ok {
|
||||
info.Node = node.Code
|
||||
}
|
||||
|
||||
latest, err := s.datasets.LatestInGroup(ctx, g.ID, &ready, &fileType)
|
||||
switch {
|
||||
case err == nil:
|
||||
summary := summaryOf(latest)
|
||||
info.Latest = &summary
|
||||
cog := "/datasets/" + latest.ID.String() + ".cog"
|
||||
info.CogURL = &cog
|
||||
if g.SldStorageKey != nil {
|
||||
sld := "/dataset-groups/" + g.ID.String() + "/sld"
|
||||
info.SldURL = &sld
|
||||
}
|
||||
case errors.Is(err, domain.ErrNotFound):
|
||||
// Nothing computed for this group yet.
|
||||
default:
|
||||
return nil, err
|
||||
}
|
||||
out = append(out, info)
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
|
||||
// MarkRunFailed records a terminal failure on a run so it does not stay stuck
|
||||
// in an in-progress status.
|
||||
func (s *LandscapeService) MarkRunFailed(ctx context.Context, runID uuid.UUID, reason string) error {
|
||||
wctx, cancel := terminalCtx(ctx)
|
||||
defer cancel()
|
||||
return s.runs.SetRunStatus(wctx, runID, domain.LPRunFailed, &reason)
|
||||
}
|
||||
|
||||
// Execute runs the whole graph for one run: stages in order, nodes within a
|
||||
// stage concurrently. The first failing node fails the run and later stages are
|
||||
// not started. It is the worker entry point.
|
||||
func (s *LandscapeService) Execute(ctx context.Context, runID uuid.UUID) error {
|
||||
run, err := s.runs.GetRun(ctx, runID)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
// A redelivered or stale message must not rerun a finished run. "running"
|
||||
// is allowed through so a run whose worker crashed mid-flight still
|
||||
// recovers on redelivery.
|
||||
if run.Status != domain.LPRunQueued && run.Status != domain.LPRunRunning {
|
||||
s.log.Info("landscape run already terminal, ignoring redelivery", "run_id", runID, "status", run.Status)
|
||||
return nil
|
||||
}
|
||||
|
||||
var params landscape.Params
|
||||
if len(run.Params) > 0 {
|
||||
if err := json.Unmarshal(run.Params, ¶ms); err != nil {
|
||||
return s.failRun(ctx, runID, fmt.Errorf("decode run params: %w", err))
|
||||
}
|
||||
}
|
||||
params = params.WithDefaults()
|
||||
|
||||
cat, err := s.categories.GetByCode(ctx, landscape.CategoryCode)
|
||||
if err != nil {
|
||||
return s.failRun(ctx, runID, fmt.Errorf("category %q: %w", landscape.CategoryCode, err))
|
||||
}
|
||||
|
||||
if err := s.runs.SetRunStatus(ctx, runID, domain.LPRunRunning, nil); err != nil {
|
||||
return err
|
||||
}
|
||||
// Re-read so started_at is available for result dataset names; on failure
|
||||
// keep the pre-transition copy and fall back to time.Now() for the names.
|
||||
if started, err := s.runs.GetRun(ctx, runID); err == nil {
|
||||
run = started
|
||||
} else {
|
||||
s.log.Warn("re-read landscape run after marking it running", "run_id", runID, "error", err)
|
||||
}
|
||||
|
||||
state := &runState{produced: map[string]uuid.UUID{}}
|
||||
for _, stage := range landscape.Stages() {
|
||||
// WithContext so the first failing node cancels its siblings: Compute
|
||||
// honours cancellation, so a doomed stage stops reading rasters
|
||||
// instead of running to completion. Terminal status writes use
|
||||
// terminalCtx, so they still land on the cancelled context.
|
||||
g, gctx := errgroup.WithContext(ctx)
|
||||
// Each node's Compute spins its own worker pool (each worker holding a
|
||||
// window buffer per input band), so peak memory is the product of the
|
||||
// two limits. Bound the node fan-out here and split the worker budget
|
||||
// over it so that product stays at opts.MaxWorkers.
|
||||
g.SetLimit(s.opts.MaxConcurrentNodes)
|
||||
for _, node := range stage {
|
||||
g.Go(func() error { return s.runNode(gctx, run, node, params, cat, state) })
|
||||
}
|
||||
// Wait for every node of the stage: each one records its own outcome,
|
||||
// and a sibling's failure must not leave a node half-recorded.
|
||||
if err := g.Wait(); err != nil {
|
||||
return s.failRun(ctx, runID, err)
|
||||
}
|
||||
}
|
||||
|
||||
return s.runs.SetRunStatus(ctx, runID, domain.LPRunSucceeded, nil)
|
||||
}
|
||||
|
||||
// terminalCtx derives a bounded context that outlives the cancellation of ctx.
|
||||
// Terminal status writes must happen even when the run failed *because* the
|
||||
// context was cancelled (shutdown, a sibling node failing the stage); on the
|
||||
// original context they would be refused and leave the run stuck in "running".
|
||||
func terminalCtx(ctx context.Context) (context.Context, context.CancelFunc) {
|
||||
return context.WithTimeout(context.WithoutCancel(ctx), terminalWriteTimeout)
|
||||
}
|
||||
|
||||
// failRun marks the run failed with cause's message and returns cause.
|
||||
func (s *LandscapeService) failRun(ctx context.Context, runID uuid.UUID, cause error) error {
|
||||
msg := cause.Error()
|
||||
wctx, cancel := terminalCtx(ctx)
|
||||
defer cancel()
|
||||
if err := s.runs.SetRunStatus(wctx, runID, domain.LPRunFailed, &msg); err != nil {
|
||||
s.log.Error("mark landscape run failed", "run_id", runID, "error", err)
|
||||
}
|
||||
return cause
|
||||
}
|
||||
|
||||
// runState carries what the run has produced so far, shared by the concurrent
|
||||
// nodes of a stage.
|
||||
type runState struct {
|
||||
mu sync.Mutex
|
||||
produced map[string]uuid.UUID
|
||||
}
|
||||
|
||||
func (r *runState) get(group string) (uuid.UUID, bool) {
|
||||
r.mu.Lock()
|
||||
defer r.mu.Unlock()
|
||||
id, ok := r.produced[group]
|
||||
return id, ok
|
||||
}
|
||||
|
||||
func (r *runState) record(refs []domain.LPDatasetRef) {
|
||||
r.mu.Lock()
|
||||
defer r.mu.Unlock()
|
||||
for _, ref := range refs {
|
||||
r.produced[ref.Group] = ref.DatasetID
|
||||
}
|
||||
}
|
||||
|
||||
// runNode resolves a node's inputs, reuses a previous result when the
|
||||
// fingerprint matches, and otherwise computes and publishes its outputs.
|
||||
func (s *LandscapeService) runNode(ctx context.Context, run domain.LPRun, node landscape.Node, params landscape.Params, cat domain.Category, state *runState) error {
|
||||
refs, inputs, err := s.resolveInputs(ctx, run, node, state)
|
||||
if err != nil {
|
||||
return s.finishNodeFailed(ctx, run.ID, node, err)
|
||||
}
|
||||
|
||||
fp := fingerprint(node, refs, params)
|
||||
if err := s.runs.SetNodeStarted(ctx, run.ID, node.Code, fp, refs); err != nil {
|
||||
// Best-effort, so the node is not left pending while the run is failed.
|
||||
return s.finishNodeFailed(ctx, run.ID, node, err)
|
||||
}
|
||||
|
||||
if run.FromStage == nil || node.Stage < *run.FromStage {
|
||||
reused, ok, err := s.reusableOutputs(ctx, run.ID, node, fp)
|
||||
if err != nil {
|
||||
return s.finishNodeFailed(ctx, run.ID, node, err)
|
||||
}
|
||||
if ok {
|
||||
state.record(reused)
|
||||
return s.runs.SetNodeFinished(ctx, run.ID, node.Code, domain.LPNodeSkipped, reused, nil)
|
||||
}
|
||||
}
|
||||
|
||||
outputs, err := s.computeNode(ctx, run, node, params, cat, refs, inputs)
|
||||
if err != nil {
|
||||
return s.finishNodeFailed(ctx, run.ID, node, err)
|
||||
}
|
||||
state.record(outputs)
|
||||
if err := s.runs.SetNodeFinished(ctx, run.ID, node.Code, domain.LPNodeSucceeded, outputs, nil); err != nil {
|
||||
// Best-effort, so the node is not left running while the run is failed.
|
||||
return s.finishNodeFailed(ctx, run.ID, node, err)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// finishNodeFailed records the failure on the node and returns the cause.
|
||||
func (s *LandscapeService) finishNodeFailed(ctx context.Context, runID uuid.UUID, node landscape.Node, cause error) error {
|
||||
msg := cause.Error()
|
||||
wctx, cancel := terminalCtx(ctx)
|
||||
defer cancel()
|
||||
if err := s.runs.SetNodeFinished(wctx, runID, node.Code, domain.LPNodeFailed, nil, &msg); err != nil {
|
||||
s.log.Error("mark landscape node failed", "run_id", runID, "node", node.Code, "error", err)
|
||||
}
|
||||
return cause
|
||||
}
|
||||
|
||||
// resolveInputs picks one dataset per input group: an explicit override first,
|
||||
// then whatever this run already produced for the group, then the freshest
|
||||
// ready raster of the group.
|
||||
func (s *LandscapeService) resolveInputs(ctx context.Context, run domain.LPRun, node landscape.Node, state *runState) ([]domain.LPDatasetRef, []domain.Dataset, error) {
|
||||
ready := domain.DatasetStatusReady
|
||||
fileType := domain.FileTypeRaster
|
||||
|
||||
refs := make([]domain.LPDatasetRef, 0, len(node.Inputs))
|
||||
datasets := make([]domain.Dataset, 0, len(node.Inputs))
|
||||
|
||||
for _, group := range node.Inputs {
|
||||
var (
|
||||
d domain.Dataset
|
||||
err error
|
||||
)
|
||||
switch id, ok := s.pinnedInput(run, group, state); {
|
||||
case ok:
|
||||
d, err = s.datasets.GetByID(ctx, id)
|
||||
default:
|
||||
g, gerr := s.groups.GetByCode(ctx, group)
|
||||
if gerr != nil {
|
||||
return nil, nil, fmt.Errorf("node %s: dataset group %q: %w", node.Code, group, gerr)
|
||||
}
|
||||
d, err = s.datasets.LatestInGroup(ctx, g.ID, &ready, &fileType)
|
||||
}
|
||||
if err != nil {
|
||||
if errors.Is(err, domain.ErrNotFound) {
|
||||
return nil, nil, fmt.Errorf("node %s: no ready raster dataset in group %q", node.Code, group)
|
||||
}
|
||||
return nil, nil, err
|
||||
}
|
||||
refs = append(refs, domain.LPDatasetRef{Group: group, DatasetID: d.ID})
|
||||
datasets = append(datasets, d)
|
||||
}
|
||||
return refs, datasets, nil
|
||||
}
|
||||
|
||||
// pinnedInput returns the dataset already decided for a group: the run's
|
||||
// override, or what an earlier node of this run produced.
|
||||
func (s *LandscapeService) pinnedInput(run domain.LPRun, group string, state *runState) (uuid.UUID, bool) {
|
||||
if id, ok := run.Overrides[group]; ok {
|
||||
return id, true
|
||||
}
|
||||
return state.get(group)
|
||||
}
|
||||
|
||||
// reusableOutputs reports whether a previous succeeded node with the same
|
||||
// fingerprint left outputs that all still exist and are ready.
|
||||
func (s *LandscapeService) reusableOutputs(ctx context.Context, runID uuid.UUID, node landscape.Node, fp string) ([]domain.LPDatasetRef, bool, error) {
|
||||
refs, err := s.runs.FindReusableOutputs(ctx, node.Code, fp, runID)
|
||||
if err != nil {
|
||||
if errors.Is(err, domain.ErrNotFound) {
|
||||
return nil, false, nil
|
||||
}
|
||||
return nil, false, err
|
||||
}
|
||||
if len(refs) != len(node.Outputs) {
|
||||
return nil, false, nil
|
||||
}
|
||||
for _, ref := range refs {
|
||||
d, err := s.datasets.GetByID(ctx, ref.DatasetID)
|
||||
if err != nil {
|
||||
if errors.Is(err, domain.ErrNotFound) {
|
||||
return nil, false, nil
|
||||
}
|
||||
return nil, false, err
|
||||
}
|
||||
if d.Status != domain.DatasetStatusReady {
|
||||
return nil, false, nil
|
||||
}
|
||||
}
|
||||
return refs, true, nil
|
||||
}
|
||||
|
||||
// computeNode downloads the node's inputs into a temp dir, computes it, and
|
||||
// publishes one raster dataset per output group. Temp files are always removed.
|
||||
func (s *LandscapeService) computeNode(
|
||||
ctx context.Context,
|
||||
run domain.LPRun,
|
||||
node landscape.Node,
|
||||
params landscape.Params,
|
||||
cat domain.Category,
|
||||
refs []domain.LPDatasetRef,
|
||||
inputs []domain.Dataset,
|
||||
) ([]domain.LPDatasetRef, error) {
|
||||
// landscape.Compute does not create outDir, so we own the whole directory.
|
||||
dir, err := os.MkdirTemp("", "lp-"+node.Code+"-*")
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("node %s: temp dir: %w", node.Code, err)
|
||||
}
|
||||
defer os.RemoveAll(dir)
|
||||
|
||||
paths := make(map[string]string, len(inputs))
|
||||
for i, d := range inputs {
|
||||
group := refs[i].Group
|
||||
// Downstream nodes read the original object, never the COG, so they
|
||||
// never wait for the conversion job.
|
||||
path := filepath.Join(dir, "in-"+group+".tif")
|
||||
if err := s.download(ctx, d.StorageKey, path); err != nil {
|
||||
return nil, fmt.Errorf("node %s: download %q: %w", node.Code, group, err)
|
||||
}
|
||||
paths[group] = path
|
||||
}
|
||||
|
||||
outPaths, err := s.computeWithWorkers(ctx, s.rio, node, paths, dir, params, s.opts.workersPerNode())
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
start, end := periodOf(inputs)
|
||||
date := time.Now()
|
||||
if run.StartedAt != nil {
|
||||
date = *run.StartedAt
|
||||
}
|
||||
|
||||
meta, err := json.Marshal(map[string]any{
|
||||
"lp_run_id": run.ID,
|
||||
"lp_node": node.Code,
|
||||
"inputs": refs,
|
||||
})
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
outputs := make([]domain.LPDatasetRef, 0, len(node.Outputs))
|
||||
for _, group := range node.Outputs {
|
||||
path, ok := outPaths[group]
|
||||
if !ok || path == "" {
|
||||
return nil, fmt.Errorf("node %s: compute produced no output for group %q", node.Code, group)
|
||||
}
|
||||
g, err := s.groups.GetByCode(ctx, group)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("node %s: dataset group %q: %w", node.Code, group, err)
|
||||
}
|
||||
title := group
|
||||
if o, ok := landscape.OutputByCode(group); ok {
|
||||
title = o.Title
|
||||
}
|
||||
d, err := s.publisher.CreateRasterFromFile(ctx, CreateRasterInput{
|
||||
CategoryID: cat.ID,
|
||||
DatasetGroupID: &g.ID,
|
||||
Name: title + " " + date.Format("2006-01-02"),
|
||||
Filename: group + ".tif",
|
||||
Path: path,
|
||||
StartDate: start,
|
||||
EndDate: end,
|
||||
Meta: meta,
|
||||
Automated: true,
|
||||
})
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("node %s: publish %q: %w", node.Code, group, err)
|
||||
}
|
||||
outputs = append(outputs, domain.LPDatasetRef{Group: group, DatasetID: d.ID})
|
||||
}
|
||||
return outputs, nil
|
||||
}
|
||||
|
||||
// download copies an object from the store to a local path.
|
||||
func (s *LandscapeService) download(ctx context.Context, key, path string) error {
|
||||
obj, err := s.store.Get(ctx, key)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
defer obj.Close()
|
||||
|
||||
f, err := os.Create(path) // #nosec G304 -- path is inside the node's os.MkdirTemp directory
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if _, err := io.Copy(f, obj); err != nil {
|
||||
f.Close()
|
||||
return err
|
||||
}
|
||||
return f.Close()
|
||||
}
|
||||
|
||||
// fingerprint is a stable hash of the node, its resolved inputs and the
|
||||
// node-relevant params.
|
||||
func fingerprint(node landscape.Node, inputs []domain.LPDatasetRef, p landscape.Params) string {
|
||||
sorted := append([]domain.LPDatasetRef(nil), inputs...)
|
||||
sort.Slice(sorted, func(i, j int) bool { return sorted[i].Group < sorted[j].Group })
|
||||
payload, _ := json.Marshal(struct {
|
||||
Node string `json:"node"`
|
||||
Inputs []domain.LPDatasetRef `json:"inputs"`
|
||||
Params landscape.Params `json:"params"`
|
||||
}{node.Code, sorted, p.ForNode(node.Code)})
|
||||
sum := sha256.Sum256(payload)
|
||||
return hex.EncodeToString(sum[:])
|
||||
}
|
||||
|
||||
// periodOf returns the min start_date and max end_date over the inputs, or nil
|
||||
// when every input leaves the respective bound unset.
|
||||
func periodOf(inputs []domain.Dataset) (*string, *string) {
|
||||
var start, end *string
|
||||
for _, d := range inputs {
|
||||
if d.StartDate != nil && (start == nil || *d.StartDate < *start) {
|
||||
start = d.StartDate
|
||||
}
|
||||
if d.EndDate != nil && (end == nil || *d.EndDate > *end) {
|
||||
end = d.EndDate
|
||||
}
|
||||
}
|
||||
return start, end
|
||||
}
|
||||
|
||||
// knownGroup reports whether a group code names an input slot or a node output.
|
||||
func knownGroup(code string) bool {
|
||||
if _, ok := landscape.SlotByCode(code); ok {
|
||||
return true
|
||||
}
|
||||
_, ok := landscape.OutputByCode(code)
|
||||
return ok
|
||||
}
|
||||
|
||||
// maxStage returns the highest stage number in the graph.
|
||||
func maxStage() int {
|
||||
highest := 0
|
||||
for _, n := range landscape.Nodes {
|
||||
if n.Stage > highest {
|
||||
highest = n.Stage
|
||||
}
|
||||
}
|
||||
return highest
|
||||
}
|
||||
|
||||
// summaryOf projects a dataset onto the summary shape used in list responses.
|
||||
func summaryOf(d domain.Dataset) domain.DatasetSummary {
|
||||
return domain.DatasetSummary{
|
||||
ID: d.ID,
|
||||
CategoryID: d.CategoryID,
|
||||
DatasetGroupID: d.DatasetGroupID,
|
||||
StartDate: d.StartDate,
|
||||
EndDate: d.EndDate,
|
||||
Name: d.Name,
|
||||
Description: d.Description,
|
||||
Unit: d.Unit,
|
||||
FileType: d.FileType,
|
||||
SizeBytes: d.SizeBytes,
|
||||
Status: d.Status,
|
||||
CreatedAt: d.CreatedAt,
|
||||
UpdatedAt: d.UpdatedAt,
|
||||
}
|
||||
}
|
||||
1530
internal/service/landscape_test.go
Normal file
1530
internal/service/landscape_test.go
Normal file
File diff suppressed because it is too large
Load Diff
63
internal/transport/http/content_disposition.go
Normal file
63
internal/transport/http/content_disposition.go
Normal file
@@ -0,0 +1,63 @@
|
||||
package http
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"strings"
|
||||
)
|
||||
|
||||
// fallbackDownloadName is used when nothing printable survives sanitisation.
|
||||
const fallbackDownloadName = "download"
|
||||
|
||||
// contentDisposition builds a Content-Disposition header value for a stored,
|
||||
// user-supplied filename following RFC 6266: an ASCII-only quoted fallback in
|
||||
// filename= and the exact UTF-8 name, percent-encoded per RFC 8187, in
|
||||
// filename*=. Neither parameter can carry quotes, backslashes, control
|
||||
// characters (CR/LF) or parameter separators taken from the filename, so a
|
||||
// crafted upload name cannot inject extra header parameters or headers.
|
||||
func contentDisposition(disposition, filename string) string {
|
||||
name := strings.Map(func(r rune) rune {
|
||||
if r < 0x20 || r == 0x7f {
|
||||
return -1 // drop control characters, including CR and LF
|
||||
}
|
||||
return r
|
||||
}, filename)
|
||||
name = strings.TrimSpace(name)
|
||||
if name == "" {
|
||||
name = fallbackDownloadName
|
||||
}
|
||||
|
||||
ascii := strings.Map(func(r rune) rune {
|
||||
if r > 0x7e || r == '"' || r == '\\' {
|
||||
return '_'
|
||||
}
|
||||
return r
|
||||
}, name)
|
||||
|
||||
return fmt.Sprintf(`%s; filename="%s"; filename*=UTF-8''%s`, disposition, ascii, encodeRFC8187(name))
|
||||
}
|
||||
|
||||
// encodeRFC8187 percent-encodes every byte of s that is not an attr-char
|
||||
// (RFC 8187 §3.2.1).
|
||||
func encodeRFC8187(s string) string {
|
||||
const hex = "0123456789ABCDEF"
|
||||
var b strings.Builder
|
||||
for i := 0; i < len(s); i++ {
|
||||
c := s[i]
|
||||
if isAttrChar(c) {
|
||||
b.WriteByte(c)
|
||||
continue
|
||||
}
|
||||
b.WriteByte('%')
|
||||
b.WriteByte(hex[c>>4])
|
||||
b.WriteByte(hex[c&0x0f])
|
||||
}
|
||||
return b.String()
|
||||
}
|
||||
|
||||
func isAttrChar(c byte) bool {
|
||||
switch {
|
||||
case 'a' <= c && c <= 'z', 'A' <= c && c <= 'Z', '0' <= c && c <= '9':
|
||||
return true
|
||||
}
|
||||
return strings.IndexByte("!#$&+-.^_`|~", c) >= 0
|
||||
}
|
||||
57
internal/transport/http/content_disposition_test.go
Normal file
57
internal/transport/http/content_disposition_test.go
Normal file
@@ -0,0 +1,57 @@
|
||||
package http
|
||||
|
||||
import (
|
||||
"mime"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestContentDisposition(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
filename string
|
||||
want string
|
||||
}{
|
||||
{"plain", "roads.geojson", `attachment; filename="roads.geojson"; filename*=UTF-8''roads.geojson`},
|
||||
{"space", "my map.tif", `attachment; filename="my map.tif"; filename*=UTF-8''my%20map.tif`},
|
||||
{"quote injection", `a.tif"; filename="evil.exe`, `attachment; filename="a.tif_; filename=_evil.exe"; filename*=UTF-8''a.tif%22%3B%20filename%3D%22evil.exe`},
|
||||
{"crlf injection", "a.tif\r\nSet-Cookie: x=1", `attachment; filename="a.tifSet-Cookie: x=1"; filename*=UTF-8''a.tifSet-Cookie%3A%20x%3D1`},
|
||||
{"backslash", `..\..\a.sld`, `attachment; filename=".._.._a.sld"; filename*=UTF-8''..%5C..%5Ca.sld`},
|
||||
{"cyrillic", "карта.tif", `attachment; filename="_____.tif"; filename*=UTF-8''%D0%BA%D0%B0%D1%80%D1%82%D0%B0.tif`},
|
||||
{"empty", "", `attachment; filename="download"; filename*=UTF-8''download`},
|
||||
{"only controls", "\r\n\t", `attachment; filename="download"; filename*=UTF-8''download`},
|
||||
}
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
got := contentDisposition("attachment", tt.filename)
|
||||
if got != tt.want {
|
||||
t.Fatalf("contentDisposition(%q)\n got %s\nwant %s", tt.filename, got, tt.want)
|
||||
}
|
||||
if strings.ContainsAny(got, "\r\n") {
|
||||
t.Fatalf("header value contains CR/LF: %q", got)
|
||||
}
|
||||
// The value must parse as a single disposition whose decoded
|
||||
// filename is the sanitised original.
|
||||
disp, params, err := mime.ParseMediaType(got)
|
||||
if err != nil {
|
||||
t.Fatalf("ParseMediaType(%q): %v", got, err)
|
||||
}
|
||||
if disp != "attachment" {
|
||||
t.Fatalf("disposition = %q, want attachment", disp)
|
||||
}
|
||||
if len(params) != 1 {
|
||||
t.Fatalf("params = %v, want exactly one decoded filename", params)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestContentDispositionDecodesUTF8Name(t *testing.T) {
|
||||
_, params, err := mime.ParseMediaType(contentDisposition("inline", "карта 2026.cog.tif"))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if got := params["filename"]; got != "карта 2026.cog.tif" {
|
||||
t.Fatalf("decoded filename = %q", got)
|
||||
}
|
||||
}
|
||||
235
internal/transport/http/dataset_group_handler.go
Normal file
235
internal/transport/http/dataset_group_handler.go
Normal file
@@ -0,0 +1,235 @@
|
||||
package http
|
||||
|
||||
import (
|
||||
"io"
|
||||
"net/http"
|
||||
"strings"
|
||||
|
||||
"gis/internal/domain"
|
||||
"gis/internal/service"
|
||||
"gis/pkg/httputil"
|
||||
|
||||
"github.com/go-chi/chi/v5"
|
||||
"github.com/go-playground/validator/v10"
|
||||
)
|
||||
|
||||
// DatasetGroupHandler serves the /dataset-groups routes.
|
||||
type DatasetGroupHandler struct {
|
||||
svc *service.DatasetGroupService
|
||||
validate *validator.Validate
|
||||
}
|
||||
|
||||
// NewDatasetGroupHandler returns a DatasetGroupHandler.
|
||||
func NewDatasetGroupHandler(svc *service.DatasetGroupService, validate *validator.Validate) *DatasetGroupHandler {
|
||||
return &DatasetGroupHandler{svc: svc, validate: validate}
|
||||
}
|
||||
|
||||
// Register mounts the dataset-group routes on r.
|
||||
func (h *DatasetGroupHandler) Register(r chi.Router) {
|
||||
r.Get("/", h.list)
|
||||
r.Post("/", h.create)
|
||||
r.Get("/{id}", h.get)
|
||||
r.Put("/{id}", h.update)
|
||||
r.Delete("/{id}", h.delete)
|
||||
r.Get("/{id}/latest", h.latest)
|
||||
r.Post("/{id}/sld", h.uploadSLD)
|
||||
r.Get("/{id}/sld", h.downloadSLD)
|
||||
}
|
||||
|
||||
type datasetGroupRequest struct {
|
||||
Code string `json:"code" validate:"required,max=255,slug"`
|
||||
Name string `json:"name" validate:"required,max=255"`
|
||||
Description string `json:"description" validate:"max=2000"`
|
||||
}
|
||||
|
||||
func (r datasetGroupRequest) toInput() service.DatasetGroupInput {
|
||||
return service.DatasetGroupInput{Code: r.Code, Name: r.Name, Description: r.Description}
|
||||
}
|
||||
|
||||
func (h *DatasetGroupHandler) create(w http.ResponseWriter, r *http.Request) {
|
||||
req, err := httputil.DecodeJSON[datasetGroupRequest](w, r)
|
||||
if err != nil {
|
||||
httputil.WriteError(w, http.StatusBadRequest, "invalid request body")
|
||||
return
|
||||
}
|
||||
if err := h.validate.Struct(req); err != nil {
|
||||
httputil.WriteValidationErrors(w, err)
|
||||
return
|
||||
}
|
||||
|
||||
group, err := h.svc.Create(r.Context(), req.toInput())
|
||||
if err != nil {
|
||||
respondDomainError(w, err)
|
||||
return
|
||||
}
|
||||
httputil.WriteJSON(w, http.StatusCreated, group)
|
||||
}
|
||||
|
||||
func (h *DatasetGroupHandler) list(w http.ResponseWriter, r *http.Request) {
|
||||
var code *string
|
||||
if v := strings.TrimSpace(r.URL.Query().Get("code")); v != "" {
|
||||
code = &v
|
||||
}
|
||||
|
||||
page, ok := parsePositiveIntQuery(w, r, "page", 1)
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
pageSize, ok := parsePositiveIntQuery(w, r, "page_size", service.DefaultPageSize)
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
|
||||
result, err := h.svc.List(r.Context(), code, page, pageSize)
|
||||
if err != nil {
|
||||
respondDomainError(w, err)
|
||||
return
|
||||
}
|
||||
httputil.WriteJSON(w, http.StatusOK, newPaginated(result.Items, result.Page, result.PageSize, result.Total))
|
||||
}
|
||||
|
||||
func (h *DatasetGroupHandler) get(w http.ResponseWriter, r *http.Request) {
|
||||
id, ok := parseUUIDParam(w, r, "id")
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
group, err := h.svc.Get(r.Context(), id)
|
||||
if err != nil {
|
||||
respondDomainError(w, err)
|
||||
return
|
||||
}
|
||||
httputil.WriteJSON(w, http.StatusOK, group)
|
||||
}
|
||||
|
||||
func (h *DatasetGroupHandler) update(w http.ResponseWriter, r *http.Request) {
|
||||
id, ok := parseUUIDParam(w, r, "id")
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
req, err := httputil.DecodeJSON[datasetGroupRequest](w, r)
|
||||
if err != nil {
|
||||
httputil.WriteError(w, http.StatusBadRequest, "invalid request body")
|
||||
return
|
||||
}
|
||||
if err := h.validate.Struct(req); err != nil {
|
||||
httputil.WriteValidationErrors(w, err)
|
||||
return
|
||||
}
|
||||
|
||||
group, err := h.svc.Update(r.Context(), id, req.toInput())
|
||||
if err != nil {
|
||||
respondDomainError(w, err)
|
||||
return
|
||||
}
|
||||
httputil.WriteJSON(w, http.StatusOK, group)
|
||||
}
|
||||
|
||||
// delete removes a group. A group still referenced by datasets yields 409.
|
||||
func (h *DatasetGroupHandler) delete(w http.ResponseWriter, r *http.Request) {
|
||||
id, ok := parseUUIDParam(w, r, "id")
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
if err := h.svc.Delete(r.Context(), id); err != nil {
|
||||
respondDomainError(w, err)
|
||||
return
|
||||
}
|
||||
w.WriteHeader(http.StatusNoContent)
|
||||
}
|
||||
|
||||
// latest returns the freshest dataset of the group (greatest end_date, NULLs
|
||||
// last, then newest), optionally narrowed by ?status= and ?file_type=.
|
||||
func (h *DatasetGroupHandler) latest(w http.ResponseWriter, r *http.Request) {
|
||||
id, ok := parseUUIDParam(w, r, "id")
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
|
||||
var status *string
|
||||
if v := strings.TrimSpace(r.URL.Query().Get("status")); v != "" {
|
||||
if !domain.ValidDatasetStatus(v) {
|
||||
httputil.WriteError(w, http.StatusBadRequest, "invalid status")
|
||||
return
|
||||
}
|
||||
status = &v
|
||||
}
|
||||
|
||||
var fileType *domain.FileType
|
||||
if v := strings.TrimSpace(r.URL.Query().Get("file_type")); v != "" {
|
||||
ft := domain.FileType(v)
|
||||
if !ft.Valid() {
|
||||
httputil.WriteError(w, http.StatusBadRequest, "file_type must be one of: vector_with_kato, vector, raster")
|
||||
return
|
||||
}
|
||||
fileType = &ft
|
||||
}
|
||||
|
||||
dataset, err := h.svc.Latest(r.Context(), id, status, fileType)
|
||||
if err != nil {
|
||||
respondDomainError(w, err)
|
||||
return
|
||||
}
|
||||
httputil.WriteJSON(w, http.StatusOK, dataset)
|
||||
}
|
||||
|
||||
func (h *DatasetGroupHandler) uploadSLD(w http.ResponseWriter, r *http.Request) {
|
||||
id, ok := parseUUIDParam(w, r, "id")
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
if !parseMultipartForm(w, r) {
|
||||
return
|
||||
}
|
||||
|
||||
file, header, err := r.FormFile("file")
|
||||
if err != nil {
|
||||
httputil.WriteError(w, http.StatusBadRequest, "file is required")
|
||||
return
|
||||
}
|
||||
defer file.Close()
|
||||
|
||||
contentType := header.Header.Get("Content-Type")
|
||||
if contentType == "" {
|
||||
contentType = "application/xml"
|
||||
}
|
||||
|
||||
group, err := h.svc.UploadSLD(r.Context(), id, service.UploadSLDInput{
|
||||
Filename: header.Filename,
|
||||
ContentType: contentType,
|
||||
Size: header.Size,
|
||||
Reader: file,
|
||||
})
|
||||
if err != nil {
|
||||
respondDomainError(w, err)
|
||||
return
|
||||
}
|
||||
httputil.WriteJSON(w, http.StatusOK, group)
|
||||
}
|
||||
|
||||
func (h *DatasetGroupHandler) downloadSLD(w http.ResponseWriter, r *http.Request) {
|
||||
id, ok := parseUUIDParam(w, r, "id")
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
group, obj, err := h.svc.DownloadSLD(r.Context(), id)
|
||||
if err != nil {
|
||||
respondDomainError(w, err)
|
||||
return
|
||||
}
|
||||
defer obj.Close()
|
||||
|
||||
contentType := "application/xml"
|
||||
if group.SldContentType != nil && *group.SldContentType != "" {
|
||||
contentType = *group.SldContentType
|
||||
}
|
||||
filename := "style.sld"
|
||||
if group.SldFilename != nil && *group.SldFilename != "" {
|
||||
filename = *group.SldFilename
|
||||
}
|
||||
w.Header().Set("Content-Type", contentType)
|
||||
w.Header().Set("Content-Disposition", contentDisposition("attachment", filename))
|
||||
if _, err := io.Copy(w, obj); err != nil {
|
||||
// Headers are already sent; nothing useful to return to the client.
|
||||
return
|
||||
}
|
||||
}
|
||||
@@ -2,6 +2,7 @@ package http
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"io"
|
||||
"net/http"
|
||||
"strconv"
|
||||
@@ -17,9 +18,31 @@ import (
|
||||
"github.com/google/uuid"
|
||||
)
|
||||
|
||||
// maxUploadBytes caps the in-memory portion of a multipart upload (64 MiB).
|
||||
// maxUploadBytes caps the in-memory portion of a multipart upload (64 MiB);
|
||||
// larger parts spill to temporary files. The total request size is capped by
|
||||
// the limitRequestBody middleware (HTTP_MAX_UPLOAD_BYTES).
|
||||
const maxUploadBytes = 64 << 20
|
||||
|
||||
// parseMultipartForm parses a multipart upload and writes the error response
|
||||
// itself when that fails: 413 when the body exceeds the router's size cap, 400
|
||||
// otherwise. It reports whether the handler may continue.
|
||||
func parseMultipartForm(w http.ResponseWriter, r *http.Request) bool {
|
||||
// #nosec G120 -- r.Body is wrapped in http.MaxBytesReader by the
|
||||
// limitRequestBody middleware that NewRouter installs on every route.
|
||||
err := r.ParseMultipartForm(maxUploadBytes)
|
||||
if err == nil {
|
||||
return true
|
||||
}
|
||||
var tooLarge *http.MaxBytesError
|
||||
if errors.As(err, &tooLarge) {
|
||||
httputil.WriteError(w, http.StatusRequestEntityTooLarge,
|
||||
"request body exceeds the "+strconv.FormatInt(tooLarge.Limit, 10)+" byte limit")
|
||||
return false
|
||||
}
|
||||
httputil.WriteError(w, http.StatusBadRequest, "request must be multipart/form-data")
|
||||
return false
|
||||
}
|
||||
|
||||
// DatasetHandler serves the /datasets routes.
|
||||
type DatasetHandler struct {
|
||||
svc *service.DatasetService
|
||||
@@ -36,9 +59,14 @@ func (h *DatasetHandler) Register(r chi.Router) {
|
||||
r.Get("/", h.list)
|
||||
r.Post("/", h.upload)
|
||||
r.Get("/{id}", h.get)
|
||||
r.Put("/{id}", h.update)
|
||||
r.Patch("/{id}", h.patch)
|
||||
r.Get("/{id}.geojson", h.geojson)
|
||||
r.Get("/{id}.cog", h.cog)
|
||||
r.Get("/{id}/status", h.status)
|
||||
r.Get("/{id}/download", h.download)
|
||||
r.Post("/{id}/sld", h.uploadSLD)
|
||||
r.Get("/{id}/sld", h.downloadSLD)
|
||||
r.Post("/{id}/mapping", h.mapping)
|
||||
r.Get("/{id}/observations", h.observations)
|
||||
r.Delete("/{id}", h.delete)
|
||||
@@ -74,8 +102,9 @@ func (h *DatasetHandler) status(w http.ResponseWriter, r *http.Request) {
|
||||
}
|
||||
|
||||
type yearColumnInput struct {
|
||||
Column string `json:"column" validate:"required"`
|
||||
Date string `json:"date" validate:"required,datetime=2006-01-02"`
|
||||
Column string `json:"column" validate:"required"`
|
||||
StartDate string `json:"start_date" validate:"required,datetime=2006-01-02"`
|
||||
EndDate *string `json:"end_date" validate:"omitempty,datetime=2006-01-02"`
|
||||
}
|
||||
|
||||
type mappingRequest struct {
|
||||
@@ -102,7 +131,7 @@ func (h *DatasetHandler) mapping(w http.ResponseWriter, r *http.Request) {
|
||||
|
||||
in := service.MappingInput{KatoColumn: req.KatoColumn}
|
||||
for _, yc := range req.YearColumns {
|
||||
in.YearColumns = append(in.YearColumns, domain.YearColumn{Column: yc.Column, Date: yc.Date})
|
||||
in.YearColumns = append(in.YearColumns, domain.YearColumn{Column: yc.Column, StartDate: yc.StartDate, EndDate: yc.EndDate})
|
||||
}
|
||||
|
||||
dataset, err := h.svc.SaveMapping(r.Context(), id, in)
|
||||
@@ -114,8 +143,7 @@ func (h *DatasetHandler) mapping(w http.ResponseWriter, r *http.Request) {
|
||||
}
|
||||
|
||||
func (h *DatasetHandler) upload(w http.ResponseWriter, r *http.Request) {
|
||||
if err := r.ParseMultipartForm(maxUploadBytes); err != nil {
|
||||
httputil.WriteError(w, http.StatusBadRequest, "request must be multipart/form-data")
|
||||
if !parseMultipartForm(w, r) {
|
||||
return
|
||||
}
|
||||
|
||||
@@ -143,6 +171,17 @@ func (h *DatasetHandler) upload(w http.ResponseWriter, r *http.Request) {
|
||||
return
|
||||
}
|
||||
|
||||
// The dataset group is optional; the period is validated by the service.
|
||||
var groupID *uuid.UUID
|
||||
if v := optionalFormValue(r, "dg_id"); v != nil {
|
||||
gid, err := uuid.Parse(*v)
|
||||
if err != nil {
|
||||
httputil.WriteError(w, http.StatusUnprocessableEntity, "dg_id must be a valid UUID")
|
||||
return
|
||||
}
|
||||
groupID = &gid
|
||||
}
|
||||
|
||||
file, header, err := r.FormFile("file")
|
||||
if err != nil {
|
||||
httputil.WriteError(w, http.StatusBadRequest, "file is required")
|
||||
@@ -156,17 +195,20 @@ func (h *DatasetHandler) upload(w http.ResponseWriter, r *http.Request) {
|
||||
}
|
||||
|
||||
dataset, err := h.svc.Upload(r.Context(), service.UploadInput{
|
||||
CategoryID: categoryID,
|
||||
Name: r.FormValue("name"),
|
||||
Description: optionalFormValue(r, "description"),
|
||||
Unit: optionalFormValue(r, "unit"),
|
||||
Meta: meta,
|
||||
Automated: automated,
|
||||
Filename: header.Filename,
|
||||
FileType: fileType,
|
||||
ContentType: contentType,
|
||||
Size: header.Size,
|
||||
Reader: file,
|
||||
CategoryID: categoryID,
|
||||
Name: r.FormValue("name"),
|
||||
Description: optionalFormValue(r, "description"),
|
||||
Unit: optionalFormValue(r, "unit"),
|
||||
Meta: meta,
|
||||
Automated: automated,
|
||||
Filename: header.Filename,
|
||||
FileType: fileType,
|
||||
ContentType: contentType,
|
||||
Size: header.Size,
|
||||
Reader: file,
|
||||
DatasetGroupID: groupID,
|
||||
StartDate: optionalFormValue(r, "start_date"),
|
||||
EndDate: optionalFormValue(r, "end_date"),
|
||||
})
|
||||
if err != nil {
|
||||
respondDomainError(w, err)
|
||||
@@ -219,7 +261,32 @@ func (h *DatasetHandler) list(w http.ResponseWriter, r *http.Request) {
|
||||
categoryCode = &v
|
||||
}
|
||||
|
||||
filter := domain.DatasetFilter{CategoryID: categoryID}
|
||||
groupID, ok := parseOptionalUUIDQuery(w, r, "dg_id")
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
|
||||
filter := domain.DatasetFilter{CategoryID: categoryID, DatasetGroupID: groupID}
|
||||
|
||||
for _, b := range []struct {
|
||||
name string
|
||||
dst **string
|
||||
}{
|
||||
{"start_date_from", &filter.StartDateFrom},
|
||||
{"start_date_to", &filter.StartDateTo},
|
||||
{"end_date_from", &filter.EndDateFrom},
|
||||
{"end_date_to", &filter.EndDateTo},
|
||||
} {
|
||||
v := strings.TrimSpace(r.URL.Query().Get(b.name))
|
||||
if v == "" {
|
||||
continue
|
||||
}
|
||||
if _, err := time.Parse("2006-01-02", v); err != nil {
|
||||
httputil.WriteError(w, http.StatusBadRequest, b.name+" must be YYYY-MM-DD")
|
||||
return
|
||||
}
|
||||
*b.dst = &v
|
||||
}
|
||||
|
||||
if v := strings.TrimSpace(r.URL.Query().Get("file_type")); v != "" {
|
||||
ft := domain.FileType(v)
|
||||
@@ -304,6 +371,222 @@ func (h *DatasetHandler) get(w http.ResponseWriter, r *http.Request) {
|
||||
httputil.WriteJSON(w, http.StatusOK, dataset)
|
||||
}
|
||||
|
||||
// datasetPutRequest is the full JSON body for PUT /datasets/{id}: every
|
||||
// editable field is required. Description and Unit are pointers so an empty
|
||||
// string is distinguishable from an absent field; a blank value clears the
|
||||
// (nullable) column. File-related and lifecycle fields are not editable here.
|
||||
type datasetPutRequest struct {
|
||||
CategoryID string `json:"category_id" validate:"required,uuid"`
|
||||
Name string `json:"name" validate:"required,max=255"`
|
||||
Description *string `json:"description" validate:"required"`
|
||||
Unit *string `json:"unit" validate:"required"`
|
||||
// DgID, StartDate and EndDate are optional; omitting them (or sending null
|
||||
// or a blank string) clears the corresponding nullable column.
|
||||
DgID *string `json:"dg_id" validate:"omitempty,uuid"`
|
||||
StartDate *string `json:"start_date" validate:"omitempty,datetime=2006-01-02"`
|
||||
EndDate *string `json:"end_date" validate:"omitempty,datetime=2006-01-02"`
|
||||
}
|
||||
|
||||
// update replaces a dataset's editable metadata (category, name, description,
|
||||
// unit, dataset group, period).
|
||||
func (h *DatasetHandler) update(w http.ResponseWriter, r *http.Request) {
|
||||
id, ok := parseUUIDParam(w, r, "id")
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
req, err := httputil.DecodeJSON[datasetPutRequest](w, r)
|
||||
if err != nil {
|
||||
httputil.WriteError(w, http.StatusBadRequest, "invalid request body")
|
||||
return
|
||||
}
|
||||
if err := h.validate.Struct(req); err != nil {
|
||||
httputil.WriteValidationErrors(w, err)
|
||||
return
|
||||
}
|
||||
categoryID, err := uuid.Parse(req.CategoryID)
|
||||
if err != nil {
|
||||
httputil.WriteError(w, http.StatusUnprocessableEntity, "category_id must be a valid UUID")
|
||||
return
|
||||
}
|
||||
|
||||
var groupID *uuid.UUID
|
||||
if raw := optionalPtr(req.DgID); raw != nil {
|
||||
gid, err := uuid.Parse(*raw)
|
||||
if err != nil {
|
||||
httputil.WriteError(w, http.StatusUnprocessableEntity, "dg_id must be a valid UUID")
|
||||
return
|
||||
}
|
||||
groupID = &gid
|
||||
}
|
||||
|
||||
dataset, err := h.svc.Update(r.Context(), id, service.DatasetUpdateInput{
|
||||
CategoryID: categoryID,
|
||||
Name: strings.TrimSpace(req.Name),
|
||||
Description: normalizeOptional(*req.Description),
|
||||
Unit: normalizeOptional(*req.Unit),
|
||||
DatasetGroupID: groupID,
|
||||
StartDate: optionalPtr(req.StartDate),
|
||||
EndDate: optionalPtr(req.EndDate),
|
||||
})
|
||||
if err != nil {
|
||||
respondDomainError(w, err)
|
||||
return
|
||||
}
|
||||
httputil.WriteJSON(w, http.StatusOK, dataset)
|
||||
}
|
||||
|
||||
// datasetPatchFields lists the JSON keys accepted by PATCH /datasets/{id}.
|
||||
var datasetPatchFields = map[string]struct{}{
|
||||
"category_id": {}, "name": {}, "description": {}, "unit": {},
|
||||
"dg_id": {}, "start_date": {}, "end_date": {},
|
||||
}
|
||||
|
||||
// patch applies a sparse update to a dataset's editable metadata (category,
|
||||
// name, description, unit, dg_id, start_date, end_date): only fields present in
|
||||
// the body are changed. Sending an explicit null for a nullable field clears
|
||||
// that column.
|
||||
func (h *DatasetHandler) patch(w http.ResponseWriter, r *http.Request) {
|
||||
id, ok := parseUUIDParam(w, r, "id")
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
raw, err := httputil.DecodeJSON[map[string]json.RawMessage](w, r)
|
||||
if err != nil {
|
||||
httputil.WriteError(w, http.StatusBadRequest, "invalid request body")
|
||||
return
|
||||
}
|
||||
if len(raw) == 0 {
|
||||
httputil.WriteError(w, http.StatusBadRequest, "request body must include at least one field")
|
||||
return
|
||||
}
|
||||
|
||||
var in service.DatasetPatchInput
|
||||
for key, val := range raw {
|
||||
if _, ok := datasetPatchFields[key]; !ok {
|
||||
httputil.WriteError(w, http.StatusUnprocessableEntity, "unknown field: "+key)
|
||||
return
|
||||
}
|
||||
switch key {
|
||||
case "category_id":
|
||||
var s string
|
||||
if err := json.Unmarshal(val, &s); err != nil || s == "" {
|
||||
httputil.WriteError(w, http.StatusUnprocessableEntity, "category_id must be a valid UUID")
|
||||
return
|
||||
}
|
||||
cid, err := uuid.Parse(s)
|
||||
if err != nil {
|
||||
httputil.WriteError(w, http.StatusUnprocessableEntity, "category_id must be a valid UUID")
|
||||
return
|
||||
}
|
||||
in.CategoryID = &cid
|
||||
case "name":
|
||||
var s string
|
||||
if err := json.Unmarshal(val, &s); err != nil || strings.TrimSpace(s) == "" {
|
||||
httputil.WriteError(w, http.StatusUnprocessableEntity, "name must not be blank")
|
||||
return
|
||||
}
|
||||
name := strings.TrimSpace(s)
|
||||
in.Name = &name
|
||||
case "description":
|
||||
desc, err := decodeOptionalPatchString(val)
|
||||
if err != nil {
|
||||
httputil.WriteError(w, http.StatusUnprocessableEntity, "description must be a string or null")
|
||||
return
|
||||
}
|
||||
in.Description, in.DescriptionSet = desc, true
|
||||
case "unit":
|
||||
unit, err := decodeOptionalPatchString(val)
|
||||
if err != nil {
|
||||
httputil.WriteError(w, http.StatusUnprocessableEntity, "unit must be a string or null")
|
||||
return
|
||||
}
|
||||
in.Unit, in.UnitSet = unit, true
|
||||
case "dg_id":
|
||||
s, err := decodeOptionalPatchString(val)
|
||||
if err != nil {
|
||||
httputil.WriteError(w, http.StatusUnprocessableEntity, "dg_id must be a valid UUID or null")
|
||||
return
|
||||
}
|
||||
if s != nil {
|
||||
gid, err := uuid.Parse(*s)
|
||||
if err != nil {
|
||||
httputil.WriteError(w, http.StatusUnprocessableEntity, "dg_id must be a valid UUID or null")
|
||||
return
|
||||
}
|
||||
in.DatasetGroupID = &gid
|
||||
}
|
||||
in.DatasetGroupIDSet = true
|
||||
case "start_date":
|
||||
date, err := decodeOptionalPatchDate(val)
|
||||
if err != nil {
|
||||
httputil.WriteError(w, http.StatusUnprocessableEntity, "start_date must be YYYY-MM-DD or null")
|
||||
return
|
||||
}
|
||||
in.StartDate, in.StartDateSet = date, true
|
||||
case "end_date":
|
||||
date, err := decodeOptionalPatchDate(val)
|
||||
if err != nil {
|
||||
httputil.WriteError(w, http.StatusUnprocessableEntity, "end_date must be YYYY-MM-DD or null")
|
||||
return
|
||||
}
|
||||
in.EndDate, in.EndDateSet = date, true
|
||||
}
|
||||
}
|
||||
|
||||
dataset, err := h.svc.Patch(r.Context(), id, in)
|
||||
if err != nil {
|
||||
respondDomainError(w, err)
|
||||
return
|
||||
}
|
||||
httputil.WriteJSON(w, http.StatusOK, dataset)
|
||||
}
|
||||
|
||||
// optionalPtr normalizes an optional request field: a nil or blank value
|
||||
// yields nil, so an omitted or empty field clears a nullable column.
|
||||
func optionalPtr(v *string) *string {
|
||||
if v == nil {
|
||||
return nil
|
||||
}
|
||||
return normalizeOptional(*v)
|
||||
}
|
||||
|
||||
// normalizeOptional trims s and returns nil when blank, so an empty string
|
||||
// clears a nullable column rather than being stored verbatim.
|
||||
func normalizeOptional(s string) *string {
|
||||
s = strings.TrimSpace(s)
|
||||
if s == "" {
|
||||
return nil
|
||||
}
|
||||
return &s
|
||||
}
|
||||
|
||||
// decodeOptionalPatchString unmarshals a PATCH field that may be a JSON string
|
||||
// or null, returning nil (clear the column) for null or a blank string.
|
||||
func decodeOptionalPatchString(raw json.RawMessage) (*string, error) {
|
||||
if string(raw) == "null" {
|
||||
return nil, nil
|
||||
}
|
||||
var s string
|
||||
if err := json.Unmarshal(raw, &s); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return normalizeOptional(s), nil
|
||||
}
|
||||
|
||||
// decodeOptionalPatchDate unmarshals a PATCH date field that may be a
|
||||
// YYYY-MM-DD JSON string or null, returning nil (clear the column) for null or
|
||||
// a blank string.
|
||||
func decodeOptionalPatchDate(raw json.RawMessage) (*string, error) {
|
||||
v, err := decodeOptionalPatchString(raw)
|
||||
if err != nil || v == nil {
|
||||
return nil, err
|
||||
}
|
||||
if _, err := time.Parse("2006-01-02", *v); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return v, nil
|
||||
}
|
||||
|
||||
// geojson returns the dataset's pre-assembled GeoJSON FeatureCollection (RFC
|
||||
// 7946), generated and stored at processing time. A vector_with_kato dataset's
|
||||
// collection joins the districts table on KATO code (one feature per KATO with
|
||||
@@ -321,7 +604,45 @@ func (h *DatasetHandler) geojson(w http.ResponseWriter, r *http.Request) {
|
||||
}
|
||||
w.Header().Set("Content-Type", "application/geo+json")
|
||||
w.WriteHeader(http.StatusOK)
|
||||
_, _ = w.Write(fc)
|
||||
_, _ = w.Write(fc) // #nosec G705 -- JSON body served as application/geo+json with nosniff, never as HTML
|
||||
}
|
||||
|
||||
// cog streams the raster dataset's Cloud-Optimized GeoTIFF. It responds 422 for
|
||||
// a non-raster dataset and 409 when the raster's COG has not been produced yet.
|
||||
func (h *DatasetHandler) cog(w http.ResponseWriter, r *http.Request) {
|
||||
id, ok := parseUUIDParam(w, r, "id")
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
dataset, obj, err := h.svc.COG(r.Context(), id)
|
||||
if err != nil {
|
||||
switch {
|
||||
case errors.Is(err, service.ErrNotRaster):
|
||||
httputil.WriteError(w, http.StatusUnprocessableEntity, "dataset is not a raster")
|
||||
case errors.Is(err, service.ErrCOGNotReady):
|
||||
httputil.WriteError(w, http.StatusConflict, "cog is not ready")
|
||||
default:
|
||||
respondDomainError(w, err)
|
||||
}
|
||||
return
|
||||
}
|
||||
defer obj.Close()
|
||||
|
||||
w.Header().Set("Content-Type", "image/tiff")
|
||||
w.Header().Set("Content-Disposition", contentDisposition("inline", cogFilename(dataset.Filename)))
|
||||
w.WriteHeader(http.StatusOK)
|
||||
if _, err := io.Copy(w, obj); err != nil {
|
||||
// Headers are already sent; nothing useful to return to the client.
|
||||
return
|
||||
}
|
||||
}
|
||||
|
||||
// cogFilename derives a .cog.tif download name from the source filename.
|
||||
func cogFilename(filename string) string {
|
||||
if i := strings.LastIndex(filename, "."); i > 0 {
|
||||
filename = filename[:i]
|
||||
}
|
||||
return filename + ".cog.tif"
|
||||
}
|
||||
|
||||
func (h *DatasetHandler) download(w http.ResponseWriter, r *http.Request) {
|
||||
@@ -337,7 +658,7 @@ func (h *DatasetHandler) download(w http.ResponseWriter, r *http.Request) {
|
||||
defer obj.Close()
|
||||
|
||||
w.Header().Set("Content-Type", dataset.ContentType)
|
||||
w.Header().Set("Content-Disposition", `attachment; filename="`+dataset.Filename+`"`)
|
||||
w.Header().Set("Content-Disposition", contentDisposition("attachment", dataset.Filename))
|
||||
if dataset.SizeBytes > 0 {
|
||||
w.Header().Set("Content-Length", strconv.FormatInt(dataset.SizeBytes, 10))
|
||||
}
|
||||
@@ -347,6 +668,73 @@ func (h *DatasetHandler) download(w http.ResponseWriter, r *http.Request) {
|
||||
}
|
||||
}
|
||||
|
||||
// uploadSLD stores (or replaces) a dataset's SLD (Styled Layer Descriptor)
|
||||
// style file. Multipart upload, mirroring POST /datasets. Applies to every
|
||||
// file_type.
|
||||
func (h *DatasetHandler) uploadSLD(w http.ResponseWriter, r *http.Request) {
|
||||
id, ok := parseUUIDParam(w, r, "id")
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
if !parseMultipartForm(w, r) {
|
||||
return
|
||||
}
|
||||
|
||||
file, header, err := r.FormFile("file")
|
||||
if err != nil {
|
||||
httputil.WriteError(w, http.StatusBadRequest, "file is required")
|
||||
return
|
||||
}
|
||||
defer file.Close()
|
||||
|
||||
contentType := header.Header.Get("Content-Type")
|
||||
if contentType == "" {
|
||||
contentType = "application/xml"
|
||||
}
|
||||
|
||||
dataset, err := h.svc.UploadSLD(r.Context(), id, service.UploadSLDInput{
|
||||
Filename: header.Filename,
|
||||
ContentType: contentType,
|
||||
Size: header.Size,
|
||||
Reader: file,
|
||||
})
|
||||
if err != nil {
|
||||
respondDomainError(w, err)
|
||||
return
|
||||
}
|
||||
httputil.WriteJSON(w, http.StatusOK, dataset)
|
||||
}
|
||||
|
||||
// downloadSLD streams a dataset's stored SLD style file. Returns 404 if none
|
||||
// has been uploaded yet.
|
||||
func (h *DatasetHandler) downloadSLD(w http.ResponseWriter, r *http.Request) {
|
||||
id, ok := parseUUIDParam(w, r, "id")
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
dataset, obj, err := h.svc.DownloadSLD(r.Context(), id)
|
||||
if err != nil {
|
||||
respondDomainError(w, err)
|
||||
return
|
||||
}
|
||||
defer obj.Close()
|
||||
|
||||
contentType := "application/xml"
|
||||
if dataset.SldContentType != nil && *dataset.SldContentType != "" {
|
||||
contentType = *dataset.SldContentType
|
||||
}
|
||||
filename := "style.sld"
|
||||
if dataset.SldFilename != nil && *dataset.SldFilename != "" {
|
||||
filename = *dataset.SldFilename
|
||||
}
|
||||
w.Header().Set("Content-Type", contentType)
|
||||
w.Header().Set("Content-Disposition", contentDisposition("attachment", filename))
|
||||
if _, err := io.Copy(w, obj); err != nil {
|
||||
// Headers are already sent; nothing useful to return to the client.
|
||||
return
|
||||
}
|
||||
}
|
||||
|
||||
func (h *DatasetHandler) delete(w http.ResponseWriter, r *http.Request) {
|
||||
id, ok := parseUUIDParam(w, r, "id")
|
||||
if !ok {
|
||||
|
||||
@@ -2,6 +2,7 @@ package http
|
||||
|
||||
import (
|
||||
"context"
|
||||
"log/slog"
|
||||
"net/http"
|
||||
|
||||
"gis/pkg/httputil"
|
||||
@@ -13,11 +14,17 @@ type ReadinessCheck func(ctx context.Context) error
|
||||
// HealthHandler serves liveness and readiness probes.
|
||||
type HealthHandler struct {
|
||||
checks map[string]ReadinessCheck
|
||||
log *slog.Logger
|
||||
}
|
||||
|
||||
// NewHealthHandler builds a HealthHandler with the given named readiness checks.
|
||||
func NewHealthHandler(checks map[string]ReadinessCheck) *HealthHandler {
|
||||
return &HealthHandler{checks: checks}
|
||||
// Failed checks are logged with their error; the response only names the
|
||||
// failing component, so connection details never reach the client.
|
||||
func NewHealthHandler(checks map[string]ReadinessCheck, log *slog.Logger) *HealthHandler {
|
||||
if log == nil {
|
||||
log = slog.Default()
|
||||
}
|
||||
return &HealthHandler{checks: checks, log: log}
|
||||
}
|
||||
|
||||
// Live reports that the process is up.
|
||||
@@ -32,7 +39,8 @@ func (h *HealthHandler) Ready(w http.ResponseWriter, r *http.Request) {
|
||||
ready := true
|
||||
for name, check := range h.checks {
|
||||
if err := check(r.Context()); err != nil {
|
||||
results[name] = "error: " + err.Error()
|
||||
h.log.Warn("readiness check failed", "component", name, "error", err)
|
||||
results[name] = "unavailable"
|
||||
ready = false
|
||||
continue
|
||||
}
|
||||
|
||||
185
internal/transport/http/landscape_handler.go
Normal file
185
internal/transport/http/landscape_handler.go
Normal file
@@ -0,0 +1,185 @@
|
||||
package http
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"io"
|
||||
"net/http"
|
||||
|
||||
"gis/internal/landscape"
|
||||
"gis/internal/service"
|
||||
"gis/pkg/httputil"
|
||||
|
||||
"github.com/go-chi/chi/v5"
|
||||
"github.com/go-playground/validator/v10"
|
||||
"github.com/google/uuid"
|
||||
)
|
||||
|
||||
// LandscapeHandler serves the /landscape routes.
|
||||
type LandscapeHandler struct {
|
||||
svc *service.LandscapeService
|
||||
validate *validator.Validate
|
||||
}
|
||||
|
||||
// NewLandscapeHandler returns a LandscapeHandler.
|
||||
func NewLandscapeHandler(svc *service.LandscapeService, validate *validator.Validate) *LandscapeHandler {
|
||||
return &LandscapeHandler{svc: svc, validate: validate}
|
||||
}
|
||||
|
||||
// Register mounts the Landscape Program routes on r.
|
||||
func (h *LandscapeHandler) Register(r chi.Router) {
|
||||
r.Get("/slots", h.slots)
|
||||
r.Post("/slots/{code}", h.uploadSlot)
|
||||
r.Get("/runs", h.listRuns)
|
||||
r.Post("/runs", h.createRun)
|
||||
r.Get("/runs/{id}", h.getRun)
|
||||
r.Get("/results", h.results)
|
||||
}
|
||||
|
||||
// runRequest is the body of POST /landscape/runs. Overrides are decoded as
|
||||
// strings so a malformed id is a 422 instead of a 400 on the whole body.
|
||||
type runRequest struct {
|
||||
FromStage *int `json:"from_stage" validate:"omitempty,gte=0,lte=3"`
|
||||
Overrides map[string]string `json:"overrides" validate:"omitempty,dive,uuid"`
|
||||
Params *landscape.Params `json:"params"`
|
||||
}
|
||||
|
||||
// slots lists every leaf input slot with the dataset currently filling it.
|
||||
func (h *LandscapeHandler) slots(w http.ResponseWriter, r *http.Request) {
|
||||
slots, err := h.svc.Slots(r.Context())
|
||||
if err != nil {
|
||||
respondDomainError(w, err)
|
||||
return
|
||||
}
|
||||
httputil.WriteJSON(w, http.StatusOK, slots)
|
||||
}
|
||||
|
||||
// uploadSlot stores a raster in the slot's dataset group under the landscape
|
||||
// category. An unknown slot code yields 404.
|
||||
func (h *LandscapeHandler) uploadSlot(w http.ResponseWriter, r *http.Request) {
|
||||
code := chi.URLParam(r, "code")
|
||||
|
||||
if !parseMultipartForm(w, r) {
|
||||
return
|
||||
}
|
||||
|
||||
file, header, err := r.FormFile("file")
|
||||
if err != nil {
|
||||
httputil.WriteError(w, http.StatusBadRequest, "file is required")
|
||||
return
|
||||
}
|
||||
defer file.Close()
|
||||
|
||||
contentType := header.Header.Get("Content-Type")
|
||||
if contentType == "" {
|
||||
contentType = "image/tiff"
|
||||
}
|
||||
|
||||
dataset, err := h.svc.UploadToSlot(r.Context(), code, service.SlotUploadInput{
|
||||
Filename: header.Filename,
|
||||
ContentType: contentType,
|
||||
Size: header.Size,
|
||||
Reader: file,
|
||||
Name: r.FormValue("name"),
|
||||
StartDate: optionalFormValue(r, "start_date"),
|
||||
EndDate: optionalFormValue(r, "end_date"),
|
||||
})
|
||||
if err != nil {
|
||||
respondDomainError(w, err)
|
||||
return
|
||||
}
|
||||
httputil.WriteJSON(w, http.StatusCreated, dataset)
|
||||
}
|
||||
|
||||
// decodeRunRequest reads the run body. The whole body is optional — every field
|
||||
// has a default — so an empty body is the zero request (a default incremental
|
||||
// run), not an error. A malformed body writes a 400 and reports false.
|
||||
func decodeRunRequest(w http.ResponseWriter, r *http.Request) (runRequest, bool) {
|
||||
req, err := httputil.DecodeJSON[runRequest](w, r)
|
||||
switch {
|
||||
case errors.Is(err, io.EOF):
|
||||
return runRequest{}, true
|
||||
case err != nil:
|
||||
httputil.WriteError(w, http.StatusBadRequest, "invalid request body")
|
||||
return runRequest{}, false
|
||||
}
|
||||
return req, true
|
||||
}
|
||||
|
||||
// createRun queues a Landscape Program run and returns it with 202 Accepted;
|
||||
// the worker executes it asynchronously.
|
||||
func (h *LandscapeHandler) createRun(w http.ResponseWriter, r *http.Request) {
|
||||
req, ok := decodeRunRequest(w, r)
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
if err := h.validate.Struct(req); err != nil {
|
||||
httputil.WriteValidationErrors(w, err)
|
||||
return
|
||||
}
|
||||
|
||||
overrides := make(map[string]uuid.UUID, len(req.Overrides))
|
||||
for group, raw := range req.Overrides {
|
||||
id, err := uuid.Parse(raw)
|
||||
if err != nil {
|
||||
httputil.WriteError(w, http.StatusUnprocessableEntity, "overrides["+group+"] must be a valid UUID")
|
||||
return
|
||||
}
|
||||
overrides[group] = id
|
||||
}
|
||||
|
||||
in := service.RunInput{FromStage: req.FromStage, Overrides: overrides}
|
||||
if req.Params != nil {
|
||||
in.Params = *req.Params
|
||||
}
|
||||
|
||||
run, err := h.svc.CreateRun(r.Context(), in)
|
||||
if err != nil {
|
||||
respondDomainError(w, err)
|
||||
return
|
||||
}
|
||||
httputil.WriteJSON(w, http.StatusAccepted, run)
|
||||
}
|
||||
|
||||
// listRuns returns a page of runs, newest first.
|
||||
func (h *LandscapeHandler) listRuns(w http.ResponseWriter, r *http.Request) {
|
||||
page, ok := parsePositiveIntQuery(w, r, "page", 1)
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
pageSize, ok := parsePositiveIntQuery(w, r, "page_size", service.DefaultPageSize)
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
|
||||
result, err := h.svc.ListRuns(r.Context(), page, pageSize)
|
||||
if err != nil {
|
||||
respondDomainError(w, err)
|
||||
return
|
||||
}
|
||||
httputil.WriteJSON(w, http.StatusOK, newPaginated(result.Items, result.Page, result.PageSize, result.Total))
|
||||
}
|
||||
|
||||
// getRun returns one run with its per-node execution state.
|
||||
func (h *LandscapeHandler) getRun(w http.ResponseWriter, r *http.Request) {
|
||||
id, ok := parseUUIDParam(w, r, "id")
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
run, err := h.svc.GetRun(r.Context(), id)
|
||||
if err != nil {
|
||||
respondDomainError(w, err)
|
||||
return
|
||||
}
|
||||
httputil.WriteJSON(w, http.StatusOK, run)
|
||||
}
|
||||
|
||||
// results lists every output group with its freshest ready dataset and the URLs
|
||||
// to render it.
|
||||
func (h *LandscapeHandler) results(w http.ResponseWriter, r *http.Request) {
|
||||
results, err := h.svc.Results(r.Context())
|
||||
if err != nil {
|
||||
respondDomainError(w, err)
|
||||
return
|
||||
}
|
||||
httputil.WriteJSON(w, http.StatusOK, results)
|
||||
}
|
||||
191
internal/transport/http/landscape_handler_test.go
Normal file
191
internal/transport/http/landscape_handler_test.go
Normal file
@@ -0,0 +1,191 @@
|
||||
package http
|
||||
|
||||
import (
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"sort"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"github.com/go-chi/chi/v5"
|
||||
"github.com/go-playground/validator/v10"
|
||||
)
|
||||
|
||||
// newLandscapeTestHandler builds a handler with a nil service: every case below
|
||||
// is rejected while parsing or validating the request, before the service is
|
||||
// reached.
|
||||
func newLandscapeTestHandler() *LandscapeHandler {
|
||||
return NewLandscapeHandler(nil, validator.New(validator.WithRequiredStructEnabled()))
|
||||
}
|
||||
|
||||
func postRun(t *testing.T, body string) *httptest.ResponseRecorder {
|
||||
t.Helper()
|
||||
req := httptest.NewRequest(http.MethodPost, "/landscape/runs", strings.NewReader(body))
|
||||
req.Header.Set("Content-Type", "application/json")
|
||||
rec := httptest.NewRecorder()
|
||||
newLandscapeTestHandler().createRun(rec, req)
|
||||
return rec
|
||||
}
|
||||
|
||||
func TestCreateRunInvalidJSON(t *testing.T) {
|
||||
rec := postRun(t, `{"from_stage":`)
|
||||
if rec.Code != http.StatusBadRequest {
|
||||
t.Fatalf("status = %d, want %d (body %s)", rec.Code, http.StatusBadRequest, rec.Body.String())
|
||||
}
|
||||
}
|
||||
|
||||
func TestCreateRunUnknownField(t *testing.T) {
|
||||
// DecodeJSON rejects unknown fields, so a typo is a 400 rather than a
|
||||
// silently ignored option.
|
||||
rec := postRun(t, `{"from_stages":1}`)
|
||||
if rec.Code != http.StatusBadRequest {
|
||||
t.Fatalf("status = %d, want %d (body %s)", rec.Code, http.StatusBadRequest, rec.Body.String())
|
||||
}
|
||||
}
|
||||
|
||||
func TestCreateRunFromStageOutOfRange(t *testing.T) {
|
||||
rec := postRun(t, `{"from_stage":7}`)
|
||||
if rec.Code != http.StatusUnprocessableEntity {
|
||||
t.Fatalf("status = %d, want %d (body %s)", rec.Code, http.StatusUnprocessableEntity, rec.Body.String())
|
||||
}
|
||||
if !strings.Contains(rec.Body.String(), "FromStage") {
|
||||
t.Fatalf("body %s does not name the offending field", rec.Body.String())
|
||||
}
|
||||
}
|
||||
|
||||
func TestCreateRunOverrideNotUUID(t *testing.T) {
|
||||
rec := postRun(t, `{"overrides":{"lp-z-lcr":"not-a-uuid"}}`)
|
||||
if rec.Code != http.StatusUnprocessableEntity {
|
||||
t.Fatalf("status = %d, want %d (body %s)", rec.Code, http.StatusUnprocessableEntity, rec.Body.String())
|
||||
}
|
||||
}
|
||||
|
||||
func TestUploadSlotRequiresMultipart(t *testing.T) {
|
||||
req := httptest.NewRequest(http.MethodPost, "/landscape/slots/lp-z-lcr", strings.NewReader(`{}`))
|
||||
req.Header.Set("Content-Type", "application/json")
|
||||
rec := httptest.NewRecorder()
|
||||
newLandscapeTestHandler().uploadSlot(rec, req)
|
||||
|
||||
if rec.Code != http.StatusBadRequest {
|
||||
t.Fatalf("status = %d, want %d (body %s)", rec.Code, http.StatusBadRequest, rec.Body.String())
|
||||
}
|
||||
}
|
||||
|
||||
func TestListRunsInvalidPage(t *testing.T) {
|
||||
req := httptest.NewRequest(http.MethodGet, "/landscape/runs?page=0", nil)
|
||||
rec := httptest.NewRecorder()
|
||||
newLandscapeTestHandler().listRuns(rec, req)
|
||||
|
||||
if rec.Code != http.StatusBadRequest {
|
||||
t.Fatalf("status = %d, want %d (body %s)", rec.Code, http.StatusBadRequest, rec.Body.String())
|
||||
}
|
||||
}
|
||||
|
||||
func TestGetRunInvalidID(t *testing.T) {
|
||||
req := httptest.NewRequest(http.MethodGet, "/landscape/runs/nope", nil)
|
||||
rec := httptest.NewRecorder()
|
||||
newLandscapeTestHandler().getRun(rec, req)
|
||||
|
||||
if rec.Code != http.StatusBadRequest {
|
||||
t.Fatalf("status = %d, want %d (body %s)", rec.Code, http.StatusBadRequest, rec.Body.String())
|
||||
}
|
||||
}
|
||||
|
||||
// TestDecodeRunRequestEmptyBody pins the "all defaults" call: POST
|
||||
// /landscape/runs with no body at all is a valid request, not a 400. Tested at
|
||||
// the helper level because the 202 path needs a live service.
|
||||
func TestDecodeRunRequestEmptyBody(t *testing.T) {
|
||||
for _, body := range []string{"", "{}"} {
|
||||
req := httptest.NewRequest(http.MethodPost, "/landscape/runs", strings.NewReader(body))
|
||||
rec := httptest.NewRecorder()
|
||||
|
||||
got, ok := decodeRunRequest(rec, req)
|
||||
if !ok {
|
||||
t.Fatalf("body %q: decodeRunRequest reported failure with status %d", body, rec.Code)
|
||||
}
|
||||
if got.FromStage != nil || len(got.Overrides) != 0 || got.Params != nil {
|
||||
t.Fatalf("body %q: got %+v, want the zero request", body, got)
|
||||
}
|
||||
if rec.Code != http.StatusOK || rec.Body.Len() != 0 {
|
||||
t.Fatalf("body %q: helper wrote a response (status %d, body %s)", body, rec.Code, rec.Body.String())
|
||||
}
|
||||
// The zero request must also survive validation, or an empty body
|
||||
// would still be rejected one step later.
|
||||
if err := validator.New(validator.WithRequiredStructEnabled()).Struct(got); err != nil {
|
||||
t.Fatalf("body %q: zero request fails validation: %v", body, err)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestDecodeRunRequestMalformedBody(t *testing.T) {
|
||||
req := httptest.NewRequest(http.MethodPost, "/landscape/runs", strings.NewReader(`{`))
|
||||
rec := httptest.NewRecorder()
|
||||
|
||||
if _, ok := decodeRunRequest(rec, req); ok {
|
||||
t.Fatal("decodeRunRequest accepted a truncated body")
|
||||
}
|
||||
if rec.Code != http.StatusBadRequest {
|
||||
t.Fatalf("status = %d, want %d", rec.Code, http.StatusBadRequest)
|
||||
}
|
||||
}
|
||||
|
||||
// TestRegisterMountsRoutes walks the route table of a router with the handler
|
||||
// registered, so every one of the six routes is checked — including GET /slots
|
||||
// and GET /results, which cannot be exercised without a live service.
|
||||
func TestRegisterMountsRoutes(t *testing.T) {
|
||||
r := chi.NewRouter()
|
||||
r.Route("/landscape", newLandscapeTestHandler().Register)
|
||||
|
||||
got := map[string]bool{}
|
||||
err := chi.Walk(r, func(method, route string, _ http.Handler, _ ...func(http.Handler) http.Handler) error {
|
||||
got[method+" "+strings.TrimSuffix(route, "/")] = true
|
||||
return nil
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("walk routes: %v", err)
|
||||
}
|
||||
|
||||
want := []string{
|
||||
"GET /landscape/slots",
|
||||
"POST /landscape/slots/{code}",
|
||||
"GET /landscape/runs",
|
||||
"POST /landscape/runs",
|
||||
"GET /landscape/runs/{id}",
|
||||
"GET /landscape/results",
|
||||
}
|
||||
for _, w := range want {
|
||||
if !got[w] {
|
||||
t.Errorf("route %q is not mounted; mounted routes: %v", w, keys(got))
|
||||
}
|
||||
}
|
||||
if len(got) != len(want) {
|
||||
t.Errorf("mounted %d routes, want %d: %v", len(got), len(want), keys(got))
|
||||
}
|
||||
}
|
||||
|
||||
// TestRoutedRequestReachesHandler sends a real request through the router so the
|
||||
// URL parameter wiring (chi.URLParam) is exercised, not just the pattern.
|
||||
func TestRoutedRequestReachesHandler(t *testing.T) {
|
||||
r := chi.NewRouter()
|
||||
r.Route("/landscape", newLandscapeTestHandler().Register)
|
||||
|
||||
req := httptest.NewRequest(http.MethodGet, "/landscape/runs/nope", nil)
|
||||
rec := httptest.NewRecorder()
|
||||
r.ServeHTTP(rec, req)
|
||||
|
||||
if rec.Code != http.StatusBadRequest {
|
||||
t.Fatalf("status = %d, want %d (body %s)", rec.Code, http.StatusBadRequest, rec.Body.String())
|
||||
}
|
||||
if !strings.Contains(rec.Body.String(), "invalid id") {
|
||||
t.Fatalf("body %s does not report the bad id", rec.Body.String())
|
||||
}
|
||||
}
|
||||
|
||||
func keys(m map[string]bool) []string {
|
||||
out := make([]string, 0, len(m))
|
||||
for k := range m {
|
||||
out = append(out, k)
|
||||
}
|
||||
sort.Strings(out)
|
||||
return out
|
||||
}
|
||||
@@ -31,3 +31,29 @@ func requestLogger(log *slog.Logger) func(http.Handler) http.Handler {
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// limitRequestBody caps every request body at maxBytes (http.MaxBytesReader),
|
||||
// so a client cannot stream an unbounded upload to memory or to the temporary
|
||||
// files multipart parsing spills to. A non-positive maxBytes disables the cap.
|
||||
// JSON endpoints apply their own, smaller cap in httputil.DecodeJSON.
|
||||
func limitRequestBody(maxBytes int64) func(http.Handler) http.Handler {
|
||||
return func(next http.Handler) http.Handler {
|
||||
if maxBytes <= 0 {
|
||||
return next
|
||||
}
|
||||
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
r.Body = http.MaxBytesReader(w, r.Body, maxBytes)
|
||||
next.ServeHTTP(w, r)
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// securityHeaders sets response headers that are safe for every route.
|
||||
// nosniff stops browsers from MIME-sniffing user-supplied content (downloads,
|
||||
// GeoJSON) into an executable type.
|
||||
func securityHeaders(next http.Handler) http.Handler {
|
||||
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
w.Header().Set("X-Content-Type-Options", "nosniff")
|
||||
next.ServeHTTP(w, r)
|
||||
})
|
||||
}
|
||||
|
||||
131
internal/transport/http/middleware_test.go
Normal file
131
internal/transport/http/middleware_test.go
Normal file
@@ -0,0 +1,131 @@
|
||||
package http
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"errors"
|
||||
"io"
|
||||
"log/slog"
|
||||
"mime/multipart"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"github.com/go-playground/validator/v10"
|
||||
)
|
||||
|
||||
// multipartBody builds a multipart/form-data body with one file part of
|
||||
// size bytes.
|
||||
func multipartBody(t *testing.T, size int) (*bytes.Buffer, string) {
|
||||
t.Helper()
|
||||
var buf bytes.Buffer
|
||||
mw := multipart.NewWriter(&buf)
|
||||
part, err := mw.CreateFormFile("file", "big.tif")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err := part.Write(bytes.Repeat([]byte{'x'}, size)); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := mw.Close(); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
return &buf, mw.FormDataContentType()
|
||||
}
|
||||
|
||||
func TestLimitRequestBodyRejectsOversizedUpload(t *testing.T) {
|
||||
body, ct := multipartBody(t, 4096)
|
||||
req := httptest.NewRequest(http.MethodPost, "/landscape/slots/lp-z-lcr", body)
|
||||
req.Header.Set("Content-Type", ct)
|
||||
rec := httptest.NewRecorder()
|
||||
|
||||
h := limitRequestBody(1024)(http.HandlerFunc(newLandscapeTestHandler().uploadSlot))
|
||||
h.ServeHTTP(rec, req)
|
||||
|
||||
if rec.Code != http.StatusRequestEntityTooLarge {
|
||||
t.Fatalf("status = %d, want %d (body %s)", rec.Code, http.StatusRequestEntityTooLarge, rec.Body.String())
|
||||
}
|
||||
if !strings.Contains(rec.Body.String(), "1024 byte limit") {
|
||||
t.Fatalf("body %s does not state the limit", rec.Body.String())
|
||||
}
|
||||
}
|
||||
|
||||
func TestLimitRequestBodyPassesSmallBody(t *testing.T) {
|
||||
var got []byte
|
||||
h := limitRequestBody(1024)(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
var err error
|
||||
got, err = io.ReadAll(r.Body)
|
||||
if err != nil {
|
||||
t.Errorf("read: %v", err)
|
||||
}
|
||||
}))
|
||||
h.ServeHTTP(httptest.NewRecorder(), httptest.NewRequest(http.MethodPost, "/", strings.NewReader("hello")))
|
||||
if string(got) != "hello" {
|
||||
t.Fatalf("body = %q, want hello", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestLimitRequestBodyDisabled(t *testing.T) {
|
||||
h := limitRequestBody(0)(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
if _, err := io.ReadAll(r.Body); err != nil {
|
||||
t.Errorf("read with cap disabled: %v", err)
|
||||
}
|
||||
}))
|
||||
h.ServeHTTP(httptest.NewRecorder(), httptest.NewRequest(http.MethodPost, "/", bytes.NewReader(make([]byte, 1<<20))))
|
||||
}
|
||||
|
||||
func newTestRouter() http.Handler {
|
||||
v := validator.New(validator.WithRequiredStructEnabled())
|
||||
return NewRouter(RouterDeps{
|
||||
Logger: slog.New(slog.NewTextHandler(io.Discard, nil)),
|
||||
Health: NewHealthHandler(nil, nil),
|
||||
Categories: NewCategoryHandler(nil, v),
|
||||
Datasets: NewDatasetHandler(nil, v),
|
||||
DatasetGroups: NewDatasetGroupHandler(nil, v),
|
||||
Landscape: NewLandscapeHandler(nil, v),
|
||||
MaxBodyBytes: 1024,
|
||||
})
|
||||
}
|
||||
|
||||
func TestRouterSetsNosniff(t *testing.T) {
|
||||
rec := httptest.NewRecorder()
|
||||
newTestRouter().ServeHTTP(rec, httptest.NewRequest(http.MethodGet, "/healthz", nil))
|
||||
if got := rec.Header().Get("X-Content-Type-Options"); got != "nosniff" {
|
||||
t.Fatalf("X-Content-Type-Options = %q, want nosniff", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRouterCapsUploadBody(t *testing.T) {
|
||||
body, ct := multipartBody(t, 4096)
|
||||
req := httptest.NewRequest(http.MethodPost, "/landscape/slots/lp-z-lcr", body)
|
||||
req.Header.Set("Content-Type", ct)
|
||||
rec := httptest.NewRecorder()
|
||||
newTestRouter().ServeHTTP(rec, req)
|
||||
if rec.Code != http.StatusRequestEntityTooLarge {
|
||||
t.Fatalf("status = %d, want %d (body %s)", rec.Code, http.StatusRequestEntityTooLarge, rec.Body.String())
|
||||
}
|
||||
}
|
||||
|
||||
func TestReadyDoesNotLeakCheckErrors(t *testing.T) {
|
||||
h := NewHealthHandler(map[string]ReadinessCheck{
|
||||
"postgres": func(context.Context) error {
|
||||
return errors.New("dial tcp 10.1.2.3:5432: connect: connection refused (user=gis)")
|
||||
},
|
||||
"s3": func(context.Context) error { return nil },
|
||||
}, slog.New(slog.NewTextHandler(io.Discard, nil)))
|
||||
|
||||
rec := httptest.NewRecorder()
|
||||
h.Ready(rec, httptest.NewRequest(http.MethodGet, "/readyz", nil))
|
||||
|
||||
if rec.Code != http.StatusServiceUnavailable {
|
||||
t.Fatalf("status = %d, want %d", rec.Code, http.StatusServiceUnavailable)
|
||||
}
|
||||
body := rec.Body.String()
|
||||
if strings.Contains(body, "10.1.2.3") || strings.Contains(body, "dial tcp") {
|
||||
t.Fatalf("readiness body leaks internal error details: %s", body)
|
||||
}
|
||||
if !strings.Contains(body, `"postgres":"unavailable"`) || !strings.Contains(body, `"s3":"ok"`) {
|
||||
t.Fatalf("unexpected readiness body: %s", body)
|
||||
}
|
||||
}
|
||||
@@ -10,21 +10,32 @@ import (
|
||||
|
||||
// RouterDeps holds the handlers and dependencies the router mounts.
|
||||
type RouterDeps struct {
|
||||
Logger *slog.Logger
|
||||
Health *HealthHandler
|
||||
Categories *CategoryHandler
|
||||
Datasets *DatasetHandler
|
||||
Logger *slog.Logger
|
||||
Health *HealthHandler
|
||||
Categories *CategoryHandler
|
||||
Datasets *DatasetHandler
|
||||
// DatasetGroups serves the /dataset-groups routes.
|
||||
DatasetGroups *DatasetGroupHandler
|
||||
// Landscape serves the /landscape routes.
|
||||
Landscape *LandscapeHandler
|
||||
OpenAPISpec []byte
|
||||
// MaxBodyBytes caps every request body (uploads included); <= 0 disables
|
||||
// the cap.
|
||||
MaxBodyBytes int64
|
||||
}
|
||||
|
||||
// NewRouter builds the application's HTTP handler with middleware and routes.
|
||||
func NewRouter(deps RouterDeps) http.Handler {
|
||||
r := chi.NewRouter()
|
||||
|
||||
// middleware.RealIP is deliberately not used: it rewrites r.RemoteAddr
|
||||
// from client-controlled headers (X-Forwarded-For, X-Real-IP,
|
||||
// True-Client-IP) and is deprecated upstream as an IP-spoofing vector.
|
||||
r.Use(middleware.RequestID)
|
||||
r.Use(middleware.RealIP)
|
||||
r.Use(requestLogger(deps.Logger))
|
||||
r.Use(middleware.Recoverer)
|
||||
r.Use(securityHeaders)
|
||||
r.Use(limitRequestBody(deps.MaxBodyBytes))
|
||||
|
||||
// Health/readiness endpoints.
|
||||
r.Get("/healthz", deps.Health.Live)
|
||||
@@ -37,6 +48,8 @@ func NewRouter(deps RouterDeps) http.Handler {
|
||||
// Domain routes. Add future feature routers here.
|
||||
r.Route("/categories", deps.Categories.Register)
|
||||
r.Route("/datasets", deps.Datasets.Register)
|
||||
r.Route("/dataset-groups", deps.DatasetGroups.Register)
|
||||
r.Route("/landscape", deps.Landscape.Register)
|
||||
|
||||
return r
|
||||
}
|
||||
|
||||
12
migrations/00008_add_sld_to_datasets.sql
Normal file
12
migrations/00008_add_sld_to_datasets.sql
Normal file
@@ -0,0 +1,12 @@
|
||||
-- +goose Up
|
||||
-- Optional SLD (Styled Layer Descriptor) style file for a dataset. Applies to
|
||||
-- every file_type; a dataset may have at most one (uploading again replaces
|
||||
-- it).
|
||||
ALTER TABLE datasets ADD COLUMN sld_storage_key TEXT;
|
||||
ALTER TABLE datasets ADD COLUMN sld_filename VARCHAR(255);
|
||||
ALTER TABLE datasets ADD COLUMN sld_content_type VARCHAR(255);
|
||||
|
||||
-- +goose Down
|
||||
ALTER TABLE datasets DROP COLUMN sld_storage_key;
|
||||
ALTER TABLE datasets DROP COLUMN sld_filename;
|
||||
ALTER TABLE datasets DROP COLUMN sld_content_type;
|
||||
15
migrations/00009_add_period_to_dataset_observations.sql
Normal file
15
migrations/00009_add_period_to_dataset_observations.sql
Normal file
@@ -0,0 +1,15 @@
|
||||
-- +goose Up
|
||||
-- Optional period end date, so a year_column mapping can represent a date
|
||||
-- range (date..date_to) instead of a single point in time. NULL preserves the
|
||||
-- pre-existing point-in-time behavior; the unique index treats a NULL date_to
|
||||
-- as equal to date itself so point observations keep deduplicating exactly as
|
||||
-- before.
|
||||
ALTER TABLE dataset_observations ADD COLUMN date_to DATE;
|
||||
|
||||
DROP INDEX uq_dataset_observations;
|
||||
CREATE UNIQUE INDEX uq_dataset_observations ON dataset_observations (dataset_id, kato_code, date, COALESCE(date_to, date));
|
||||
|
||||
-- +goose Down
|
||||
DROP INDEX uq_dataset_observations;
|
||||
ALTER TABLE dataset_observations DROP COLUMN date_to;
|
||||
CREATE UNIQUE INDEX uq_dataset_observations ON dataset_observations (dataset_id, kato_code, date);
|
||||
67
migrations/00010_dataset_groups_and_dates.sql
Normal file
67
migrations/00010_dataset_groups_and_dates.sql
Normal file
@@ -0,0 +1,67 @@
|
||||
-- +goose Up
|
||||
-- Dataset groups: "одноимённые" datasets versioned by period. A group may carry a
|
||||
-- shared SLD used as a fallback for its datasets.
|
||||
CREATE TABLE dataset_groups (
|
||||
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
|
||||
code VARCHAR(255) NOT NULL UNIQUE,
|
||||
name VARCHAR(255) NOT NULL,
|
||||
description TEXT NOT NULL DEFAULT '',
|
||||
sld_storage_key TEXT,
|
||||
sld_filename VARCHAR(255),
|
||||
sld_content_type VARCHAR(255),
|
||||
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
|
||||
updated_at TIMESTAMPTZ NOT NULL DEFAULT now()
|
||||
);
|
||||
|
||||
ALTER TABLE datasets ADD COLUMN dg_id UUID REFERENCES dataset_groups (id) ON DELETE RESTRICT;
|
||||
ALTER TABLE datasets ADD COLUMN start_date DATE;
|
||||
ALTER TABLE datasets ADD COLUMN end_date DATE;
|
||||
ALTER TABLE datasets ADD CONSTRAINT chk_datasets_period
|
||||
CHECK (start_date IS NULL OR end_date IS NULL OR end_date >= start_date);
|
||||
CREATE INDEX idx_datasets_dg_id_end_date ON datasets (dg_id, end_date DESC NULLS LAST, created_at DESC);
|
||||
CREATE INDEX idx_datasets_start_date ON datasets (start_date);
|
||||
CREATE INDEX idx_datasets_end_date ON datasets (end_date);
|
||||
|
||||
-- Period columns are renamed everywhere: observations, and the year_columns
|
||||
-- mapping JSON ({column, date, date_to} -> {column, start_date, end_date}).
|
||||
ALTER TABLE dataset_observations RENAME COLUMN date TO start_date;
|
||||
ALTER TABLE dataset_observations RENAME COLUMN date_to TO end_date;
|
||||
DROP INDEX uq_dataset_observations;
|
||||
DROP INDEX idx_dataset_observations_kato_date;
|
||||
CREATE UNIQUE INDEX uq_dataset_observations ON dataset_observations (dataset_id, kato_code, start_date, COALESCE(end_date, start_date));
|
||||
CREATE INDEX idx_dataset_observations_kato_date ON dataset_observations (kato_code, start_date);
|
||||
|
||||
UPDATE datasets
|
||||
SET year_columns = (
|
||||
SELECT jsonb_agg(
|
||||
(elem - 'date' - 'date_to')
|
||||
|| jsonb_build_object('start_date', elem->'date')
|
||||
|| CASE WHEN elem ? 'date_to' AND elem->'date_to' IS NOT NULL
|
||||
THEN jsonb_build_object('end_date', elem->'date_to') ELSE '{}'::jsonb END)
|
||||
FROM jsonb_array_elements(year_columns) AS elem)
|
||||
WHERE year_columns IS NOT NULL AND jsonb_typeof(year_columns) = 'array' AND jsonb_array_length(year_columns) > 0;
|
||||
|
||||
-- +goose Down
|
||||
UPDATE datasets
|
||||
SET year_columns = (
|
||||
SELECT jsonb_agg(
|
||||
(elem - 'start_date' - 'end_date')
|
||||
|| jsonb_build_object('date', elem->'start_date')
|
||||
|| CASE WHEN elem ? 'end_date' AND elem->'end_date' IS NOT NULL
|
||||
THEN jsonb_build_object('date_to', elem->'end_date') ELSE '{}'::jsonb END)
|
||||
FROM jsonb_array_elements(year_columns) AS elem)
|
||||
WHERE year_columns IS NOT NULL AND jsonb_typeof(year_columns) = 'array' AND jsonb_array_length(year_columns) > 0;
|
||||
DROP INDEX uq_dataset_observations;
|
||||
DROP INDEX idx_dataset_observations_kato_date;
|
||||
ALTER TABLE dataset_observations RENAME COLUMN start_date TO date;
|
||||
ALTER TABLE dataset_observations RENAME COLUMN end_date TO date_to;
|
||||
CREATE UNIQUE INDEX uq_dataset_observations ON dataset_observations (dataset_id, kato_code, date, COALESCE(date_to, date));
|
||||
CREATE INDEX idx_dataset_observations_kato_date ON dataset_observations (kato_code, date);
|
||||
DROP INDEX idx_datasets_dg_id_end_date;
|
||||
DROP INDEX idx_datasets_start_date;
|
||||
DROP INDEX idx_datasets_end_date;
|
||||
ALTER TABLE datasets DROP CONSTRAINT chk_datasets_period;
|
||||
ALTER TABLE datasets DROP COLUMN dg_id;
|
||||
ALTER TABLE datasets DROP COLUMN start_date;
|
||||
ALTER TABLE datasets DROP COLUMN end_date;
|
||||
DROP TABLE dataset_groups;
|
||||
51
migrations/00011_landscape.sql
Normal file
51
migrations/00011_landscape.sql
Normal file
@@ -0,0 +1,51 @@
|
||||
-- +goose Up
|
||||
-- Landscape Program: seed the category and dataset groups (input slots and
|
||||
-- node outputs; codes must match internal/landscape/graph.go), and store runs.
|
||||
INSERT INTO categories (code, name, description)
|
||||
VALUES ('landscape', 'Ландшафтная программа', 'Входные и результирующие растры Ландшафтной программы')
|
||||
ON CONFLICT (code) DO NOTHING;
|
||||
|
||||
INSERT INTO dataset_groups (code, name) VALUES
|
||||
('lp-z-biodiversity', 'Z Biodiversity'), ('lp-z-lcr', 'Z LCR'), ('lp-z-soil-protection', 'Z Soil Protection'), ('lp-z-wr', 'Z Water Resources'),
|
||||
('lp-u-surf-water', 'U Surface Water'), ('lp-u-soil', 'U Soil'), ('lp-u-crop', 'U Crop'), ('lp-u-veg', 'U Vegetation'),
|
||||
('lp-p-irrigation', 'P Irrigation'), ('lp-p-snow', 'P Snow'), ('lp-p-crop', 'P Crop'), ('lp-p-pasture', 'P Pasture'), ('lp-p-tourism', 'P Tourism'),
|
||||
('lp-z-integrated', 'Z Integrated'), ('lp-u-integrated', 'U Integrated'), ('lp-z-max', 'Z Max'), ('lp-u-max', 'U Max'),
|
||||
('lp-e', 'E Ecological Constraint'),
|
||||
('lp-c-irrigation', 'C Irrigation'), ('lp-c-snow', 'C Snow'), ('lp-c-crop', 'C Crop'), ('lp-c-pasture', 'C Pasture'), ('lp-c-tourism', 'C Tourism'),
|
||||
('lp-p-max', 'P Max'),
|
||||
('lp-v1-conservative', 'V1 Conservative'), ('lp-v2-balanced', 'V2 Balanced'), ('lp-v3-development', 'V3 Development'), ('lp-c-max', 'C Max')
|
||||
ON CONFLICT (code) DO NOTHING;
|
||||
|
||||
CREATE TABLE lp_runs (
|
||||
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
|
||||
status VARCHAR(32) NOT NULL,
|
||||
from_stage SMALLINT,
|
||||
params JSONB NOT NULL,
|
||||
overrides JSONB NOT NULL DEFAULT '{}'::jsonb,
|
||||
error TEXT,
|
||||
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
|
||||
started_at TIMESTAMPTZ,
|
||||
finished_at TIMESTAMPTZ
|
||||
);
|
||||
CREATE INDEX idx_lp_runs_created_at ON lp_runs (created_at DESC);
|
||||
|
||||
CREATE TABLE lp_run_nodes (
|
||||
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
|
||||
run_id UUID NOT NULL REFERENCES lp_runs (id) ON DELETE CASCADE,
|
||||
node_code VARCHAR(8) NOT NULL,
|
||||
stage SMALLINT NOT NULL,
|
||||
status VARCHAR(32) NOT NULL,
|
||||
fingerprint TEXT NOT NULL DEFAULT '',
|
||||
input_datasets JSONB NOT NULL DEFAULT '[]'::jsonb,
|
||||
output_datasets JSONB,
|
||||
error TEXT,
|
||||
started_at TIMESTAMPTZ,
|
||||
finished_at TIMESTAMPTZ,
|
||||
UNIQUE (run_id, node_code)
|
||||
);
|
||||
CREATE INDEX idx_lp_run_nodes_lookup ON lp_run_nodes (node_code, fingerprint, status, finished_at DESC);
|
||||
|
||||
-- +goose Down
|
||||
DROP TABLE lp_run_nodes;
|
||||
DROP TABLE lp_runs;
|
||||
-- Seeded category/groups are left in place (datasets may reference them).
|
||||
2
test/acceptance/.gitignore
vendored
Normal file
2
test/acceptance/.gitignore
vendored
Normal file
@@ -0,0 +1,2 @@
|
||||
results/
|
||||
__pycache__/
|
||||
320
test/acceptance/README.md
Normal file
320
test/acceptance/README.md
Normal file
@@ -0,0 +1,320 @@
|
||||
# Пакет приёмочных испытаний (Этап 8)
|
||||
|
||||
Пакет позволяет Заказчику самостоятельно воспроизвести то, что заявлено в
|
||||
документах Этапа 8, на изолированном испытательном стенде, развёрнутом из
|
||||
исходного кода этой ветки:
|
||||
|
||||
| № | Что проверяется | Скрипт | Документ |
|
||||
|---|---|---|---|
|
||||
| 1 | 89 функциональных проверок испытательного стенда (F01–F38, N01–N38, L02–L05, L07–L15) | `functional.sh` | Приложение 7 «Протокол функционального тестирования», раздел 5 |
|
||||
| 2 | Попиксельная сверка расчёта Ландшафтной программы (ЛП) с независимой эталонной реализацией | `lp_gen.py`, `lp_run.sh`, `lp_verify.py` | ЛП техописание (правила пикселя, граф узлов); Приложение 7, L07/L08/L12 |
|
||||
| 3 | Нагрузочные испытания: 1000 одновременных клиентов, сценарии S1–S6 | `load.sh` (+ `seed.sh`) | Приложение 3 «Протокол нагрузочного тестирования» |
|
||||
| 4 | Повторный анализ безопасности: 0 уязвимостей High/Critical | `scan.sh` | Приложение 4 «Отчет об устранении уязвимостей» |
|
||||
|
||||
Все скрипты — bash (macOS и Linux, `set -euo pipefail`, совместимы с bash 3.2),
|
||||
curl, jq и Python 3. Результаты прогонов сохраняются в `test/acceptance/results/`
|
||||
(каталог не версионируется).
|
||||
|
||||
> **Внимание.** `functional.sh`, `lp_run.sh`, `seed.sh` и сценарий S5 из
|
||||
> `load.sh` **создают данные** (в том числе заполняют слоты ЛП). Их следует
|
||||
> запускать только на изолированном стенде `gis-acceptance`. На рабочем стенде
|
||||
> https://dssgis.dwh.kz допускается только `lp_verify.py` — он выполняет
|
||||
> исключительно GET-запросы (`lp_run.sh` и `seed.sh` отказываются работать с
|
||||
> этим адресом).
|
||||
|
||||
## Состав пакета
|
||||
|
||||
| Файл | Назначение |
|
||||
|---|---|
|
||||
| `compose.acceptance.yml` | Надстройка над `deployments/docker-compose.yml` для изолированного стенда (проект `gis-acceptance`) |
|
||||
| `stand.sh` | Развёртывание (`up`: сборка, миграции, справочник районов, ожидание `/readyz`) и удаление (`down`) стенда |
|
||||
| `functional.sh` | 89 функциональных проверок, построчный вывод `ID \| описание \| PASS/FAIL \| detail` |
|
||||
| `lp_gen.py` | Генерация 13 синтетических входных растров ЛП и одного невыровненного растра |
|
||||
| `lp_run.sh` | Загрузка входов в 13 слотов → прогон `{"from_stage":0}` → `lp_verify.py` |
|
||||
| `lp_verify.py` | Независимая (numpy) реализация всех узлов ЛП и попиксельная сверка 15 выходов; только GET |
|
||||
| `load.sh` | Сценарии нагрузки S1–S6 (hey 0.1.5), сводная таблица, CSV/JSON |
|
||||
| `seed.sh` | Наполнение стенда данными для нагрузочных испытаний (`seed.sh clean` — удаление) |
|
||||
| `scan.sh` | govulncheck (исходный код и бинарный файл), gosec, trivy fs, trivy image, golangci-lint |
|
||||
| `data/` | Тестовые векторные данные (участки, население и поголовье КРС по 14 районам СКО с реальными кодами КАТО), SLD-стили |
|
||||
| `lib/` | Вспомогательные скрипты: генерация NDVI, анализ вывода hey, Dockerfile генератора нагрузки, пересоздание `data/` |
|
||||
| `requirements.txt` | Python-зависимости (`numpy`, `rasterio`) |
|
||||
|
||||
## Требования
|
||||
|
||||
* Docker Engine с Docker Compose v2.24+ (используется тег `!override` для портов).
|
||||
* `curl`, `jq`, Python 3.10+ с `numpy` и `rasterio`:
|
||||
```sh
|
||||
python3 -m venv .venv && .venv/bin/pip install -r test/acceptance/requirements.txt
|
||||
export PYTHON=$PWD/.venv/bin/python # скрипты используют $PYTHON (по умолчанию python3)
|
||||
```
|
||||
* Для `scan.sh`: Go (версия toolchain берётся из `go.mod`), `govulncheck`,
|
||||
`gosec`, `trivy`; `golangci-lint` — по желанию.
|
||||
|
||||
## 0. Испытательный стенд
|
||||
|
||||
```sh
|
||||
git clone https://dssgit.dwh.kz/infureal/gis.git && cd gis && git checkout stage8/security-fixes
|
||||
test/acceptance/stand.sh up # сборка образа, запуск, migrate up, import-districts, ожидание /readyz
|
||||
```
|
||||
|
||||
`stand.sh up` выполняет те же шаги, что и вручную:
|
||||
|
||||
```sh
|
||||
C="docker compose -p gis-acceptance -f deployments/docker-compose.yml -f test/acceptance/compose.acceptance.yml"
|
||||
$C up -d --build --wait postgres minio rabbitmq
|
||||
$C run --rm --no-deps app migrate up # 11 миграций 00001…00011
|
||||
$C run --rm --no-deps app import-districts /stubs/districts.geojson # 14 районов СКО
|
||||
$C up -d app worker
|
||||
curl http://localhost:18080/readyz # {"components":{"postgres":"ok","rabbitmq":"ok","s3":"ok"},"ready":true}
|
||||
```
|
||||
|
||||
Порты стенда: API — 18080, PostgreSQL — 15432, MinIO — 19000 (API) / 19001
|
||||
(консоль), RabbitMQ — 15672 (только веб-интерфейс управления). Сеть контейнеров —
|
||||
`gis-acceptance_default`. Учётные данные — значения по умолчанию из
|
||||
`deployments/docker-compose.yml` (стенд изолированный).
|
||||
|
||||
Переменные окружения стенда:
|
||||
|
||||
| Переменная | По умолчанию | Назначение |
|
||||
|---|---|---|
|
||||
| `POSTGIS_PLATFORM` | `linux/amd64` | Образ `postgis/postgis:17-3.5` публикуется только для amd64. На Apple Silicon (arm64) он запускается в эмуляции, на amd64-хостах — нативно, поэтому значение по умолчанию работает на обеих платформах |
|
||||
| `MINIO_IMAGE` | `minio/minio:latest` | Если образ `minio/minio` недоступен в реестре, подойдёт любой образ с сервером `minio` и клиентом `mc`, например `MINIO_IMAGE=cgr.dev/chainguard/minio:latest` |
|
||||
| `APP_IMAGE` | `gis-acceptance-app:local` | Тег локально собранного образа приложения |
|
||||
| `HTTP_MAX_UPLOAD_BYTES` | `2147483648` | Предельный размер тела запроса (2 ГиБ) |
|
||||
| `LP_MAX_WORKERS` / `LP_MAX_NODES` | `8` / `2` | Параллелизм расчёта ЛП |
|
||||
|
||||
Удаление стенда вместе с томами: `test/acceptance/stand.sh down`
|
||||
(= `docker compose -p gis-acceptance down -v`).
|
||||
|
||||
## 1. Функциональные проверки — `functional.sh`
|
||||
|
||||
```sh
|
||||
test/acceptance/functional.sh # стенд http://localhost:18080
|
||||
BASE_URL=http://host:18080 LOCAL_CHECKS=0 test/acceptance/functional.sh # удалённый испытательный стенд
|
||||
```
|
||||
|
||||
| Переменная | По умолчанию | Назначение |
|
||||
|---|---|---|
|
||||
| `BASE_URL` | `http://localhost:18080` | Адрес API |
|
||||
| `LOCAL_CHECKS` | `auto` | `1` — подтверждение в PostGIS и MinIO через контейнеры проекта (`docker exec … psql`, `mc` в контейнере MinIO): F35 (часть MinIO), F36, F37, F38, ресурсы worker в L15. `0` — эти проверки пропускаются (`SKIP`). `auto` — `1`, если запущены контейнеры проекта `gis-acceptance` |
|
||||
| `COMPOSE_PROJECT` | `gis-acceptance` | Проект Compose для `LOCAL_CHECKS` |
|
||||
| `KEEP_DATA` | `0` | `1` — не удалять созданные данные |
|
||||
| `WAIT_TIMEOUT` | `300` | Ожидание асинхронной обработки, с |
|
||||
|
||||
Скрипт создаёт собственные тестовые данные (все коды имеют уникальный префикс
|
||||
`acc<время>-`), никогда не изменяет и не удаляет существовавшие объекты и в
|
||||
конце удаляет всё созданное: наборы данных (в том числе входы и выходы ЛП своих
|
||||
прогонов), группы, категории. Записи прогонов ЛП остаются — API не
|
||||
предусматривает их удаления. Порядок и смысл проверок совпадают с таблицами
|
||||
раздела 5 Приложения 7.
|
||||
|
||||
Ожидаемый вывод (фрагмент):
|
||||
|
||||
```
|
||||
F01 | Создание корневой категории POST /categories | PASS | 201; acc1792084680-ecology «Экология», parent_id null
|
||||
...
|
||||
N38 | Загрузка 100 МБ мусора с сигнатурой TIFF | PASS | 201 (104 857 604 байт за 0.5 с) → status failed, parse_error от gdal_translate; /readyz 200, сервис доступен
|
||||
...
|
||||
L07 | Попиксельная сверка 15 выходов с эталонной реализацией | PASS | 15 выходов × 6 000 000 пикселей; «TOTAL mismatching pixels: 0 -> PASS» …
|
||||
...
|
||||
ИТОГО: 89 проверок — PASS 89, FAIL 0, SKIP 0
|
||||
RESULT: 89/89 PASS
|
||||
```
|
||||
|
||||
Код возврата ненулевой при любом `FAIL`. Журнал и вспомогательные выводы
|
||||
(`L07_lp_verify.txt`, `L08_points.txt`, `L12_lp_verify.txt`, `F36.txt`) —
|
||||
в `results/functional-<время>/`.
|
||||
|
||||
### Соответствие Приложению 7 и уточнения методики
|
||||
|
||||
| Проверки | Раздел Приложения 7 | Как проверяется в пакете |
|
||||
|---|---|---|
|
||||
| F01–F09 | 5.2 Категории | HTTP-код и поля ответа, контроль цикла, дубликат, удаление |
|
||||
| F10–F34, N01–N13 | 5.3 Импорт/экспорт, SLD, группы, фильтры | Загрузка `data/plots_nko.geojson` (vector), `data/population_kato.geojson` и `data/livestock_kato.gpkg` (vector_with_kato), растра NDVI 600×400 EPSG:4326 Float32 (генерируется `lib/helpers.py`) |
|
||||
| N14–N38, F35 | 5.4 Негативные проверки | 12 некорректных загрузок, 10 эндпоинтов с несуществующим id, 4 — с некорректным id, некорректные тела JSON, параметры ЛП, 100 МБ мусора |
|
||||
| F36–F38 | 5.5 PostGIS и MinIO | SQL `ST_GeometryType/ST_SRID/ST_IsValid`, соединение наблюдений с `districts`, `mc stat` объектов |
|
||||
| L02–L15 | 5.6 Ландшафтная программа | 13 растров `lp_gen.py`, прогоны, `lp_verify.py`, инкрементальный пересчёт, `from_stage`, параметры, невыровненная сетка |
|
||||
|
||||
Уточнения относительно протокола (смысл и ожидаемые результаты проверок не
|
||||
изменены):
|
||||
|
||||
* **Коды объектов** имеют префикс прогона (`acc…-ecology` вместо `ecology`),
|
||||
чтобы повторный запуск не конфликтовал с данными прошлых прогонов.
|
||||
* **F33, F34** — выборки ограничены собственной категорией (`category_id`), чтобы
|
||||
результат не зависел от других данных стенда; во втором фильтре F34
|
||||
(`file_type + status`) добавлен `category_id`.
|
||||
* **F12** — охват проверяется по координатам выгруженного GeoJSON (вместо
|
||||
`ogrinfo`); **F22, L05** — формат COG проверяется через rasterio/GDAL
|
||||
(`LAYOUT=COG`, сжатие, размер, EPSG, тип, NoData) вместо `gdalinfo`;
|
||||
**L08** — 9 точек проверяются `lp_verify.py --points` вместо `gdallocationinfo`.
|
||||
* **F38** — проверяется наличие всех 7 объектов, созданных скриптом (4 исходных
|
||||
файла, COG, SLD набора, SLD группы), а не общее число объектов в бакете.
|
||||
* **F16, F17** — эталонные значения берутся из `data/population_kato.geojson`
|
||||
(значения, приведённые в протоколе, в файле сохранены).
|
||||
* **L05** — перед чтением `/landscape/results` скрипт дожидается статуса `ready`
|
||||
всех 15 выходов: конвертация выходов в COG выполняется worker'ом асинхронно
|
||||
после завершения прогона, а `/landscape/results` возвращает только готовые слои.
|
||||
* **N38 и ограничение размера тела запроса.** В этой ветке (исправления по
|
||||
Приложению 4) тело запроса ограничено `HTTP_MAX_UPLOAD_BYTES` (по умолчанию
|
||||
2 ГиБ); загрузка сверх предела отклоняется с **413** `request body exceeds
|
||||
the … byte limit`. Файл N38 (100 МБ) меньше предела, поэтому ожидание N38
|
||||
не изменилось: 201, затем `status = failed`. Проверить 413 можно вручную,
|
||||
запустив стенд с `HTTP_MAX_UPLOAD_BYTES=1048576` и загрузив файл больше 1 МиБ.
|
||||
* **`/readyz`** в этой ветке не раскрывает текст ошибок зависимостей: при сбое
|
||||
компонент помечается `"unavailable"` (ранее — `"error: <текст>"`). На исправном
|
||||
стенде ответ не изменился: `200 {"components":{"postgres":"ok","rabbitmq":"ok","s3":"ok"},"ready":true}`;
|
||||
N38 проверяет `200` и `ready = true`.
|
||||
* **L15** — PASS определяется успешным полным прогоном (202, `succeeded`, 9 узлов
|
||||
`succeeded`); время CPU и `memory.peak` worker (cgroup v2) выводятся
|
||||
справочно и включают COG-конвертацию выходов.
|
||||
|
||||
## 2. Ландшафтная программа: попиксельная сверка
|
||||
|
||||
### Входные данные — `lp_gen.py`
|
||||
|
||||
```sh
|
||||
$PYTHON test/acceptance/lp_gen.py --out /tmp/lp-in [--width 3000 --height 2000]
|
||||
```
|
||||
|
||||
13 растров `<код слота>.tif`: Int16, EPSG:32642, пиксель 100 м, 3000×2000,
|
||||
классы 1…5 из гладких тригонометрических полей (у каждого слота своё),
|
||||
рамка NoData −9999 шириной 40 пикселей. Особые случаи для правил пикселя:
|
||||
в `lp-u-crop` — блок NoData 200×200 (U считается по water, soil и veg),
|
||||
в `lp-u-veg` — блок NoData 100×100, наполовину перекрывающий его (там нет ни
|
||||
crop, ни veg → U = NoData), в `lp-p-snow` — блоки со значениями 0 и 6
|
||||
(не NoData, но вне 1…5 → недопустимы). Плюс `misaligned-lp-u-veg.tif`
|
||||
(1500×1000, 200 м) для проверки L13. Генерация детерминирована.
|
||||
|
||||
### Прогон на стенде — `lp_run.sh`, режим (a)
|
||||
|
||||
```sh
|
||||
test/acceptance/lp_run.sh # генерация → 13 слотов → {"from_stage":0} → lp_verify.py
|
||||
```
|
||||
|
||||
Входы и выходы остаются на стенде, прогон можно перепроверить:
|
||||
`$PYTHON test/acceptance/lp_verify.py --base-url http://localhost:18080 --run <run-id>`.
|
||||
|
||||
### Проверка любого стенда (только чтение) — `lp_verify.py`, режим (b)
|
||||
|
||||
```sh
|
||||
$PYTHON test/acceptance/lp_verify.py --base-url https://dssgis.dwh.kz --run <run-id> [--cache DIR]
|
||||
```
|
||||
|
||||
Скрипт выполняет только GET-запросы:
|
||||
|
||||
1. `GET /landscape/runs/{id}` — параметры прогона, для каждого узла `inputs` и
|
||||
`outputs` (группа → набор данных);
|
||||
2. `GET /datasets/{id}.cog` — 13 входов (наборы слотов, которые прогон
|
||||
фактически использовал) и 15 выходов (`--inputs-from download` читает
|
||||
входы через `/datasets/{id}/download` — исходный файл, который читал узел).
|
||||
|
||||
Ожидаемые значения рассчитываются по всей цепочке **только из 13 входов**
|
||||
(E — из пересчитанных Z и U, C и V — из пересчитанного E), поэтому ошибка в
|
||||
промежуточном слое не может скрыть ошибку в последующем. Дополнительно
|
||||
проверяется согласованность цепочки самого прогона (узел 10 читал выход узла
|
||||
00 этого прогона и т. д.) и совпадение сетки всех 28 растров. Растры
|
||||
обрабатываются окнами 512×4096 пикселей (выровнены по тайлам COG 512), поэтому
|
||||
расход памяти ограничен (~0,5 ГБ) и для растров рабочего стенда 22219×14977.
|
||||
|
||||
Правила пикселя (источник истины — `internal/landscape/kernels.go`, они же —
|
||||
в ЛП техописании): значение допустимо при `1 ≤ v ≤ 5`; `classify` в float32
|
||||
(`≤1.8→1, ≤2.6→2, ≤3.4→3, ≤4.2→4, ≤5.0→5`); z_integrated — среднее четырёх Z с
|
||||
весом 0,25; u_integrated — water ∧ soil ∧ (crop ∨ veg), веса 0,25 только у
|
||||
допустимых crop/veg; z_max/u_max — максимум; E = max(Z, U); C_j = min(P_j, E);
|
||||
p_max/c_max — максимум по допустимым; func_zoning V1–V3 с порогами
|
||||
`v1_min_s_e12`, `v1_min_s_e3`, `v2_min_s_e3`, `v2_min_s_e4` и `tie_priority`
|
||||
из параметров прогона.
|
||||
|
||||
Вывод: по каждому из 15 выходов — число пикселей, допустимых (по эталону) и
|
||||
**число несовпадающих пикселей** (с координатами первого расхождения), итоговая
|
||||
строка `TOTAL mismatching pixels: N -> PASS/FAIL`. Код возврата: 0 — полное
|
||||
совпадение, 1 — есть расхождения, 2 — ошибка данных (прогон не `succeeded`,
|
||||
не найден вход/выход, разные сетки).
|
||||
|
||||
## 3. Нагрузочные испытания — `load.sh`
|
||||
|
||||
```sh
|
||||
test/acceptance/load.sh # S1–S6, генератор внутри сети стенда
|
||||
SCENARIOS="S1 S3 S6" test/acceptance/load.sh # часть сценариев
|
||||
GEN=host TARGET_URL=http://localhost:18080 test/acceptance/load.sh # hey на хосте, через порт стенда
|
||||
NETWORK=bridge TARGET_URL=https://host BASE_URL=https://host SCENARIOS="S1 S3" test/acceptance/load.sh
|
||||
```
|
||||
|
||||
Генератор — hey 0.1.5, собирается из исходного кода в образ
|
||||
`gis-acceptance-hey:0.1.5` (`lib/hey.Dockerfile`, на тех же закреплённых образах
|
||||
Go/Alpine, что и приложение) и по умолчанию запускается **внутри сети стенда**
|
||||
(`docker run --network gis-acceptance_default …`, цель `http://app:8080`).
|
||||
Перед нагрузкой `seed.sh` (идемпотентно) наполняет стенд: 25 vector, 25
|
||||
vector_with_kato (с маппингом КАТО, 14 районов × 3 года) и 50 raster в
|
||||
категории `acc-load`; если прогонов ЛП ещё нет, выполняется `lp_run.sh`.
|
||||
На удалённом стенде без `ALLOW_WRITES=1` данные не создаются: id для
|
||||
эндпоинтов определяются GET-запросами, сценарий S5 пропускается.
|
||||
|
||||
Эндпоинты (табл. 5 Приложения 3): `/datasets?page_size=20`, `/datasets/{id}`,
|
||||
`/datasets/{id}.geojson`, `/datasets/{id}/observations`, `/categories`,
|
||||
`/dataset-groups/{id}/latest`, `/landscape/slots`, `/landscape/results`,
|
||||
`/landscape/runs/{id}`, `/readyz`.
|
||||
|
||||
| Сценарий | Параметры hey |
|
||||
|---|---|
|
||||
| S1 | `-n 100 -c 1` последовательно по 10 эндпоинтам (1000 запросов) |
|
||||
| S2 | 1000 запросов, 100 клиентов: 10 параллельных процессов `-n 100 -c 10` |
|
||||
| S3.1 | `-n 1000 -c 1000 /datasets?page_size=20`, 3 прогона |
|
||||
| S3.2 | `-n 1000 -c 1000 /landscape/results`, 3 прогона |
|
||||
| S4 | `-z 60s -q 1`, 1000 клиентов в 8 процессах: 300 `/datasets?page_size=20`, 150 `/datasets/{id}`, по 100 `.geojson`, `/observations`, `/categories`, `/latest`, `/landscape/results`, 50 `/readyz` |
|
||||
| S5 | `-z 30s -c 10` × 10 эндпоинтов + 20 параллельных `POST /datasets` (10 GeoJSON + 10 GeoTIFF), все загрузки должны дойти до `ready` (затем удаляются) |
|
||||
| S6 | Повтор S1 и `GET /readyz` после нагрузки |
|
||||
|
||||
S3.3 протокола (1000 одновременных запросов через опубликованный порт)
|
||||
воспроизводится запуском `GEN=host TARGET_URL=http://localhost:18080 SCENARIOS=S3`.
|
||||
|
||||
Вывод: сводная таблица (запросы, клиенты, длительность, RPS, доля успешных
|
||||
ответов, min/avg/P50/P90/P95/P99/max в мс, коды ответов, транспортные ошибки,
|
||||
итог по критериям), таблицы по эндпоинтам/прогонам для S2–S5, результаты
|
||||
загрузок S5, пиковое потребление CPU/памяти контейнерами (docker stats во время
|
||||
S4/S5) и критерий K4. Как и в протоколе, перцентили сценариев из нескольких
|
||||
процессов/прогонов — средневзвешенные по числу ответов. Файлы:
|
||||
`results/load-<время>/summary.csv`, `summary.json`, `detail.csv`, `raw/*.txt`
|
||||
(исходные выводы hey). Критерии (табл. 8 Приложения 3): K1 — успешных ответов
|
||||
≥ 99 %, K2 — ответов 5xx нет, K3 — P95 ≤ 2000 мс, K4 — после нагрузки `/readyz`
|
||||
200 и P50 S6 не хуже P50 S1 (допуск ×2 + 1 мс). Код возврата 1 при
|
||||
невыполнении любого критерия.
|
||||
|
||||
Генератор и стенд работают на одной машине и делят CPU, поэтому абсолютные
|
||||
значения RPS зависят от оборудования (протокол фиксирует их как нижнюю оценку).
|
||||
|
||||
## 4. Анализ безопасности — `scan.sh`
|
||||
|
||||
```sh
|
||||
test/acceptance/scan.sh
|
||||
```
|
||||
|
||||
| Инструмент | Что проверяет | Что считается High/Critical |
|
||||
|---|---|---|
|
||||
| `govulncheck ./...` | Уязвимости Go-модулей и стандартной библиотеки, достижимые из кода | Каждая уязвимость с цепочкой вызовов из кода (у govulncheck нет шкалы severity) |
|
||||
| `govulncheck -mode=binary` | Бинарный файл `/usr/local/bin/gis` из собранного образа | Каждая уязвимость с затронутыми символами в бинарнике |
|
||||
| `gosec ./...` | Статический анализ кода | Severity `HIGH` |
|
||||
| `trivy fs` | Зависимости (`go.mod`), секреты, конфигурации (Dockerfile, compose) | `HIGH`, `CRITICAL` |
|
||||
| `trivy image` | Образ приложения (Alpine + GDAL + бинарник), собирается из `build/package/Dockerfile` | `HIGH`, `CRITICAL` |
|
||||
| `golangci-lint` | Качество кода (если установлен) | Справочно: число замечаний (шкалы severity нет) |
|
||||
|
||||
Скрипт печатает таблицу «инструмент | цель | High | Critical | примечания» и
|
||||
итог `RESULT: 0 High / 0 Critical -> PASS`; код возврата 1 при наличии
|
||||
High/Critical. Полные отчёты (JSON) — `results/scan-<время>/`. Без Docker:
|
||||
`SKIP_IMAGE=1` (бинарник собирается локально с cgo и GDAL, `trivy image`
|
||||
пропускается).
|
||||
|
||||
## Порядок полного воспроизведения
|
||||
|
||||
```sh
|
||||
test/acceptance/stand.sh up
|
||||
test/acceptance/functional.sh | tee functional.txt # RESULT: 89/89 PASS
|
||||
test/acceptance/lp_run.sh | tee lp_local.txt # TOTAL mismatching pixels: 0 -> PASS
|
||||
$PYTHON test/acceptance/lp_verify.py --base-url https://dssgis.dwh.kz --run <run-id> | tee lp_prod.txt
|
||||
test/acceptance/load.sh | tee load.txt # критерии K1–K4 выполнены
|
||||
test/acceptance/scan.sh | tee scan.txt # RESULT: 0 High / 0 Critical -> PASS
|
||||
test/acceptance/stand.sh down
|
||||
```
|
||||
|
||||
Тестовые векторные данные `data/` пересоздаются командой
|
||||
`python3 test/acceptance/lib/make_fixtures.py` (нужен `ogr2ogr`).
|
||||
55
test/acceptance/compose.acceptance.yml
Normal file
55
test/acceptance/compose.acceptance.yml
Normal file
@@ -0,0 +1,55 @@
|
||||
# Isolated acceptance-test stand (Приложения 3, 4, 7 к отчёту Этапа 8).
|
||||
#
|
||||
# Overlay on deployments/docker-compose.yml. Always use the project name
|
||||
# gis-acceptance so the stand never collides with a development or production
|
||||
# stack on the same host:
|
||||
#
|
||||
# docker compose -p gis-acceptance \
|
||||
# -f deployments/docker-compose.yml \
|
||||
# -f test/acceptance/compose.acceptance.yml up -d --build
|
||||
#
|
||||
# Host ports: app 18080, postgres 15432, minio 19000 (API) / 19001 (console),
|
||||
# rabbitmq 15672 (management UI only; AMQP stays inside the docker network).
|
||||
#
|
||||
# Environment knobs (all optional):
|
||||
# POSTGIS_PLATFORM platform of postgis/postgis (default linux/amd64: the image
|
||||
# has no arm64 build; Apple Silicon runs it under emulation,
|
||||
# amd64 hosts run it natively, so the default works on both)
|
||||
# MINIO_IMAGE MinIO image (default minio/minio:latest). Any image that
|
||||
# ships the `minio` server and the `mc` client works, e.g.
|
||||
# cgr.dev/chainguard/minio:latest.
|
||||
# APP_IMAGE tag for the locally built application image
|
||||
# HTTP_MAX_UPLOAD_BYTES, LP_MAX_WORKERS, LP_MAX_NODES — passed to app/worker.
|
||||
name: gis-acceptance
|
||||
|
||||
services:
|
||||
app:
|
||||
image: ${APP_IMAGE:-gis-acceptance-app:local}
|
||||
environment:
|
||||
HTTP_MAX_UPLOAD_BYTES: ${HTTP_MAX_UPLOAD_BYTES:-2147483648}
|
||||
LP_MAX_WORKERS: ${LP_MAX_WORKERS:-8}
|
||||
LP_MAX_NODES: ${LP_MAX_NODES:-2}
|
||||
ports: !override
|
||||
- "18080:8080"
|
||||
|
||||
worker:
|
||||
image: ${APP_IMAGE:-gis-acceptance-app:local}
|
||||
environment:
|
||||
HTTP_MAX_UPLOAD_BYTES: ${HTTP_MAX_UPLOAD_BYTES:-2147483648}
|
||||
LP_MAX_WORKERS: ${LP_MAX_WORKERS:-8}
|
||||
LP_MAX_NODES: ${LP_MAX_NODES:-2}
|
||||
|
||||
postgres:
|
||||
platform: ${POSTGIS_PLATFORM:-linux/amd64}
|
||||
ports: !override
|
||||
- "15432:5432"
|
||||
|
||||
minio:
|
||||
image: ${MINIO_IMAGE:-minio/minio:latest}
|
||||
ports: !override
|
||||
- "19000:9000"
|
||||
- "19001:9001"
|
||||
|
||||
rabbitmq:
|
||||
ports: !override
|
||||
- "15672:15672"
|
||||
19
test/acceptance/data/group_style.sld
Normal file
19
test/acceptance/data/group_style.sld
Normal file
@@ -0,0 +1,19 @@
|
||||
<?xml version="1.0" encoding="UTF-8"?>
|
||||
<sld:StyledLayerDescriptor version="1.0.0" xmlns:sld="http://www.opengis.net/sld" xmlns:ogc="http://www.opengis.net/ogc">
|
||||
<sld:NamedLayer>
|
||||
<sld:Name>group</sld:Name>
|
||||
<sld:UserStyle>
|
||||
<sld:Title>Group fallback style</sld:Title>
|
||||
<sld:FeatureTypeStyle>
|
||||
<sld:Rule>
|
||||
<sld:RasterSymbolizer>
|
||||
<sld:ColorMap>
|
||||
<sld:ColorMapEntry color="#f7fbff" quantity="0"/>
|
||||
<sld:ColorMapEntry color="#08306b" quantity="100000"/>
|
||||
</sld:ColorMap>
|
||||
</sld:RasterSymbolizer>
|
||||
</sld:Rule>
|
||||
</sld:FeatureTypeStyle>
|
||||
</sld:UserStyle>
|
||||
</sld:NamedLayer>
|
||||
</sld:StyledLayerDescriptor>
|
||||
BIN
test/acceptance/data/livestock_kato.gpkg
Normal file
BIN
test/acceptance/data/livestock_kato.gpkg
Normal file
Binary file not shown.
145
test/acceptance/data/plots_nko.geojson
Normal file
145
test/acceptance/data/plots_nko.geojson
Normal file
@@ -0,0 +1,145 @@
|
||||
{
|
||||
"type": "FeatureCollection",
|
||||
"features": [
|
||||
{
|
||||
"type": "Feature",
|
||||
"properties": {
|
||||
"plot_id": 1,
|
||||
"name": "Участок 1",
|
||||
"area_ha": 1680.0
|
||||
},
|
||||
"geometry": {
|
||||
"type": "Polygon",
|
||||
"coordinates": [
|
||||
[
|
||||
[
|
||||
69.0,
|
||||
53.6
|
||||
],
|
||||
[
|
||||
69.6,
|
||||
53.6
|
||||
],
|
||||
[
|
||||
69.6,
|
||||
54.0
|
||||
],
|
||||
[
|
||||
69.0,
|
||||
54.0
|
||||
],
|
||||
[
|
||||
69.0,
|
||||
53.6
|
||||
]
|
||||
]
|
||||
]
|
||||
}
|
||||
},
|
||||
{
|
||||
"type": "Feature",
|
||||
"properties": {
|
||||
"plot_id": 2,
|
||||
"name": "Участок 2",
|
||||
"area_ha": 2100.0
|
||||
},
|
||||
"geometry": {
|
||||
"type": "Polygon",
|
||||
"coordinates": [
|
||||
[
|
||||
[
|
||||
69.9,
|
||||
54.2
|
||||
],
|
||||
[
|
||||
70.5,
|
||||
54.2
|
||||
],
|
||||
[
|
||||
70.5,
|
||||
54.7
|
||||
],
|
||||
[
|
||||
69.9,
|
||||
54.7
|
||||
],
|
||||
[
|
||||
69.9,
|
||||
54.2
|
||||
]
|
||||
]
|
||||
]
|
||||
}
|
||||
},
|
||||
{
|
||||
"type": "Feature",
|
||||
"properties": {
|
||||
"plot_id": 3,
|
||||
"name": "Участок 3",
|
||||
"area_ha": 1400.0
|
||||
},
|
||||
"geometry": {
|
||||
"type": "Polygon",
|
||||
"coordinates": [
|
||||
[
|
||||
[
|
||||
70.6,
|
||||
53.7
|
||||
],
|
||||
[
|
||||
71.1,
|
||||
53.7
|
||||
],
|
||||
[
|
||||
71.1,
|
||||
54.1
|
||||
],
|
||||
[
|
||||
70.6,
|
||||
54.1
|
||||
],
|
||||
[
|
||||
70.6,
|
||||
53.7
|
||||
]
|
||||
]
|
||||
]
|
||||
}
|
||||
},
|
||||
{
|
||||
"type": "Feature",
|
||||
"properties": {
|
||||
"plot_id": 4,
|
||||
"name": "Участок 4",
|
||||
"area_ha": 1750.0
|
||||
},
|
||||
"geometry": {
|
||||
"type": "Polygon",
|
||||
"coordinates": [
|
||||
[
|
||||
[
|
||||
71.2,
|
||||
54.5
|
||||
],
|
||||
[
|
||||
71.7,
|
||||
54.5
|
||||
],
|
||||
[
|
||||
71.7,
|
||||
55.0
|
||||
],
|
||||
[
|
||||
71.2,
|
||||
55.0
|
||||
],
|
||||
[
|
||||
71.2,
|
||||
54.5
|
||||
]
|
||||
]
|
||||
]
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
243
test/acceptance/data/population_kato.geojson
Normal file
243
test/acceptance/data/population_kato.geojson
Normal file
@@ -0,0 +1,243 @@
|
||||
{
|
||||
"type": "FeatureCollection",
|
||||
"features": [
|
||||
{
|
||||
"type": "Feature",
|
||||
"properties": {
|
||||
"kato": "591000000",
|
||||
"district": "Петропавловск Г.А.",
|
||||
"Y2022": 47567.6,
|
||||
"Y2023": 48311.2,
|
||||
"Y2024": 49120.5
|
||||
},
|
||||
"geometry": {
|
||||
"type": "Point",
|
||||
"coordinates": [
|
||||
69.14998,
|
||||
54.89148
|
||||
]
|
||||
}
|
||||
},
|
||||
{
|
||||
"type": "Feature",
|
||||
"properties": {
|
||||
"kato": "593200000",
|
||||
"district": "Айыртауский район",
|
||||
"Y2022": 21802.7,
|
||||
"Y2023": 22615.1,
|
||||
"Y2024": 23427.5
|
||||
},
|
||||
"geometry": {
|
||||
"type": "Point",
|
||||
"coordinates": [
|
||||
68.01276,
|
||||
53.19148
|
||||
]
|
||||
}
|
||||
},
|
||||
{
|
||||
"type": "Feature",
|
||||
"properties": {
|
||||
"kato": "593400000",
|
||||
"district": "Акжарский район",
|
||||
"Y2022": 23605.4,
|
||||
"Y2023": 24417.8,
|
||||
"Y2024": 25230.2
|
||||
},
|
||||
"geometry": {
|
||||
"type": "Point",
|
||||
"coordinates": [
|
||||
71.74765,
|
||||
53.78951
|
||||
]
|
||||
}
|
||||
},
|
||||
{
|
||||
"type": "Feature",
|
||||
"properties": {
|
||||
"kato": "595800000",
|
||||
"district": "Аккайынский район",
|
||||
"Y2022": 25264.0,
|
||||
"Y2023": 26076.4,
|
||||
"Y2024": 26888.8
|
||||
},
|
||||
"geometry": {
|
||||
"type": "Point",
|
||||
"coordinates": [
|
||||
69.55798,
|
||||
54.45755
|
||||
]
|
||||
}
|
||||
},
|
||||
{
|
||||
"type": "Feature",
|
||||
"properties": {
|
||||
"kato": "594200000",
|
||||
"district": "Есильский район",
|
||||
"Y2022": 27066.7,
|
||||
"Y2023": 27879.1,
|
||||
"Y2024": 28691.5
|
||||
},
|
||||
"geometry": {
|
||||
"type": "Point",
|
||||
"coordinates": [
|
||||
68.32766,
|
||||
54.11517
|
||||
]
|
||||
}
|
||||
},
|
||||
{
|
||||
"type": "Feature",
|
||||
"properties": {
|
||||
"kato": "594600000",
|
||||
"district": "Жамбылский район",
|
||||
"Y2022": 28869.4,
|
||||
"Y2023": 29681.8,
|
||||
"Y2024": 30494.2
|
||||
},
|
||||
"geometry": {
|
||||
"type": "Point",
|
||||
"coordinates": [
|
||||
66.98273,
|
||||
54.4391
|
||||
]
|
||||
}
|
||||
},
|
||||
{
|
||||
"type": "Feature",
|
||||
"properties": {
|
||||
"kato": "595000000",
|
||||
"district": "Кызылжарский район",
|
||||
"Y2022": 46862.7,
|
||||
"Y2023": 48268.6,
|
||||
"Y2024": 49205.8
|
||||
},
|
||||
"geometry": {
|
||||
"type": "Point",
|
||||
"coordinates": [
|
||||
69.25907,
|
||||
54.89353
|
||||
]
|
||||
}
|
||||
},
|
||||
{
|
||||
"type": "Feature",
|
||||
"properties": {
|
||||
"kato": "595200000",
|
||||
"district": "Мамлютский район",
|
||||
"Y2022": 32330.7,
|
||||
"Y2023": 33143.1,
|
||||
"Y2024": 33955.5
|
||||
},
|
||||
"geometry": {
|
||||
"type": "Point",
|
||||
"coordinates": [
|
||||
68.21021,
|
||||
54.81946
|
||||
]
|
||||
}
|
||||
},
|
||||
{
|
||||
"type": "Feature",
|
||||
"properties": {
|
||||
"kato": "596000000",
|
||||
"district": "Тайыншинский район",
|
||||
"Y2022": 34133.4,
|
||||
"Y2023": 34945.8,
|
||||
"Y2024": 35758.2
|
||||
},
|
||||
"geometry": {
|
||||
"type": "Point",
|
||||
"coordinates": [
|
||||
69.89444,
|
||||
53.80885
|
||||
]
|
||||
}
|
||||
},
|
||||
{
|
||||
"type": "Feature",
|
||||
"properties": {
|
||||
"kato": "596200000",
|
||||
"district": "Тимирязевский район",
|
||||
"Y2022": 35792.0,
|
||||
"Y2023": 36604.4,
|
||||
"Y2024": 37416.8
|
||||
},
|
||||
"geometry": {
|
||||
"type": "Point",
|
||||
"coordinates": [
|
||||
66.56047,
|
||||
53.66856
|
||||
]
|
||||
}
|
||||
},
|
||||
{
|
||||
"type": "Feature",
|
||||
"properties": {
|
||||
"kato": "596400000",
|
||||
"district": "Уалихановский район",
|
||||
"Y2022": 37594.7,
|
||||
"Y2023": 38407.1,
|
||||
"Y2024": 39219.5
|
||||
},
|
||||
"geometry": {
|
||||
"type": "Point",
|
||||
"coordinates": [
|
||||
72.82078,
|
||||
53.36512
|
||||
]
|
||||
}
|
||||
},
|
||||
{
|
||||
"type": "Feature",
|
||||
"properties": {
|
||||
"kato": "593600000",
|
||||
"district": "Район Магжана Жумабаева",
|
||||
"Y2022": 39253.3,
|
||||
"Y2023": 40065.7,
|
||||
"Y2024": 40878.1
|
||||
},
|
||||
"geometry": {
|
||||
"type": "Point",
|
||||
"coordinates": [
|
||||
70.5439,
|
||||
54.63954
|
||||
]
|
||||
}
|
||||
},
|
||||
{
|
||||
"type": "Feature",
|
||||
"properties": {
|
||||
"kato": "595600000",
|
||||
"district": "Район Шал акына",
|
||||
"Y2022": 41056.0,
|
||||
"Y2023": 41868.4,
|
||||
"Y2024": 42680.8
|
||||
},
|
||||
"geometry": {
|
||||
"type": "Point",
|
||||
"coordinates": [
|
||||
67.40455,
|
||||
53.80847
|
||||
]
|
||||
}
|
||||
},
|
||||
{
|
||||
"type": "Feature",
|
||||
"properties": {
|
||||
"kato": "596600000",
|
||||
"district": "Район им. Габита Мусрепова",
|
||||
"Y2022": 42858.7,
|
||||
"Y2023": 43671.1,
|
||||
"Y2024": 44483.5
|
||||
},
|
||||
"geometry": {
|
||||
"type": "Point",
|
||||
"coordinates": [
|
||||
67.02892,
|
||||
52.78419
|
||||
]
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
20
test/acceptance/data/style.sld
Normal file
20
test/acceptance/data/style.sld
Normal file
@@ -0,0 +1,20 @@
|
||||
<?xml version="1.0" encoding="UTF-8"?>
|
||||
<sld:StyledLayerDescriptor version="1.0.0" xmlns:sld="http://www.opengis.net/sld" xmlns:ogc="http://www.opengis.net/ogc">
|
||||
<sld:NamedLayer>
|
||||
<sld:Name>ndvi</sld:Name>
|
||||
<sld:UserStyle>
|
||||
<sld:Title>NDVI gradient</sld:Title>
|
||||
<sld:FeatureTypeStyle>
|
||||
<sld:Rule>
|
||||
<sld:RasterSymbolizer>
|
||||
<sld:ColorMap>
|
||||
<sld:ColorMapEntry color="#a50026" quantity="0.0"/>
|
||||
<sld:ColorMapEntry color="#ffffbf" quantity="0.5"/>
|
||||
<sld:ColorMapEntry color="#006837" quantity="1.0"/>
|
||||
</sld:ColorMap>
|
||||
</sld:RasterSymbolizer>
|
||||
</sld:Rule>
|
||||
</sld:FeatureTypeStyle>
|
||||
</sld:UserStyle>
|
||||
</sld:NamedLayer>
|
||||
</sld:StyledLayerDescriptor>
|
||||
1
test/acceptance/data/style.txt
Normal file
1
test/acceptance/data/style.txt
Normal file
@@ -0,0 +1 @@
|
||||
not an SLD file
|
||||
758
test/acceptance/functional.sh
Executable file
758
test/acceptance/functional.sh
Executable file
@@ -0,0 +1,758 @@
|
||||
#!/usr/bin/env bash
|
||||
# Functional acceptance test of the GIS service on a test stand — the 89 checks
|
||||
# of section 5 of «Протокол функционального тестирования» (Приложение 7):
|
||||
# F01–F38 positive scenarios (categories, datasets, groups, SLD, filters,
|
||||
# PostGIS / MinIO confirmation)
|
||||
# N01–N38 negative scenarios
|
||||
# L02–L05, L07–L15 Landscape Program (ЛП) on synthetic rasters, incl. the
|
||||
# pixel-by-pixel reference check (lp_verify.py)
|
||||
#
|
||||
# Usage:
|
||||
# test/acceptance/functional.sh # stand at http://localhost:18080
|
||||
# BASE_URL=http://host:port test/acceptance/functional.sh
|
||||
#
|
||||
# Environment:
|
||||
# BASE_URL API address (default http://localhost:18080)
|
||||
# LOCAL_CHECKS 1 = confirm in PostGIS/MinIO through the docker containers of
|
||||
# the compose project (F35 MinIO part, F36, F37, F38, L15
|
||||
# resource figures); 0 = skip those (reported as SKIP);
|
||||
# auto (default) = 1 when the gis-acceptance containers run
|
||||
# COMPOSE_PROJECT compose project of the stand (default gis-acceptance)
|
||||
# PYTHON python3 with numpy + rasterio (default python3)
|
||||
# KEEP_DATA 1 = do not delete the created test data at the end
|
||||
# WAIT_TIMEOUT seconds to wait for asynchronous processing (default 300)
|
||||
#
|
||||
# Output: one line per check "ID | описание | PASS/FAIL/SKIP | detail", then a
|
||||
# summary. Exit code 0 only when there is no FAIL. Every object the script
|
||||
# creates carries a unique run prefix and is deleted at the end (unless
|
||||
# KEEP_DATA=1); pre-existing objects are never modified or deleted. The script
|
||||
# writes the ЛП slots, so run it on an isolated test stand, never on production.
|
||||
set -euo pipefail
|
||||
|
||||
HERE="$(cd "$(dirname "$0")" && pwd)"
|
||||
DATA="$HERE/data"
|
||||
BASE_URL="${BASE_URL:-http://localhost:18080}"
|
||||
BASE_URL="${BASE_URL%/}"
|
||||
COMPOSE_PROJECT="${COMPOSE_PROJECT:-gis-acceptance}"
|
||||
PYTHON="${PYTHON:-python3}"
|
||||
KEEP_DATA="${KEEP_DATA:-0}"
|
||||
WAIT_TIMEOUT="${WAIT_TIMEOUT:-300}"
|
||||
LOCAL_CHECKS="${LOCAL_CHECKS:-auto}"
|
||||
|
||||
for bin in curl jq "$PYTHON"; do
|
||||
command -v "$bin" >/dev/null 2>&1 || { echo "missing required tool: $bin" >&2; exit 2; }
|
||||
done
|
||||
"$PYTHON" -c 'import numpy, rasterio' 2>/dev/null || { echo "$PYTHON needs numpy and rasterio (pip install -r $HERE/requirements.txt)" >&2; exit 2; }
|
||||
|
||||
container_of() { # service -> container id of the compose project (empty if none)
|
||||
docker ps -q --filter "label=com.docker.compose.project=$COMPOSE_PROJECT" \
|
||||
--filter "label=com.docker.compose.service=$1" 2>/dev/null | head -n1
|
||||
}
|
||||
if [ "$LOCAL_CHECKS" = auto ]; then
|
||||
LOCAL_CHECKS=0
|
||||
if command -v docker >/dev/null 2>&1 && [ -n "$(container_of postgres)" ] && [ -n "$(container_of minio)" ]; then
|
||||
LOCAL_CHECKS=1
|
||||
fi
|
||||
fi
|
||||
PG_C=""; MINIO_C=""; WORKER_C=""
|
||||
if [ "$LOCAL_CHECKS" = 1 ]; then
|
||||
PG_C="$(container_of postgres)"; MINIO_C="$(container_of minio)"; WORKER_C="$(container_of worker)"
|
||||
fi
|
||||
|
||||
STAMP="$(date +%Y%m%d-%H%M%S)"
|
||||
P="acc$(date +%s)" # unique slug prefix of every object this run creates
|
||||
TMP="$(mktemp -d "${TMPDIR:-/tmp}/gis-func.XXXXXX")"
|
||||
RESULTS="$HERE/results/functional-$STAMP"
|
||||
mkdir -p "$RESULTS"
|
||||
BODY="$TMP/body"
|
||||
JSON=(-H 'Content-Type: application/json')
|
||||
|
||||
# ------------------------------------------------------------------ reporting
|
||||
PASSN=0; FAILN=0; SKIPN=0
|
||||
CUR_ID=""; CUR_DESC=""; CUR_OK=1; CUR_MSG=""
|
||||
begin() { CUR_ID="$1"; CUR_DESC="$2"; CUR_OK=1; CUR_MSG=""; }
|
||||
bad() { CUR_OK=0; CUR_MSG="${CUR_MSG:+$CUR_MSG; }$1"; }
|
||||
line() { printf '%-4s | %s | %s | %s\n' "$1" "$2" "$3" "$4" | tee -a "$RESULTS/functional.txt"; }
|
||||
end() { # detail
|
||||
if [ "$CUR_OK" = 1 ]; then PASSN=$((PASSN + 1)); line "$CUR_ID" "$CUR_DESC" PASS "$1"
|
||||
else FAILN=$((FAILN + 1)); line "$CUR_ID" "$CUR_DESC" FAIL "$CUR_MSG"; fi
|
||||
}
|
||||
skip() { SKIPN=$((SKIPN + 1)); line "$CUR_ID" "$CUR_DESC" SKIP "$1"; }
|
||||
snippet() { head -c 220 "$BODY" 2>/dev/null | tr '\n\r' ' '; }
|
||||
|
||||
# ----------------------------------------------------------------------- HTTP
|
||||
CODE=000; TIME=0
|
||||
http() { # METHOD PATH [curl args...] -> $CODE, $TIME (s), body in $BODY
|
||||
local m="$1" p="$2" out; shift 2
|
||||
out="$(curl -sS -o "$BODY" -w '%{http_code} %{time_total}' -X "$m" "$BASE_URL$p" "$@" 2>"$TMP/curl.err")" || out="000 0"
|
||||
CODE="${out%% *}"; TIME="${out#* }"
|
||||
}
|
||||
jb() { jq -r "$1" "$BODY" 2>/dev/null || true; }
|
||||
jbc() { jq -c "${1:-.}" "$BODY" 2>/dev/null || snippet; }
|
||||
need_code() { [ "$CODE" = "$1" ] || bad "HTTP $CODE (ожидалось $1): $(snippet)"; }
|
||||
need_eq() { [ "$1" = "$2" ] || bad "$3: «$1» ≠ «$2»"; }
|
||||
need_has() { case "$1" in *"$2"*) ;; *) bad "$3: нет «$2» в «${1:0:200}»";; esac; }
|
||||
need_true() { [ "$1" = true ] || bad "$2"; }
|
||||
|
||||
sha256() { if command -v sha256sum >/dev/null 2>&1; then sha256sum "$1" | cut -d' ' -f1; else shasum -a 256 "$1" | cut -d' ' -f1; fi; }
|
||||
|
||||
# dataset status polling: wait_ds ID STATUS... -> $WS (final status)
|
||||
WS=""
|
||||
wait_ds() {
|
||||
local id="$1"; shift
|
||||
local deadline=$(( $(date +%s) + WAIT_TIMEOUT ))
|
||||
while :; do
|
||||
WS="$(curl -sS "$BASE_URL/datasets/$id" | jq -r '.status' 2>/dev/null || echo "?")"
|
||||
for s in "$@"; do [ "$WS" = "$s" ] && return 0; done
|
||||
[ "$WS" = failed ] && return 0
|
||||
[ "$(date +%s)" -ge "$deadline" ] && return 0
|
||||
sleep 1
|
||||
done
|
||||
}
|
||||
# run polling: wait_run ID -> $RS final status; run JSON in $TMP/run-ID.json
|
||||
RS=""
|
||||
wait_run() {
|
||||
local id="$1" deadline=$(( $(date +%s) + WAIT_TIMEOUT ))
|
||||
while :; do
|
||||
curl -sS "$BASE_URL/landscape/runs/$id" -o "$TMP/run-$id.json" || true
|
||||
RS="$(jq -r '.status' "$TMP/run-$id.json" 2>/dev/null || echo "?")"
|
||||
case "$RS" in succeeded|failed) return 0;; esac
|
||||
[ "$(date +%s)" -ge "$deadline" ] && return 0
|
||||
sleep 1
|
||||
done
|
||||
}
|
||||
node_statuses() { jq -r '[.nodes[] | "\(.node_code):\(.status)"] | sort | join(" ")' "$TMP/run-$1.json" 2>/dev/null || true; }
|
||||
wait_outputs() { # RUN: wait until every output dataset of the run is ready (COG done)
|
||||
local id; for id in $(run_outputs "$1" | jq -r '.[].dataset_id'); do wait_ds "$id" ready; done
|
||||
}
|
||||
run_outputs() { jq -c '[.nodes[] | .outputs[]?] | sort_by(.group)' "$TMP/run-$1.json" 2>/dev/null || echo '[]'; }
|
||||
|
||||
# ------------------------------------------------------------- cleanup ledger
|
||||
DATASETS=""; GRPS=""; CATS=""; RUNS=""
|
||||
track_ds() { [ -n "$1" ] && [ "$1" != null ] && DATASETS="$DATASETS $1"; return 0; }
|
||||
cleanup() {
|
||||
local rc=$?
|
||||
if [ "$KEEP_DATA" = 1 ]; then
|
||||
echo "KEEP_DATA=1: test data kept (prefix $P)"
|
||||
else
|
||||
local ids="$DATASETS" r d
|
||||
for r in $RUNS; do # outputs computed by our runs (never reused foreign ones)
|
||||
[ -f "$TMP/run-$r.json" ] || continue
|
||||
ids="$ids $(jq -r '[.nodes[] | select(.status=="succeeded") | .outputs[]?.dataset_id] | join(" ")' "$TMP/run-$r.json" 2>/dev/null || true)"
|
||||
done
|
||||
for d in $ids; do curl -sS -o /dev/null -X DELETE "$BASE_URL/datasets/$d" || true; done
|
||||
for d in $GRPS; do curl -sS -o /dev/null -X DELETE "$BASE_URL/dataset-groups/$d" || true; done
|
||||
for d in $CATS; do curl -sS -o /dev/null -X DELETE "$BASE_URL/categories/$d" || true; done
|
||||
echo "cleanup: removed test datasets, groups and categories with prefix $P (ЛП run records are kept: the API has no run deletion)"
|
||||
fi
|
||||
rm -rf "$TMP"
|
||||
exit "$rc"
|
||||
}
|
||||
trap cleanup EXIT
|
||||
|
||||
echo "functional acceptance test: $BASE_URL prefix=$P local_checks=$LOCAL_CHECKS log=$RESULTS"
|
||||
: > "$RESULTS/functional.txt"
|
||||
|
||||
# ====================================================================== 5.2
|
||||
begin F01 "Создание корневой категории POST /categories"
|
||||
http POST /categories "${JSON[@]}" -d "{\"code\":\"$P-ecology\",\"name\":\"Экология\",\"description\":\"Приёмочный тест\"}"
|
||||
need_code 201; CAT_ECO="$(jb .id)"; need_eq "$(jb .parent_id)" null "parent_id"
|
||||
[ -n "$CAT_ECO" ] && [ "$CAT_ECO" != null ] && CATS="$CAT_ECO $CATS"
|
||||
end "201; $P-ecology «Экология», parent_id null"
|
||||
|
||||
begin F02 "Создание дочерних категорий (parent_id)"
|
||||
http POST /categories "${JSON[@]}" -d "{\"code\":\"$P-ecology-soil\",\"name\":\"Почвы\",\"parent_id\":\"$CAT_ECO\"}"
|
||||
need_code 201; CAT_SOIL="$(jb .id)"; need_eq "$(jb .parent_id)" "$CAT_ECO" "parent_id soil"
|
||||
[ -n "$CAT_SOIL" ] && [ "$CAT_SOIL" != null ] && CATS="$CAT_SOIL $CATS"
|
||||
c1="$CODE"
|
||||
http POST /categories "${JSON[@]}" -d "{\"code\":\"$P-ecology-soil-humus\",\"name\":\"Гумус\",\"parent_id\":\"$CAT_SOIL\"}"
|
||||
need_code 201; CAT_HUM="$(jb .id)"; need_eq "$(jb .parent_id)" "$CAT_SOIL" "parent_id humus"
|
||||
[ -n "$CAT_HUM" ] && [ "$CAT_HUM" != null ] && CATS="$CAT_HUM $CATS"
|
||||
end "$c1 / $CODE: трёхуровневая иерархия ecology → soil → humus"
|
||||
|
||||
begin F03 "Список категорий с фильтром parent_id"
|
||||
http GET "/categories?parent_id=$CAT_ECO"
|
||||
need_code 200; need_eq "$(jb 'length')" 1 "число дочерних"; need_eq "$(jb '.[0].id')" "$CAT_SOIL" "дочерняя категория"
|
||||
end "200; возвращена дочерняя категория $P-ecology-soil"
|
||||
|
||||
begin F04 "Получение категории по id"
|
||||
http GET "/categories/$CAT_SOIL"
|
||||
need_code 200
|
||||
need_eq "$(jb '[has("id"),has("parent_id"),has("code"),has("name"),has("description")]|all')" true "состав полей"
|
||||
SOIL_UPDATED="$(jb .updated_at)"
|
||||
end "200; поля id, parent_id, code, name, description"
|
||||
|
||||
begin F05 "Обновление категории (PUT)"
|
||||
sleep 1
|
||||
http PUT "/categories/$CAT_SOIL" "${JSON[@]}" -d "{\"code\":\"$P-ecology-soil\",\"name\":\"Почвы и грунты\",\"description\":\"Обновлено\",\"parent_id\":\"$CAT_ECO\"}"
|
||||
need_code 200; need_eq "$(jb .name)" "Почвы и грунты" name; need_eq "$(jb .description)" "Обновлено" description
|
||||
[ "$(jb .updated_at)" != "$SOIL_UPDATED" ] || bad "updated_at не изменился"
|
||||
end "200; name «Почвы и грунты», description «Обновлено», updated_at обновлён"
|
||||
|
||||
begin F06 "Цикл в иерархии (корень → внук)"
|
||||
http PUT "/categories/$CAT_ECO" "${JSON[@]}" -d "{\"code\":\"$P-ecology\",\"name\":\"Экология\",\"parent_id\":\"$CAT_HUM\"}"
|
||||
need_code 422; need_has "$(jb .error)" "own ancestor" сообщение
|
||||
end "422 «$(jb .error)»"
|
||||
|
||||
begin F07 "Категория-родитель самой себя"
|
||||
http PUT "/categories/$CAT_ECO" "${JSON[@]}" -d "{\"code\":\"$P-ecology\",\"name\":\"Экология\",\"parent_id\":\"$CAT_ECO\"}"
|
||||
need_code 422; need_has "$(jb .error)" "own ancestor" сообщение
|
||||
end "422 «$(jb .error)»"
|
||||
|
||||
begin F08 "Дубликат кода категории"
|
||||
http POST /categories "${JSON[@]}" -d "{\"code\":\"$P-ecology\",\"name\":\"Дубликат\"}"
|
||||
need_code 409
|
||||
[ "$CODE" = 201 ] && CATS="$(jb .id) $CATS"
|
||||
end "409 «$(jb .error)»"
|
||||
|
||||
begin F09 "Удаление категории"
|
||||
http POST /categories "${JSON[@]}" -d "{\"code\":\"$P-tmp\",\"name\":\"Временная\"}"
|
||||
need_code 201; tmpc="$(jb .id)"; c1="$CODE"
|
||||
http DELETE "/categories/$tmpc"; need_code 204; c2="$CODE"
|
||||
http GET "/categories/$tmpc"; need_code 404
|
||||
end "$c1 (создана) → $c2 (удалена) → повторный GET $CODE"
|
||||
|
||||
# ====================================================================== 5.3
|
||||
begin F10 "Создание группы наборов POST /dataset-groups"
|
||||
http POST /dataset-groups "${JSON[@]}" -d "{\"code\":\"$P-population-nko\",\"name\":\"Численность населения СКО\",\"description\":\"Приёмочный тест\"}"
|
||||
need_code 201; GRP="$(jb .id)"
|
||||
[ -n "$GRP" ] && [ "$GRP" != null ] && GRPS="$GRP $GRPS"
|
||||
end "201; группа $P-population-nko «Численность населения СКО»"
|
||||
|
||||
begin F11 "Загрузка набора vector (GeoJSON)"
|
||||
http POST /datasets -F "file=@$DATA/plots_nko.geojson" -F file_type=vector -F "category_id=$CAT_ECO" \
|
||||
-F "name=Участки СКО" -F "description=Полигоны участков" -F "unit=ha"
|
||||
need_code 201; DS_PLOTS="$(jb .id)"; track_ds "$DS_PLOTS"; s0="$(jb .status)"; key="$(jb .storage_key)"
|
||||
need_has "$key" "/plots_nko.geojson" storage_key
|
||||
wait_ds "$DS_PLOTS" ready; need_eq "$WS" ready status
|
||||
end "201, status $s0 → $WS; storage_key …/plots_nko.geojson"
|
||||
|
||||
begin F12 "Выгрузка vector в GeoJSON"
|
||||
http GET "/datasets/$DS_PLOTS.geojson"
|
||||
need_code 200; need_eq "$(jb .type)" FeatureCollection type; need_eq "$(jb '.features|length')" 4 "число объектов"
|
||||
need_eq "$(jb '[.features[].geometry.type]|unique|join(",")')" Polygon "тип геометрии"
|
||||
ext="$(jb '[.. | arrays | select(length==2 and (.[0]|type)=="number")] as $p | "\([$p[][0]]|min),\([$p[][1]]|min) – \([$p[][0]]|max),\([$p[][1]]|max)"')"
|
||||
need_eq "$(jb '[.. | arrays | select(length==2 and (.[0]|type)=="number")] as $p | ([$p[][0]]|min)==69 and ([$p[][1]]|min)==53.6 and ([$p[][0]]|max)==71.7 and ([$p[][1]]|max)==55')" true "охват (69.0, 53.6) – (71.7, 55.0): $ext"
|
||||
end "200; FeatureCollection, 4 объекта Polygon; охват ($ext)"
|
||||
|
||||
begin F13 "Загрузка vector_with_kato (GeoJSON, реальные коды КАТО)"
|
||||
http POST /datasets -F "file=@$DATA/population_kato.geojson" -F file_type=vector_with_kato -F "category_id=$CAT_ECO" \
|
||||
-F "name=Численность населения" -F unit=чел -F "dg_id=$GRP" -F start_date=2022-01-01 -F end_date=2024-12-31
|
||||
need_code 201; DS_POP="$(jb .id)"; track_ds "$DS_POP"; s0="$(jb .status)"; need_eq "$(jb .dg_id)" "$GRP" dg_id
|
||||
wait_ds "$DS_POP" awaiting_mapping; need_eq "$WS" awaiting_mapping status
|
||||
end "201, status $s0 → $WS, dg_id = группа $P-population-nko"
|
||||
|
||||
begin F14 "Обнаружение колонок"
|
||||
http GET "/datasets/$DS_POP"
|
||||
need_code 200; need_eq "$(jb .status)" awaiting_mapping status
|
||||
cols="$(jb '[.attribute_columns[].name]|join(",")')"
|
||||
need_eq "$cols" "kato,district,Y2022,Y2023,Y2024" колонки
|
||||
need_eq "$(jb '[.attribute_columns[]|(.samples|length)>0]|all')" true "примеры значений"
|
||||
end "200; колонки $cols с примерами значений"
|
||||
|
||||
MAP="$TMP/map.json"
|
||||
mapping() { http POST "/datasets/$DS_POP/mapping" "${JSON[@]}" -d "$1"; }
|
||||
begin N01 "Маппинг: несуществующая колонка года"
|
||||
mapping '{"kato_column":"kato","year_columns":[{"column":"Y1999","start_date":"2022-01-01"}]}'
|
||||
need_code 422; need_has "$(jb .error)" 'year column "Y1999" is not among the detected columns' сообщение
|
||||
end "422 «$(jb .error)»"
|
||||
begin N02 "Маппинг: несуществующая колонка КАТО"
|
||||
mapping '{"kato_column":"KATO_X","year_columns":[{"column":"Y2022","start_date":"2022-01-01"}]}'
|
||||
need_code 422; need_has "$(jb .error)" 'kato_column "KATO_X" is not among the detected columns' сообщение
|
||||
end "422 «$(jb .error)»"
|
||||
begin N03 "Маппинг: end_date раньше start_date"
|
||||
mapping '{"kato_column":"kato","year_columns":[{"column":"Y2022","start_date":"2022-12-31","end_date":"2022-01-01"}]}'
|
||||
need_code 422; need_has "$(jb .error)" 'is before start_date' сообщение
|
||||
end "422 «$(jb .error)»"
|
||||
begin N04 "Маппинг: некорректная дата 2022-13-45"
|
||||
mapping '{"kato_column":"kato","year_columns":[{"column":"Y2022","start_date":"2022-13-45"}]}'
|
||||
need_code 422; need_eq "$(jb .errors.StartDate)" "is invalid" errors.StartDate
|
||||
end "422 $(jbc)"
|
||||
begin N05 "Маппинг: пустой список year_columns"
|
||||
mapping '{"kato_column":"kato","year_columns":[]}'
|
||||
need_code 422; need_has "$(jb .errors.YearColumns)" "at least 1" errors.YearColumns
|
||||
end "422 $(jbc)"
|
||||
|
||||
begin F15 "Сохранение маппинга КАТО и годовых колонок с периодами"
|
||||
mapping '{"kato_column":"kato","year_columns":[{"column":"Y2022","start_date":"2022-01-01","end_date":"2022-12-31"},{"column":"Y2023","start_date":"2023-01-01","end_date":"2023-12-31"},{"column":"Y2024","start_date":"2024-01-01","end_date":"2024-12-31"}]}'
|
||||
need_code 200; s0="$(jb .status)"; need_eq "$(jb '.year_columns|length')" 3 year_columns; need_eq "$(jb .kato_column)" kato kato_column
|
||||
wait_ds "$DS_POP" ready; need_eq "$WS" ready status
|
||||
end "200, status $s0 → $WS; 3 колонки Y2022–Y2024 с периодами"
|
||||
|
||||
fixval() { jq -r --arg k "$1" --arg y "$2" '.features[]|select(.properties.kato==$k)|.properties[$y]' "$DATA/population_kato.geojson"; }
|
||||
begin F16 "Наблюдения (long format)"
|
||||
http GET "/datasets/$DS_POP/observations?page_size=100"
|
||||
need_code 200; need_eq "$(jb .total)" 42 total
|
||||
exp="$(fixval 591000000 Y2022)"
|
||||
got="$(jb '.data[]|select(.kato_code=="591000000" and .start_date=="2022-01-01")|.value')"
|
||||
need_eq "$got" "$exp" "591000000 / 2022"
|
||||
end "200; total $(jb .total) (14 районов × 3 года), 591000000 / 2022 → $got"
|
||||
|
||||
begin F17 "Фильтр наблюдений по kato_code"
|
||||
http GET "/datasets/$DS_POP/observations?kato_code=595000000"
|
||||
need_code 200; need_eq "$(jb .total)" 3 total
|
||||
got="$(jb '[.data|sort_by(.start_date)[]|.value]|join(" / ")')"
|
||||
exp="$(fixval 595000000 Y2022) / $(fixval 595000000 Y2023) / $(fixval 595000000 Y2024)"
|
||||
need_eq "$got" "$exp" "значения 595000000"
|
||||
end "200; для 595000000 — $(jb .total) наблюдения: $got"
|
||||
|
||||
begin F18 "GeoJSON vector_with_kato: соединение с districts"
|
||||
http GET "/datasets/$DS_POP.geojson"
|
||||
need_code 200; need_eq "$(jb '.features|length')" 14 "число объектов"
|
||||
need_eq "$(jb '[.features[].geometry.type]|unique|join(",")')" MultiPolygon "тип геометрии"
|
||||
need_eq "$(jb '.features[0].properties|keys|join(",")')" "data,kato,name,periods" "свойства"
|
||||
end "200; 14 объектов MultiPolygon, свойства data, kato, name, periods"
|
||||
|
||||
begin F19 "Загрузка vector_with_kato (GeoPackage)"
|
||||
http POST /datasets -F "file=@$DATA/livestock_kato.gpkg" -F file_type=vector_with_kato -F "category_id=$CAT_ECO" \
|
||||
-F "name=Поголовье КРС" -F unit=голов
|
||||
need_code 201; DS_LIVE="$(jb .id)"; track_ds "$DS_LIVE"; c1="$CODE"
|
||||
wait_ds "$DS_LIVE" awaiting_mapping; need_eq "$WS" awaiting_mapping status
|
||||
http GET "/datasets/$DS_LIVE"; cols="$(jb '[.attribute_columns[].name]|join(",")')"
|
||||
need_eq "$cols" "fid,KATO_CODE,district,H_2022,H_2023,H_2024" колонки
|
||||
http POST "/datasets/$DS_LIVE/mapping" "${JSON[@]}" -d '{"kato_column":"KATO_CODE","year_columns":[{"column":"H_2022","start_date":"2022-01-01","end_date":"2022-12-31"},{"column":"H_2023","start_date":"2023-01-01","end_date":"2023-12-31"},{"column":"H_2024","start_date":"2024-01-01","end_date":"2024-12-31"}]}'
|
||||
need_code 200; c2="$CODE"
|
||||
wait_ds "$DS_LIVE" ready; need_eq "$WS" ready status
|
||||
end "$c1 → awaiting_mapping; колонки $cols; маппинг $c2, статус $WS"
|
||||
|
||||
NDVI="$TMP/ndvi_nko.tif"
|
||||
"$PYTHON" "$HERE/lib/helpers.py" ndvi "$NDVI"
|
||||
begin F20 "Загрузка растра GeoTIFF (600×400, EPSG:4326, Float32)"
|
||||
http POST /datasets -F "file=@$NDVI" -F file_type=raster -F "category_id=$CAT_ECO" -F "name=NDVI СКО" -F unit=NDVI
|
||||
need_code 201; DS_NDVI="$(jb .id)"; track_ds "$DS_NDVI"; s0="$(jb .status)"
|
||||
wait_ds "$DS_NDVI" ready; need_eq "$WS" ready status
|
||||
end "201, status $s0 → $WS"
|
||||
|
||||
begin F21 "Растр: статус, bbox"
|
||||
http GET "/datasets/$DS_NDVI"
|
||||
need_code 200; need_eq "$(jb .status)" ready status
|
||||
need_has "$(jb .cog_storage_key)" "/cog/ndvi_nko.tif" cog_storage_key
|
||||
bbox="$(jb '.bbox|map(.*1000|round/1000)|join(", ")')"
|
||||
need_eq "$bbox" "68.5, 51.5, 74.5, 55.5" bbox
|
||||
end "200; ready, cog_storage_key …/cog/ndvi_nko.tif, bbox [$bbox]"
|
||||
|
||||
begin F22 "Получение COG и проверка LAYOUT=COG"
|
||||
ctype="$(curl -sS -o "$TMP/ndvi.cog.tif" -w '%{http_code} %{content_type}' "$BASE_URL/datasets/$DS_NDVI.cog" || echo 000)"
|
||||
need_eq "$ctype" "200 image/tiff" "код и Content-Type"
|
||||
ri="$("$PYTHON" "$HERE/lib/helpers.py" rinfo "$TMP/ndvi.cog.tif" 2>&1 || true)"
|
||||
need_eq "$(echo "$ri" | jq -r '"\(.driver) \(.width)x\(.height) EPSG:\(.epsg) \(.layout) \(.compression)"' 2>/dev/null)" "GTiff 600x400 EPSG:4326 COG DEFLATE" "формат"
|
||||
end "$ctype, $(wc -c < "$TMP/ndvi.cog.tif" | tr -d ' ') байт; $(echo "$ri" | jq -r '"\(.driver), \(.width)×\(.height), EPSG:\(.epsg), LAYOUT=\(.layout), \(.compression)"' 2>/dev/null)"
|
||||
|
||||
begin F23 "Скачивание оригинала /download"
|
||||
dl="$(curl -sS -o "$TMP/ndvi.dl.tif" -w '%{http_code}' "$BASE_URL/datasets/$DS_NDVI/download" || echo 000)"
|
||||
need_eq "$dl" 200 "HTTP"
|
||||
h1="$(sha256 "$NDVI")"; h2="$(sha256 "$TMP/ndvi.dl.tif")"
|
||||
need_eq "$h2" "$h1" "SHA-256"
|
||||
end "$dl, $(wc -c < "$TMP/ndvi.dl.tif" | tr -d ' ') байт; SHA-256 совпадает (${h1:0:8}…${h1: -5})"
|
||||
|
||||
begin F24 "Загрузка SLD для набора"
|
||||
http GET "/datasets/$DS_NDVI/sld"; need_code 404; c0="$CODE"
|
||||
http POST "/datasets/$DS_NDVI/sld" -F "file=@$DATA/style.sld"; need_code 200; need_eq "$(jb .sld_filename)" style.sld sld_filename; c1="$CODE"
|
||||
http GET "/datasets/$DS_NDVI/sld"; need_code 200; need_has "$(cat "$BODY")" "NDVI gradient" "SLD"
|
||||
end "до загрузки GET /sld → $c0; POST → $c1, sld_filename style.sld; GET /sld → $CODE («NDVI gradient»)"
|
||||
|
||||
begin F25 "Общий SLD группы и наследование набором"
|
||||
http POST "/dataset-groups/$GRP/sld" -F "file=@$DATA/group_style.sld"; need_code 200; c1="$CODE"
|
||||
http GET "/dataset-groups/$GRP/sld"; need_code 200; need_has "$(cat "$BODY")" "Group fallback style" "SLD группы"
|
||||
http GET "/datasets/$DS_POP/sld"; need_code 200; need_has "$(cat "$BODY")" "Group fallback style" "SLD набора без собственного"
|
||||
end "POST группы → $c1; GET /sld группы и набора без собственного SLD → «Group fallback style»"
|
||||
|
||||
begin N06 "SLD: недопустимое расширение файла"
|
||||
http POST "/datasets/$DS_NDVI/sld" -F "file=@$DATA/style.txt"
|
||||
need_code 422; need_has "$(jb .error)" "sld file must have one of these extensions" сообщение
|
||||
end "422 «$(jb .error)»"
|
||||
|
||||
begin F26 "PATCH набора: dg_id и период"
|
||||
http PATCH "/datasets/$DS_NDVI" "${JSON[@]}" -d "{\"dg_id\":\"$GRP\",\"start_date\":\"2026-01-01\",\"end_date\":\"2026-03-31\",\"description\":\"NDVI за I квартал 2026\"}"
|
||||
need_code 200; need_eq "$(jb .dg_id)" "$GRP" dg_id; need_eq "$(jb .start_date)" 2026-01-01 start_date
|
||||
need_eq "$(jb .end_date)" 2026-03-31 end_date; need_eq "$(jb .description)" "NDVI за I квартал 2026" description
|
||||
end "200; dg_id, start_date 2026-01-01, end_date 2026-03-31, description обновлены"
|
||||
|
||||
begin F27 "PUT набора: полная замена метаданных"
|
||||
http PUT "/datasets/$DS_PLOTS" "${JSON[@]}" -d "{\"category_id\":\"$CAT_ECO\",\"name\":\"Участки СКО (ред.)\",\"description\":\"Полигоны участков\",\"unit\":\"га\",\"dg_id\":\"$GRP\",\"start_date\":\"2025-01-01\",\"end_date\":\"2025-12-31\"}"
|
||||
need_code 200; need_eq "$(jb .name)" "Участки СКО (ред.)" name; need_eq "$(jb .unit)" "га" unit
|
||||
need_eq "$(jb .dg_id)" "$GRP" dg_id; need_eq "$(jb .end_date)" 2025-12-31 end_date
|
||||
end "200; name «Участки СКО (ред.)», unit га, dg_id, период 2025"
|
||||
|
||||
begin N07 "PATCH: end_date раньше start_date"
|
||||
http PATCH "/datasets/$DS_PLOTS" "${JSON[@]}" -d '{"start_date":"2025-12-31","end_date":"2025-01-01"}'
|
||||
need_code 422; need_has "$(jb .error)" 'end_date "2025-01-01" is before start_date "2025-12-31"' сообщение
|
||||
end "422 «$(jb .error)»"
|
||||
begin N08 "PATCH: нередактируемое поле file_type"
|
||||
http PATCH "/datasets/$DS_PLOTS" "${JSON[@]}" -d '{"file_type":"raster"}'
|
||||
need_code 422; need_eq "$(jb .error)" "unknown field: file_type" сообщение
|
||||
end "422 «$(jb .error)»"
|
||||
begin N09 "PUT: отсутствуют обязательные поля"
|
||||
http PUT "/datasets/$DS_PLOTS" "${JSON[@]}" -d '{"name":"Только имя"}'
|
||||
need_code 422
|
||||
need_eq "$(jb '[.errors.CategoryID,.errors.Description,.errors.Unit]|join(",")')" "is required,is required,is required" errors
|
||||
end "422 CategoryID, Description, Unit «is required»"
|
||||
|
||||
begin F28 "Свежайший набор группы /latest (по max end_date, фильтры file_type, status)"
|
||||
http GET "/dataset-groups/$GRP/latest"; need_code 200; need_eq "$(jb .id)" "$DS_NDVI" "latest"; l1="$(jb .end_date)"
|
||||
http GET "/dataset-groups/$GRP/latest?file_type=vector_with_kato"; need_code 200; need_eq "$(jb .id)" "$DS_POP" "latest vector_with_kato"; l2="$(jb .end_date)"
|
||||
http GET "/dataset-groups/$GRP/latest?status=failed"; need_code 404
|
||||
end "200; NDVI (end_date $l1); с file_type=vector_with_kato — набор населения ($l2); status=failed → $CODE"
|
||||
|
||||
begin F29 "Обновление группы"
|
||||
http PUT "/dataset-groups/$GRP" "${JSON[@]}" -d "{\"code\":\"$P-population-nko\",\"name\":\"Население и NDVI СКО\",\"description\":\"Приёмочный тест\"}"
|
||||
need_code 200; need_eq "$(jb .name)" "Население и NDVI СКО" name
|
||||
end "200; name «Население и NDVI СКО»"
|
||||
|
||||
begin F30 "Удаление группы, на которую ссылаются наборы"
|
||||
http DELETE "/dataset-groups/$GRP"
|
||||
need_code 409
|
||||
end "409 «$(jb .error)»"
|
||||
|
||||
begin F31 "Список групп с фильтром code"
|
||||
http GET "/dataset-groups?code=$P-population-nko"
|
||||
need_code 200; need_eq "$(jb .total)" 1 total; need_eq "$(jb '.data[0].id')" "$GRP" группа
|
||||
end "200; total $(jb .total)"
|
||||
|
||||
begin F32 "Удаление пустой группы"
|
||||
http POST /dataset-groups "${JSON[@]}" -d "{\"code\":\"$P-empty\",\"name\":\"Пустая\"}"
|
||||
need_code 201; eg="$(jb .id)"; c1="$CODE"
|
||||
http DELETE "/dataset-groups/$eg"; need_code 204
|
||||
[ "$CODE" = 204 ] || GRPS="$eg $GRPS"
|
||||
end "$c1 → $CODE"
|
||||
|
||||
begin N10 "Группа: код не slug («Bad Code!»)"
|
||||
http POST /dataset-groups "${JSON[@]}" -d '{"code":"Bad Code!","name":"Плохой код"}'
|
||||
need_code 422; need_eq "$(jb .errors.Code)" "is invalid" errors.Code
|
||||
[ "$CODE" = 201 ] && GRPS="$(jb .id) $GRPS"
|
||||
end "422 $(jbc)"
|
||||
|
||||
begin F33 "Список наборов: пагинация"
|
||||
http GET "/datasets?category_id=$CAT_ECO&page_size=2&page=1"
|
||||
need_code 200; need_eq "$(jb .total)" 4 total; need_eq "$(jb .page_size)" 2 page_size; need_eq "$(jb .total_pages)" 2 total_pages
|
||||
p1="$(jb '[.data[].id]|join(" ")')"
|
||||
http GET "/datasets?category_id=$CAT_ECO&page_size=2&page=2"
|
||||
need_code 200; p2="$(jb '[.data[].id]|join(" ")')"
|
||||
all="$(printf '%s\n' $p1 $p2 | sort -u | wc -l | tr -d ' ')"
|
||||
need_eq "$all" 4 "уникальных записей на 2 страницах"
|
||||
end "200; total 4, page_size 2, total_pages 2; страницы 1 и 2 без пересечений"
|
||||
|
||||
begin F34 "Фильтры dg_id + end_date_from, file_type + status, category_id + start_date_to"
|
||||
http GET "/datasets?dg_id=$GRP&end_date_from=2025-01-01"; need_code 200; n1="$(jb .total)"
|
||||
need_eq "$(jb '[.data[].id]|sort|join(" ")')" "$(printf '%s\n' "$DS_PLOTS" "$DS_NDVI" | sort | tr '\n' ' ' | sed 's/ $//')" "dg_id+end_date_from"
|
||||
http GET "/datasets?category_id=$CAT_ECO&file_type=vector_with_kato&status=ready"; need_code 200; n2="$(jb .total)"
|
||||
need_eq "$(jb '[.data[].id]|sort|join(" ")')" "$(printf '%s\n' "$DS_POP" "$DS_LIVE" | sort | tr '\n' ' ' | sed 's/ $//')" "file_type+status"
|
||||
http GET "/datasets?category_id=$CAT_ECO&start_date_to=2024-01-01"; need_code 200; n3="$(jb .total)"
|
||||
need_eq "$(jb '[.data[].id]|join(" ")')" "$DS_POP" "category_id+start_date_to"
|
||||
end "200; $n1 / $n2 / $n3 записи соответственно, состав соответствует условиям"
|
||||
|
||||
begin N11 "Фильтр: некорректная дата 2025-31-12"
|
||||
http GET "/datasets?start_date_from=2025-31-12"
|
||||
need_code 400; need_eq "$(jb .error)" "start_date_from must be YYYY-MM-DD" сообщение
|
||||
end "400 «$(jb .error)»"
|
||||
begin N12 "Пагинация: page_size=0"
|
||||
http GET "/datasets?page_size=0"
|
||||
need_code 400; need_eq "$(jb .error)" "invalid page_size" сообщение
|
||||
end "400 «$(jb .error)»"
|
||||
begin N13 "Фильтр: file_type=shapefile"
|
||||
http GET "/datasets?file_type=shapefile"
|
||||
need_code 400; need_has "$(jb .error)" "file_type must be one of" сообщение
|
||||
end "400 «$(jb .error)»"
|
||||
|
||||
# ====================================================================== 5.4
|
||||
printf 'just text\n' > "$TMP/notes.txt"
|
||||
printf 'this is not a tiff\n' > "$TMP/fake.tif"
|
||||
printf 'not json at all\n' > "$TMP/bad.geojson"
|
||||
up() { http POST /datasets "$@"; [ "$CODE" = 201 ] && track_ds "$(jb .id)"; return 0; }
|
||||
negup() { # ID desc expected-code expected-substring curl-args...
|
||||
local id="$1" desc="$2" ec="$3" sub="$4"; shift 4
|
||||
begin "$id" "$desc"; up "$@"; need_code "$ec"; need_has "$(jb .error)" "$sub" сообщение
|
||||
end "$CODE «$(jb .error)»"
|
||||
}
|
||||
negup N14 "Загрузка: расширение .txt" 422 'extension ".txt" is not allowed for file_type "vector"' \
|
||||
-F "file=@$TMP/notes.txt" -F file_type=vector -F "category_id=$CAT_SOIL" -F name=x
|
||||
negup N15 "Загрузка: .tif без TIFF-сигнатуры" 422 "file is not a valid TIFF/BigTIFF GeoTIFF" \
|
||||
-F "file=@$TMP/fake.tif" -F file_type=raster -F "category_id=$CAT_SOIL" -F name=x
|
||||
negup N16 "Загрузка: .geojson не JSON" 422 "file is not valid GeoJSON" \
|
||||
-F "file=@$TMP/bad.geojson" -F file_type=vector -F "category_id=$CAT_SOIL" -F name=x
|
||||
negup N17 "Загрузка: неизвестный file_type" 422 "file_type must be one of: vector_with_kato, vector, raster" \
|
||||
-F "file=@$DATA/plots_nko.geojson" -F file_type=shapefile -F "category_id=$CAT_SOIL" -F name=x
|
||||
negup N18 "Загрузка: .tif для типа vector" 422 'extension ".tif" is not allowed for file_type "vector"' \
|
||||
-F "file=@$NDVI" -F file_type=vector -F "category_id=$CAT_SOIL" -F name=x
|
||||
negup N19 "Загрузка: нет файла" 400 "file is required" \
|
||||
-F file_type=vector -F "category_id=$CAT_SOIL" -F name=x
|
||||
negup N20 "Загрузка: нет category_id" 422 "category_id must be a valid UUID" \
|
||||
-F "file=@$DATA/plots_nko.geojson" -F file_type=vector -F name=x
|
||||
|
||||
begin N21 "Загрузка без name (поле необязательно по OpenAPI)"
|
||||
up -F "file=@$DATA/plots_nko.geojson" -F file_type=vector -F "category_id=$CAT_SOIL"
|
||||
need_code 201; DS_N21="$(jb .id)"; N21_KEY="$(jb .storage_key)"; need_eq "$(jb .name)" plots_nko.geojson name
|
||||
end "201; name = имя файла ($(jb .name))"
|
||||
|
||||
negup N22 "Загрузка: end_date раньше start_date" 422 'end_date "2025-01-01" is before start_date "2025-12-31"' \
|
||||
-F "file=@$DATA/plots_nko.geojson" -F file_type=vector -F "category_id=$CAT_SOIL" -F name=x -F start_date=2025-12-31 -F end_date=2025-01-01
|
||||
negup N23 "Загрузка: несуществующая категория" 422 "category does not exist" \
|
||||
-F "file=@$DATA/plots_nko.geojson" -F file_type=vector -F "category_id=00000000-0000-4000-8000-000000000000" -F name=x
|
||||
negup N24 "Загрузка: невалидный JSON в meta" 422 "meta must be valid JSON" \
|
||||
-F "file=@$DATA/plots_nko.geojson" -F file_type=vector -F "category_id=$CAT_SOIL" -F name=x -F 'meta={bad json'
|
||||
begin N25 "Загрузка: не multipart"
|
||||
http POST /datasets "${JSON[@]}" -d '{"file_type":"vector"}'
|
||||
need_code 400; need_eq "$(jb .error)" "request must be multipart/form-data" сообщение
|
||||
end "400 «$(jb .error)»"
|
||||
|
||||
begin N26 "Несуществующие id на 10 эндпоинтах"
|
||||
X=7d3c9a52-1f4e-4b8a-9c2d-5e6f7a8b9c0d; codes=""
|
||||
for p in "/datasets/$X" "/datasets/$X.geojson" "/datasets/$X.cog" "/datasets/$X/download" "/datasets/$X/observations" \
|
||||
"/datasets/$X/sld" "/categories/$X" "/dataset-groups/$X" "/dataset-groups/$X/latest" "/landscape/runs/$X"; do
|
||||
http GET "$p"; codes="$codes $CODE"; [ "$CODE" = 404 ] || bad "$p → $CODE"
|
||||
done
|
||||
end "404 на всех 10 эндпоинтах:$codes"
|
||||
|
||||
begin N27 "Некорректный формат id (4 эндпоинта)"
|
||||
codes=""
|
||||
for p in /datasets/not-a-uuid /categories/not-a-uuid /dataset-groups/not-a-uuid /landscape/runs/not-a-uuid; do
|
||||
http GET "$p"; codes="$codes $CODE"; [ "$CODE" = 400 ] || bad "$p → $CODE"; [ "$(jb .error)" = "invalid id" ] || bad "$p: $(jb .error)"
|
||||
done
|
||||
end "400 «invalid id» на всех 4:$codes"
|
||||
|
||||
begin N28 "Битый JSON"
|
||||
http POST /categories "${JSON[@]}" -d '{"code":"x","name":'
|
||||
need_code 400; need_eq "$(jb .error)" "invalid request body" сообщение
|
||||
end "400 «$(jb .error)»"
|
||||
begin N29 "Мусорное тело 200 КБ"
|
||||
head -c 204800 /dev/urandom > "$TMP/garbage.bin"
|
||||
http POST /categories "${JSON[@]}" --data-binary "@$TMP/garbage.bin"
|
||||
need_code 400; need_eq "$(jb .error)" "invalid request body" сообщение
|
||||
end "400 «$(jb .error)»"
|
||||
begin N30 "Неизвестное поле в JSON"
|
||||
http POST /categories "${JSON[@]}" -d "{\"code\":\"$P-unknown\",\"name\":\"x\",\"color\":\"red\"}"
|
||||
need_code 400; need_eq "$(jb .error)" "invalid request body" сообщение
|
||||
[ "$CODE" = 201 ] && CATS="$(jb .id) $CATS"
|
||||
end "400 «$(jb .error)»"
|
||||
begin N37 "JSON-тело 2 200 048 байт (> 1 МиБ)"
|
||||
{ printf '{"code":"%s-big","name":"' "$P"; head -c $((2200048 - 25 - ${#P})) /dev/zero | tr '\0' 'a'; printf '"}'; } > "$TMP/big.json"
|
||||
http POST /categories "${JSON[@]}" --data-binary "@$TMP/big.json"
|
||||
need_code 400; need_eq "$(jb .error)" "invalid request body" сообщение
|
||||
[ "$CODE" = 201 ] && CATS="$(jb .id) $CATS"
|
||||
end "400 «$(jb .error)» (тело $(wc -c < "$TMP/big.json" | tr -d ' ') байт)"
|
||||
|
||||
runreq() { http POST /landscape/runs "${JSON[@]}" -d "$1"; [ "$CODE" = 202 ] && RUNS="$RUNS $(jb .id)"; return 0; }
|
||||
begin N31 "ЛП: from_stage=7"
|
||||
runreq '{"from_stage":7}'; need_code 422; need_eq "$(jb .errors.FromStage)" "must be 3 or less" errors.FromStage
|
||||
end "422 $(jbc)"
|
||||
begin N32 "ЛП: tie_priority не перестановка"
|
||||
runreq '{"params":{"tie_priority":[1,1,2,3,4]}}'; need_code 422; need_has "$(jb .error)" "tie_priority must be a permutation of 1..5" сообщение
|
||||
end "422 «$(jb .error)»"
|
||||
begin N33 "ЛП: override с некорректным UUID"
|
||||
runreq '{"overrides":{"lp-z-lcr":"not-a-uuid"}}'; need_code 422; need_eq "$(jb '.errors["Overrides[lp-z-lcr]"]')" "must be a valid UUID" errors
|
||||
end "422 $(jbc)"
|
||||
begin N34 "ЛП: неизвестный слот"
|
||||
http POST /landscape/slots/lp-unknown-slot -F "file=@$NDVI"; need_code 404
|
||||
[ "$CODE" = 201 ] && track_ds "$(jb .id)"
|
||||
end "404 «$(jb .error)»"
|
||||
begin N35 "ЛП: не-GeoTIFF в слот"
|
||||
http POST /landscape/slots/lp-z-lcr -F "file=@$DATA/plots_nko.geojson"; need_code 422
|
||||
need_has "$(jb .error)" 'extension ".geojson" is not allowed for file_type "raster"' сообщение
|
||||
[ "$CODE" = 201 ] && track_ds "$(jb .id)"
|
||||
end "422 «$(jb .error)»"
|
||||
begin N36 "ЛП: window_size=100"
|
||||
runreq '{"params":{"window_size":100}}'; need_code 422; need_has "$(jb .error)" "window_size must be in 256..8192" сообщение
|
||||
end "422 «$(jb .error)»"
|
||||
|
||||
begin N38 "Загрузка 100 МБ мусора с сигнатурой TIFF"
|
||||
{ printf 'II*\000'; head -c 104857600 /dev/urandom; } > "$TMP/garbage.tif"
|
||||
up -F "file=@$TMP/garbage.tif" -F file_type=raster -F "category_id=$CAT_SOIL" -F "name=$P garbage"
|
||||
need_code 201; DS_G="$(jb .id)"; up_time="$TIME"
|
||||
wait_ds "$DS_G" ready; need_eq "$WS" failed status
|
||||
http GET "/datasets/$DS_G"; pe="$(jb .parse_error)"; need_has "$pe" gdal_translate parse_error
|
||||
http GET /readyz; need_code 200; need_eq "$(jb .ready)" true "/readyz ready"
|
||||
rm -f "$TMP/garbage.tif"
|
||||
end "201 (104 857 604 байт за ${up_time} с) → status $WS, parse_error от gdal_translate; /readyz 200, сервис доступен"
|
||||
|
||||
begin F35 "Удаление набора (строка и объект в S3)"
|
||||
http DELETE "/datasets/$DS_N21"; need_code 204; c1="$CODE"
|
||||
http GET "/datasets/$DS_N21"; need_code 404; c2="$CODE"
|
||||
detail="$c1; GET → $c2"
|
||||
if [ "$LOCAL_CHECKS" = 1 ]; then
|
||||
prefix="${N21_KEY%/*}"
|
||||
n="$(docker exec "$MINIO_C" sh -c "MC_CONFIG_DIR=/tmp/.mc MC_HOST_local=http://\$MINIO_ROOT_USER:\$MINIO_ROOT_PASSWORD@127.0.0.1:9000 mc ls --recursive local/geofiles/$prefix/ 2>/dev/null | wc -l" | tr -d ' ' || echo "?")"
|
||||
need_eq "$n" 0 "объектов в префиксе MinIO"
|
||||
detail="$detail; объектов в префиксе MinIO $prefix/ — $n"
|
||||
else
|
||||
detail="$detail; проверка MinIO пропущена (LOCAL_CHECKS=0)"
|
||||
fi
|
||||
end "$detail"
|
||||
|
||||
# ====================================================================== 5.5
|
||||
psql_q() { docker exec -i "$PG_C" psql -U gis -d gis -At -F '|' -c "$1"; }
|
||||
begin F36 "Геометрии наборов в PostGIS: тип, SRID, ST_IsValid"
|
||||
if [ "$LOCAL_CHECKS" = 1 ]; then
|
||||
out="$(psql_q "SELECT id, ST_GeometryType(geometry), ST_SRID(geometry), ST_IsValid(geometry) FROM datasets WHERE id IN ('$DS_PLOTS','$DS_POP','$DS_LIVE','$DS_NDVI')" 2>&1 || true)"
|
||||
echo "$out" > "$RESULTS/F36.txt"
|
||||
for pair in "$DS_PLOTS:ST_MultiPolygon" "$DS_POP:ST_MultiPoint" "$DS_LIVE:ST_MultiPoint" "$DS_NDVI:ST_Polygon"; do
|
||||
id="${pair%%:*}"; want="${pair#*:}"
|
||||
row="$(echo "$out" | grep "^$id|" || true)"
|
||||
need_eq "$(echo "$row" | cut -d'|' -f2-4)" "$want|4326|t" "геометрия $id"
|
||||
done
|
||||
end "участки ST_MultiPolygon; население и КРС ST_MultiPoint; NDVI ST_Polygon; у всех SRID 4326, valid = t"
|
||||
else skip "LOCAL_CHECKS=0 (нет доступа к контейнеру PostGIS)"; fi
|
||||
|
||||
begin F37 "Наблюдения dataset_observations соединяются с districts по КАТО"
|
||||
if [ "$LOCAL_CHECKS" = 1 ]; then
|
||||
out="$(psql_q "SELECT count(*), count(DISTINCT o.kato_code), count(*) FILTER (WHERE d.kato IS NULL) FROM dataset_observations o LEFT JOIN districts d ON d.kato = o.kato_code WHERE o.dataset_id IN ('$DS_POP','$DS_LIVE')" 2>&1 || true)"
|
||||
need_eq "$out" "84|14|0" "наблюдений|кодов КАТО|сирот"
|
||||
kz="$(psql_q "SELECT d.name || ' (' || d.kato || ') — ' || string_agg(o.value::text, ' / ' ORDER BY o.start_date) FROM dataset_observations o JOIN districts d ON d.kato = o.kato_code WHERE o.dataset_id = '$DS_POP' AND o.kato_code = '595000000' GROUP BY d.name, d.kato" 2>&1 || true)"
|
||||
end "84 наблюдения, 14 кодов КАТО, orphan_kato 0 ($out); $kz"
|
||||
else skip "LOCAL_CHECKS=0 (нет доступа к контейнеру PostGIS)"; fi
|
||||
|
||||
begin F38 "Объекты в бакете geofiles MinIO"
|
||||
if [ "$LOCAL_CHECKS" = 1 ]; then
|
||||
keys=""
|
||||
for d in "$DS_PLOTS" "$DS_POP" "$DS_LIVE" "$DS_NDVI"; do
|
||||
http GET "/datasets/$d"; keys="$keys $(jb '[.storage_key, .cog_storage_key, .sld_storage_key] | map(select(. != null)) | join(" ")')"
|
||||
done
|
||||
http GET "/dataset-groups/$GRP"; keys="$keys $(jb .sld_storage_key)"
|
||||
found=0; total=0
|
||||
for k in $keys; do
|
||||
total=$((total + 1))
|
||||
if docker exec "$MINIO_C" sh -c "MC_CONFIG_DIR=/tmp/.mc MC_HOST_local=http://\$MINIO_ROOT_USER:\$MINIO_ROOT_PASSWORD@127.0.0.1:9000 mc stat local/geofiles/$k" >/dev/null 2>&1; then
|
||||
found=$((found + 1))
|
||||
else bad "нет объекта $k"; fi
|
||||
done
|
||||
du="$(docker exec "$MINIO_C" sh -c 'MC_CONFIG_DIR=/tmp/.mc MC_HOST_local=http://$MINIO_ROOT_USER:$MINIO_ROOT_PASSWORD@127.0.0.1:9000 mc du local/geofiles' 2>/dev/null | tr -s ' \t' ' ' || true)"
|
||||
[ "$total" -eq 7 ] || bad "ожидалось 7 ключей (4 исходных файла, COG, SLD набора, SLD группы), получено $total"
|
||||
end "$found/$total объектов найдено: plots_nko.geojson, population_kato.geojson, livestock_kato.gpkg, ndvi_nko.tif, cog/ndvi_nko.tif, sld/style.sld, groups/…/sld/group_style.sld; бакет: $du"
|
||||
else skip "LOCAL_CHECKS=0 (нет доступа к контейнеру MinIO)"; fi
|
||||
|
||||
# ====================================================================== 5.6 ЛП
|
||||
LPIN="$TMP/lpin"; LPCACHE="$TMP/lpcache"; mkdir -p "$LPCACHE"
|
||||
"$PYTHON" "$HERE/lp_gen.py" --out "$LPIN" > "$RESULTS/lp_gen.txt"
|
||||
SLOTS="lp-z-biodiversity lp-z-lcr lp-z-soil-protection lp-z-wr lp-u-surf-water lp-u-soil lp-u-crop lp-u-veg lp-p-irrigation lp-p-snow lp-p-crop lp-p-pasture lp-p-tourism"
|
||||
slot_id_file() { echo "$TMP/slot-$1"; }
|
||||
|
||||
begin L02 "Загрузка 13 входных растров в слоты POST /landscape/slots/{code}"
|
||||
codes=""
|
||||
for s in $SLOTS; do
|
||||
http POST "/landscape/slots/$s" -F "file=@$LPIN/$s.tif" -F "name=$P $s" -F start_date=2025-01-01 -F end_date=2025-12-31
|
||||
codes="$codes $CODE"; [ "$CODE" = 201 ] || bad "$s → $CODE: $(snippet)"
|
||||
id="$(jb .id)"; echo "$id" > "$(slot_id_file "$s")"; track_ds "$id"
|
||||
done
|
||||
for s in $SLOTS; do wait_ds "$(cat "$(slot_id_file "$s")")" ready; [ "$WS" = ready ] || bad "$s: status $WS"; done
|
||||
http GET /landscape/slots
|
||||
for s in $SLOTS; do
|
||||
got="$(jq -r --arg s "$s" '.[]|select(.code==$s)|"\(.latest.id) \(.latest.status)"' "$BODY")"
|
||||
need_eq "$got" "$(cat "$(slot_id_file "$s")") ready" "слот $s"
|
||||
done
|
||||
end "201 для всех 13 слотов; GET /landscape/slots — у всех слотов latest = загруженный растр, status ready"
|
||||
|
||||
begin L03 "Запуск полного прогона {\"from_stage\":0}"
|
||||
runreq '{"from_stage":0}'; need_code 202; RUN1="$(jb .id)"; need_eq "$(jb .status)" queued status
|
||||
need_eq "$(jq -S -c .params "$BODY")" '{"tie_priority":[1,2,3,4,5],"v1_min_s_e12":3,"v1_min_s_e3":4,"v2_min_s_e3":3,"v2_min_s_e4":5,"window_size":2048}' params
|
||||
end "202; run ${RUN1:0:8} поставлен в очередь, params по умолчанию (window_size 2048, tie_priority [1..5], пороги 3/4/3/5)"
|
||||
|
||||
begin L04 "Завершение прогона"
|
||||
wait_run "$RUN1"; need_eq "$RS" succeeded status; ns="$(node_statuses "$RUN1")"
|
||||
need_eq "$ns" "00:succeeded 01:succeeded 02:succeeded 03:succeeded 10:succeeded 20:succeeded 21:succeeded 30:succeeded 31:succeeded" узлы
|
||||
need_eq "$(jq -r .error "$TMP/run-$RUN1.json")" null error
|
||||
end "$RS за $("$PYTHON" "$HERE/lib/helpers.py" durations "$TMP/run-$RUN1.json"); error null; все 9 узлов succeeded"
|
||||
|
||||
begin L05 "Результаты GET /landscape/results"
|
||||
outs="$(run_outputs "$RUN1")"
|
||||
# Results list only ready layers; the COG conversion of the outputs is asynchronous.
|
||||
for id in $(jq -r '.[].dataset_id' <<< "$outs"); do wait_ds "$id" ready; [ "$WS" = ready ] || bad "выход $id: status $WS"; done
|
||||
http GET /landscape/results; need_code 200; cp "$BODY" "$TMP/results.json"
|
||||
need_eq "$(jb 'length')" 15 "число групп"
|
||||
for g in $(jq -r '.[].group' <<< "$outs"); do
|
||||
want="$(jq -r --arg g "$g" '.[]|select(.group==$g)|.dataset_id' <<< "$outs")"
|
||||
got="$(jq -r --arg g "$g" '.[]|select(.group==$g)|"\(.latest.id) \(.latest.status) \(.cog_url)"' "$TMP/results.json")"
|
||||
need_eq "$got" "$want ready /datasets/$want.cog" "результат $g"
|
||||
done
|
||||
first="$(jq -r '.[0].dataset_id' <<< "$outs")"
|
||||
http GET "/datasets/$first"; need_eq "$(jb .automated)" true automated; need_eq "$(jb .meta.lp_run_id)" "$RUN1" meta.lp_run_id
|
||||
[ "$(jb .meta.lp_node)" != null ] || bad "meta.lp_node"; [ "$(jb '.meta.inputs|length')" -ge 1 ] 2>/dev/null || bad "meta.inputs"
|
||||
fmt_ok=0
|
||||
for id in $(jq -r '.[].dataset_id' <<< "$outs"); do
|
||||
curl -fsS -o "$LPCACHE/$id.cog.tif" "$BASE_URL/datasets/$id.cog" || rm -f "$LPCACHE/$id.cog.tif"
|
||||
ri="$("$PYTHON" "$HERE/lib/helpers.py" rinfo "$LPCACHE/$id.cog.tif" 2>/dev/null || echo '{}')"
|
||||
f="$(jq -r '"\(.width)x\(.height) EPSG:\(.epsg) \(.layout) \(.dtype) \(.nodata|floor)"' <<< "$ri" 2>/dev/null || true)"
|
||||
if [ "$f" = "3000x2000 EPSG:32642 COG int16 -9999" ]; then fmt_ok=$((fmt_ok + 1)); else bad "COG $id: $f"; fi
|
||||
done
|
||||
end "200; 15 групп (узлы 00…31), все ready, cog_url /datasets/{id}.cog; выходной набор: automated true, meta.lp_run_id, lp_node, inputs. $fmt_ok/15 COG: 3000×2000, EPSG:32642, LAYOUT=COG, Int16, NoData −9999"
|
||||
|
||||
begin L07 "Попиксельная сверка 15 выходов с эталонной реализацией"
|
||||
if "$PYTHON" "$HERE/lp_verify.py" --base-url "$BASE_URL" --run "$RUN1" --cache "$LPCACHE" > "$RESULTS/L07_lp_verify.txt" 2>&1; then :; else bad "lp_verify: $(tail -n 3 "$RESULTS/L07_lp_verify.txt" | tr '\n' ' ')"; fi
|
||||
tot="$(grep 'TOTAL mismatching' "$RESULTS/L07_lp_verify.txt" || echo 'нет итога')"
|
||||
end "15 выходов × 6 000 000 пикселей; «$tot» (журнал: results/functional-$STAMP/L07_lp_verify.txt)"
|
||||
|
||||
begin L08 "Выборочная проверка 9 пикселей (Z_int, U_int, E, C_k, V1–V3)"
|
||||
PTS="20,20;1450,1000;1575,1000;350,350;520,320;500,1500;1000,700;2200,1500;2900,1900"
|
||||
if "$PYTHON" "$HERE/lp_verify.py" --base-url "$BASE_URL" --run "$RUN1" --cache "$LPCACHE" --points "$PTS" > "$RESULTS/L08_points.txt" 2>&1; then :; else bad "$(tail -n 1 "$RESULTS/L08_points.txt")"; fi
|
||||
end "$(tail -n 1 "$RESULTS/L08_points.txt"): NoData (20,20), дыра NoData в U crop (1450,1000), crop и veg NoData (1575,1000), P snow = 0 и 6"
|
||||
|
||||
begin L09 "Повторный прогон {} без изменений входов"
|
||||
runreq '{}'; need_code 202; RUN2="$(jb .id)"; wait_run "$RUN2"; need_eq "$RS" succeeded status
|
||||
ns="$(node_statuses "$RUN2")"
|
||||
need_eq "$ns" "00:skipped 01:skipped 02:skipped 03:skipped 10:skipped 20:skipped 21:skipped 30:skipped 31:skipped" узлы
|
||||
need_eq "$(run_outputs "$RUN2")" "$(run_outputs "$RUN1")" "выходные наборы"
|
||||
end "202; run ${RUN2:0:8} $RS за $("$PYTHON" "$HERE/lib/helpers.py" durations "$TMP/run-$RUN2.json"); все 9 узлов skipped, выходные наборы совпадают с первым прогоном"
|
||||
|
||||
begin L10 "Прогон {\"from_stage\":3}"
|
||||
runreq '{"from_stage":3}'; need_code 202; RUN3="$(jb .id)"; wait_run "$RUN3"; need_eq "$RS" succeeded status
|
||||
ns="$(node_statuses "$RUN3")"
|
||||
need_eq "$ns" "00:skipped 01:skipped 02:skipped 03:skipped 10:skipped 20:skipped 21:skipped 30:succeeded 31:succeeded" узлы
|
||||
new="$(jq -r '[.nodes[]|select(.node_code=="30" or .node_code=="31")|.outputs[]?.dataset_id]|join(" ")' "$TMP/run-$RUN3.json" || true)"
|
||||
for id in $new; do run_outputs "$RUN1" | grep -q "$id" && bad "узел стадии 3 переиспользовал $id"; done
|
||||
end "202; run ${RUN3:0:8} $RS за $("$PYTHON" "$HERE/lib/helpers.py" durations "$TMP/run-$RUN3.json"); узлы 00–21 skipped, узлы 30 и 31 succeeded (новые выходные наборы)"
|
||||
|
||||
begin L11 "Прогон с параметрами tie_priority [5,4,3,2,1], пороги 4/5/4/4"
|
||||
runreq '{"params":{"tie_priority":[5,4,3,2,1],"v1_min_s_e12":4,"v1_min_s_e3":5,"v2_min_s_e3":4,"v2_min_s_e4":4}}'
|
||||
need_code 202; RUN4="$(jb .id)"; wait_run "$RUN4"; need_eq "$RS" succeeded status
|
||||
ns="$(node_statuses "$RUN4")"
|
||||
need_eq "$ns" "00:skipped 01:skipped 02:skipped 03:skipped 10:skipped 20:skipped 21:skipped 30:succeeded 31:skipped" узлы
|
||||
end "202; run ${RUN4:0:8} $RS за $("$PYTHON" "$HERE/lib/helpers.py" durations "$TMP/run-$RUN4.json"); пересчитан только узел 30 (V1–V3), остальные skipped"
|
||||
|
||||
begin L12 "Сверка V1–V3 с пользовательскими параметрами"
|
||||
wait_outputs "$RUN4"
|
||||
if "$PYTHON" "$HERE/lp_verify.py" --base-url "$BASE_URL" --run "$RUN4" --cache "$LPCACHE" > "$RESULTS/L12_lp_verify.txt" 2>&1; then :; else bad "lp_verify: $(tail -n 3 "$RESULTS/L12_lp_verify.txt" | tr '\n' ' ')"; fi
|
||||
v="$(grep -E '^lp-v[123]-' "$RESULTS/L12_lp_verify.txt" | awk '{printf "%s mismatch %s; ", $1, $6}' || true)"
|
||||
end "$v; «$(grep 'TOTAL mismatching' "$RESULTS/L12_lp_verify.txt" || echo 'нет итога')» (все 15 выходов)"
|
||||
|
||||
begin L13 "Невыровненный растр (1500×1000, 200 м) в слоте lp-u-veg"
|
||||
http POST /landscape/slots/lp-u-veg -F "file=@$LPIN/misaligned-lp-u-veg.tif" -F "name=$P misaligned" -F start_date=2025-01-01 -F end_date=2025-12-31
|
||||
need_code 201; DS_MIS="$(jb .id)"; track_ds "$DS_MIS"; wait_ds "$DS_MIS" ready; need_eq "$WS" ready "status растра"
|
||||
runreq '{}'; need_code 202; RUN5="$(jb .id)"; wait_run "$RUN5"; need_eq "$RS" failed status
|
||||
err="$(jq -r .error "$TMP/run-$RUN5.json" || true)"
|
||||
need_has "$err" "input lp-u-veg grid (1500x1000" "сообщение"; need_has "$err" "inputs must be aligned" "сообщение"
|
||||
later="$(jq -r '[.nodes[]|select(.stage>=1)|.status]|unique|join(",")' "$TMP/run-$RUN5.json" || true)"
|
||||
need_eq "$later" pending "узлы стадий 1–3"
|
||||
end "run ${RUN5:0:8} $RS: «${err:0:160}…»; последующие узлы не выполняются ($later)"
|
||||
|
||||
begin L14 "Восстановление: удаление растра, повторный прогон"
|
||||
http DELETE "/datasets/$DS_MIS"; need_code 204; c1="$CODE"
|
||||
http GET /landscape/slots
|
||||
need_eq "$(jq -r '.[]|select(.code=="lp-u-veg")|.latest.id' "$BODY")" "$(cat "$(slot_id_file lp-u-veg)")" "слот lp-u-veg"
|
||||
runreq '{}'; need_code 202; RUN6="$(jb .id)"; wait_run "$RUN6"; need_eq "$RS" succeeded status
|
||||
ns="$(node_statuses "$RUN6")"
|
||||
need_eq "$ns" "00:skipped 01:skipped 02:skipped 03:skipped 10:skipped 20:skipped 21:skipped 30:skipped 31:skipped" узлы
|
||||
end "DELETE $c1; в слоте снова исходный растр; run ${RUN6:0:8} $RS, все узлы skipped"
|
||||
|
||||
begin L15 "Повторный полный прогон с замером ресурсов worker"
|
||||
cpu0=""
|
||||
if [ "$LOCAL_CHECKS" = 1 ] && [ -n "$WORKER_C" ]; then
|
||||
cpu0="$(docker exec "$WORKER_C" sh -c "awk '/usage_usec/{print \$2}' /sys/fs/cgroup/cpu.stat" 2>/dev/null || true)"
|
||||
fi
|
||||
runreq '{"from_stage":0}'; need_code 202; RUN7="$(jb .id)"; c1="$CODE"; wait_run "$RUN7"; need_eq "$RS" succeeded status
|
||||
ns="$(node_statuses "$RUN7")"
|
||||
need_eq "$ns" "00:succeeded 01:succeeded 02:succeeded 03:succeeded 10:succeeded 20:succeeded 21:succeeded 30:succeeded 31:succeeded" узлы
|
||||
res=""
|
||||
if [ -n "$cpu0" ]; then
|
||||
cpu1="$(docker exec "$WORKER_C" sh -c "awk '/usage_usec/{print \$2}' /sys/fs/cgroup/cpu.stat" 2>/dev/null || echo "$cpu0")"
|
||||
peak="$(docker exec "$WORKER_C" sh -c 'cat /sys/fs/cgroup/memory.peak 2>/dev/null || echo 0')"
|
||||
res="; CPU worker $(awk -v a="$cpu0" -v b="$cpu1" 'BEGIN{printf "%.2f", (b-a)/1e6}') с (включая COG-конвертацию), memory.peak $(awk -v p="$peak" 'BEGIN{printf "%.2f", p/1e9}') ГБ (с учётом страничного кэша)"
|
||||
fi
|
||||
end "$c1; run ${RUN7:0:8} $RS за $("$PYTHON" "$HERE/lib/helpers.py" durations "$TMP/run-$RUN7.json")$res"
|
||||
|
||||
# ==================================================================== summary
|
||||
TOTAL=$((PASSN + FAILN + SKIPN))
|
||||
{
|
||||
echo "-----------------------------------------------------------------"
|
||||
echo "ИТОГО: $TOTAL проверок — PASS $PASSN, FAIL $FAILN, SKIP $SKIPN"
|
||||
echo "RESULT: $PASSN/$TOTAL PASS"
|
||||
echo "ЛП прогоны: RUN1=${RUN1:-} (from_stage 0), RUN4=${RUN4:-} (пользовательские параметры)"
|
||||
} | tee -a "$RESULTS/functional.txt"
|
||||
[ "$FAILN" -eq 0 ]
|
||||
78
test/acceptance/lib/helpers.py
Normal file
78
test/acceptance/lib/helpers.py
Normal file
@@ -0,0 +1,78 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Small helpers for functional.sh (numpy + rasterio).
|
||||
|
||||
helpers.py ndvi OUT.tif write the 600x400 EPSG:4326 Float32 NDVI test raster
|
||||
helpers.py rinfo FILE.tif print raster facts as JSON (driver, size, crs, dtype,
|
||||
nodata, layout, compression, overviews)
|
||||
helpers.py durations RUN.json print "total=<s> nodes=<min>..<max>ms" for a run
|
||||
helpers.py elapsed DATASET.json print updated_at - created_at of a dataset, seconds
|
||||
"""
|
||||
import json
|
||||
import sys
|
||||
from datetime import datetime
|
||||
|
||||
|
||||
def ndvi(out):
|
||||
import numpy as np
|
||||
import rasterio
|
||||
from rasterio.transform import from_origin
|
||||
|
||||
w, h = 600, 400 # 0.01 deg pixels, bbox [68.5, 51.5, 74.5, 55.5]
|
||||
y, x = np.mgrid[0:h, 0:w].astype(np.float32)
|
||||
v = 0.45 + 0.35 * np.sin(x / 47.0) * np.cos(y / 31.0) + 0.1 * np.sin((x + y) / 13.0)
|
||||
v = np.clip(v, -0.2, 1.0).astype(np.float32)
|
||||
v[:5, :] = -9999
|
||||
profile = dict(driver="GTiff", width=w, height=h, count=1, dtype="float32", crs="EPSG:4326",
|
||||
transform=from_origin(68.5, 55.5, 0.01, 0.01), nodata=-9999)
|
||||
with rasterio.open(out, "w", **profile) as dst:
|
||||
dst.write(v, 1)
|
||||
|
||||
|
||||
def rinfo(path):
|
||||
import rasterio
|
||||
|
||||
with rasterio.open(path) as d:
|
||||
st = d.tags(ns="IMAGE_STRUCTURE")
|
||||
print(json.dumps({
|
||||
"driver": d.driver, "width": d.width, "height": d.height,
|
||||
"epsg": d.crs.to_epsg() if d.crs else None, "dtype": d.dtypes[0], "nodata": d.nodata,
|
||||
"layout": st.get("LAYOUT"), "compression": st.get("COMPRESSION"),
|
||||
"overviews": len(d.overviews(1)), "bounds": [round(b, 6) for b in d.bounds],
|
||||
}))
|
||||
|
||||
|
||||
def ts(s):
|
||||
if not s:
|
||||
return None
|
||||
s = s.replace("Z", "+00:00")
|
||||
if "." in s: # trim nanoseconds to microseconds for fromisoformat
|
||||
head, rest = s.split(".", 1)
|
||||
frac = "".join(ch for ch in rest if ch.isdigit())
|
||||
tz = rest[len(frac):]
|
||||
s = "%s.%s%s" % (head, frac[:6].ljust(6, "0"), tz)
|
||||
return datetime.fromisoformat(s)
|
||||
|
||||
|
||||
def durations(path):
|
||||
with open(path, encoding="utf-8") as f:
|
||||
run = json.load(f)
|
||||
total = None
|
||||
if run.get("started_at") and run.get("finished_at"):
|
||||
total = (ts(run["finished_at"]) - ts(run["started_at"])).total_seconds()
|
||||
nd = [(ts(n["finished_at"]) - ts(n["started_at"])).total_seconds() * 1000
|
||||
for n in run.get("nodes", []) if n.get("started_at") and n.get("finished_at")]
|
||||
out = "total=%.2fs" % total if total is not None else "total=?"
|
||||
if nd:
|
||||
out += " nodes=%d..%dms" % (min(nd), max(nd))
|
||||
print(out)
|
||||
|
||||
|
||||
def elapsed(path):
|
||||
with open(path, encoding="utf-8") as f:
|
||||
d = json.load(f)
|
||||
print("%.2f" % (ts(d["updated_at"]) - ts(d["created_at"])).total_seconds())
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
cmd, arg = sys.argv[1], sys.argv[2]
|
||||
{"ndvi": ndvi, "rinfo": rinfo, "durations": durations, "elapsed": elapsed}[cmd](arg)
|
||||
11
test/acceptance/lib/hey.Dockerfile
Normal file
11
test/acceptance/lib/hey.Dockerfile
Normal file
@@ -0,0 +1,11 @@
|
||||
# Load generator image for load.sh: hey v0.1.5 (github.com/rakyll/hey) built
|
||||
# from source on the same pinned Go/Alpine images as the application.
|
||||
FROM golang:1.26.9-alpine3.24 AS build
|
||||
RUN CGO_ENABLED=0 go install github.com/rakyll/hey@v0.1.5
|
||||
FROM alpine:3.24
|
||||
COPY --from=build /go/bin/hey /usr/local/bin/hey
|
||||
RUN adduser -D -u 10001 hey
|
||||
USER hey
|
||||
# One-shot CLI container: nothing to health-check.
|
||||
HEALTHCHECK NONE
|
||||
ENTRYPOINT ["hey"]
|
||||
200
test/acceptance/lib/load_analyze.py
Normal file
200
test/acceptance/lib/load_analyze.py
Normal file
@@ -0,0 +1,200 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Aggregates the hey outputs of load.sh into the tables of Приложение 3.
|
||||
|
||||
load_analyze.py RESULTS_DIR
|
||||
|
||||
RESULTS_DIR/manifest.csv rows: scenario,mode,label,clients,file
|
||||
mode par = the files of the scenario ran at the same time (duration = max)
|
||||
seq = they ran one after another (duration = sum)
|
||||
Each file is a default (summary) hey output. Per scenario the program reports
|
||||
requests, clients, duration, RPS, success %, status codes, transport errors,
|
||||
min/avg/P50/P90/P95/P99/max (ms). As in the protocol, percentiles of a
|
||||
scenario made of several hey processes/runs are averages weighted by the
|
||||
number of responses; min/max are exact.
|
||||
|
||||
Criteria (Приложение 3, табл. 8): K1 success >= 99 %, K2 no 5xx, K3 P95 <=
|
||||
2000 ms; K4 (S6 vs S1): /readyz 200 and S6 P50 <= 2 x S1 P50 + 1 ms.
|
||||
|
||||
Writes summary.csv, summary.json, detail.csv into RESULTS_DIR and prints the
|
||||
tables. Exit code 1 when a criterion fails.
|
||||
"""
|
||||
import csv
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import sys
|
||||
|
||||
PCTS = ["10", "25", "50", "75", "90", "95", "99"]
|
||||
|
||||
|
||||
def parse_hey(path):
|
||||
txt = open(path, encoding="utf-8", errors="replace").read()
|
||||
|
||||
def num(label):
|
||||
m = re.search(r"^\s*%s:\s+([0-9.]+)" % re.escape(label), txt, re.M)
|
||||
return float(m.group(1)) if m else None
|
||||
|
||||
r = {"total_s": num("Total"), "slowest_s": num("Slowest"), "fastest_s": num("Fastest"),
|
||||
"average_s": num("Average"), "rps": num("Requests/sec"), "codes": {}, "errors": 0, "pct": {}}
|
||||
for p, v in re.findall(r"^\s*(\d+)%+ in ([0-9.]+) secs", txt, re.M):
|
||||
if p != "0": # hey 0.1.5 prints "%%" and a bogus "0%% in 0.0000" row for tiny samples
|
||||
r["pct"][p] = float(v)
|
||||
sec = txt.split("Status code distribution:", 1)
|
||||
if len(sec) == 2:
|
||||
for code, n in re.findall(r"^\s*\[(\d+)\]\s+(\d+) responses", sec[1].split("Error distribution:")[0], re.M):
|
||||
r["codes"][code] = r["codes"].get(code, 0) + int(n)
|
||||
sec = txt.split("Error distribution:", 1)
|
||||
if len(sec) == 2:
|
||||
r["errors"] = sum(int(n) for n in re.findall(r"^\s*\[(\d+)\]\s", sec[1], re.M))
|
||||
r["responses"] = sum(r["codes"].values())
|
||||
r["requests"] = r["responses"] + r["errors"]
|
||||
return r
|
||||
|
||||
|
||||
def aggregate(rows, mode):
|
||||
parts = [x["hey"] for x in rows]
|
||||
resp = sum(p["responses"] for p in parts)
|
||||
req = sum(p["requests"] for p in parts)
|
||||
durs = [p["total_s"] or 0 for p in parts]
|
||||
dur = max(durs) if mode == "par" else sum(durs)
|
||||
codes = {}
|
||||
for p in parts:
|
||||
for c, n in p["codes"].items():
|
||||
codes[c] = codes.get(c, 0) + n
|
||||
|
||||
def wavg(get):
|
||||
num = sum(get(p) * p["responses"] for p in parts if get(p) is not None and p["responses"])
|
||||
den = sum(p["responses"] for p in parts if get(p) is not None and p["responses"])
|
||||
return num / den if den else None
|
||||
|
||||
ms = lambda v: None if v is None else v * 1000.0
|
||||
ok2xx = sum(n for c, n in codes.items() if c.startswith("2"))
|
||||
fast = [p["fastest_s"] for p in parts if p["fastest_s"] is not None]
|
||||
slow = [p["slowest_s"] for p in parts if p["slowest_s"] is not None]
|
||||
return {
|
||||
"requests": req, "responses": resp, "errors": sum(p["errors"] for p in parts),
|
||||
"clients": sum(int(x["clients"]) for x in rows) if mode == "par" else max(int(x["clients"]) for x in rows),
|
||||
"duration_s": dur, "rps": (resp / dur) if dur else None,
|
||||
"success_pct": (100.0 * ok2xx / req) if req else 0.0, "codes": codes,
|
||||
"n5xx": sum(n for c, n in codes.items() if c.startswith("5")),
|
||||
"min_ms": ms(min(fast)) if fast else None, "avg_ms": ms(wavg(lambda p: p["average_s"])),
|
||||
**{"p%s_ms" % k: ms(wavg(lambda p, k=k: p["pct"].get(k))) for k in ("50", "90", "95", "99")},
|
||||
"max_ms": ms(max(slow)) if slow else None,
|
||||
}
|
||||
|
||||
|
||||
def f1(v, nd=1):
|
||||
return "-" if v is None else ("%.*f" % (nd, v))
|
||||
|
||||
|
||||
def codes_str(c):
|
||||
return ", ".join("%s: %d" % (k, v) for k, v in sorted(c.items())) or "-"
|
||||
|
||||
|
||||
def main():
|
||||
d = sys.argv[1]
|
||||
rows = []
|
||||
with open(os.path.join(d, "manifest.csv"), encoding="utf-8") as f:
|
||||
for x in csv.DictReader(f):
|
||||
x["hey"] = parse_hey(os.path.join(d, "raw", x["file"]))
|
||||
rows.append(x)
|
||||
order, by = [], {}
|
||||
for x in rows:
|
||||
if x["scenario"] not in by:
|
||||
order.append(x["scenario"])
|
||||
by[x["scenario"]] = []
|
||||
by[x["scenario"]].append(x)
|
||||
|
||||
summary = []
|
||||
for s in order:
|
||||
agg = aggregate(by[s], by[s][0]["mode"])
|
||||
agg["scenario"] = s
|
||||
agg["k1"] = agg["success_pct"] >= 99.0
|
||||
agg["k2"] = agg["n5xx"] == 0
|
||||
agg["k3"] = agg["p95_ms"] is not None and agg["p95_ms"] <= 2000.0
|
||||
summary.append(agg)
|
||||
|
||||
extra = {}
|
||||
xp = os.path.join(d, "extra.json")
|
||||
if os.path.exists(xp):
|
||||
extra = json.load(open(xp, encoding="utf-8"))
|
||||
k4 = None
|
||||
s1 = next((x for x in summary if x["scenario"] == "S1"), None)
|
||||
s6 = next((x for x in summary if x["scenario"] == "S6"), None)
|
||||
if s1 and s6:
|
||||
k4 = (extra.get("readyz_after") == 200 and s6["p50_ms"] is not None and s1["p50_ms"] is not None
|
||||
and s6["p50_ms"] <= 2 * s1["p50_ms"] + 1.0)
|
||||
|
||||
hdr = ["Сцен.", "Запр.", "Клиентов", "Длит.,с", "RPS", "Успех,%", "min", "avg", "P50", "P90", "P95", "P99", "max", "Коды", "Ошибки", "Итог"]
|
||||
lines = []
|
||||
for a in summary:
|
||||
ok = a["k1"] and a["k2"] and a["k3"]
|
||||
lines.append([a["scenario"], str(a["requests"]), str(a["clients"]), f1(a["duration_s"]), f1(a["rps"], 0),
|
||||
f1(a["success_pct"], 2), f1(a["min_ms"]), f1(a["avg_ms"]), f1(a["p50_ms"]), f1(a["p90_ms"]),
|
||||
f1(a["p95_ms"]), f1(a["p99_ms"]), f1(a["max_ms"]), codes_str(a["codes"]), str(a["errors"]),
|
||||
"PASS" if ok else "FAIL"])
|
||||
print_table("Сводные результаты (время — мс)", hdr, lines)
|
||||
|
||||
for s in order:
|
||||
if len(by[s]) > 1 and s not in ("S1", "S6"):
|
||||
dl = []
|
||||
for x in by[s]:
|
||||
a = aggregate([x], "par")
|
||||
dl.append([x["label"], x["clients"], str(a["responses"]), f1(a["rps"], 1), f1(a["p50_ms"]),
|
||||
f1(a["p95_ms"]), f1(a["p99_ms"]), f1(a["max_ms"]), codes_str(a["codes"]), str(a["errors"])])
|
||||
print_table("%s по эндпоинтам / прогонам (время — мс)" % s,
|
||||
["Эндпоинт / прогон", "Клиентов", "Ответов", "RPS", "P50", "P95", "P99", "max", "Коды", "Ошибки"], dl)
|
||||
|
||||
if extra.get("uploads"):
|
||||
u = extra["uploads"]
|
||||
print("S5 загрузки: %d запросов POST /datasets, коды %s, время ответа %s–%s мс; обработано до ready: %d/%d, макс. время обработки %s с"
|
||||
% (u["count"], u["codes"], f1(u.get("min_ms")), f1(u.get("max_ms")), u.get("ready", 0), u["count"], f1(u.get("max_processing_s"), 2)))
|
||||
if extra.get("stats_peak"):
|
||||
print("\nПиковое потребление ресурсов (docker stats, сценарии S4/S5):")
|
||||
for name, v in sorted(extra["stats_peak"].items()):
|
||||
print(" %-34s CPU %6.1f %% память %8.1f МиБ" % (name, v["cpu"], v["mem_mib"]))
|
||||
if k4 is not None:
|
||||
print("\nK4 (после нагрузки): /readyz %s; S6 P50/P95 = %s/%s мс против S1 %s/%s мс -> %s" % (
|
||||
extra.get("readyz_after"), f1(s6["p50_ms"]), f1(s6["p95_ms"]), f1(s1["p50_ms"]), f1(s1["p95_ms"]),
|
||||
"PASS" if k4 else "FAIL"))
|
||||
|
||||
with open(os.path.join(d, "summary.csv"), "w", newline="", encoding="utf-8") as f:
|
||||
w = csv.writer(f)
|
||||
w.writerow(["scenario", "requests", "clients", "duration_s", "rps", "success_pct", "min_ms", "avg_ms",
|
||||
"p50_ms", "p90_ms", "p95_ms", "p99_ms", "max_ms", "codes", "errors", "result"])
|
||||
for ln, a in zip(lines, summary):
|
||||
w.writerow([a["scenario"], a["requests"], a["clients"], a["duration_s"], a["rps"], a["success_pct"],
|
||||
a["min_ms"], a["avg_ms"], a["p50_ms"], a["p90_ms"], a["p95_ms"], a["p99_ms"], a["max_ms"],
|
||||
codes_str(a["codes"]), a["errors"], ln[-1]])
|
||||
with open(os.path.join(d, "detail.csv"), "w", newline="", encoding="utf-8") as f:
|
||||
w = csv.writer(f)
|
||||
w.writerow(["scenario", "label", "clients", "requests", "responses", "errors", "duration_s", "rps",
|
||||
"min_ms", "avg_ms", "p50_ms", "p90_ms", "p95_ms", "p99_ms", "max_ms", "codes", "file"])
|
||||
for x in rows:
|
||||
a = aggregate([x], "par")
|
||||
w.writerow([x["scenario"], x["label"], x["clients"], a["requests"], a["responses"], a["errors"],
|
||||
a["duration_s"], a["rps"], a["min_ms"], a["avg_ms"], a["p50_ms"], a["p90_ms"], a["p95_ms"],
|
||||
a["p99_ms"], a["max_ms"], codes_str(a["codes"]), x["file"]])
|
||||
with open(os.path.join(d, "summary.json"), "w", encoding="utf-8") as f:
|
||||
json.dump({"scenarios": summary, "k4": k4, "extra": extra}, f, ensure_ascii=False, indent=2)
|
||||
|
||||
failed = [a["scenario"] for a in summary if not (a["k1"] and a["k2"] and a["k3"])]
|
||||
if k4 is False:
|
||||
failed.append("K4")
|
||||
print("\nКритерии K1 (успех >= 99 %%), K2 (5xx = 0), K3 (P95 <= 2000 мс)%s: %s" % (
|
||||
", K4" if k4 is not None else "", "все выполнены" if not failed else "НЕ выполнены: " + ", ".join(failed)))
|
||||
print("Файлы: %s/{summary.csv,summary.json,detail.csv,raw/}" % d)
|
||||
return 1 if failed else 0
|
||||
|
||||
|
||||
def print_table(title, hdr, lines):
|
||||
widths = [max(len(str(r[i])) for r in [hdr] + lines) for i in range(len(hdr))]
|
||||
print("\n" + title)
|
||||
print(" | ".join(h.ljust(w) for h, w in zip(hdr, widths)))
|
||||
print("-+-".join("-" * w for w in widths))
|
||||
for r in lines:
|
||||
print(" | ".join(str(c).ljust(w) for c, w in zip(r, widths)))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
150
test/acceptance/lib/make_fixtures.py
Normal file
150
test/acceptance/lib/make_fixtures.py
Normal file
@@ -0,0 +1,150 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Regenerates the committed vector/SLD fixtures in test/acceptance/data/.
|
||||
|
||||
The fixtures are committed, so this script is only needed to rebuild them.
|
||||
Requires the GDAL command-line tool ogr2ogr (GeoPackage conversion).
|
||||
|
||||
python3 test/acceptance/lib/make_fixtures.py
|
||||
|
||||
Values are deterministic: district KATO codes and names come from
|
||||
stubs/districts.geojson (14 districts of North Kazakhstan region), the
|
||||
indicator values follow a fixed formula, so re-running yields identical files.
|
||||
"""
|
||||
import json
|
||||
import os
|
||||
import subprocess
|
||||
import sys
|
||||
import tempfile
|
||||
|
||||
HERE = os.path.dirname(os.path.abspath(__file__))
|
||||
ROOT = os.path.abspath(os.path.join(HERE, "..", "..", ".."))
|
||||
DATA = os.path.abspath(os.path.join(HERE, "..", "data"))
|
||||
|
||||
|
||||
def ring_bbox(coords):
|
||||
xs, ys = [], []
|
||||
|
||||
def walk(c):
|
||||
if isinstance(c[0], (int, float)):
|
||||
xs.append(c[0])
|
||||
ys.append(c[1])
|
||||
else:
|
||||
for x in c:
|
||||
walk(x)
|
||||
|
||||
walk(coords)
|
||||
return min(xs), min(ys), max(xs), max(ys)
|
||||
|
||||
|
||||
def population(i, kato, year):
|
||||
# Values quoted in the protocol (Приложение 7, F16/F17) are kept verbatim.
|
||||
fixed = {
|
||||
("591000000", 2022): 47567.6,
|
||||
("591000000", 2023): 48311.2,
|
||||
("591000000", 2024): 49120.5,
|
||||
("595000000", 2022): 46862.7,
|
||||
("595000000", 2023): 48268.6,
|
||||
("595000000", 2024): 49205.8,
|
||||
}
|
||||
if (kato, year) in fixed:
|
||||
return fixed[(kato, year)]
|
||||
return round(20000 + 1750.3 * i + 812.4 * (year - 2022) + (i * 37 % 11) * 13.1, 1)
|
||||
|
||||
|
||||
def livestock(i, year):
|
||||
return int(30000 + 2311 * i + 977 * (year - 2022) + (i * 53 % 17) * 41)
|
||||
|
||||
|
||||
def main():
|
||||
with open(os.path.join(ROOT, "stubs", "districts.geojson"), encoding="utf-8") as f:
|
||||
districts = json.load(f)["features"]
|
||||
os.makedirs(DATA, exist_ok=True)
|
||||
|
||||
# 1. plain vector: 4 plot polygons spanning lon 69.0..71.7, lat 53.6..55.0
|
||||
plots = []
|
||||
boxes = [
|
||||
(69.0, 53.6, 69.6, 54.0),
|
||||
(69.9, 54.2, 70.5, 54.7),
|
||||
(70.6, 53.7, 71.1, 54.1),
|
||||
(71.2, 54.5, 71.7, 55.0),
|
||||
]
|
||||
for n, (x0, y0, x1, y1) in enumerate(boxes, 1):
|
||||
plots.append({
|
||||
"type": "Feature",
|
||||
"properties": {"plot_id": n, "name": "Участок %d" % n, "area_ha": round((x1 - x0) * (y1 - y0) * 7000, 1)},
|
||||
"geometry": {"type": "Polygon", "coordinates": [[[x0, y0], [x1, y0], [x1, y1], [x0, y1], [x0, y0]]]},
|
||||
})
|
||||
write_json("plots_nko.geojson", {"type": "FeatureCollection", "features": plots})
|
||||
|
||||
# 2. vector_with_kato (GeoJSON): population per district, Y2022..Y2024
|
||||
pop = []
|
||||
live = []
|
||||
for i, d in enumerate(districts):
|
||||
p = d["properties"]
|
||||
kato = str(p["kato"])
|
||||
x0, y0, x1, y1 = ring_bbox(d["geometry"]["coordinates"])
|
||||
pt = {"type": "Point", "coordinates": [round((x0 + x1) / 2, 5), round((y0 + y1) / 2, 5)]}
|
||||
props = {"kato": kato, "district": p["name_ru"]}
|
||||
for y in (2022, 2023, 2024):
|
||||
props["Y%d" % y] = population(i, kato, y)
|
||||
pop.append({"type": "Feature", "properties": props, "geometry": pt})
|
||||
lprops = {"KATO_CODE": kato, "district": p["name_ru"]}
|
||||
for y in (2022, 2023, 2024):
|
||||
lprops["H_%d" % y] = livestock(i, y)
|
||||
live.append({"type": "Feature", "properties": lprops, "geometry": pt})
|
||||
write_json("population_kato.geojson", {"type": "FeatureCollection", "features": pop})
|
||||
|
||||
# 3. vector_with_kato (GeoPackage): cattle per district, H_2022..H_2024
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
src = os.path.join(tmp, "livestock.geojson")
|
||||
with open(src, "w", encoding="utf-8") as f:
|
||||
json.dump({"type": "FeatureCollection", "features": live}, f, ensure_ascii=False)
|
||||
dst = os.path.join(DATA, "livestock_kato.gpkg")
|
||||
if os.path.exists(dst):
|
||||
os.remove(dst)
|
||||
subprocess.run(["ogr2ogr", "-f", "GPKG", "-nln", "livestock", "-a_srs", "EPSG:4326", dst, src], check=True)
|
||||
|
||||
# 4. SLD styles
|
||||
write_text("style.sld", sld("ndvi", "NDVI gradient", [("0.0", "#a50026"), ("0.5", "#ffffbf"), ("1.0", "#006837")]))
|
||||
write_text("group_style.sld", sld("group", "Group fallback style", [("0", "#f7fbff"), ("100000", "#08306b")]))
|
||||
write_text("style.txt", "not an SLD file\n")
|
||||
print("fixtures written to", DATA)
|
||||
|
||||
|
||||
def sld(name, title, stops):
|
||||
entries = "\n".join(
|
||||
' <sld:ColorMapEntry color="%s" quantity="%s"/>' % (c, q) for q, c in stops)
|
||||
return """<?xml version="1.0" encoding="UTF-8"?>
|
||||
<sld:StyledLayerDescriptor version="1.0.0" xmlns:sld="http://www.opengis.net/sld" xmlns:ogc="http://www.opengis.net/ogc">
|
||||
<sld:NamedLayer>
|
||||
<sld:Name>%s</sld:Name>
|
||||
<sld:UserStyle>
|
||||
<sld:Title>%s</sld:Title>
|
||||
<sld:FeatureTypeStyle>
|
||||
<sld:Rule>
|
||||
<sld:RasterSymbolizer>
|
||||
<sld:ColorMap>
|
||||
%s
|
||||
</sld:ColorMap>
|
||||
</sld:RasterSymbolizer>
|
||||
</sld:Rule>
|
||||
</sld:FeatureTypeStyle>
|
||||
</sld:UserStyle>
|
||||
</sld:NamedLayer>
|
||||
</sld:StyledLayerDescriptor>
|
||||
""" % (name, title, entries)
|
||||
|
||||
|
||||
def write_json(name, obj):
|
||||
with open(os.path.join(DATA, name), "w", encoding="utf-8") as f:
|
||||
json.dump(obj, f, ensure_ascii=False, indent=1)
|
||||
f.write("\n")
|
||||
|
||||
|
||||
def write_text(name, text):
|
||||
with open(os.path.join(DATA, name), "w", encoding="utf-8") as f:
|
||||
f.write(text)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
227
test/acceptance/load.sh
Executable file
227
test/acceptance/load.sh
Executable file
@@ -0,0 +1,227 @@
|
||||
#!/usr/bin/env bash
|
||||
# Load test of the GIS API — scenarios S1–S6 of «Протокол нагрузочного
|
||||
# тестирования» (Приложение 3), generator hey 0.1.5:
|
||||
# S1 baseline: 10 endpoints one after another, hey -n 100 -c 1 each
|
||||
# S2 mixed: 1000 requests, 100 concurrent clients over the 10 endpoints
|
||||
# (10 parallel hey processes, -n 100 -c 10 each)
|
||||
# S3.1 1000 simultaneous requests: hey -n 1000 -c 1000 /datasets?page_size=20, 3 runs
|
||||
# S3.2 1000 simultaneous requests: hey -n 1000 -c 1000 /landscape/results, 3 runs
|
||||
# S4 sustained ~1000 req/s for 60 s: 1000 clients at -q 1 split over 8
|
||||
# endpoints (300/150/100/100/100/100/100/50), 8 parallel hey processes
|
||||
# S5 30 s of reads (10 endpoints x 10 clients, -z 30s) while 20 uploads
|
||||
# (10 GeoJSON + 10 GeoTIFF) run in parallel; uploads must reach ready
|
||||
# S6 recovery: S1 again + /readyz
|
||||
#
|
||||
# By default the generator runs INSIDE the stand's docker network
|
||||
# (docker run --network gis-acceptance_default gis-acceptance-hey:0.1.5,
|
||||
# target http://app:8080), so host port forwarding is not measured.
|
||||
#
|
||||
# Environment:
|
||||
# BASE_URL API address for setup calls (default http://localhost:18080)
|
||||
# TARGET_URL address the generator loads (default http://app:8080 inside
|
||||
# the network; any URL with GEN=host or NETWORK=bridge)
|
||||
# GEN docker (default) | host (use a locally installed hey)
|
||||
# NETWORK docker network of the generator (default gis-acceptance_default)
|
||||
# SCENARIOS subset to run (default "S1 S2 S3 S4 S5 S6")
|
||||
# S4_DURATION / S5_DURATION default 60s / 30s
|
||||
# PYTHON python3 with numpy + rasterio (S5 raster uploads, seeding)
|
||||
# ALLOW_WRITES 1 = allow seeding and S5 uploads on a non-local BASE_URL
|
||||
#
|
||||
# Output: summary table on stdout; test/acceptance/results/load-<timestamp>/
|
||||
# {summary.csv, summary.json, detail.csv, raw/*.txt}. Exit code 1 when a
|
||||
# criterion K1–K4 fails.
|
||||
set -euo pipefail
|
||||
|
||||
HERE="$(cd "$(dirname "$0")" && pwd)"
|
||||
BASE_URL="${BASE_URL:-http://localhost:18080}"; BASE_URL="${BASE_URL%/}"
|
||||
GEN="${GEN:-docker}"
|
||||
NETWORK="${NETWORK:-gis-acceptance_default}"
|
||||
HEY_IMAGE="${HEY_IMAGE:-gis-acceptance-hey:0.1.5}"
|
||||
SCENARIOS="${SCENARIOS:-S1 S2 S3 S4 S5 S6}"
|
||||
S4_DURATION="${S4_DURATION:-60s}"
|
||||
S5_DURATION="${S5_DURATION:-30s}"
|
||||
PYTHON="${PYTHON:-python3}"
|
||||
ALLOW_WRITES="${ALLOW_WRITES:-0}"
|
||||
if [ -z "${TARGET_URL:-}" ]; then
|
||||
if [ "$GEN" = docker ] && [ "$NETWORK" = gis-acceptance_default ]; then TARGET_URL=http://app:8080; else TARGET_URL="$BASE_URL"; fi
|
||||
fi
|
||||
TARGET_URL="${TARGET_URL%/}"
|
||||
LOCAL=0
|
||||
case "$BASE_URL" in http://localhost:*|http://127.0.0.1:*) LOCAL=1;; esac
|
||||
case "$BASE_URL" in *dssgis.dwh.kz*) ALLOW_WRITES=0;; esac
|
||||
|
||||
for bin in curl jq python3; do command -v "$bin" >/dev/null || { echo "missing $bin" >&2; exit 2; }; done
|
||||
STAMP="$(date +%Y%m%d-%H%M%S)"
|
||||
OUT="$HERE/results/load-$STAMP"; RAW="$OUT/raw"; mkdir -p "$RAW"
|
||||
echo "scenario,mode,label,clients,file" > "$OUT/manifest.csv"
|
||||
EXTRA="$OUT/extra.json"; echo '{}' > "$EXTRA"
|
||||
log() { echo "[$(date +%H:%M:%S)] $*"; }
|
||||
extra_set() { local t; t="$(mktemp)"; jq "$1" "$EXTRA" > "$t" && mv "$t" "$EXTRA"; }
|
||||
|
||||
# ---------------------------------------------------------------- generator
|
||||
if [ "$GEN" = docker ]; then
|
||||
if ! docker image inspect "$HEY_IMAGE" >/dev/null 2>&1; then
|
||||
log "building $HEY_IMAGE"
|
||||
docker build -q -t "$HEY_IMAGE" -f "$HERE/lib/hey.Dockerfile" "$HERE/lib" >/dev/null
|
||||
fi
|
||||
NETARG="--network $NETWORK"; [ "$NETWORK" = bridge ] && NETARG=""
|
||||
fi
|
||||
# run_batch SCEN MODE "label|clients|file|hey args and path" ...
|
||||
# MODE par: all hey processes start together; seq: one after another.
|
||||
run_batch() {
|
||||
local scen="$1" mode="$2"; shift 2
|
||||
local script="" spec label clients file args sep=" & "
|
||||
[ "$mode" = seq ] && sep=" ; "
|
||||
for spec in "$@"; do
|
||||
IFS='|' read -r label clients file args <<< "$spec"
|
||||
echo "$scen,$mode,\"$label\",$clients,$file" >> "$OUT/manifest.csv"
|
||||
script="${script}hey $args > /out/$file 2>&1${sep}"
|
||||
done
|
||||
script="${script} wait"
|
||||
if [ "$GEN" = docker ]; then
|
||||
# shellcheck disable=SC2086
|
||||
docker run --rm $NETARG --ulimit nofile=65536:65536 -v "$RAW:/out" --entrypoint sh "$HEY_IMAGE" -c "$script"
|
||||
else
|
||||
( cd "$RAW" && ulimit -n 65536 2>/dev/null || true; sh -c "${script//\/out\//$RAW/}" )
|
||||
fi
|
||||
}
|
||||
|
||||
# --------------------------------------------------------------- endpoints
|
||||
if [ "$LOCAL" = 1 ] || [ "$ALLOW_WRITES" = 1 ]; then
|
||||
log "seed (idempotent): $BASE_URL"
|
||||
BASE_URL="$BASE_URL" PYTHON="$PYTHON" "$HERE/seed.sh" | tail -n 4
|
||||
# shellcheck disable=SC1091
|
||||
. "$HERE/results/seed.env"
|
||||
else
|
||||
log "read-only discovery of endpoint ids on $BASE_URL"
|
||||
VWK_ID="$(curl -fsS "$BASE_URL/datasets?file_type=vector_with_kato&status=ready&page_size=1" | jq -r '.data[0].id')"
|
||||
GROUP_ID="$(curl -fsS "$BASE_URL/datasets/$VWK_ID" | jq -r '.dg_id // empty')"
|
||||
CAT_ID=""
|
||||
fi
|
||||
RUN_ID="$(curl -fsS "$BASE_URL/landscape/runs?page_size=1" | jq -r '.data[0].id // empty')"
|
||||
if [ -z "$RUN_ID" ] && { [ "$LOCAL" = 1 ] || [ "$ALLOW_WRITES" = 1 ]; }; then
|
||||
log "no ЛП run yet: running lp_run.sh"
|
||||
BASE_URL="$BASE_URL" PYTHON="$PYTHON" "$HERE/lp_run.sh" > "$OUT/lp_run.txt" 2>&1
|
||||
RUN_ID="$(curl -fsS "$BASE_URL/landscape/runs?page_size=1" | jq -r '.data[0].id')"
|
||||
fi
|
||||
for v in VWK_ID GROUP_ID RUN_ID; do
|
||||
eval "val=\${$v:-}"
|
||||
# shellcheck disable=SC2154
|
||||
[ -n "$val" ] && [ "$val" != null ] || { echo "cannot resolve $v on $BASE_URL" >&2; exit 2; }
|
||||
done
|
||||
EP_PATHS="/datasets?page_size=20 /datasets/$VWK_ID /datasets/$VWK_ID.geojson /datasets/$VWK_ID/observations /categories /dataset-groups/$GROUP_ID/latest /landscape/slots /landscape/results /landscape/runs/$RUN_ID /readyz"
|
||||
label_of() { # path -> protocol label
|
||||
echo "$1" | sed -e "s#$VWK_ID#{id}#g" -e "s#$GROUP_ID#{id}#g" -e "s#$RUN_ID#{id}#g"
|
||||
}
|
||||
fname() { echo "$1" | sed -e 's#^/##' -e 's#[/?=.{}]#_#g'; }
|
||||
for p in $EP_PATHS; do # every endpoint must answer 200 before loading it
|
||||
c="$(curl -sS -o /dev/null -w '%{http_code}' "$BASE_URL$p")"
|
||||
[ "$c" = 200 ] || { echo "endpoint $p answers $c, fix the stand first" >&2; exit 2; }
|
||||
done
|
||||
log "target $TARGET_URL (generator: $GEN${NETARG:+, $NETARG}); results $OUT"
|
||||
curl -fsS "$BASE_URL/datasets?status=ready&page_size=1" | jq -r '"ready datasets on the stand: \(.total)"'
|
||||
|
||||
stats_start() { # docker stats sampler for the compose project (local stand only)
|
||||
[ "$LOCAL" = 1 ] || return 0
|
||||
( while :; do docker stats --no-stream --format '{{.Name}},{{.CPUPerc}},{{.MemUsage}}' \
|
||||
$(docker ps -q --filter label=com.docker.compose.project=gis-acceptance) >> "$OUT/stats.csv" 2>/dev/null || true
|
||||
sleep 3; done ) &
|
||||
STATS_PID=$!
|
||||
}
|
||||
stats_stop() { [ -n "${STATS_PID:-}" ] && kill "$STATS_PID" 2>/dev/null || true; STATS_PID=""; }
|
||||
trap 'stats_stop' EXIT
|
||||
|
||||
baseline() { # S1 / S6
|
||||
local scen="$1" specs=() p
|
||||
for p in $EP_PATHS; do specs+=("$(label_of "$p")|1|$scen-$(fname "$p").txt|-n 100 -c 1 '$TARGET_URL$p'"); done
|
||||
log "$scen: 10 endpoints x hey -n 100 -c 1 (sequential)"
|
||||
run_batch "$scen" seq "${specs[@]}"
|
||||
}
|
||||
|
||||
for S in $SCENARIOS; do
|
||||
case "$S" in
|
||||
S1) baseline S1 ;;
|
||||
S2)
|
||||
specs=()
|
||||
for p in $EP_PATHS; do specs+=("$(label_of "$p")|10|S2-$(fname "$p").txt|-n 100 -c 10 '$TARGET_URL$p'"); done
|
||||
log "S2: 1000 requests, 100 clients over 10 endpoints"
|
||||
run_batch S2 par "${specs[@]}" ;;
|
||||
S3)
|
||||
for i in 1 2 3; do
|
||||
log "S3.1 run $i: hey -n 1000 -c 1000 /datasets?page_size=20"
|
||||
run_batch S3.1 seq "/datasets?page_size=20 №$i|1000|S3.1-run$i.txt|-n 1000 -c 1000 '$TARGET_URL/datasets?page_size=20'"
|
||||
done
|
||||
for i in 1 2 3; do
|
||||
log "S3.2 run $i: hey -n 1000 -c 1000 /landscape/results"
|
||||
run_batch S3.2 seq "/landscape/results №$i|1000|S3.2-run$i.txt|-n 1000 -c 1000 '$TARGET_URL/landscape/results'"
|
||||
done ;;
|
||||
S4)
|
||||
specs=()
|
||||
for pair in "/datasets?page_size=20:300" "/datasets/$VWK_ID:150" "/datasets/$VWK_ID.geojson:100" "/datasets/$VWK_ID/observations:100" \
|
||||
"/categories:100" "/dataset-groups/$GROUP_ID/latest:100" "/landscape/results:100" "/readyz:50"; do
|
||||
p="${pair%:*}"; c="${pair##*:}"
|
||||
specs+=("$(label_of "$p")|$c|S4-$(fname "$p").txt|-z $S4_DURATION -q 1 -c $c '$TARGET_URL$p'")
|
||||
done
|
||||
log "S4: 1000 clients at 1 req/s each for $S4_DURATION (8 hey processes)"
|
||||
stats_start; run_batch S4 par "${specs[@]}"; stats_stop ;;
|
||||
S5)
|
||||
if [ "$LOCAL" != 1 ] && [ "$ALLOW_WRITES" != 1 ]; then log "S5 skipped: uploads need a local stand or ALLOW_WRITES=1"; continue; fi
|
||||
TMP5="$(mktemp -d "${TMPDIR:-/tmp}/gis-load.XXXXXX")"
|
||||
"$PYTHON" "$HERE/lib/helpers.py" ndvi "$TMP5/ndvi_s5.tif"
|
||||
specs=()
|
||||
for p in $EP_PATHS; do specs+=("$(label_of "$p")|10|S5-$(fname "$p").txt|-z $S5_DURATION -c 10 '$TARGET_URL$p'"); done
|
||||
log "S5: reads 10 x 10 clients for $S5_DURATION + 20 parallel uploads"
|
||||
stats_start
|
||||
run_batch S5 par "${specs[@]}" &
|
||||
READ_PID=$!
|
||||
sleep 5
|
||||
for i in $(seq 1 20); do
|
||||
if [ "$i" -le 10 ]; then f="$HERE/data/plots_nko.geojson"; t=vector; else f="$TMP5/ndvi_s5.tif"; t=raster; fi
|
||||
( curl -sS -o "$TMP5/up$i.json" -w '%{http_code} %{time_total}\n' -X POST "$BASE_URL/datasets" \
|
||||
-F "file=@$f" -F "file_type=$t" -F "category_id=$CAT_ID" -F "name=S5 upload $i" > "$TMP5/up$i.code"; date +%s > "$TMP5/up$i.t" ) &
|
||||
done
|
||||
for i in $(seq 1 20); do while [ ! -s "$TMP5/up$i.t" ]; do sleep 0.2; done; done
|
||||
wait "$READ_PID" || true
|
||||
stats_stop
|
||||
# processing: every upload must reach ready
|
||||
codes=""; ready=0; maxproc=0; mins=999999; maxs=0
|
||||
for i in $(seq 1 20); do
|
||||
read -r code tt < "$TMP5/up$i.code"; codes="$codes $code"
|
||||
ms="$(awk -v t="$tt" 'BEGIN{printf "%.1f", t*1000}')"
|
||||
mins="$(awk -v a="$mins" -v b="$ms" 'BEGIN{print (b<a)?b:a}')"; maxs="$(awk -v a="$maxs" -v b="$ms" 'BEGIN{print (b>a)?b:a}')"
|
||||
id="$(jq -r '.id // empty' "$TMP5/up$i.json")"; [ -n "$id" ] || continue
|
||||
for _ in $(seq 1 300); do
|
||||
st="$(curl -fsS "$BASE_URL/datasets/$id" | jq -r .status)"
|
||||
[ "$st" = ready ] || [ "$st" = failed ] && break; sleep 1
|
||||
done
|
||||
if [ "$st" = ready ]; then
|
||||
ready=$((ready + 1))
|
||||
# server-side processing time: updated_at (ready) - created_at (upload)
|
||||
curl -fsS "$BASE_URL/datasets/$id" -o "$TMP5/ds$i.json"
|
||||
proc="$(python3 "$HERE/lib/helpers.py" elapsed "$TMP5/ds$i.json" 2>/dev/null || echo 0)"
|
||||
maxproc="$(awk -v a="$maxproc" -v b="$proc" 'BEGIN{print (b>a)?b:a}')"
|
||||
fi
|
||||
curl -sS -o /dev/null -X DELETE "$BASE_URL/datasets/$id"
|
||||
done
|
||||
cs="$(echo $codes | tr ' ' '\n' | sort | uniq -c | awk '{printf "%s%s: %s", (NR>1?", ":""), $2, $1}')"
|
||||
extra_set ".uploads = {count: 20, codes: \"$cs\", min_ms: $mins, max_ms: $maxs, ready: $ready, max_processing_s: $maxproc}"
|
||||
log "S5 uploads: $cs; ready $ready/20 (max processing ${maxproc}s = updated_at - created_at); uploads deleted"
|
||||
rm -rf "$TMP5" ;;
|
||||
S6)
|
||||
baseline S6
|
||||
rz="$(curl -sS -o "$OUT/readyz_after.json" -w '%{http_code}' "$BASE_URL/readyz")"
|
||||
extra_set ".readyz_after = $rz | .readyz_body = $(cat "$OUT/readyz_after.json")"
|
||||
log "S6: /readyz $rz $(cat "$OUT/readyz_after.json")" ;;
|
||||
*) echo "unknown scenario $S" >&2; exit 2 ;;
|
||||
esac
|
||||
done
|
||||
|
||||
if [ -s "$OUT/stats.csv" ]; then # peak CPU % / memory MiB per container
|
||||
peaks="$(awk -F, '{gsub("%","",$2); split($3,m," / "); v=m[1]; mult=1;
|
||||
if (v ~ /GiB$/) {mult=1024} else if (v ~ /KiB$/) {mult=1/1024} else if (v ~ /kB$/) {mult=1/1024} else if (v ~ /MB$/) {mult=1};
|
||||
gsub(/[A-Za-z]+$/,"",v); mem=v*mult; if ($2>cpu[$1]) cpu[$1]=$2; if (mem>mm[$1]) mm[$1]=mem }
|
||||
END { printf "{"; n=0; for (k in cpu) { printf "%s\"%s\":{\"cpu\":%s,\"mem_mib\":%.1f}", (n++?",":""), k, cpu[k], mm[k] } printf "}" }' "$OUT/stats.csv")"
|
||||
extra_set ".stats_peak = $peaks"
|
||||
fi
|
||||
|
||||
python3 "$HERE/lib/load_analyze.py" "$OUT"
|
||||
111
test/acceptance/lp_gen.py
Executable file
111
test/acceptance/lp_gen.py
Executable file
@@ -0,0 +1,111 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Generates the synthetic input rasters of the Landscape Program (ЛП) test.
|
||||
|
||||
Writes 13 grid-aligned GeoTIFFs, one per input slot (file name = slot code),
|
||||
plus one deliberately misaligned raster for the negative test (L13):
|
||||
|
||||
<out>/lp-z-biodiversity.tif ... <out>/lp-p-tourism.tif (13 files)
|
||||
<out>/misaligned-lp-u-veg.tif (half size, 2x pixel)
|
||||
|
||||
Every input raster: int16, EPSG:32642 (UTM 42N), 100 m pixels, default
|
||||
3000 x 2000 pixels, classes 1..5 from smooth trigonometric patterns (a
|
||||
different pattern per slot, so ties and every class combination occur), and a
|
||||
NoData (-9999) frame 40 pixels wide. Extra edge cases exercise the pixel rules:
|
||||
|
||||
* lp-u-crop: a 200 x 200 NoData block — U is computed from water, soil and
|
||||
veg only (the crop/veg union rule of u_integrated / u_max);
|
||||
* lp-u-veg: a 100 x 100 NoData block overlapping half of the crop block —
|
||||
where both crop and veg are missing, U must be NoData;
|
||||
* lp-p-snow: a 120 x 120 block of 0 and a 60 x 60 block of 6 — values that
|
||||
are not NoData but outside 1..5, hence invalid (the "valid iff 1 <= v <= 5"
|
||||
rule).
|
||||
|
||||
Output is deterministic (no random numbers), so the same command always
|
||||
produces byte-identical pixel values.
|
||||
|
||||
python lp_gen.py --out /tmp/lp-inputs [--width 3000 --height 2000]
|
||||
"""
|
||||
import argparse
|
||||
import os
|
||||
import sys
|
||||
|
||||
import numpy as np
|
||||
import rasterio
|
||||
from rasterio.transform import from_origin
|
||||
|
||||
SLOTS = [
|
||||
"lp-z-biodiversity", "lp-z-lcr", "lp-z-soil-protection", "lp-z-wr",
|
||||
"lp-u-surf-water", "lp-u-soil", "lp-u-crop", "lp-u-veg",
|
||||
"lp-p-irrigation", "lp-p-snow", "lp-p-crop", "lp-p-pasture", "lp-p-tourism",
|
||||
]
|
||||
NODATA = -9999
|
||||
CRS = "EPSG:32642"
|
||||
ORIGIN_X, ORIGIN_Y = 300000.0, 6100000.0 # UTM 42N, North Kazakhstan region
|
||||
PIXEL = 100.0
|
||||
FRAME = 40
|
||||
|
||||
|
||||
def pattern(k, width, height):
|
||||
"""Smooth class field 1..5 for slot index k."""
|
||||
y, x = np.mgrid[0:height, 0:width].astype(np.float64)
|
||||
fx = 2 * np.pi * (1.0 + 0.37 * k) / width
|
||||
fy = 2 * np.pi * (0.8 + 0.29 * ((k * 5) % 7)) / height
|
||||
v = (np.sin(fx * x + 0.9 * k) * np.cos(fy * y + 0.4 * k)
|
||||
+ 0.5 * np.sin(fx * 2.3 * x + fy * 1.7 * y + k))
|
||||
v = (v + 1.5) / 3.0 # ~0..1
|
||||
cls = np.floor(np.clip(v, 0, 0.999999) * 5).astype(np.int16) + 1
|
||||
return cls
|
||||
|
||||
|
||||
def block(arr, r0, c0, h, w, value):
|
||||
arr[r0:r0 + h, c0:c0 + w] = value
|
||||
|
||||
|
||||
def write(path, arr, transform):
|
||||
profile = dict(driver="GTiff", width=arr.shape[1], height=arr.shape[0], count=1,
|
||||
dtype="int16", crs=CRS, transform=transform, nodata=NODATA,
|
||||
tiled=True, blockxsize=256, blockysize=256, compress="DEFLATE")
|
||||
with rasterio.open(path, "w", **profile) as dst:
|
||||
dst.write(arr, 1)
|
||||
|
||||
|
||||
def main():
|
||||
ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
|
||||
ap.add_argument("--out", required=True, help="output directory")
|
||||
ap.add_argument("--width", type=int, default=3000)
|
||||
ap.add_argument("--height", type=int, default=2000)
|
||||
args = ap.parse_args()
|
||||
w, h = args.width, args.height
|
||||
if w < 400 or h < 400:
|
||||
sys.exit("width/height must be >= 400")
|
||||
os.makedirs(args.out, exist_ok=True)
|
||||
transform = from_origin(ORIGIN_X, ORIGIN_Y, PIXEL, PIXEL)
|
||||
|
||||
for k, slot in enumerate(SLOTS):
|
||||
arr = pattern(k, w, h)
|
||||
arr[:FRAME, :] = NODATA
|
||||
arr[-FRAME:, :] = NODATA
|
||||
arr[:, :FRAME] = NODATA
|
||||
arr[:, -FRAME:] = NODATA
|
||||
cr, cc = h // 2 - 100, w // 2 - 100 # centre block anchor
|
||||
if slot == "lp-u-crop":
|
||||
block(arr, cr, cc, 200, 200, NODATA)
|
||||
if slot == "lp-u-veg":
|
||||
block(arr, cr + 50, cc + 150, 100, 100, NODATA)
|
||||
if slot == "lp-p-snow":
|
||||
block(arr, 300, 300, 120, 120, 0)
|
||||
block(arr, 300, 500, 60, 60, 6)
|
||||
write(os.path.join(args.out, slot + ".tif"), arr, transform)
|
||||
print("wrote", slot + ".tif")
|
||||
|
||||
# Misaligned raster for L13: half the size, double pixel => same footprint,
|
||||
# different grid.
|
||||
mis = pattern(7, w // 2, h // 2)
|
||||
write(os.path.join(args.out, "misaligned-lp-u-veg.tif"), mis,
|
||||
from_origin(ORIGIN_X, ORIGIN_Y, PIXEL * 2, PIXEL * 2))
|
||||
print("wrote misaligned-lp-u-veg.tif (%dx%d, %d m)" % (w // 2, h // 2, int(PIXEL * 2)))
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
72
test/acceptance/lp_run.sh
Executable file
72
test/acceptance/lp_run.sh
Executable file
@@ -0,0 +1,72 @@
|
||||
#!/usr/bin/env bash
|
||||
# Landscape Program (ЛП) end-to-end check on the acceptance stand:
|
||||
# 1. lp_gen.py — 13 synthetic input rasters (default 3000 x 2000)
|
||||
# 2. upload — POST /landscape/slots/{code} for the 13 slots
|
||||
# 3. run — POST /landscape/runs {"from_stage":0}, wait for it
|
||||
# 4. lp_verify.py — pixel-by-pixel check of all 15 outputs (mode a)
|
||||
#
|
||||
# test/acceptance/lp_run.sh # BASE_URL defaults to http://localhost:18080
|
||||
# LP_WIDTH=6000 LP_HEIGHT=4000 test/acceptance/lp_run.sh
|
||||
#
|
||||
# The uploaded inputs and the run's outputs are left in place so the run can be
|
||||
# re-verified later (python lp_verify.py --run <id>); they are ordinary datasets
|
||||
# of the "landscape" category. WRITES to the stand: never point it at production.
|
||||
set -euo pipefail
|
||||
|
||||
HERE="$(cd "$(dirname "$0")" && pwd)"
|
||||
BASE_URL="${BASE_URL:-http://localhost:18080}"
|
||||
BASE_URL="${BASE_URL%/}"
|
||||
PYTHON="${PYTHON:-python3}"
|
||||
WAIT_TIMEOUT="${WAIT_TIMEOUT:-600}"
|
||||
LP_WIDTH="${LP_WIDTH:-3000}"
|
||||
LP_HEIGHT="${LP_HEIGHT:-2000}"
|
||||
case "$BASE_URL" in
|
||||
*dssgis.dwh.kz*) echo "refusing to write to $BASE_URL (production); use lp_verify.py for read-only checks" >&2; exit 2;;
|
||||
esac
|
||||
|
||||
TMP="$(mktemp -d "${TMPDIR:-/tmp}/gis-lp.XXXXXX")"
|
||||
trap 'rm -rf "$TMP"' EXIT
|
||||
STAMP="$(date +%Y%m%d-%H%M%S)"
|
||||
SLOTS="lp-z-biodiversity lp-z-lcr lp-z-soil-protection lp-z-wr lp-u-surf-water lp-u-soil lp-u-crop lp-u-veg lp-p-irrigation lp-p-snow lp-p-crop lp-p-pasture lp-p-tourism"
|
||||
|
||||
echo "== 1. generate inputs (${LP_WIDTH}x${LP_HEIGHT})"
|
||||
"$PYTHON" "$HERE/lp_gen.py" --out "$TMP/in" --width "$LP_WIDTH" --height "$LP_HEIGHT"
|
||||
|
||||
echo "== 2. upload into 13 slots ($BASE_URL)"
|
||||
ids=""
|
||||
for s in $SLOTS; do
|
||||
out="$(curl -sS -w ' %{http_code}' -X POST "$BASE_URL/landscape/slots/$s" -F "file=@$TMP/in/$s.tif" \
|
||||
-F "name=ЛП приёмка $s $STAMP" -F start_date=2025-01-01 -F end_date=2025-12-31)"
|
||||
code="${out##* }"; body="${out% *}"
|
||||
[ "$code" = 201 ] || { echo "upload $s failed: HTTP $code $body" >&2; exit 1; }
|
||||
id="$(jq -r .id <<< "$body")"; ids="$ids $id"
|
||||
printf ' %-22s %s HTTP %s\n' "$s" "$id" "$code"
|
||||
done
|
||||
for id in $ids; do
|
||||
deadline=$(( $(date +%s) + WAIT_TIMEOUT ))
|
||||
until [ "$(curl -sS "$BASE_URL/datasets/$id" | jq -r .status)" = ready ]; do
|
||||
[ "$(date +%s)" -lt "$deadline" ] || { echo "dataset $id not ready" >&2; exit 1; }
|
||||
sleep 1
|
||||
done
|
||||
done
|
||||
echo " all 13 inputs ready"
|
||||
|
||||
echo "== 3. run {\"from_stage\":0}"
|
||||
run="$(curl -sS -X POST "$BASE_URL/landscape/runs" -H 'Content-Type: application/json' -d '{"from_stage":0}')"
|
||||
RUN="$(jq -r .id <<< "$run")"
|
||||
[ -n "$RUN" ] && [ "$RUN" != null ] || { echo "run not created: $run" >&2; exit 1; }
|
||||
echo " run $RUN queued"
|
||||
deadline=$(( $(date +%s) + WAIT_TIMEOUT ))
|
||||
while :; do
|
||||
st="$(curl -sS "$BASE_URL/landscape/runs/$RUN" | jq -r .status)"
|
||||
case "$st" in succeeded|failed) break;; esac
|
||||
[ "$(date +%s)" -lt "$deadline" ] || { echo "run $RUN still $st" >&2; exit 1; }
|
||||
sleep 1
|
||||
done
|
||||
curl -sS "$BASE_URL/landscape/runs/$RUN" > "$TMP/run.json"
|
||||
echo " run $RUN $st: $(jq -r '[.nodes[] | "\(.node_code):\(.status)"] | sort | join(" ")' "$TMP/run.json")"
|
||||
"$PYTHON" "$HERE/lib/helpers.py" durations "$TMP/run.json" | sed 's/^/ /'
|
||||
[ "$st" = succeeded ] || { echo "run failed: $(jq -r .error "$TMP/run.json")" >&2; exit 1; }
|
||||
|
||||
echo "== 4. lp_verify.py --base-url $BASE_URL --run $RUN"
|
||||
"$PYTHON" "$HERE/lp_verify.py" --base-url "$BASE_URL" --run "$RUN" --cache "$TMP/cache"
|
||||
436
test/acceptance/lp_verify.py
Executable file
436
test/acceptance/lp_verify.py
Executable file
@@ -0,0 +1,436 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Independent pixel-by-pixel check of a Landscape Program (ЛП) run.
|
||||
|
||||
Re-implements every node of the ЛП graph in numpy — 00 z_integrated,
|
||||
01 u_integrated, 02 z_max, 03 u_max, 10 e_ecological_constraint,
|
||||
20 c_conflicts, 21 p_max, 30 func_zoning (V1–V3 with the run's thresholds and
|
||||
tie_priority), 31 c_max — from the 13 leaf input rasters the run actually used,
|
||||
and compares the result with all 15 output rasters the run published.
|
||||
|
||||
The program is READ-ONLY: it issues only HTTP GET requests, so it can be
|
||||
pointed at any stand, production included:
|
||||
|
||||
python lp_verify.py --base-url http://localhost:18080 --run <run-id>
|
||||
python lp_verify.py --base-url https://dssgis.dwh.kz --run <run-id>
|
||||
|
||||
Data flow:
|
||||
GET /landscape/runs/{id} run params, per-node inputs and outputs
|
||||
GET /datasets/{id}.cog every input (13) and output (15) raster
|
||||
(or /datasets/{id}/download for inputs with --inputs-from download)
|
||||
|
||||
The leaf inputs are taken from the run's node `inputs` (group = slot code), the
|
||||
outputs from the node `outputs`. Expected values are computed through the whole
|
||||
chain from the leaf inputs only (E from the re-computed Z/U, C and V from the
|
||||
re-computed E, ...), so a wrong intermediate output cannot hide a later error.
|
||||
The run's own chain references (node 10 reading node 00's output, ...) are
|
||||
checked for consistency as well.
|
||||
|
||||
Large rasters (production: 22219 x 14977) are processed in windows of
|
||||
--block-rows x --block-cols pixels (default 512 x 4096, aligned with the
|
||||
512-pixel COG tiles), so memory stays bounded (~0.5 GB).
|
||||
|
||||
Pixel rules (source of truth: internal/landscape/kernels.go):
|
||||
valid(v) 1 <= v <= 5 (NoData and anything else is invalid); class
|
||||
values are used as int16(v) (truncation)
|
||||
classify(x) float32 x: < 1 -> NoData, <= 1.8 -> 1, <= 2.6 -> 2,
|
||||
<= 3.4 -> 3, <= 4.2 -> 4, <= 5.0 -> 5, else NoData
|
||||
z_integrated all Z valid: classify(0.25*Z1 + 0.25*Z2 + 0.25*Z3 + 0.25*Z4)
|
||||
(float32, summed left to right)
|
||||
u_integrated water & soil & (crop | veg): classify(num/den),
|
||||
num = 0.25*water + 0.25*soil [+ 0.25*crop] [+ 0.25*veg],
|
||||
den = 0.5 [+ 0.25] [+ 0.25] (float32)
|
||||
z_max all Z valid: max; u_max: U domain as above, max of valid
|
||||
e Z_int & U_int valid: max(Z, U)
|
||||
c_conflicts P_j & E valid: min(P_j, E)
|
||||
p_max, c_max max over valid inputs; NoData when none is valid
|
||||
func_zoning E valid & any P_k valid; candidate k: P_k valid & rule(E, P_k)
|
||||
V1: E<=2 & S>=v1_min_s_e12 | E==3 & S>=v1_min_s_e3
|
||||
V2: E<=2 | E==3 & S>=v2_min_s_e3 | E==4 & S>=v2_min_s_e4
|
||||
V3: E<=4
|
||||
no candidate -> 1; else sector with max S (ties by
|
||||
tie_priority) + 1
|
||||
|
||||
Exit code: 0 when every output matches pixel for pixel, 1 on any mismatch,
|
||||
2 on a usage / data error.
|
||||
"""
|
||||
import argparse
|
||||
import json
|
||||
import os
|
||||
import shutil
|
||||
import sys
|
||||
import tempfile
|
||||
import time
|
||||
import urllib.error
|
||||
import urllib.request
|
||||
|
||||
import numpy as np
|
||||
|
||||
try:
|
||||
import rasterio
|
||||
from rasterio.windows import Window
|
||||
except ImportError: # pragma: no cover
|
||||
sys.exit("lp_verify.py needs numpy and rasterio: pip install numpy rasterio")
|
||||
|
||||
NODATA = np.int16(-9999)
|
||||
SLOTS = [
|
||||
"lp-z-biodiversity", "lp-z-lcr", "lp-z-soil-protection", "lp-z-wr",
|
||||
"lp-u-surf-water", "lp-u-soil", "lp-u-crop", "lp-u-veg",
|
||||
"lp-p-irrigation", "lp-p-snow", "lp-p-crop", "lp-p-pasture", "lp-p-tourism",
|
||||
]
|
||||
Z = SLOTS[0:4]
|
||||
U = SLOTS[4:8] # water, soil, crop, veg
|
||||
P = SLOTS[8:13] # sector order 1..5
|
||||
C = ["lp-c-irrigation", "lp-c-snow", "lp-c-crop", "lp-c-pasture", "lp-c-tourism"]
|
||||
OUTPUTS = [
|
||||
"lp-z-integrated", "lp-u-integrated", "lp-z-max", "lp-u-max", "lp-e",
|
||||
*C, "lp-p-max", "lp-v1-conservative", "lp-v2-balanced", "lp-v3-development", "lp-c-max",
|
||||
]
|
||||
OUTPUT_NODE = {
|
||||
"lp-z-integrated": "00", "lp-u-integrated": "01", "lp-z-max": "02", "lp-u-max": "03", "lp-e": "10",
|
||||
**{c: "20" for c in C}, "lp-p-max": "21",
|
||||
"lp-v1-conservative": "30", "lp-v2-balanced": "30", "lp-v3-development": "30", "lp-c-max": "31",
|
||||
}
|
||||
DEFAULT_PARAMS = {"tie_priority": [1, 2, 3, 4, 5], "v1_min_s_e12": 3, "v1_min_s_e3": 4,
|
||||
"v2_min_s_e3": 3, "v2_min_s_e4": 5}
|
||||
|
||||
F32 = np.float32
|
||||
|
||||
|
||||
# ----------------------------------------------------------------- pixel rules
|
||||
def valid(v):
|
||||
return (v >= 1) & (v <= 5)
|
||||
|
||||
|
||||
def cls(v, ok):
|
||||
"""int16(v) for valid pixels (Go truncation), 0 elsewhere."""
|
||||
return np.where(ok, np.trunc(np.where(ok, v, 0)), 0).astype(np.int16)
|
||||
|
||||
|
||||
def classify(x):
|
||||
x = x.astype(np.float32, copy=False)
|
||||
return np.select(
|
||||
[x < F32(1.0), x <= F32(1.8), x <= F32(2.6), x <= F32(3.4), x <= F32(4.2), x <= F32(5.0)],
|
||||
[NODATA, 1, 2, 3, 4, 5], default=NODATA).astype(np.int16)
|
||||
|
||||
|
||||
def quarter(v):
|
||||
# float32(float32(0.25) * float32(v)), as in the Go kernels
|
||||
return F32(0.25) * v.astype(np.float32)
|
||||
|
||||
|
||||
def z_integrated(z):
|
||||
ok = valid(z[0]) & valid(z[1]) & valid(z[2]) & valid(z[3])
|
||||
score = ((quarter(z[0]) + quarter(z[1])) + quarter(z[2])) + quarter(z[3])
|
||||
return np.where(ok, classify(score), NODATA).astype(np.int16)
|
||||
|
||||
|
||||
def u_domain(u):
|
||||
water, soil, crop, veg = u
|
||||
crop_ok, veg_ok = valid(crop), valid(veg)
|
||||
return valid(water) & valid(soil) & (crop_ok | veg_ok), crop_ok, veg_ok
|
||||
|
||||
|
||||
def u_integrated(u):
|
||||
water, soil, crop, veg = u
|
||||
ok, crop_ok, veg_ok = u_domain(u)
|
||||
num = quarter(water) + quarter(soil)
|
||||
den = np.full(num.shape, F32(0.5), dtype=np.float32)
|
||||
num = np.where(crop_ok, num + quarter(crop), num).astype(np.float32)
|
||||
den = np.where(crop_ok, den + F32(0.25), den).astype(np.float32)
|
||||
num = np.where(veg_ok, num + quarter(veg), num).astype(np.float32)
|
||||
den = np.where(veg_ok, den + F32(0.25), den).astype(np.float32)
|
||||
with np.errstate(invalid="ignore", divide="ignore"):
|
||||
score = (num / den).astype(np.float32)
|
||||
return np.where(ok, classify(score), NODATA).astype(np.int16)
|
||||
|
||||
|
||||
def z_max(z):
|
||||
ok = valid(z[0]) & valid(z[1]) & valid(z[2]) & valid(z[3])
|
||||
m = np.maximum.reduce([cls(v, ok) for v in z])
|
||||
return np.where(ok, m, NODATA).astype(np.int16)
|
||||
|
||||
|
||||
def u_max(u):
|
||||
water, soil, crop, veg = u
|
||||
ok, crop_ok, veg_ok = u_domain(u)
|
||||
m = np.maximum(cls(water, ok), cls(soil, ok))
|
||||
m = np.where(crop_ok, np.maximum(m, cls(crop, crop_ok)), m)
|
||||
m = np.where(veg_ok, np.maximum(m, cls(veg, veg_ok)), m)
|
||||
return np.where(ok, m, NODATA).astype(np.int16)
|
||||
|
||||
|
||||
def e_constraint(zi, ui):
|
||||
ok = valid(zi) & valid(ui)
|
||||
return np.where(ok, np.maximum(zi, ui), NODATA).astype(np.int16)
|
||||
|
||||
|
||||
def c_conflict(p, e):
|
||||
ok = valid(p) & valid(e)
|
||||
return np.where(ok, np.minimum(cls(p, ok), e), NODATA).astype(np.int16)
|
||||
|
||||
|
||||
def max_any_valid(layers):
|
||||
oks = [valid(v) for v in layers]
|
||||
m = np.maximum.reduce([cls(v, ok) for v, ok in zip(layers, oks)])
|
||||
anyok = np.logical_or.reduce(oks)
|
||||
return np.where(anyok, m, NODATA).astype(np.int16)
|
||||
|
||||
|
||||
def func_zoning(e, p, params):
|
||||
eok = valid(e)
|
||||
ev = np.where(eok, e, 0).astype(np.int16)
|
||||
base = [valid(v) for v in p]
|
||||
s = [cls(v, ok) for v, ok in zip(p, base)]
|
||||
any_base = np.logical_or.reduce(base)
|
||||
t = params
|
||||
rules = [
|
||||
lambda e_, s_: ((e_ <= 2) & (s_ >= t["v1_min_s_e12"])) | ((e_ == 3) & (s_ >= t["v1_min_s_e3"])),
|
||||
lambda e_, s_: (e_ <= 2) | ((e_ == 3) & (s_ >= t["v2_min_s_e3"])) | ((e_ == 4) & (s_ >= t["v2_min_s_e4"])),
|
||||
lambda e_, s_: e_ <= 4,
|
||||
]
|
||||
outs = []
|
||||
for rule in rules:
|
||||
cand = [b & rule(ev, sk) for b, sk in zip(base, s)]
|
||||
any_cand = np.logical_or.reduce(cand)
|
||||
max_s = np.maximum.reduce([np.where(ck, sk, 0) for ck, sk in zip(cand, s)]).astype(np.int16)
|
||||
sector = np.zeros(ev.shape, dtype=np.int16)
|
||||
done = np.zeros(ev.shape, dtype=bool)
|
||||
for code in t["tie_priority"]:
|
||||
k = code - 1
|
||||
hit = cand[k] & (s[k] == max_s) & ~done
|
||||
sector[hit] = code
|
||||
done |= hit
|
||||
v = np.where(any_cand, sector + 1, 1).astype(np.int16)
|
||||
outs.append(np.where(eok & any_base, v, NODATA).astype(np.int16))
|
||||
return outs
|
||||
|
||||
|
||||
def expected_block(inp, params):
|
||||
"""inp: dict slot -> float64 array. Returns dict output group -> int16."""
|
||||
z = [inp[g] for g in Z]
|
||||
u = [inp[g] for g in U]
|
||||
p = [inp[g] for g in P]
|
||||
out = {
|
||||
"lp-z-integrated": z_integrated(z),
|
||||
"lp-u-integrated": u_integrated(u),
|
||||
"lp-z-max": z_max(z),
|
||||
"lp-u-max": u_max(u),
|
||||
}
|
||||
e = e_constraint(out["lp-z-integrated"], out["lp-u-integrated"])
|
||||
out["lp-e"] = e
|
||||
for cg, pk in zip(C, p):
|
||||
out[cg] = c_conflict(pk, e)
|
||||
out["lp-p-max"] = max_any_valid(p)
|
||||
v1, v2, v3 = func_zoning(e, p, params)
|
||||
out["lp-v1-conservative"], out["lp-v2-balanced"], out["lp-v3-development"] = v1, v2, v3
|
||||
out["lp-c-max"] = max_any_valid([out[c] for c in C])
|
||||
return out
|
||||
|
||||
|
||||
# ------------------------------------------------------------------- HTTP (GET)
|
||||
def http_get(url, timeout=120, retries=3):
|
||||
req = urllib.request.Request(url, method="GET", headers={"User-Agent": "lp_verify/1.0"})
|
||||
last = None
|
||||
for attempt in range(retries):
|
||||
try:
|
||||
return urllib.request.urlopen(req, timeout=timeout)
|
||||
except urllib.error.HTTPError as e:
|
||||
if e.code < 500:
|
||||
raise
|
||||
last = e
|
||||
except (urllib.error.URLError, TimeoutError, ConnectionError) as e:
|
||||
last = e
|
||||
time.sleep(2 * (attempt + 1))
|
||||
raise last
|
||||
|
||||
|
||||
def get_json(url):
|
||||
with http_get(url) as r:
|
||||
return json.loads(r.read().decode("utf-8"))
|
||||
|
||||
|
||||
def download(url, path, wait=0):
|
||||
"""GET url into path (reused when already cached). A 409 means the COG is
|
||||
still being produced (conversion is asynchronous): retry for up to `wait`
|
||||
seconds."""
|
||||
if os.path.exists(path) and os.path.getsize(path) > 0:
|
||||
return "cached"
|
||||
tmp = path + ".part"
|
||||
deadline = time.time() + wait
|
||||
while True:
|
||||
try:
|
||||
r = http_get(url, timeout=600)
|
||||
break
|
||||
except urllib.error.HTTPError as e:
|
||||
if e.code != 409 or time.time() >= deadline:
|
||||
raise
|
||||
time.sleep(2)
|
||||
with r, open(tmp, "wb") as f:
|
||||
shutil.copyfileobj(r, f, length=4 << 20)
|
||||
os.replace(tmp, path)
|
||||
return "%.1f MB" % (os.path.getsize(path) / 1e6)
|
||||
|
||||
|
||||
# ----------------------------------------------------------------------- main
|
||||
def parse_points(s):
|
||||
pts = []
|
||||
for part in filter(None, (x.strip() for x in s.split(";"))):
|
||||
c, r = part.split(",")
|
||||
pts.append((int(c), int(r)))
|
||||
return pts
|
||||
|
||||
|
||||
def main():
|
||||
ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
|
||||
ap.add_argument("--base-url", default=os.environ.get("BASE_URL", "http://localhost:18080"))
|
||||
ap.add_argument("--run", required=True, help="landscape run id")
|
||||
ap.add_argument("--cache", help="directory for downloaded rasters (kept; existing files are reused)")
|
||||
ap.add_argument("--inputs-from", choices=["cog", "download"], default="cog",
|
||||
help="read inputs from their COG (default) or the original upload (what the node read)")
|
||||
ap.add_argument("--block-rows", type=int, default=512)
|
||||
ap.add_argument("--block-cols", type=int, default=4096)
|
||||
ap.add_argument("--wait", type=int, default=180,
|
||||
help="seconds to keep retrying a COG that is not ready yet (HTTP 409), default 180")
|
||||
ap.add_argument("--points", help='spot-check pixels "col,row;col,row;..." and print their values')
|
||||
args = ap.parse_args()
|
||||
base = args.base_url.rstrip("/")
|
||||
|
||||
run = get_json("%s/landscape/runs/%s" % (base, args.run))
|
||||
params = dict(DEFAULT_PARAMS)
|
||||
raw_params = run.get("params") or {}
|
||||
for k in DEFAULT_PARAMS:
|
||||
if raw_params.get(k):
|
||||
params[k] = raw_params[k]
|
||||
print("run %s status=%s from_stage=%s" % (run["id"], run["status"], run.get("from_stage")))
|
||||
print("params %s" % json.dumps(params))
|
||||
nodes = sorted(run.get("nodes") or [], key=lambda n: n["node_code"])
|
||||
print("nodes %s" % " ".join("%s:%s" % (n["node_code"], n["status"]) for n in nodes))
|
||||
if run["status"] != "succeeded":
|
||||
print("ERROR: run is not succeeded; nothing to verify")
|
||||
return 2
|
||||
|
||||
errors = []
|
||||
slot_ids, out_ids = {}, {}
|
||||
for n in nodes:
|
||||
for ref in n.get("inputs") or []:
|
||||
g, did = ref["group"], ref["dataset_id"]
|
||||
if g in SLOTS:
|
||||
if slot_ids.setdefault(g, did) != did:
|
||||
errors.append("slot %s resolved to two datasets (%s, %s)" % (g, slot_ids[g], did))
|
||||
for ref in n.get("outputs") or []:
|
||||
out_ids[ref["group"]] = ref["dataset_id"]
|
||||
if OUTPUT_NODE.get(ref["group"]) != n["node_code"]:
|
||||
errors.append("output %s reported by node %s" % (ref["group"], n["node_code"]))
|
||||
# chain consistency: an intermediate input must be what this run produced
|
||||
for n in nodes:
|
||||
for ref in n.get("inputs") or []:
|
||||
g = ref["group"]
|
||||
if g in OUTPUTS and out_ids.get(g) != ref["dataset_id"]:
|
||||
errors.append("node %s read %s=%s but the run's output is %s" % (n["node_code"], g, ref["dataset_id"], out_ids.get(g)))
|
||||
missing = [g for g in SLOTS if g not in slot_ids] + [g for g in OUTPUTS if g not in out_ids]
|
||||
if missing:
|
||||
errors.append("run does not reference: %s" % ", ".join(missing))
|
||||
if errors:
|
||||
for e in errors:
|
||||
print("ERROR:", e)
|
||||
return 2
|
||||
|
||||
cache = args.cache or tempfile.mkdtemp(prefix="lp_verify_")
|
||||
os.makedirs(cache, exist_ok=True)
|
||||
files = {}
|
||||
t0 = time.time()
|
||||
for g in SLOTS + OUTPUTS:
|
||||
did = slot_ids.get(g) or out_ids[g]
|
||||
if g in SLOTS and args.inputs_from == "download":
|
||||
url, path = "%s/datasets/%s/download" % (base, did), os.path.join(cache, did + ".orig.tif")
|
||||
else:
|
||||
url, path = "%s/datasets/%s.cog" % (base, did), os.path.join(cache, did + ".cog.tif")
|
||||
try:
|
||||
info = download(url, path, args.wait)
|
||||
except urllib.error.HTTPError as e:
|
||||
print("ERROR: GET %s -> HTTP %d (output not ready or missing?)" % (url, e.code))
|
||||
return 2
|
||||
files[g] = path
|
||||
print("fetch %-22s %s %s" % (g, did, info))
|
||||
print("downloaded in %.1f s to %s" % (time.time() - t0, cache))
|
||||
|
||||
with rasterio.Env(GDAL_CACHEMAX=512):
|
||||
ds = {g: rasterio.open(files[g]) for g in SLOTS + OUTPUTS}
|
||||
ref = ds[SLOTS[0]]
|
||||
for g, d in ds.items():
|
||||
if (d.width, d.height) != (ref.width, ref.height) or not d.transform.almost_equals(ref.transform) or d.crs != ref.crs:
|
||||
errors.append("%s grid %dx%d %s differs from %s %dx%d %s" % (
|
||||
g, d.width, d.height, d.crs, SLOTS[0], ref.width, ref.height, ref.crs))
|
||||
for g in OUTPUTS:
|
||||
if ds[g].dtypes[0] != "int16" or ds[g].nodata != -9999:
|
||||
errors.append("output %s: dtype %s nodata %s (want int16, -9999)" % (g, ds[g].dtypes[0], ds[g].nodata))
|
||||
if errors:
|
||||
for e in errors:
|
||||
print("ERROR:", e)
|
||||
return 2
|
||||
W, H = ref.width, ref.height
|
||||
print("grid %dx%d %s pixel %.3f" % (W, H, ref.crs, ref.transform.a))
|
||||
|
||||
if args.points:
|
||||
return spot_check(ds, parse_points(args.points), params)
|
||||
|
||||
mism = {g: 0 for g in OUTPUTS}
|
||||
validn = {g: 0 for g in OUTPUTS}
|
||||
first = {}
|
||||
t0 = time.time()
|
||||
for r0 in range(0, H, args.block_rows):
|
||||
h = min(args.block_rows, H - r0)
|
||||
for c0 in range(0, W, args.block_cols):
|
||||
w = min(args.block_cols, W - c0)
|
||||
win = Window(c0, r0, w, h)
|
||||
inp = {g: ds[g].read(1, window=win, masked=False).astype(np.float64) for g in SLOTS}
|
||||
exp = expected_block(inp, params)
|
||||
for g in OUTPUTS:
|
||||
act = ds[g].read(1, window=win, masked=False)
|
||||
bad = exp[g] != act
|
||||
nbad = int(bad.sum())
|
||||
validn[g] += int((exp[g] != NODATA).sum())
|
||||
if nbad:
|
||||
mism[g] += nbad
|
||||
if g not in first:
|
||||
rr, cc = np.argwhere(bad)[0]
|
||||
first[g] = (c0 + int(cc), r0 + int(rr), int(exp[g][rr, cc]), int(act[rr, cc]))
|
||||
print("compared in %.1f s" % (time.time() - t0))
|
||||
for d in ds.values():
|
||||
d.close()
|
||||
|
||||
print()
|
||||
print("%-20s %-4s %-38s %12s %12s %10s" % ("output", "node", "dataset_id", "pixels", "valid(exp)", "mismatch"))
|
||||
total = 0
|
||||
for g in OUTPUTS:
|
||||
total += mism[g]
|
||||
line = "%-20s %-4s %-38s %12d %12d %10d" % (g, OUTPUT_NODE[g], out_ids[g], W * H, validn[g], mism[g])
|
||||
if g in first:
|
||||
line += " first at col=%d row=%d expected=%d actual=%d" % first[g]
|
||||
print(line)
|
||||
print()
|
||||
print("TOTAL mismatching pixels: %d -> %s" % (total, "PASS" if total == 0 else "FAIL"))
|
||||
return 0 if total == 0 else 1
|
||||
|
||||
|
||||
def spot_check(ds, points, params):
|
||||
bad = 0
|
||||
shown = ["lp-z-integrated", "lp-u-integrated", "lp-e", *C, "lp-v1-conservative", "lp-v2-balanced", "lp-v3-development"]
|
||||
for col, row in points:
|
||||
win = Window(col, row, 1, 1)
|
||||
inp = {g: ds[g].read(1, window=win, masked=False).astype(np.float64) for g in SLOTS}
|
||||
exp = expected_block(inp, params)
|
||||
vals = " ".join("%s=%d" % (g.replace("lp-", ""), int(inp[g][0, 0])) for g in SLOTS)
|
||||
print("point (%d,%d) inputs: %s" % (col, row, vals))
|
||||
for g in shown:
|
||||
act = int(ds[g].read(1, window=win, masked=False)[0, 0])
|
||||
e = int(exp[g][0, 0])
|
||||
ok = act == e
|
||||
bad += 0 if ok else 1
|
||||
print(" %-20s expected=%6d actual=%6d %s" % (g, e, act, "OK" if ok else "MISMATCH"))
|
||||
print("SPOT CHECK: %d points, %d mismatches -> %s" % (len(points), bad, "PASS" if bad == 0 else "FAIL"))
|
||||
return 0 if bad == 0 else 1
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
2
test/acceptance/requirements.txt
Normal file
2
test/acceptance/requirements.txt
Normal file
@@ -0,0 +1,2 @@
|
||||
numpy>=1.24
|
||||
rasterio>=1.3
|
||||
131
test/acceptance/scan.sh
Executable file
131
test/acceptance/scan.sh
Executable file
@@ -0,0 +1,131 @@
|
||||
#!/usr/bin/env bash
|
||||
# Security re-scan (Приложение 4 «Отчет об устранении уязвимостей»):
|
||||
# 1. govulncheck, source mode — vulnerabilities reachable from the code
|
||||
# 2. govulncheck -mode=binary — the binary shipped in the image
|
||||
# 3. gosec — static analysis (HIGH severity counted)
|
||||
# 4. trivy fs — dependencies, secrets, IaC misconfig
|
||||
# 5. trivy image — the application image (built here)
|
||||
# 6. golangci-lint (if installed) — code quality, informational only
|
||||
#
|
||||
# test/acceptance/scan.sh
|
||||
#
|
||||
# Environment: SCAN_IMAGE (default gis-scan:local, built from
|
||||
# build/package/Dockerfile), SKIP_IMAGE=1 (no docker: skip 2 via image and 5;
|
||||
# the binary for 2 is then built locally with cgo + GDAL).
|
||||
# govulncheck has no severity scale: every vulnerability reachable from the
|
||||
# code (source) or present in the binary symbol table is counted as High.
|
||||
# Prints High/Critical per tool; exit code 1 if any High/Critical is found.
|
||||
# Raw reports: test/acceptance/results/scan-<timestamp>/.
|
||||
set -euo pipefail
|
||||
|
||||
HERE="$(cd "$(dirname "$0")" && pwd)"
|
||||
ROOT="$(cd "$HERE/../.." && pwd)"
|
||||
STAMP="$(date +%Y%m%d-%H%M%S)"
|
||||
OUT="$HERE/results/scan-$STAMP"; mkdir -p "$OUT"
|
||||
SCAN_IMAGE="${SCAN_IMAGE:-gis-scan:local}"
|
||||
SKIP_IMAGE="${SKIP_IMAGE:-0}"
|
||||
GOBIN_DIR="$(go env GOPATH 2>/dev/null)/bin"
|
||||
PATH="$PATH:$GOBIN_DIR"
|
||||
cd "$ROOT"
|
||||
|
||||
ROWS=""; FAIL=0
|
||||
row() { # tool target high critical note
|
||||
ROWS="${ROWS}$(printf '%-26s | %-40s | %5s | %8s | %s' "$1" "$2" "$3" "$4" "$5")"$'\n'
|
||||
case "$3$4" in *[!0-9]*) ;; *) [ "$3" -eq 0 ] && [ "$4" -eq 0 ] || FAIL=1;; esac
|
||||
}
|
||||
have() { command -v "$1" >/dev/null 2>&1; }
|
||||
log() { echo "[$(date +%H:%M:%S)] $*"; }
|
||||
|
||||
log "commit $(git rev-parse --short HEAD 2>/dev/null || echo '?'), $(go version)"
|
||||
|
||||
# --- govulncheck (source) -------------------------------------------------
|
||||
gv_count() { # json stream -> number of distinct OSV ids with a call trace (reachable)
|
||||
jq -rs '[.[] | select(.finding != null) | .finding | select((.trace // []) | length > 0)
|
||||
| select(.trace[0].function != null) | .osv] | unique | length' "$1"
|
||||
}
|
||||
gv_ids() { jq -rs '[.[] | select(.finding != null) | .finding | select(.trace[0].function != null) | .osv] | unique | join(", ")' "$1"; }
|
||||
if have govulncheck; then
|
||||
log "govulncheck ./... (source)"
|
||||
govulncheck -format json ./... > "$OUT/govulncheck-source.json" 2> "$OUT/govulncheck-source.err" || true
|
||||
n="$(gv_count "$OUT/govulncheck-source.json")"
|
||||
mods="$(jq -rs '[.[] | select(.finding != null) | .finding.osv] | unique | length' "$OUT/govulncheck-source.json")"
|
||||
row "govulncheck (source)" "./..." "$n" 0 "reachable: ${n}$( [ "$n" -gt 0 ] && echo " ($(gv_ids "$OUT/govulncheck-source.json"))"); in dependencies but not called: $((mods - n))"
|
||||
else row "govulncheck (source)" "./..." "n/a" "n/a" "not installed (go install golang.org/x/vuln/cmd/govulncheck@latest)"; fi
|
||||
|
||||
# --- application image + binary -------------------------------------------
|
||||
BIN=""
|
||||
if [ "$SKIP_IMAGE" != 1 ] && have docker; then
|
||||
log "docker build -t $SCAN_IMAGE -f build/package/Dockerfile ."
|
||||
docker build -q -t "$SCAN_IMAGE" -f build/package/Dockerfile . > "$OUT/docker-build.txt"
|
||||
cid="$(docker create "$SCAN_IMAGE")"
|
||||
docker cp "$cid:/usr/local/bin/gis" "$OUT/gis" >/dev/null; docker rm "$cid" >/dev/null
|
||||
BIN="$OUT/gis"; BIN_SRC="image $SCAN_IMAGE:/usr/local/bin/gis"
|
||||
else
|
||||
log "go build (cgo + GDAL) for binary-mode scan"
|
||||
CGO_ENABLED=1 go build -trimpath -o "$OUT/gis" ./cmd/gis; BIN="$OUT/gis"; BIN_SRC="local build"
|
||||
fi
|
||||
|
||||
# --- govulncheck (binary) -------------------------------------------------
|
||||
if have govulncheck; then
|
||||
log "govulncheck -mode=binary ($BIN_SRC)"
|
||||
govulncheck -mode=binary -format json "$BIN" > "$OUT/govulncheck-binary.json" 2> "$OUT/govulncheck-binary.err" || true
|
||||
n="$(gv_count "$OUT/govulncheck-binary.json")"
|
||||
gov="$(go version "$BIN" 2>/dev/null | awk '{print $2}')"
|
||||
row "govulncheck (binary)" "$BIN_SRC" "$n" 0 "built with $gov; vulnerable symbols present: $n$( [ "$n" -gt 0 ] && echo " ($(gv_ids "$OUT/govulncheck-binary.json"))")"
|
||||
fi
|
||||
|
||||
# --- gosec -----------------------------------------------------------------
|
||||
if have gosec; then
|
||||
log "gosec ./..."
|
||||
gosec -quiet -fmt json -out "$OUT/gosec.json" ./... 2> "$OUT/gosec.err" || true
|
||||
hi="$(jq '[.Issues[]? | select(.severity=="HIGH")] | length' "$OUT/gosec.json")"
|
||||
me="$(jq '[.Issues[]? | select(.severity=="MEDIUM")] | length' "$OUT/gosec.json")"
|
||||
lo="$(jq '[.Issues[]? | select(.severity=="LOW")] | length' "$OUT/gosec.json")"
|
||||
ns="$(jq '.Stats.nosec // 0' "$OUT/gosec.json")"
|
||||
row "gosec" "./... ($(jq '.Stats.files' "$OUT/gosec.json") files)" "$hi" 0 "MEDIUM $me, LOW $lo; #nosec-annotated $ns"
|
||||
else row "gosec" "./..." "n/a" "n/a" "not installed"; fi
|
||||
|
||||
# --- trivy ------------------------------------------------------------------
|
||||
trivy_counts() { # json -> "high critical"
|
||||
jq -r '[.Results[]? | (.Vulnerabilities[]?, .Secrets[]?, .Misconfigurations[]?) | .Severity] as $s
|
||||
| "\([$s[] | select(.=="HIGH")] | length) \([$s[] | select(.=="CRITICAL")] | length)"' "$1"
|
||||
}
|
||||
if have trivy; then
|
||||
log "trivy fs (vuln, secret, misconfig)"
|
||||
trivy fs --quiet --scanners vuln,secret,misconfig --format json --output "$OUT/trivy-fs.json" \
|
||||
--skip-dirs test/acceptance/results --skip-dirs graft . 2> "$OUT/trivy-fs.err" || true
|
||||
read -r h c <<< "$(trivy_counts "$OUT/trivy-fs.json")"
|
||||
all="$(jq '[.Results[]? | (.Vulnerabilities[]?, .Secrets[]?, .Misconfigurations[]?)] | length' "$OUT/trivy-fs.json")"
|
||||
row "trivy fs" "repository (go.mod, Dockerfile, compose)" "$h" "$c" "all severities: $all"
|
||||
if [ -n "$BIN" ] && [ "$SKIP_IMAGE" != 1 ] && have docker; then
|
||||
log "trivy image $SCAN_IMAGE"
|
||||
trivy image --quiet --format json --output "$OUT/trivy-image.json" "$SCAN_IMAGE" 2> "$OUT/trivy-image.err" || true
|
||||
read -r h c <<< "$(trivy_counts "$OUT/trivy-image.json")"
|
||||
all="$(jq '[.Results[]? | .Vulnerabilities[]?] | length' "$OUT/trivy-image.json")"
|
||||
os="$(jq -r '.Metadata.OS | "\(.Family) \(.Name)"' "$OUT/trivy-image.json")"
|
||||
row "trivy image" "$SCAN_IMAGE ($os)" "$h" "$c" "all severities: $all"
|
||||
fi
|
||||
else row "trivy" "-" "n/a" "n/a" "not installed"; fi
|
||||
|
||||
# --- golangci-lint (informational) ------------------------------------------
|
||||
if have golangci-lint; then
|
||||
log "golangci-lint run ./..."
|
||||
note=""
|
||||
golangci-lint run --output.json.path "$OUT/golangci-lint.json" ./... > "$OUT/golangci-lint.txt" 2>&1 || true
|
||||
if grep -q "unsupported version of the configuration" "$OUT/golangci-lint.txt"; then
|
||||
# .golangci.yml is in the v1 format; a v2 binary refuses it — use the default linters
|
||||
note=" (.golangci.yml is v1 format, $(golangci-lint version 2>/dev/null | awk '{print $4}') ran with --no-config)"
|
||||
golangci-lint run --no-config --output.json.path "$OUT/golangci-lint.json" ./... > "$OUT/golangci-lint.txt" 2>&1 || true
|
||||
fi
|
||||
n="$(jq '[.Issues[]?] | length' "$OUT/golangci-lint.json" 2>/dev/null || echo '?')"
|
||||
row "golangci-lint" "./..." "-" "-" "issues: $n$note; no severity scale, informational"
|
||||
fi
|
||||
|
||||
echo
|
||||
printf '%-26s | %-40s | %5s | %8s | %s\n' "Tool" "Target" "High" "Critical" "Notes"
|
||||
printf '%s\n' "---------------------------+------------------------------------------+-------+----------+------------------------------"
|
||||
printf '%s' "$ROWS"
|
||||
echo
|
||||
if [ "$FAIL" = 0 ]; then echo "RESULT: 0 High / 0 Critical -> PASS"; else echo "RESULT: High/Critical findings present -> FAIL"; fi
|
||||
echo "raw reports: $OUT"
|
||||
exit "$FAIL"
|
||||
101
test/acceptance/seed.sh
Executable file
101
test/acceptance/seed.sh
Executable file
@@ -0,0 +1,101 @@
|
||||
#!/usr/bin/env bash
|
||||
# Seeds the acceptance stand with datasets for the load test (Приложение 3,
|
||||
# п. 3.3), so the read endpoints work on a realistic catalogue.
|
||||
#
|
||||
# test/acceptance/seed.sh create (idempotent) category "acc-load":
|
||||
# SEED_VECTOR (25) vector GeoJSON,
|
||||
# SEED_KATO (25) vector_with_kato GeoJSON with
|
||||
# KATO mapping (14 districts x 3 years each),
|
||||
# SEED_RASTER (50) GeoTIFF rasters; the
|
||||
# vector_with_kato datasets form the group
|
||||
# "acc-load-population"
|
||||
# test/acceptance/seed.sh clean delete everything seed.sh created
|
||||
#
|
||||
# Writes the ids load.sh needs to test/acceptance/results/seed.env.
|
||||
# WRITES to the stand: never point it at production.
|
||||
set -euo pipefail
|
||||
|
||||
HERE="$(cd "$(dirname "$0")" && pwd)"
|
||||
BASE_URL="${BASE_URL:-http://localhost:18080}"
|
||||
BASE_URL="${BASE_URL%/}"
|
||||
PYTHON="${PYTHON:-python3}"
|
||||
SEED_VECTOR="${SEED_VECTOR:-25}"
|
||||
SEED_KATO="${SEED_KATO:-25}"
|
||||
SEED_RASTER="${SEED_RASTER:-50}"
|
||||
CAT_CODE=acc-load
|
||||
GRP_CODE=acc-load-population
|
||||
ENV_OUT="$HERE/results/seed.env"
|
||||
mkdir -p "$HERE/results"
|
||||
case "$BASE_URL" in
|
||||
*dssgis.dwh.kz*) echo "refusing to write to $BASE_URL (production)" >&2; exit 2;;
|
||||
esac
|
||||
J=(-H 'Content-Type: application/json')
|
||||
|
||||
cat_id() { curl -fsS "$BASE_URL/categories" | jq -r --arg c "$CAT_CODE" '.[]|select(.code==$c)|.id' | head -n1; }
|
||||
grp_id() { curl -fsS "$BASE_URL/dataset-groups?code=$GRP_CODE" | jq -r '.data[0].id // empty'; }
|
||||
ids_in_category() { # all dataset ids of a category (paginated)
|
||||
local cat="$1" page=1 out
|
||||
while :; do
|
||||
out="$(curl -fsS "$BASE_URL/datasets?category_id=$cat&page_size=100&page=$page")"
|
||||
jq -r '.data[].id' <<< "$out"
|
||||
[ "$page" -lt "$(jq -r .total_pages <<< "$out")" ] || break
|
||||
page=$((page + 1))
|
||||
done
|
||||
}
|
||||
|
||||
if [ "${1:-}" = clean ]; then
|
||||
CAT="$(cat_id || true)"; GRP="$(grp_id || true)"
|
||||
if [ -n "$CAT" ]; then
|
||||
n=0; for id in $(ids_in_category "$CAT"); do curl -sS -o /dev/null -X DELETE "$BASE_URL/datasets/$id"; n=$((n + 1)); done
|
||||
echo "deleted $n datasets"
|
||||
fi
|
||||
[ -n "$GRP" ] && curl -sS -o /dev/null -w "group delete %{http_code}\n" -X DELETE "$BASE_URL/dataset-groups/$GRP"
|
||||
[ -n "$CAT" ] && curl -sS -o /dev/null -w "category delete %{http_code}\n" -X DELETE "$BASE_URL/categories/$CAT"
|
||||
rm -f "$ENV_OUT"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
CAT="$(cat_id || true)"
|
||||
if [ -z "$CAT" ]; then
|
||||
CAT="$(curl -fsS -X POST "$BASE_URL/categories" "${J[@]}" -d "{\"code\":\"$CAT_CODE\",\"name\":\"Нагрузочное тестирование\",\"description\":\"Данные для test/acceptance/load.sh\"}" | jq -r .id)"
|
||||
fi
|
||||
GRP="$(grp_id || true)"
|
||||
if [ -z "$GRP" ]; then
|
||||
GRP="$(curl -fsS -X POST "$BASE_URL/dataset-groups" "${J[@]}" -d "{\"code\":\"$GRP_CODE\",\"name\":\"Население СКО (нагрузочный тест)\"}" | jq -r .id)"
|
||||
fi
|
||||
|
||||
want=$((SEED_VECTOR + SEED_KATO + SEED_RASTER))
|
||||
have="$(curl -fsS "$BASE_URL/datasets?category_id=$CAT&status=ready&page_size=1" | jq -r .total)"
|
||||
if [ "$have" -lt "$want" ]; then
|
||||
echo "seeding $BASE_URL: $SEED_VECTOR vector, $SEED_KATO vector_with_kato, $SEED_RASTER raster (category $CAT_CODE)"
|
||||
TMP="$(mktemp -d "${TMPDIR:-/tmp}/gis-seed.XXXXXX")"; trap 'rm -rf "$TMP"' EXIT
|
||||
"$PYTHON" "$HERE/lib/helpers.py" ndvi "$TMP/ndvi_seed.tif"
|
||||
upload() { curl -fsS -X POST "$BASE_URL/datasets" "$@" | jq -r .id; }
|
||||
for i in $(seq 1 "$SEED_VECTOR"); do
|
||||
upload -F "file=@$HERE/data/plots_nko.geojson" -F file_type=vector -F "category_id=$CAT" -F "name=Участки $i" >/dev/null
|
||||
done
|
||||
kato_ids=""
|
||||
for i in $(seq 1 "$SEED_KATO"); do
|
||||
y=$((2000 + i))
|
||||
kato_ids="$kato_ids $(upload -F "file=@$HERE/data/population_kato.geojson" -F file_type=vector_with_kato -F "category_id=$CAT" \
|
||||
-F "name=Население $i" -F "dg_id=$GRP" -F "start_date=$y-01-01" -F "end_date=$y-12-31")"
|
||||
done
|
||||
for i in $(seq 1 "$SEED_RASTER"); do
|
||||
upload -F "file=@$TMP/ndvi_seed.tif" -F file_type=raster -F "category_id=$CAT" -F "name=NDVI $i" >/dev/null
|
||||
done
|
||||
MAP='{"kato_column":"kato","year_columns":[{"column":"Y2022","start_date":"2022-01-01","end_date":"2022-12-31"},{"column":"Y2023","start_date":"2023-01-01","end_date":"2023-12-31"},{"column":"Y2024","start_date":"2024-01-01","end_date":"2024-12-31"}]}'
|
||||
for id in $kato_ids; do
|
||||
for _ in $(seq 1 120); do
|
||||
[ "$(curl -fsS "$BASE_URL/datasets/$id" | jq -r .status)" = awaiting_mapping ] && break; sleep 1
|
||||
done
|
||||
curl -fsS -o /dev/null -X POST "$BASE_URL/datasets/$id/mapping" "${J[@]}" -d "$MAP"
|
||||
done
|
||||
for _ in $(seq 1 300); do
|
||||
have="$(curl -fsS "$BASE_URL/datasets?category_id=$CAT&status=ready&page_size=1" | jq -r .total)"
|
||||
[ "$have" -ge "$want" ] && break; sleep 1
|
||||
done
|
||||
fi
|
||||
VWK="$(curl -fsS "$BASE_URL/datasets?category_id=$CAT&file_type=vector_with_kato&status=ready&page_size=1" | jq -r '.data[0].id')"
|
||||
printf 'CAT_ID=%s\nGROUP_ID=%s\nVWK_ID=%s\n' "$CAT" "$GRP" "$VWK" > "$ENV_OUT"
|
||||
echo "seed: category $CAT_CODE has $have ready datasets (wanted $want); ids in $ENV_OUT"
|
||||
cat "$ENV_OUT"
|
||||
54
test/acceptance/stand.sh
Executable file
54
test/acceptance/stand.sh
Executable file
@@ -0,0 +1,54 @@
|
||||
#!/usr/bin/env bash
|
||||
# Brings the isolated acceptance stand (docker compose project gis-acceptance)
|
||||
# up or down.
|
||||
#
|
||||
# test/acceptance/stand.sh up build the image from this checkout, start
|
||||
# all services, apply migrations, import the
|
||||
# 14 districts, wait for /readyz
|
||||
# test/acceptance/stand.sh down stop and remove the stand with its volumes
|
||||
# test/acceptance/stand.sh ps|logs pass-through to docker compose
|
||||
#
|
||||
# Environment: MINIO_IMAGE, POSTGIS_PLATFORM, APP_IMAGE, HTTP_MAX_UPLOAD_BYTES,
|
||||
# LP_MAX_WORKERS, LP_MAX_NODES (see compose.acceptance.yml).
|
||||
set -euo pipefail
|
||||
|
||||
HERE="$(cd "$(dirname "$0")" && pwd)"
|
||||
ROOT="$(cd "$HERE/../.." && pwd)"
|
||||
PROJECT=gis-acceptance
|
||||
compose() {
|
||||
docker compose -p "$PROJECT" -f "$ROOT/deployments/docker-compose.yml" -f "$HERE/compose.acceptance.yml" "$@"
|
||||
}
|
||||
|
||||
cmd="${1:-up}"
|
||||
shift || true
|
||||
case "$cmd" in
|
||||
up)
|
||||
echo "== build application image from $(git -C "$ROOT" rev-parse --short HEAD 2>/dev/null || echo 'this checkout')"
|
||||
compose build app worker
|
||||
echo "== start infrastructure ($PROJECT)"
|
||||
compose up -d --wait postgres minio rabbitmq
|
||||
echo "== migrate up"
|
||||
compose run --rm --no-deps app migrate up
|
||||
echo "== import districts"
|
||||
compose run --rm --no-deps app import-districts /stubs/districts.geojson
|
||||
compose up -d app worker
|
||||
echo "== wait for http://localhost:18080/readyz"
|
||||
for _ in $(seq 1 60); do
|
||||
if curl -fsS http://localhost:18080/readyz >/dev/null 2>&1; then
|
||||
curl -sS http://localhost:18080/readyz; echo
|
||||
compose ps --format 'table {{.Service}}\t{{.Image}}\t{{.Status}}\t{{.Ports}}'
|
||||
exit 0
|
||||
fi
|
||||
sleep 2
|
||||
done
|
||||
echo "stand did not become ready" >&2
|
||||
compose logs --tail 50 app worker >&2
|
||||
exit 1
|
||||
;;
|
||||
down)
|
||||
compose down -v --remove-orphans
|
||||
;;
|
||||
*)
|
||||
compose "$cmd" "$@"
|
||||
;;
|
||||
esac
|
||||
Reference in New Issue
Block a user