gis/README.md

# gis

A Go service scaffold following [golang-standards/project-layout](https://github.com/golang-standards/project-layout),
with cleanly separated layers: HTTP transport → services → repositories, plus
RabbitMQ messaging and embedded database migrations. Single binary, three
subcommands.

## Layout

```
cmd/gis/                 binary entrypoint
internal/
  cli/                   cobra commands: serve, worker, migrate
  config/                env-based configuration
  app/                   composition root (wires all dependencies)
  domain/                entities, enums, sentinel errors
  repository/postgres/   pgx-backed repositories
  service/               business logic
  transport/http/        chi router, middleware, handlers
  storage/s3/            MinIO/S3 object storage
  messaging/rabbitmq/    connection, publisher, consumer
  landscape/             Landscape Program graph, kernels, window executor
  raster/                GDAL raster I/O (COG conversion + godal engine)
  platform/logger/       slog setup
pkg/httputil/            generic JSON/validation HTTP helpers
migrations/              embedded goose SQL migrations
configs/                 .env.example
deployments/             docker-compose (postgres, minio, rabbitmq)
build/package/           Dockerfile
api/openapi.yaml         OpenAPI 3.1.1 spec (embedded + served at /openapi.yaml)
```

## Domain

- **Category** — hierarchical (self-referencing `parent_id`). Full CRUD; cycle-safe
  on update.
- **DatasetGroup** — a named group of "одноимённые" datasets (see *Dataset
  groups* below). Full CRUD; deleting a group still referenced by datasets is
  rejected with 409.
- **Dataset** — a geo file uploaded to S3/MinIO (`file_type`: `vector_with_kato |
  vector | raster`), belonging to one Category. Carries `code`/`name`/`description`/
  `unit` metadata, a user-defined `meta` (JSONB) blob, an `automated` flag, a
  `status` lifecycle field (defaults to `pending`), `properties` (JSONB, populated
  from the file's attribute table), and a PostGIS `geometry` footprint stored in
  EPSG:4326 (returned as GeoJSON, with a STAC-style `bbox` array for rasters).
  Upload / list / get / download / delete (delete also removes the stored object).
Uploads are validated three ways before being stored: the `file_type` enum, the
file **extension** (must be allowed for the type), and a **content** magic-byte
check (TIFF for `.tif`, ZIP for `.zip`, SQLite for `.gpkg`, JSON for `.geojson`)
so mislabeled files are rejected with 422 up front.

Every uploaded file is then processed asynchronously by the worker, dispatched by
`file_type`:

- **`vector`** — the attribute table is parsed and stored (as a JSON array of row
  objects) in `properties` (`status` `processing` → `ready`).
- **`raster`** — converted to a **Cloud-Optimized GeoTIFF** via `gdal_translate
  -of COG` (`processing` → `ready`); the COG is stored under `cog_storage_key`
  (the original is kept) and the footprint `geometry` + `bbox` are read from the
  raster extent. Requires GDAL in the worker image (`gdal-tools`).
- **`vector_with_kato`** — the column-selection flow below (`parsing` →
  `awaiting_mapping` → `extracting` → `ready`).
- **events** + the example RabbitMQ consumer/publisher are a generic messaging
  scaffold kept alongside the real async flows.

### vector_with_kato two-phase flow

Uploading a `vector_with_kato` file (zipped shapefile, GeoJSON, or GeoPackage)
triggers asynchronous parsing of its attribute table, after which the user maps
the KATO column and the year columns:

1. `POST /datasets` with `file_type=vector_with_kato` → dataset created with
   `status=parsing`; a `dataset.parse` job is published to RabbitMQ.
2. The **worker** consumes the job, parses the file's columns (with sample
   values; CP1251/Cyrillic aware for shapefiles) and stores them in
   `attribute_columns`; `status` → `awaiting_mapping` (or `failed` with
   `parse_error`).
3. The client polls `GET /datasets/{id}` until `awaiting_mapping`, then submits
   `POST /datasets/{id}/mapping` with the chosen `kato_column` and a
   `year_columns` map (each `{column, start_date}`, optionally `end_date` for
   a period). Validated against the detected columns; `status` → `extracting`.
4. A second worker job **unpivots** the attribute table into long-format
   `dataset_observations` — one row per `(kato_code, start_date)` with a numeric
   `value` (or `value_text` for non-numeric cells); `status` → `ready`. Read
   them via `GET /datasets/{id}/observations` (paginated, optional
   `?kato_code=`).

```sh
curl -X POST https://dssgis.dwh.kz/datasets/<id>/mapping -H 'Content-Type: application/json' -d '{
  "kato_column": "като",
  "year_columns": [
    {"column": "F_2023", "start_date": "2023-01-01"},
    {"column": "D_2025", "start_date": "2025-01-01"}
  ]
}'
```

### Dataset periods

A dataset carries an optional period — `start_date` and `end_date`
(`YYYY-MM-DD`, both nullable, `end_date` inclusive and never before
`start_date`) — plus an optional `dg_id` linking it to a *dataset group*: the
"одноимённые" datasets that describe the same thing for different periods. All
three are accepted on `POST /datasets` (form fields), `PUT /datasets/{id}` and
`PATCH /datasets/{id}` (an explicit `null`, or omission on PUT, clears them).

`GET /datasets` filters on them with `?dg_id=`, and with the inclusive bounds
`?start_date_from=`, `?start_date_to=`, `?end_date_from=` and `?end_date_to=`
(each `YYYY-MM-DD`).

### Dataset groups

A **dataset group** (`/dataset-groups`) collects the datasets that describe the
same indicator for different periods — the "одноимённые" datasets. A group has a
slug `code` (unique), a `name` and a `description`, and datasets join it through
their `dg_id`.

**Freshness.** `GET /dataset-groups/{id}/latest` returns the group's freshest
dataset: the greatest `end_date` wins, datasets without an `end_date` come last,
and ties are broken by the newest `created_at`. Narrow the candidates with
`?status=` and `?file_type=` (for example `?status=ready&file_type=raster` to
get the latest published raster). A group with no matching dataset yields 404.

**Shared SLD.** A group can carry one SLD (Styled Layer Descriptor) style file
(`POST`/`GET /dataset-groups/{id}/sld`, multipart `file`, `.sld` or `.xml`),
which acts as the fallback style for every dataset in the group:
`GET /datasets/{id}/sld` returns the dataset's own SLD when it has one, and
otherwise falls back to its group's — 404 only when neither exists. Deleting a
group also removes its stored SLD object.

## Landscape Program (ЛП)

The **Landscape Program** is a DAG of per-pixel raster computations that turns
ecological value (Z), vulnerability (U) and sector suitability (P) class rasters
into an ecological constraint (E), per-sector conflicts (C) and three functional
zoning scenarios (V1–V3). Every input and output of every node is a *dataset
group*; the leaf inputs are the **slots** users upload to, and every result is an
ordinary `raster` dataset (COG + SLD) served by the existing dataset endpoints.

The graph lives in Go (`internal/landscape/graph.go`); the groups and the
`landscape` category are seeded by migration `00011_landscape.sql`. A node code
`XY` reads as `X` = stage, `Y` = function within the stage: a stage starts only
after every node of all lower stages finished, and nodes of one stage run
concurrently.

| Node | Name | Inputs (group codes) | Outputs (group codes) |
|---|---|---|---|
| 00 | z_integrated | `lp-z-biodiversity`, `lp-z-lcr`, `lp-z-soil-protection`, `lp-z-wr` | `lp-z-integrated` |
| 01 | u_integrated | `lp-u-surf-water`, `lp-u-soil`, `lp-u-crop`, `lp-u-veg` | `lp-u-integrated` |
| 02 | z_max | Z1..Z4 as above | `lp-z-max` |
| 03 | u_max | U as above | `lp-u-max` |
| 10 | e_ecological_constraint | `lp-z-integrated`, `lp-u-integrated` | `lp-e` |
| 20 | c_conflicts | `lp-p-irrigation`, `lp-p-snow`, `lp-p-crop`, `lp-p-pasture`, `lp-p-tourism`, `lp-e` | `lp-c-irrigation`, `lp-c-snow`, `lp-c-crop`, `lp-c-pasture`, `lp-c-tourism` |
| 21 | p_max | five P groups | `lp-p-max` |
| 30 | func_zoning | `lp-e`, five C groups | `lp-v1-conservative`, `lp-v2-balanced`, `lp-v3-development` |
| 31 | c_max | five C groups | `lp-c-max` |

Sector order everywhere: 1 irrigation, 2 snow, 3 crop, 4 pasture, 5 tourism.

### Pixel rules

A pixel value is **valid** iff `1 <= v <= 5`; anything else (nodata included) is
invalid. Output rasters are `int16` with nodata `-9999`. `classify(x)` maps a
float32 score to a class: `<= 1.8 → 1`, `<= 2.6 → 2`, `<= 3.4 → 3`, `<= 4.2 → 4`,
`<= 5.0 → 5` (and requires `x >= 1.0`).

| Node | Valid domain (pixel) | Value |
|---|---|---|
| z_integrated | all four Z valid | `classify(0.25*Z1 + 0.25*Z2 + 0.25*Z3 + 0.25*Z4)` |
| u_integrated | water ∧ soil ∧ (crop ∨ veg) | `classify(num/den)` with crop/veg contributing `0.25` each only when valid |
| z_max / u_max | as above | `max` over the valid inputs |
| e_ecological_constraint | Z ∧ U valid | `max(Z, U)` |
| c_conflicts | `P_j` ∧ E valid | `min(P_j, E)` per sector |
| p_max / c_max | at least one input valid | `max` over the valid ones |
| func_zoning | E valid ∧ at least one `C_k` valid | winning sector + 1, else 1 |

Functional zoning takes `S_k = C_k`. A sector is a candidate when `S_k` and `E`
are valid and the scenario rule holds — V1: `E <= 2 && S >= v1_min_s_e12` or
`E == 3 && S >= v1_min_s_e3`; V2: `E <= 2`, or `E == 3 && S >= v2_min_s_e3`, or
`E == 4 && S >= v2_min_s_e4`; V3: `E <= 4`. With no candidate the pixel is class
1 (regulated); otherwise the candidate with the greatest `S` wins, ties broken by
`tie_priority`, and the class is `sector + 1` (2 irrigation, 3 snow, 4 crop,
5 pasture, 6 tourism).

Inputs are **not** resampled or reprojected: a node whose inputs do not share one
grid (size, geotransform, CRS) fails with a clear message, and so does the run.

### Uploading inputs, running, reading results

```sh
# 1. fill a slot (repeat per input group; see GET /landscape/slots for the codes)
curl -X POST https://dssgis.dwh.kz/landscape/slots/lp-z-biodiversity \
  -F file=@z_biodiversity.tif -F name="Z Biodiversity 2025" \
  -F start_date=2025-01-01 -F end_date=2025-12-31

# 2. queue a run (202); the worker executes the graph asynchronously
curl -X POST https://dssgis.dwh.kz/landscape/runs \
  -H 'Content-Type: application/json' -d '{"from_stage": 0}'

# 3. follow it, then read the layers
curl https://dssgis.dwh.kz/landscape/runs/<run-id>
curl https://dssgis.dwh.kz/landscape/results
```

`GET /landscape/slots` lists every leaf slot with the dataset currently filling
it (any status), so the UI can show what is still missing. The upload is the
regular raster path: `.tif`/`.tiff` with a TIFF magic-byte check, stored in the
slot's group under the `landscape` category, COG conversion enqueued.

**Input resolution.** For each input group a node takes, in order: the run's
`overrides[group]`, else whatever this run already produced for that group, else
the freshest `ready` raster of the group (greatest `end_date`, NULLs last, newest
`created_at` first). A missing input fails the node and the run.

**Incremental recompute.** Each node is fingerprinted over its code, its resolved
inputs and the params that actually affect it. Without `from_stage`, a node whose
fingerprint matches an earlier `succeeded` node — with all of its output datasets
still present and `ready` — is marked `skipped` and its outputs reused. With
`from_stage: N`, every node of stage `N` and above is recomputed regardless;
`from_stage: 0` recomputes everything. `overrides` pins a specific dataset
(existing, `raster`, `ready`) for a group instead of resolving the freshest one.
Run params: `window_size` (256..8192, default 2048), `tie_priority` (a
permutation of 1..5, default `[1,2,3,4,5]`), `v1_min_s_e12`/`v1_min_s_e3`
(defaults 3/4) and `v2_min_s_e3`/`v2_min_s_e4` (defaults 3/5).

**Results.** Every output is a normal dataset in its group, named
`<Output name> <run date>`, `automated = true`, with
`meta = {lp_run_id, lp_node, inputs}` and the min/max period of its inputs.
`GET /landscape/results` returns one entry per output group with its freshest
`ready` dataset, a `cog_url` (`/datasets/{id}.cog`) and, when the group carries
one, an `sld_url` (`/dataset-groups/{id}/sld`). A node failure fails its stage;
later stages are not started and the run ends `failed` with the message. Jobs are
nacked without requeue, so there is no automatic retry — start a new run.

### Memory and parallelism

A run has two nested levels of parallelism: nodes of a stage run concurrently,
and each node processes its windows with its own worker pool. Every worker holds
one window buffer per input **and** per output, so the worker's peak resident set
is roughly

```
nodes × workers-per-node × window_size² × (inputs + outputs) × 8 bytes
```

(the widest node, `20 c_conflicts`, has 6 inputs and 5 outputs). Two environment
variables bound that product, independently of how many cores the host has:

| Variable | Default | Meaning |
| --- | --- | --- |
| `LP_MAX_WORKERS` | `8` | Total window-worker budget shared by all nodes of a run. |
| `LP_MAX_NODES` | `2` | How many graph nodes of one stage run at the same time. |

Each node gets `max(1, LP_MAX_WORKERS / LP_MAX_NODES)` workers, further capped by
`GOMAXPROCS` and by the number of windows, so the total stays at
`LP_MAX_WORKERS`. At the defaults and `window_size: 2048` that is about
8 × 2048² × 11 × 8 B ≈ 2.9 GiB worst case — lower `LP_MAX_WORKERS` (or
`window_size`) on a small host, raise it on a large one.

### Build requirements

The raster math uses [godal](https://github.com/airbusgeo/godal), a **cgo**
binding to GDAL, so the binary must be built with `CGO_ENABLED=1` against GDAL
headers and libraries:

```sh
brew install gdal          # macOS; Linux: libgdal-dev / gdal-dev
go build ./...             # cgo is on by default for a native build
```

`build/package/Dockerfile` already handles the image: the builder stage adds
`gdal-dev gcc g++ musl-dev pkgconfig` and builds with `CGO_ENABLED=1`, and the
runtime stage installs `gdal` (the shared library godal needs) plus `gdal-tools`
(`gdal_translate`/`gdalinfo` for the COG conversion).

## Getting started

```sh
cp configs/.env.example .env
docker compose -f deployments/docker-compose.yml up -d postgres minio rabbitmq

go run ./cmd/gis migrate up        # apply migrations
go run ./cmd/gis serve             # HTTP server on :8080
go run ./cmd/gis worker --publish-example   # consume (and seed one message)
```

Health: `GET /healthz` (liveness), `GET /readyz` (DB + S3 + RabbitMQ).

### HTTP API

The API is described by an **OpenAPI 3.1.1** spec at
[`api/openapi.yaml`](api/openapi.yaml), embedded into the binary. While the
server runs it is served at `/openapi.yaml`, with an interactive **Redoc** UI at
`/docs`.

| Method | Path                       | Description                          |
|--------|----------------------------|--------------------------------------|
| GET    | `/categories`              | list (optional `?parent_id=`)        |
| POST   | `/categories`              | create (`name`, `description`, `parent_id?`) |
| GET    | `/categories/{id}`         | get                                  |
| PUT    | `/categories/{id}`         | update                               |
| DELETE | `/categories/{id}`         | delete                               |
| GET    | `/datasets`                | paginated list of summaries (`?page=`, `?page_size=`, `?category_id=`, `?dg_id=`, `?start_date_from=`, `?start_date_to=`, `?end_date_from=`, `?end_date_to=`) |
| POST   | `/datasets`                | upload (multipart: `file`, `file_type`, `category_id`, `code`, `name`, `description?`, `unit?`, `meta?` (JSON), `automated?` (bool), `dg_id?`, `start_date?`, `end_date?`) |
| GET    | `/datasets/{id}`           | full dataset (geometry as GeoJSON, `bbox` for rasters) |
| GET    | `/datasets/{id}.geojson`   | GeoJSON `FeatureCollection`; plain `vector` returns its geometry as a single feature with the extracted attribute table as top-level properties; `vector_with_kato` always ignores its own geometry and joins the `districts` table by KATO, mapping observations onto each polygon |
| GET    | `/datasets/{id}/status`    | processing status; long-polls with `?current=<status>` (holds up to `?wait=` secs, default 25, max 60) |
| GET    | `/datasets/{id}/download`  | download the stored file             |
| POST   | `/datasets/{id}/mapping`   | set KATO column + year→period map (vector_with_kato) |
| GET    | `/datasets/{id}/observations` | paginated unpivoted values (`?kato_code=`, `?page=`, `?page_size=`) |
| POST   | `/datasets/{id}/sld`       | upload/replace the dataset's SLD (multipart `file`, `.sld`/`.xml`) |
| GET    | `/datasets/{id}/sld`       | download the dataset's SLD, falling back to its group's |
| DELETE | `/datasets/{id}`           | delete (row + object)                |
| GET    | `/dataset-groups`          | paginated list (`?code=`, `?page=`, `?page_size=`) |
| POST   | `/dataset-groups`          | create (`code`, `name`, `description?`) |
| GET    | `/dataset-groups/{id}`     | get                                  |
| PUT    | `/dataset-groups/{id}`     | update                               |
| DELETE | `/dataset-groups/{id}`     | delete (409 while datasets reference it) |
| GET    | `/dataset-groups/{id}/latest` | freshest dataset of the group (`?status=`, `?file_type=`) |
| POST   | `/dataset-groups/{id}/sld` | upload/replace the group's shared SLD |
| GET    | `/dataset-groups/{id}/sld` | download the group's shared SLD      |
| GET    | `/landscape/slots`         | ЛП input slots + the dataset filling each |
| POST   | `/landscape/slots/{code}`  | upload a GeoTIFF into a slot (multipart: `file`, `name?`, `start_date?`, `end_date?`) |
| GET    | `/landscape/runs`          | paginated runs, newest first (`?page=`, `?page_size=`) |
| POST   | `/landscape/runs`          | queue a run (`from_stage?`, `overrides?`, `params?`); 202 |
| GET    | `/landscape/runs/{id}`     | run with per-node status, inputs, outputs, timings |
| GET    | `/landscape/results`       | per output group: latest layer, `cog_url`, `sld_url` |

Example upload:

```sh
curl -X POST https://dssgis.dwh.kz/datasets \
  -F file=@sample.geojson -F file_type=vector -F category_id=<uuid> \
  -F code=POP_2026 -F name=Population -F description="Resident population" -F unit=people
```

## Migrations

Embedded via goose and run through the binary. The first migration enables the
PostGIS extension (the database runs the `postgis/postgis` image), so a PostGIS-
capable Postgres is required.

```sh
go run ./cmd/gis migrate up|down|status|reset
go run ./cmd/gis migrate fresh    # drop everything in the schema and re-run
```

> On Apple Silicon, `postgis/postgis` has no native arm64 build, so the compose
> file pins `platform: linux/amd64` (Docker Desktop emulates it). Remove that line
> on amd64 hosts.

## Development

Common tasks are wrapped in the `Makefile` (run `make help` for the full list):

```sh
make up            # start postgres, minio, rabbitmq
make migrate-fresh # drop the schema and re-apply migrations
make run           # run the HTTP server
make check         # go vet + go test
make lint          # golangci-lint (if installed)
```

CI (`.github/workflows/ci.yml`) runs build, vet, `go test -race`, and golangci-lint
on every push and pull request.

Repository tests in `internal/repository/postgres` run against a real database
when `TEST_DB_URL` points at one with migrations applied, and are skipped
otherwise. They catch Postgres-only failures (parameter type deduction, casts,
constraints) that in-memory stubs cannot:

```sh
docker run -d --name gis-test-pg -e POSTGRES_USER=gis -e POSTGRES_PASSWORD=gis -e POSTGRES_DB=gis -p 5434:5432 postgis/postgis:17-3.5
DB_URL='postgres://gis:gis@127.0.0.1:5434/gis?sslmode=disable' S3_ENDPOINT=x S3_ACCESS_KEY=x S3_SECRET_KEY=x RABBITMQ_URL=amqp://x go run ./cmd/gis migrate up
TEST_DB_URL='postgres://gis:gis@127.0.0.1:5434/gis?sslmode=disable' go test ./internal/repository/postgres/ -v
```

## Adding a feature

Each new domain is one vertical slice mirroring Category/Dataset:
`domain/` → `repository/postgres/` → `service/` → `transport/http/`
(+ `messaging/rabbitmq/` if it needs async processing), wired in `internal/app`.