gis/README.md

gis

A Go service scaffold following golang-standards/project-layout, with cleanly separated layers: HTTP transport → services → repositories, plus RabbitMQ messaging and embedded database migrations. Single binary, three subcommands.

Layout

cmd/gis/                 binary entrypoint
internal/
  cli/                   cobra commands: serve, worker, migrate
  config/                env-based configuration
  app/                   composition root (wires all dependencies)
  domain/                entities, enums, sentinel errors
  repository/postgres/   pgx-backed repositories
  service/               business logic
  transport/http/        chi router, middleware, handlers
  storage/s3/            MinIO/S3 object storage
  messaging/rabbitmq/    connection, publisher, consumer
  landscape/             Landscape Program graph, kernels, window executor
  raster/                GDAL raster I/O (COG conversion + godal engine)
  platform/logger/       slog setup
pkg/httputil/            generic JSON/validation HTTP helpers
migrations/              embedded goose SQL migrations
configs/                 .env.example
deployments/             docker-compose (postgres, minio, rabbitmq)
build/package/           Dockerfile
api/openapi.yaml         OpenAPI 3.1.1 spec (embedded + served at /openapi.yaml)

Domain

• Category — hierarchical (self-referencing parent_id). Full CRUD; cycle-safe on update.
• DatasetGroup — a named group of "одноимённые" datasets (see Dataset groups below). Full CRUD; deleting a group still referenced by datasets is rejected with 409.
• Dataset — a geo file uploaded to S3/MinIO (file_type: vector_with_kato | vector | raster), belonging to one Category. Carries code/name/description/ unit metadata, a user-defined meta (JSONB) blob, an automated flag, a status lifecycle field (defaults to pending), properties (JSONB, populated from the file's attribute table), and a PostGIS geometry footprint stored in EPSG:4326 (returned as GeoJSON, with a STAC-style bbox array for rasters). Upload / list / get / download / delete (delete also removes the stored object). Uploads are validated three ways before being stored: the file_type enum, the file extension (must be allowed for the type), and a content magic-byte check (TIFF for .tif, ZIP for .zip, SQLite for .gpkg, JSON for .geojson) so mislabeled files are rejected with 422 up front.

Every uploaded file is then processed asynchronously by the worker, dispatched by file_type:

• vector — the attribute table is parsed and stored (as a JSON array of row objects) in properties (status processing → ready).
• raster — converted to a Cloud-Optimized GeoTIFF via gdal_translate -of COG (processing → ready); the COG is stored under cog_storage_key (the original is kept) and the footprint geometry + bbox are read from the raster extent. Requires GDAL in the worker image (gdal-tools).
• vector_with_kato — the column-selection flow below (parsing → awaiting_mapping → extracting → ready).
• events + the example RabbitMQ consumer/publisher are a generic messaging scaffold kept alongside the real async flows.

vector_with_kato two-phase flow

Uploading a vector_with_kato file (zipped shapefile, GeoJSON, or GeoPackage) triggers asynchronous parsing of its attribute table, after which the user maps the KATO column and the year columns:

1. POST /datasets with file_type=vector_with_kato → dataset created with status=parsing; a dataset.parse job is published to RabbitMQ.
2. The worker consumes the job, parses the file's columns (with sample values; CP1251/Cyrillic aware for shapefiles) and stores them in attribute_columns; status → awaiting_mapping (or failed with parse_error).
3. The client polls GET /datasets/{id} until awaiting_mapping, then submits POST /datasets/{id}/mapping with the chosen kato_column and a year_columns map (each {column, start_date}, optionally end_date for a period). Validated against the detected columns; status → extracting.
4. A second worker job unpivots the attribute table into long-format dataset_observations — one row per (kato_code, start_date) with a numeric value (or value_text for non-numeric cells); status → ready. Read them via GET /datasets/{id}/observations (paginated, optional ?kato_code=).

curl -X POST https://dssgis.dwh.kz/datasets/<id>/mapping -H 'Content-Type: application/json' -d '{
  "kato_column": "като",
  "year_columns": [
    {"column": "F_2023", "start_date": "2023-01-01"},
    {"column": "D_2025", "start_date": "2025-01-01"}
  ]
}'

Dataset periods

A dataset carries an optional period — start_date and end_date (YYYY-MM-DD, both nullable, end_date inclusive and never before start_date) — plus an optional dg_id linking it to a dataset group: the "одноимённые" datasets that describe the same thing for different periods. All three are accepted on POST /datasets (form fields), PUT /datasets/{id} and PATCH /datasets/{id} (an explicit null, or omission on PUT, clears them).

GET /datasets filters on them with ?dg_id=, and with the inclusive bounds ?start_date_from=, ?start_date_to=, ?end_date_from= and ?end_date_to= (each YYYY-MM-DD).

Dataset groups

A dataset group (/dataset-groups) collects the datasets that describe the same indicator for different periods — the "одноимённые" datasets. A group has a slug code (unique), a name and a description, and datasets join it through their dg_id.

Freshness. GET /dataset-groups/{id}/latest returns the group's freshest dataset: the greatest end_date wins, datasets without an end_date come last, and ties are broken by the newest created_at. Narrow the candidates with ?status= and ?file_type= (for example ?status=ready&file_type=raster to get the latest published raster). A group with no matching dataset yields 404.

Shared SLD. A group can carry one SLD (Styled Layer Descriptor) style file (POST/GET /dataset-groups/{id}/sld, multipart file, .sld or .xml), which acts as the fallback style for every dataset in the group: GET /datasets/{id}/sld returns the dataset's own SLD when it has one, and otherwise falls back to its group's — 404 only when neither exists. Deleting a group also removes its stored SLD object.

Landscape Program (ЛП)

The Landscape Program is a DAG of per-pixel raster computations that turns ecological value (Z), vulnerability (U) and sector suitability (P) class rasters into an ecological constraint (E), per-sector conflicts (C) and three functional zoning scenarios (V1–V3). Every input and output of every node is a dataset group; the leaf inputs are the slots users upload to, and every result is an ordinary raster dataset (COG + SLD) served by the existing dataset endpoints.

The graph lives in Go (internal/landscape/graph.go); the groups and the landscape category are seeded by migration 00011_landscape.sql. A node code XY reads as X = stage, Y = function within the stage: a stage starts only after every node of all lower stages finished, and nodes of one stage run concurrently.

Node	Name	Inputs (group codes)	Outputs (group codes)
00	z_integrated	lp-z-biodiversity, lp-z-lcr, lp-z-soil-protection, lp-z-wr	lp-z-integrated
01	u_integrated	lp-u-surf-water, lp-u-soil, lp-u-crop, lp-u-veg	lp-u-integrated
02	z_max	Z1..Z4 as above	lp-z-max
03	u_max	U as above	lp-u-max
10	e_ecological_constraint	lp-z-integrated, lp-u-integrated	lp-e
20	c_conflicts	lp-p-irrigation, lp-p-snow, lp-p-crop, lp-p-pasture, lp-p-tourism, lp-e	lp-c-irrigation, lp-c-snow, lp-c-crop, lp-c-pasture, lp-c-tourism
21	p_max	five P groups	lp-p-max
30	func_zoning	lp-e, five P groups (original suitability)	lp-v1-conservative, lp-v2-balanced, lp-v3-development
31	c_max	five C groups	lp-c-max

Sector order everywhere: 1 irrigation, 2 snow, 3 crop, 4 pasture, 5 tourism.

Pixel rules

A pixel value is valid iff 1 <= v <= 5; anything else (nodata included) is invalid. Output rasters are int16 with nodata -9999. classify(x) maps a float32 score to a class: <= 1.8 → 1, <= 2.6 → 2, <= 3.4 → 3, <= 4.2 → 4, <= 5.0 → 5 (and requires x >= 1.0).

Node	Valid domain (pixel)	Value
z_integrated	all four Z valid	classify(0.25*Z1 + 0.25*Z2 + 0.25*Z3 + 0.25*Z4)
u_integrated	water ∧ soil ∧ (crop ∨ veg)	classify(num/den) with crop/veg contributing 0.25 each only when valid
z_max / u_max	as above	max over the valid inputs
e_ecological_constraint	Z ∧ U valid	max(Z, U)
c_conflicts	P_j ∧ E valid	min(P_j, E) per sector
p_max / c_max	at least one input valid	max over the valid ones
func_zoning	E valid ∧ at least one P_k valid	winning sector + 1, else 1

Functional zoning takes S_k = P_k, the original suitability rasters (the conflict layers C_k are diagnostics and are not zoning inputs). A sector is a candidate when S_k and E are valid and the scenario rule holds — V1: E <= 2 && S >= v1_min_s_e12 or E == 3 && S >= v1_min_s_e3; V2: E <= 2, or E == 3 && S >= v2_min_s_e3, or E == 4 && S >= v2_min_s_e4; V3: E <= 4. With no candidate the pixel is class 1 (regulated); otherwise the candidate with the greatest S wins, ties broken by tie_priority, and the class is sector + 1 (2 irrigation, 3 snow, 4 crop, 5 pasture, 6 tourism).

Inputs are not resampled or reprojected: a node whose inputs do not share one grid (size, geotransform, CRS) fails with a clear message, and so does the run.

Uploading inputs, running, reading results

# 1. fill a slot (repeat per input group; see GET /landscape/slots for the codes)
curl -X POST https://dssgis.dwh.kz/landscape/slots/lp-z-biodiversity \
  -F file=@z_biodiversity.tif -F name="Z Biodiversity 2025" \
  -F start_date=2025-01-01 -F end_date=2025-12-31

# 2. queue a run (202); the worker executes the graph asynchronously
curl -X POST https://dssgis.dwh.kz/landscape/runs \
  -H 'Content-Type: application/json' -d '{"from_stage": 0}'

# 3. follow it, then read the layers
curl https://dssgis.dwh.kz/landscape/runs/<run-id>
curl https://dssgis.dwh.kz/landscape/results

GET /landscape/slots lists every leaf slot with the dataset currently filling it (any status), so the UI can show what is still missing. The upload is the regular raster path: .tif/.tiff with a TIFF magic-byte check, stored in the slot's group under the landscape category, COG conversion enqueued.

Input resolution. For each input group a node takes, in order: the run's overrides[group], else whatever this run already produced for that group, else the freshest ready raster of the group (greatest end_date, NULLs last, newest created_at first). A missing input fails the node and the run.

Incremental recompute. Each node is fingerprinted over its code, its resolved inputs and the params that actually affect it. Without from_stage, a node whose fingerprint matches an earlier succeeded node — with all of its output datasets still present and ready — is marked skipped and its outputs reused. With from_stage: N, every node of stage N and above is recomputed regardless; from_stage: 0 recomputes everything. overrides pins a specific dataset (existing, raster, ready) for a group instead of resolving the freshest one. Run params: window_size (256..8192, default 2048), tie_priority (a permutation of 1..5, default [1,2,3,4,5]), v1_min_s_e12/v1_min_s_e3 (defaults 3/4) and v2_min_s_e3/v2_min_s_e4 (defaults 3/5).

Results. Every output is a normal dataset in its group, named <Output name> <run date>, automated = true, with meta = {lp_run_id, lp_node, inputs} and the min/max period of its inputs. GET /landscape/results returns one entry per output group with its freshest ready dataset, a cog_url (/datasets/{id}.cog) and, when the group carries one, an sld_url (/dataset-groups/{id}/sld). A node failure fails its stage; later stages are not started and the run ends failed with the message. Jobs are nacked without requeue, so there is no automatic retry — start a new run.

Memory and parallelism

A run has two nested levels of parallelism: nodes of a stage run concurrently, and each node processes its windows with its own worker pool. Every worker holds one window buffer per input and per output, so the worker's peak resident set is roughly

nodes × workers-per-node × window_size² × (inputs + outputs) × 8 bytes

(the widest node, 20 c_conflicts, has 6 inputs and 5 outputs). Two environment variables bound that product, independently of how many cores the host has:

Variable	Default	Meaning
LP_MAX_WORKERS	8	Total window-worker budget shared by all nodes of a run.
LP_MAX_NODES	2	How many graph nodes of one stage run at the same time.

Each node gets max(1, LP_MAX_WORKERS / LP_MAX_NODES) workers, further capped by GOMAXPROCS and by the number of windows, so the total stays at LP_MAX_WORKERS. At the defaults and window_size: 2048 that is about 8 × 2048² × 11 × 8 B ≈ 2.9 GiB worst case — lower LP_MAX_WORKERS (or window_size) on a small host, raise it on a large one.

Build requirements

The raster math uses godal, a cgo binding to GDAL, so the binary must be built with CGO_ENABLED=1 against GDAL headers and libraries:

brew install gdal          # macOS; Linux: libgdal-dev / gdal-dev
go build ./...             # cgo is on by default for a native build

build/package/Dockerfile already handles the image: the builder stage adds gdal-dev gcc g++ musl-dev pkgconfig and builds with CGO_ENABLED=1, and the runtime stage installs gdal (the shared library godal needs) plus gdal-tools (gdal_translate/gdalinfo for the COG conversion).

Getting started

cp configs/.env.example .env
docker compose -f deployments/docker-compose.yml up -d postgres minio rabbitmq

go run ./cmd/gis migrate up        # apply migrations
go run ./cmd/gis serve             # HTTP server on :8080
go run ./cmd/gis worker --publish-example   # consume (and seed one message)

Health: GET /healthz (liveness), GET /readyz (DB + S3 + RabbitMQ).

HTTP API

The API is described by an OpenAPI 3.1.1 spec at api/openapi.yaml, embedded into the binary. While the server runs it is served at /openapi.yaml, with an interactive Redoc UI at /docs.

Method	Path	Description
GET	/categories	list (optional ?parent_id=)
POST	/categories	create (name, description, parent_id?)
GET	/categories/{id}	get
PUT	/categories/{id}	update
DELETE	/categories/{id}	delete
GET	/datasets	paginated list of summaries (?page=, ?page_size=, ?category_id=, ?dg_id=, ?start_date_from=, ?start_date_to=, ?end_date_from=, ?end_date_to=)
POST	/datasets	upload (multipart: file, file_type, category_id, code, name, description?, unit?, meta? (JSON), automated? (bool), dg_id?, start_date?, end_date?)
GET	/datasets/{id}	full dataset (geometry as GeoJSON, bbox for rasters)
GET	/datasets/{id}.geojson	GeoJSON FeatureCollection; plain vector returns its geometry as a single feature with the extracted attribute table as top-level properties; vector_with_kato always ignores its own geometry and joins the districts table by KATO, mapping observations onto each polygon
GET	/datasets/{id}/status	processing status; long-polls with ?current=<status> (holds up to ?wait= secs, default 25, max 60)
GET	/datasets/{id}/download	download the stored file
POST	/datasets/{id}/mapping	set KATO column + year→period map (vector_with_kato)
GET	/datasets/{id}/observations	paginated unpivoted values (?kato_code=, ?page=, ?page_size=)
POST	/datasets/{id}/sld	upload/replace the dataset's SLD (multipart file, .sld/.xml)
GET	/datasets/{id}/sld	download the dataset's SLD, falling back to its group's
DELETE	/datasets/{id}	delete (row + object)
GET	/dataset-groups	paginated list (?code=, ?page=, ?page_size=)
POST	/dataset-groups	create (code, name, description?)
GET	/dataset-groups/{id}	get
PUT	/dataset-groups/{id}	update
DELETE	/dataset-groups/{id}	delete (409 while datasets reference it)
GET	/dataset-groups/{id}/latest	freshest dataset of the group (?status=, ?file_type=)
POST	/dataset-groups/{id}/sld	upload/replace the group's shared SLD
GET	/dataset-groups/{id}/sld	download the group's shared SLD
GET	/landscape/slots	ЛП input slots + the dataset filling each
POST	/landscape/slots/{code}	upload a GeoTIFF into a slot (multipart: file, name?, start_date?, end_date?)
GET	/landscape/runs	paginated runs, newest first (?page=, ?page_size=)
POST	/landscape/runs	queue a run (from_stage?, overrides?, params?); 202
GET	/landscape/runs/{id}	run with per-node status, inputs, outputs, timings
GET	/landscape/results	per output group: latest layer, cog_url, sld_url

Example upload:

curl -X POST https://dssgis.dwh.kz/datasets \
  -F file=@sample.geojson -F file_type=vector -F category_id=<uuid> \
  -F code=POP_2026 -F name=Population -F description="Resident population" -F unit=people

Migrations

Embedded via goose and run through the binary. The first migration enables the PostGIS extension (the database runs the postgis/postgis image), so a PostGIS- capable Postgres is required.

go run ./cmd/gis migrate up|down|status|reset
go run ./cmd/gis migrate fresh    # drop everything in the schema and re-run

On Apple Silicon, postgis/postgis has no native arm64 build, so the compose file pins platform: linux/amd64 (Docker Desktop emulates it). Remove that line on amd64 hosts.

Development

Common tasks are wrapped in the Makefile (run make help for the full list):

make up            # start postgres, minio, rabbitmq
make migrate-fresh # drop the schema and re-apply migrations
make run           # run the HTTP server
make check         # go vet + go test
make lint          # golangci-lint (if installed)

CI (.github/workflows/ci.yml) runs build, vet, go test -race, and golangci-lint on every push and pull request.

Repository tests in internal/repository/postgres run against a real database when TEST_DB_URL points at one with migrations applied, and are skipped otherwise. They catch Postgres-only failures (parameter type deduction, casts, constraints) that in-memory stubs cannot:

docker run -d --name gis-test-pg -e POSTGRES_USER=gis -e POSTGRES_PASSWORD=gis -e POSTGRES_DB=gis -p 5434:5432 postgis/postgis:17-3.5
DB_URL='postgres://gis:gis@127.0.0.1:5434/gis?sslmode=disable' S3_ENDPOINT=x S3_ACCESS_KEY=x S3_SECRET_KEY=x RABBITMQ_URL=amqp://x go run ./cmd/gis migrate up
TEST_DB_URL='postgres://gis:gis@127.0.0.1:5434/gis?sslmode=disable' go test ./internal/repository/postgres/ -v

Adding a feature

Each new domain is one vertical slice mirroring Category/Dataset: domain/ → repository/postgres/ → service/ → transport/http/ (+ messaging/rabbitmq/ if it needs async processing), wired in internal/app.