gis

A Go service scaffold following golang-standards/project-layout, with cleanly separated layers: HTTP transport → services → repositories, plus RabbitMQ messaging and embedded database migrations. Single binary, three subcommands.

Layout

cmd/gis/                 binary entrypoint
internal/
  cli/                   cobra commands: serve, worker, migrate
  config/                env-based configuration
  app/                   composition root (wires all dependencies)
  domain/                entities, enums, sentinel errors
  repository/postgres/   pgx-backed repositories
  service/               business logic
  transport/http/        chi router, middleware, handlers
  storage/s3/            MinIO/S3 object storage
  messaging/rabbitmq/    connection, publisher, consumer
  platform/logger/       slog setup
pkg/httputil/            generic JSON/validation HTTP helpers
migrations/              embedded goose SQL migrations
configs/                 .env.example
deployments/             docker-compose (postgres, minio, rabbitmq)
build/package/           Dockerfile
api/openapi.yaml         OpenAPI 3.1.1 spec (embedded + served at /openapi.yaml)

Domain

• Category — hierarchical (self-referencing parent_id). Full CRUD; cycle-safe on update.
• DatasetGroup — a named group of "одноимённые" datasets (see Dataset groups below). Full CRUD; deleting a group still referenced by datasets is rejected with 409.
• Dataset — a geo file uploaded to S3/MinIO (file_type: vector_with_kato | vector | raster), belonging to one Category. Carries code/name/description/ unit metadata, a user-defined meta (JSONB) blob, an automated flag, a status lifecycle field (defaults to pending), properties (JSONB, populated from the file's attribute table), and a PostGIS geometry footprint stored in EPSG:4326 (returned as GeoJSON, with a STAC-style bbox array for rasters). Upload / list / get / download / delete (delete also removes the stored object). Uploads are validated three ways before being stored: the file_type enum, the file extension (must be allowed for the type), and a content magic-byte check (TIFF for .tif, ZIP for .zip, SQLite for .gpkg, JSON for .geojson) so mislabeled files are rejected with 422 up front.

Every uploaded file is then processed asynchronously by the worker, dispatched by file_type:

• vector — the attribute table is parsed and stored (as a JSON array of row objects) in properties (status processing → ready).
• raster — converted to a Cloud-Optimized GeoTIFF via gdal_translate -of COG (processing → ready); the COG is stored under cog_storage_key (the original is kept) and the footprint geometry + bbox are read from the raster extent. Requires GDAL in the worker image (gdal-tools).
• vector_with_kato — the column-selection flow below (parsing → awaiting_mapping → extracting → ready).
• events + the example RabbitMQ consumer/publisher are a generic messaging scaffold kept alongside the real async flows.

vector_with_kato two-phase flow

Uploading a vector_with_kato file (zipped shapefile, GeoJSON, or GeoPackage) triggers asynchronous parsing of its attribute table, after which the user maps the KATO column and the year columns:

1. POST /datasets with file_type=vector_with_kato → dataset created with status=parsing; a dataset.parse job is published to RabbitMQ.
2. The worker consumes the job, parses the file's columns (with sample values; CP1251/Cyrillic aware for shapefiles) and stores them in attribute_columns; status → awaiting_mapping (or failed with parse_error).
3. The client polls GET /datasets/{id} until awaiting_mapping, then submits POST /datasets/{id}/mapping with the chosen kato_column and a year_columns map (each {column, start_date}, optionally end_date for a period). Validated against the detected columns; status → extracting.
4. A second worker job unpivots the attribute table into long-format dataset_observations — one row per (kato_code, start_date) with a numeric value (or value_text for non-numeric cells); status → ready. Read them via GET /datasets/{id}/observations (paginated, optional ?kato_code=).

curl -X POST https://dssgis.dwh.kz/datasets/<id>/mapping -H 'Content-Type: application/json' -d '{
  "kato_column": "като",
  "year_columns": [
    {"column": "F_2023", "start_date": "2023-01-01"},
    {"column": "D_2025", "start_date": "2025-01-01"}
  ]
}'

Dataset periods

A dataset carries an optional period — start_date and end_date (YYYY-MM-DD, both nullable, end_date inclusive and never before start_date) — plus an optional dg_id linking it to a dataset group: the "одноимённые" datasets that describe the same thing for different periods. All three are accepted on POST /datasets (form fields), PUT /datasets/{id} and PATCH /datasets/{id} (an explicit null, or omission on PUT, clears them).

GET /datasets filters on them with ?dg_id=, and with the inclusive bounds ?start_date_from=, ?start_date_to=, ?end_date_from= and ?end_date_to= (each YYYY-MM-DD).

Dataset groups

A dataset group (/dataset-groups) collects the datasets that describe the same indicator for different periods — the "одноимённые" datasets. A group has a slug code (unique), a name and a description, and datasets join it through their dg_id.

Freshness. GET /dataset-groups/{id}/latest returns the group's freshest dataset: the greatest end_date wins, datasets without an end_date come last, and ties are broken by the newest created_at. Narrow the candidates with ?status= and ?file_type= (for example ?status=ready&file_type=raster to get the latest published raster). A group with no matching dataset yields 404.

Shared SLD. A group can carry one SLD (Styled Layer Descriptor) style file (POST/GET /dataset-groups/{id}/sld, multipart file, .sld or .xml), which acts as the fallback style for every dataset in the group: GET /datasets/{id}/sld returns the dataset's own SLD when it has one, and otherwise falls back to its group's — 404 only when neither exists. Deleting a group also removes its stored SLD object.

Getting started

cp configs/.env.example .env
docker compose -f deployments/docker-compose.yml up -d postgres minio rabbitmq

go run ./cmd/gis migrate up        # apply migrations
go run ./cmd/gis serve             # HTTP server on :8080
go run ./cmd/gis worker --publish-example   # consume (and seed one message)

Health: GET /healthz (liveness), GET /readyz (DB + S3 + RabbitMQ).

HTTP API

The API is described by an OpenAPI 3.1.1 spec at api/openapi.yaml, embedded into the binary. While the server runs it is served at /openapi.yaml, with an interactive Redoc UI at /docs.

Method	Path	Description
GET	/categories	list (optional ?parent_id=)
POST	/categories	create (name, description, parent_id?)
GET	/categories/{id}	get
PUT	/categories/{id}	update
DELETE	/categories/{id}	delete
GET	/datasets	paginated list of summaries (?page=, ?page_size=, ?category_id=, ?dg_id=, ?start_date_from=, ?start_date_to=, ?end_date_from=, ?end_date_to=)
POST	/datasets	upload (multipart: file, file_type, category_id, code, name, description?, unit?, meta? (JSON), automated? (bool), dg_id?, start_date?, end_date?)
GET	/datasets/{id}	full dataset (geometry as GeoJSON, bbox for rasters)
GET	/datasets/{id}.geojson	GeoJSON FeatureCollection; plain vector returns its geometry as a single feature with the extracted attribute table as top-level properties; vector_with_kato always ignores its own geometry and joins the districts table by KATO, mapping observations onto each polygon
GET	/datasets/{id}/status	processing status; long-polls with ?current=<status> (holds up to ?wait= secs, default 25, max 60)
GET	/datasets/{id}/download	download the stored file
POST	/datasets/{id}/mapping	set KATO column + year→period map (vector_with_kato)
GET	/datasets/{id}/observations	paginated unpivoted values (?kato_code=, ?page=, ?page_size=)
POST	/datasets/{id}/sld	upload/replace the dataset's SLD (multipart file, .sld/.xml)
GET	/datasets/{id}/sld	download the dataset's SLD, falling back to its group's
DELETE	/datasets/{id}	delete (row + object)
GET	/dataset-groups	paginated list (?code=, ?page=, ?page_size=)
POST	/dataset-groups	create (code, name, description?)
GET	/dataset-groups/{id}	get
PUT	/dataset-groups/{id}	update
DELETE	/dataset-groups/{id}	delete (409 while datasets reference it)
GET	/dataset-groups/{id}/latest	freshest dataset of the group (?status=, ?file_type=)
POST	/dataset-groups/{id}/sld	upload/replace the group's shared SLD
GET	/dataset-groups/{id}/sld	download the group's shared SLD

Example upload:

curl -X POST https://dssgis.dwh.kz/datasets \
  -F file=@sample.geojson -F file_type=vector -F category_id=<uuid> \
  -F code=POP_2026 -F name=Population -F description="Resident population" -F unit=people

Migrations

Embedded via goose and run through the binary. The first migration enables the PostGIS extension (the database runs the postgis/postgis image), so a PostGIS- capable Postgres is required.

go run ./cmd/gis migrate up|down|status|reset
go run ./cmd/gis migrate fresh    # drop everything in the schema and re-run

On Apple Silicon, postgis/postgis has no native arm64 build, so the compose file pins platform: linux/amd64 (Docker Desktop emulates it). Remove that line on amd64 hosts.

Development

Common tasks are wrapped in the Makefile (run make help for the full list):

make up            # start postgres, minio, rabbitmq
make migrate-fresh # drop the schema and re-apply migrations
make run           # run the HTTP server
make check         # go vet + go test
make lint          # golangci-lint (if installed)

CI (.github/workflows/ci.yml) runs build, vet, go test -race, and golangci-lint on every push and pull request.

Adding a feature

Each new domain is one vertical slice mirroring Category/Dataset: domain/ → repository/postgres/ → service/ → transport/http/ (+ messaging/rabbitmq/ if it needs async processing), wired in internal/app.