# Landscape Program + Dataset Groups Implementation Plan > **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. **Goal:** Add dataset groups + dataset periods (with the `date`→`start_date` rename), and the Landscape Program raster pipeline (godal-backed, incremental, results stored as raster datasets). **Architecture:** Part 1 extends the existing dataset stack (domain → postgres repo → service → chi handler) with a `dataset_groups` entity and period columns. Part 2 adds a pure-Go compute engine (`internal/landscape`: graph, per-window kernels, `Compute`) behind a swappable `raster.IO` interface (godal implementation), and a `LandscapeService` that plans runs (fingerprint-based skip/reuse), resolves inputs from dataset groups, downloads inputs, computes, and publishes outputs as raster datasets that the existing COG conversion picks up. A RabbitMQ consumer executes runs in the worker. **Tech Stack:** Go 1.26 (toolchain go1.26.2 pinned), chi, pgx, goose, MinIO, RabbitMQ, GDAL via `github.com/airbusgeo/godal` v0.0.18 (cgo) + existing GDAL CLI tools. **Spec:** `docs/superpowers/specs/2026-09-05-landscape-program-design.md` ## Global Constraints - Build with `CGO_ENABLED=1`; local GDAL is Homebrew 3.13.3 (`pkg-config --modversion gdal`). The local Go SDK is incomplete; `go.mod` pins `toolchain go1.26.2`, so plain `go test ./...` works. - Follow existing layering: `internal/domain` (no internal deps) → `internal/repository/postgres` → `internal/service` (interfaces declared in service) → `internal/transport/http`. Wiring only in `internal/app/app.go`. - Errors: repositories return `domain.ErrNotFound` / `domain.ErrConflict` / `domain.ErrValidation` (wrapped with `%w`); handlers use `respondDomainError`. - Dates in JSON are strings `YYYY-MM-DD` (`time.Parse("2006-01-02", …)`). - Output rasters: `int16`, nodata `-9999`, `TILED=YES`, `BLOCKXSIZE=512`, `BLOCKYSIZE=512`, `COMPRESS=DEFLATE`, `PREDICTOR=2`, `BIGTIFF=IF_SAFER`. - Pixel validity: a value is valid iff `1 <= v <= 5`. Anything else (including nodata) is invalid. Never resample; grids must match exactly or the node fails. - Group codes (slots and outputs) are fixed strings listed in Task 4; they are the single source of truth for both the graph and the seed migration. - Tests: table-driven `testing` package, stubs like `internal/service/dataset_test.go`. Run only your own packages while other agents work in parallel (`go test ./internal/landscape/...`), not `go test ./...`, unless you are the integration task. - **Parallel execution rule:** several agents edit the same working tree on branch `feature/landscape-program`. Only touch the files listed in your task. Never run `git add -A`/`git add .`; add explicit paths. If `git commit` fails with `index.lock`, wait 2 s and retry. ## Execution waves | Wave | Tasks (parallel within a wave) | Depends on | |---|---|---| | 1 | Task 1 → Task 2 (same lane, sequential); Task 3; Task 4; Task 5 | — | | 2 | Task 6; Task 7 | 3, 4 (Task 6); 2, 4, 5 (Task 7) | | 3 | Task 8 | 6, 7 | --- ## File structure **Part 1** - `migrations/00010_dataset_groups_and_dates.sql` — new table, dataset columns, observation rename, year_columns key rename. - `internal/domain/dataset.go` — modify: rename fields, add group/date fields, `DatasetFields`, filter fields. - `internal/domain/dataset_group.go` — create: `DatasetGroup`. - `internal/repository/postgres/dataset.go` — modify: columns, filter clause, `UpdateFields(DatasetFields)`, `LatestInGroup`, observation columns. - `internal/repository/postgres/dataset_group.go` — create: group repository. - `internal/service/dataset.go` — modify: rename, upload/update inputs, `CreateRasterFromFile`, SLD fallback. - `internal/service/dataset_group.go` — create: group service. - `internal/service/dataset_test.go` — modify: stubs + tests. `internal/service/dataset_group_test.go` — create. - `internal/transport/http/dataset_handler.go` — modify. `internal/transport/http/dataset_group_handler.go` — create. `internal/transport/http/router.go`, `internal/app/app.go` — modify (wiring). - `api/openapi.yaml`, `README.md` — modify. **Part 2** - `internal/raster/io.go` (interfaces), `internal/raster/godal.go` (+ `godal_test.go`), `build/package/Dockerfile`, `go.mod`. - `internal/landscape/graph.go`, `params.go`, `kernels.go` (+ tests), `compute.go` (+ `compute_test.go` with a fake IO). - `internal/domain/landscape.go`, `migrations/00011_landscape.sql`, `internal/repository/postgres/landscape.go`. - `internal/service/landscape.go` (+ test), `internal/messaging/rabbitmq/landscape_consumer.go`. - `internal/transport/http/landscape_handler.go`, `router.go`, `app.go`, `api/openapi.yaml`, `README.md`. --- ### Task 1: Dataset periods — rename `date`→`start_date`, add dataset `start_date`/`end_date`, list filters, upload/PUT/PATCH fields **Files:** - Create: `migrations/00010_dataset_groups_and_dates.sql` (the *whole* migration incl. the `dataset_groups` table so Task 2 does not need another migration) - Modify: `internal/domain/dataset.go`, `internal/repository/postgres/dataset.go`, `internal/service/dataset.go`, `internal/service/dataset_test.go`, `internal/transport/http/dataset_handler.go`, `api/openapi.yaml`, `README.md` **Interfaces:** - Produces: `domain.Observation{StartDate string; EndDate *string}`, `domain.YearColumn{Column, StartDate string; EndDate *string}`, `domain.Dataset/DatasetSummary{DatasetGroupID *uuid.UUID; StartDate, EndDate *string}`, `domain.DatasetFilter{…, DatasetGroupID *uuid.UUID; StartDateFrom, StartDateTo, EndDateFrom, EndDateTo *string}`, `domain.DatasetFields`, repo `UpdateFields(ctx, id uuid.UUID, f domain.DatasetFields) (domain.Dataset, error)`, service `UploadInput{…; DatasetGroupID *uuid.UUID; StartDate, EndDate *string}`, `DatasetUpdateInput`/`DatasetPatchInput` with the same three fields (+ `DatasetGroupIDSet`, `StartDateSet`, `EndDateSet` on patch). - Task 2 relies on `dg_id` column + `DatasetGroupID` field existing after this task, and on `domain.ValidateDatePeriod`. - [ ] **Step 1: Write the migration** ```sql -- migrations/00010_dataset_groups_and_dates.sql -- +goose Up -- Dataset groups: "одноимённые" datasets versioned by period. A group may carry a -- shared SLD used as a fallback for its datasets. CREATE TABLE dataset_groups ( id UUID PRIMARY KEY DEFAULT gen_random_uuid(), code VARCHAR(255) NOT NULL UNIQUE, name VARCHAR(255) NOT NULL, description TEXT NOT NULL DEFAULT '', sld_storage_key TEXT, sld_filename VARCHAR(255), sld_content_type VARCHAR(255), created_at TIMESTAMPTZ NOT NULL DEFAULT now(), updated_at TIMESTAMPTZ NOT NULL DEFAULT now() ); ALTER TABLE datasets ADD COLUMN dg_id UUID REFERENCES dataset_groups (id) ON DELETE RESTRICT; ALTER TABLE datasets ADD COLUMN start_date DATE; ALTER TABLE datasets ADD COLUMN end_date DATE; ALTER TABLE datasets ADD CONSTRAINT chk_datasets_period CHECK (start_date IS NULL OR end_date IS NULL OR end_date >= start_date); CREATE INDEX idx_datasets_dg_id_end_date ON datasets (dg_id, end_date DESC NULLS LAST, created_at DESC); CREATE INDEX idx_datasets_start_date ON datasets (start_date); CREATE INDEX idx_datasets_end_date ON datasets (end_date); -- Period columns are renamed everywhere: observations, and the year_columns -- mapping JSON ({column, date, date_to} -> {column, start_date, end_date}). ALTER TABLE dataset_observations RENAME COLUMN date TO start_date; ALTER TABLE dataset_observations RENAME COLUMN date_to TO end_date; DROP INDEX uq_dataset_observations; DROP INDEX idx_dataset_observations_kato_date; CREATE UNIQUE INDEX uq_dataset_observations ON dataset_observations (dataset_id, kato_code, start_date, COALESCE(end_date, start_date)); CREATE INDEX idx_dataset_observations_kato_date ON dataset_observations (kato_code, start_date); UPDATE datasets SET year_columns = ( SELECT jsonb_agg( (elem - 'date' - 'date_to') || jsonb_build_object('start_date', elem->'date') || CASE WHEN elem ? 'date_to' AND elem->'date_to' IS NOT NULL THEN jsonb_build_object('end_date', elem->'date_to') ELSE '{}'::jsonb END) FROM jsonb_array_elements(year_columns) AS elem) WHERE year_columns IS NOT NULL AND jsonb_typeof(year_columns) = 'array' AND jsonb_array_length(year_columns) > 0; -- +goose Down UPDATE datasets SET year_columns = ( SELECT jsonb_agg( (elem - 'start_date' - 'end_date') || jsonb_build_object('date', elem->'start_date') || CASE WHEN elem ? 'end_date' AND elem->'end_date' IS NOT NULL THEN jsonb_build_object('date_to', elem->'end_date') ELSE '{}'::jsonb END) FROM jsonb_array_elements(year_columns) AS elem) WHERE year_columns IS NOT NULL AND jsonb_typeof(year_columns) = 'array' AND jsonb_array_length(year_columns) > 0; DROP INDEX uq_dataset_observations; DROP INDEX idx_dataset_observations_kato_date; ALTER TABLE dataset_observations RENAME COLUMN start_date TO date; ALTER TABLE dataset_observations RENAME COLUMN end_date TO date_to; CREATE UNIQUE INDEX uq_dataset_observations ON dataset_observations (dataset_id, kato_code, date, COALESCE(date_to, date)); CREATE INDEX idx_dataset_observations_kato_date ON dataset_observations (kato_code, date); DROP INDEX idx_datasets_dg_id_end_date; DROP INDEX idx_datasets_start_date; DROP INDEX idx_datasets_end_date; ALTER TABLE datasets DROP CONSTRAINT chk_datasets_period; ALTER TABLE datasets DROP COLUMN dg_id; ALTER TABLE datasets DROP COLUMN start_date; ALTER TABLE datasets DROP COLUMN end_date; DROP TABLE dataset_groups; ``` - [ ] **Step 2: Domain — rename and add fields** (`internal/domain/dataset.go`) Rename `Observation.Date`→`StartDate` (`json:"start_date"`), `Observation.DateTo`→`EndDate` (`json:"end_date,omitempty"`), `YearColumn.Date`→`StartDate` (`json:"start_date"`), `YearColumn.DateTo`→`EndDate` (`json:"end_date,omitempty"`). Update doc comments accordingly. Add to both `Dataset` and `DatasetSummary` (after `CategoryID`): ```go // DatasetGroupID links the dataset to its dataset group ("одноимённые" // datasets versioned by period). Nullable. DatasetGroupID *uuid.UUID `json:"dg_id"` // StartDate / EndDate is the period the dataset describes (YYYY-MM-DD). // Both nullable; EndDate drives the "freshest in group" rule. StartDate *string `json:"start_date"` EndDate *string `json:"end_date"` ``` Add to `DatasetFilter` (document: dates are inclusive `YYYY-MM-DD` bounds): ```go DatasetGroupID *uuid.UUID StartDateFrom *string StartDateTo *string EndDateFrom *string EndDateTo *string ``` Add: ```go // DatasetFields is the set of editable metadata columns written by // UpdateFields. Nil pointers clear the corresponding nullable column. type DatasetFields struct { CategoryID uuid.UUID Name string Description *string Unit *string DatasetGroupID *uuid.UUID StartDate *string EndDate *string } // ValidateDatePeriod checks that start/end (YYYY-MM-DD, either may be nil) // parse and that end is not before start. Returns an ErrValidation-wrapped // error naming the offending field. func ValidateDatePeriod(start, end *string) error { var from, to time.Time var err error if start != nil { if from, err = time.Parse("2006-01-02", *start); err != nil { return fmt.Errorf("%w: start_date %q is not YYYY-MM-DD", ErrValidation, *start) } } if end != nil { if to, err = time.Parse("2006-01-02", *end); err != nil { return fmt.Errorf("%w: end_date %q is not YYYY-MM-DD", ErrValidation, *end) } } if start != nil && end != nil && to.Before(from) { return fmt.Errorf("%w: end_date %q is before start_date %q", ErrValidation, *end, *start) } return nil } ``` - [ ] **Step 3: Domain test** (`internal/domain/dataset_test.go`, append) ```go func TestValidateDatePeriod(t *testing.T) { s := func(v string) *string { return &v } cases := []struct { name string start, end *string wantErr bool }{ {"both nil", nil, nil, false}, {"start only", s("2020-01-01"), nil, false}, {"ordered", s("2020-01-01"), s("2020-12-31"), false}, {"same day", s("2020-01-01"), s("2020-01-01"), false}, {"reversed", s("2020-12-31"), s("2020-01-01"), true}, {"bad start", s("01.01.2020"), nil, true}, {"bad end", nil, s("2020-13-01"), true}, } for _, tc := range cases { t.Run(tc.name, func(t *testing.T) { err := ValidateDatePeriod(tc.start, tc.end) if (err != nil) != tc.wantErr { t.Fatalf("err=%v wantErr=%v", err, tc.wantErr) } if err != nil && !errors.Is(err, ErrValidation) { t.Fatalf("expected ErrValidation, got %v", err) } }) } } ``` Run: `go test ./internal/domain/ -run TestValidateDatePeriod -v` → PASS after Step 2. - [ ] **Step 4: Repository** (`internal/repository/postgres/dataset.go`) `datasetColumns`: insert `dg_id, to_char(start_date, 'YYYY-MM-DD') AS start_date, to_char(end_date, 'YYYY-MM-DD') AS end_date,` right after `category_id,`; `scanDataset` scans `&d.DatasetGroupID, &d.StartDate, &d.EndDate` in the same position. Same for `datasetSummaryColumns`/`scanDatasetSummary`. `Create`: insert `dg_id, start_date, end_date` (`$14,$15,$16`, values `d.DatasetGroupID, d.StartDate, d.EndDate`; pgx maps `*string` → `date` text fine, use `$15::date, $16::date` casts). Replace `UpdateFields` with: ```go // UpdateFields overwrites a dataset's editable metadata columns. Nil pointers // clear the nullable columns. Returns domain.ErrNotFound if the dataset does // not exist, or domain.ErrConflict if the category/group does not exist. func (r *DatasetRepository) UpdateFields(ctx context.Context, id uuid.UUID, f domain.DatasetFields) (domain.Dataset, error) { row := r.pool.QueryRow(ctx, `UPDATE datasets SET category_id = $2, name = $3, description = $4, unit = $5, dg_id = $6, start_date = $7::date, end_date = $8::date, updated_at = now() WHERE id = $1 RETURNING `+datasetColumns, id, f.CategoryID, f.Name, f.Description, f.Unit, f.DatasetGroupID, f.StartDate, f.EndDate, ) out, err := scanDataset(row) return out, mapError(err) } ``` `datasetFilterClause`: append ```go if f.DatasetGroupID != nil { args = append(args, *f.DatasetGroupID) conds = append(conds, fmt.Sprintf("dg_id = $%d", len(args))) } for _, b := range []struct { col, op string val *string }{ {"start_date", ">=", f.StartDateFrom}, {"start_date", "<=", f.StartDateTo}, {"end_date", ">=", f.EndDateFrom}, {"end_date", "<=", f.EndDateTo}, } { if b.val != nil { args = append(args, *b.val) conds = append(conds, fmt.Sprintf("%s %s $%d::date", b.col, b.op, len(args))) } } ``` Add: ```go // LatestInGroup returns the freshest dataset of a group: greatest end_date // (NULL end_date sorts last), ties broken by created_at DESC. status and // fileType optionally restrict candidates. Returns domain.ErrNotFound when no // dataset matches. func (r *DatasetRepository) LatestInGroup(ctx context.Context, groupID uuid.UUID, status *string, fileType *domain.FileType) (domain.Dataset, error) { where, args := datasetFilterClause(domain.DatasetFilter{DatasetGroupID: &groupID, Status: status, FileType: fileType}, nil) row := r.pool.QueryRow(ctx, `SELECT `+datasetColumns+` FROM datasets`+where+ ` ORDER BY end_date DESC NULLS LAST, created_at DESC LIMIT 1`, args...) out, err := scanDataset(row) return out, mapError(err) } ``` Observations: `ReplaceObservations` uses `o.StartDate`/`o.EndDate`, copies into columns `"start_date", "end_date"`; `observationColumns` becomes `… to_char(start_date, 'YYYY-MM-DD') AS start_date, to_char(end_date, 'YYYY-MM-DD') AS end_date, …`; `scanObservation` scans `&o.StartDate, &o.EndDate`; `ORDER BY kato_code, date` → `ORDER BY kato_code, start_date` (three places: ListObservations ×2, ListAllObservations in `district.go`). - [ ] **Step 5: Service** (`internal/service/dataset.go`) - Interface `DatasetRepository`: `UpdateFields(ctx, id uuid.UUID, f domain.DatasetFields) (domain.Dataset, error)`; add `LatestInGroup(ctx context.Context, groupID uuid.UUID, status *string, fileType *domain.FileType) (domain.Dataset, error)`. - `UploadInput`: add `DatasetGroupID *uuid.UUID; StartDate *string; EndDate *string`. In `Upload`, after category check: `if err := domain.ValidateDatePeriod(in.StartDate, in.EndDate); err != nil { return domain.Dataset{}, err }` and pass the three fields into `repo.Create`. - `DatasetUpdateInput`: add the three fields; `Update` validates the period and calls `s.repo.UpdateFields(ctx, id, domain.DatasetFields{CategoryID: in.CategoryID, Name: in.Name, Description: in.Description, Unit: in.Unit, DatasetGroupID: in.DatasetGroupID, StartDate: in.StartDate, EndDate: in.EndDate})`. - `DatasetPatchInput`: add `DatasetGroupID *uuid.UUID; DatasetGroupIDSet bool; StartDate *string; StartDateSet bool; EndDate *string; EndDateSet bool`; `Patch` merges like description/unit, validates the merged period, calls `UpdateFields` with `DatasetFields`. - Rename all `yc.Date`/`yc.DateTo`/`o.Date`/`o.DateTo` to `StartDate`/`EndDate`; error messages say `start_date`/`end_date`. GeoJSON `data`/`periods` maps are keyed by `o.StartDate` and the period value is `*o.EndDate` (behaviour unchanged). - [ ] **Step 6: Handler** (`internal/transport/http/dataset_handler.go`) - `yearColumnInput`: `StartDate string \`json:"start_date" validate:"required,datetime=2006-01-02"\``, `EndDate *string \`json:"end_date" validate:"omitempty,datetime=2006-01-02"\``; build `domain.YearColumn{Column, StartDate, EndDate}`. - `upload`: parse optional form values `dg_id` (UUID → 422 "dg_id must be a valid UUID"), `start_date`, `end_date` (blank → nil; format checked by the service) and pass into `UploadInput`. - `list`: parse `dg_id` with `parseOptionalUUIDQuery`; for each of `start_date_from`, `start_date_to`, `end_date_from`, `end_date_to`: trim; if non-empty, `time.Parse("2006-01-02", v)` else 400 `" must be YYYY-MM-DD"`; set the filter pointer. - `datasetPutRequest`: add `DgID *string \`json:"dg_id" validate:"omitempty,uuid"\``, `StartDate *string \`json:"start_date" validate:"omitempty,datetime=2006-01-02"\``, `EndDate *string \`json:"end_date" validate:"omitempty,datetime=2006-01-02"\``; map into `DatasetUpdateInput` (parse `dg_id` UUID). - `patch`: extend the raw-map handling used for description/unit: keys `dg_id`, `start_date`, `end_date`; `null` → `Set=true, nil`; string → parsed (`dg_id` must be UUID → 422; dates `datetime` → 422 "start_date must be YYYY-MM-DD"). - [ ] **Step 7: Update tests** (`internal/service/dataset_test.go`) Update the stub repo: `UpdateFields(_ context.Context, id uuid.UUID, f domain.DatasetFields)` assigns all seven fields; add `LatestInGroup` to the stub: ```go func (r *stubDatasetRepo) LatestInGroup(_ context.Context, groupID uuid.UUID, status *string, fileType *domain.FileType) (domain.Dataset, error) { var best *domain.Dataset for _, d := range r.store { d := d if d.DatasetGroupID == nil || *d.DatasetGroupID != groupID { continue } if status != nil && d.Status != *status { continue } if fileType != nil && d.FileType != *fileType { continue } if best == nil || laterPeriod(d, *best) { best = &d } } if best == nil { return domain.Dataset{}, domain.ErrNotFound } return *best, nil } // laterPeriod reports whether a is fresher than b: greater end_date, NULL // end_date last, then newer created_at. func laterPeriod(a, b domain.Dataset) bool { switch { case a.EndDate != nil && b.EndDate == nil: return true case a.EndDate == nil && b.EndDate != nil: return false case a.EndDate != nil && b.EndDate != nil && *a.EndDate != *b.EndDate: return *a.EndDate > *b.EndDate default: return a.CreatedAt.After(b.CreatedAt) } } ``` Rename every `Date:`/`DateTo:` literal and `.Date`/`.DateTo` access to `StartDate`/`EndDate`. Add tests: ```go func TestDatasetService_Upload_StoresPeriodAndGroup(t *testing.T) { repo := newStubDatasetRepo() svc := newTestService(repo) // use the existing helper in this file that builds a service with stub store/categories/jobs gid := uuid.New() start, end := "2020-01-01", "2020-12-31" ds, err := svc.Upload(context.Background(), validRasterUpload(t, func(in *UploadInput) { in.DatasetGroupID = &gid in.StartDate = &start in.EndDate = &end })) if err != nil { t.Fatal(err) } if ds.DatasetGroupID == nil || *ds.DatasetGroupID != gid || *ds.StartDate != start || *ds.EndDate != end { t.Fatalf("period/group not stored: %+v", ds) } } func TestDatasetService_Upload_RejectsReversedPeriod(t *testing.T) { svc := newTestService(newStubDatasetRepo()) start, end := "2021-01-01", "2020-12-31" _, err := svc.Upload(context.Background(), validRasterUpload(t, func(in *UploadInput) { in.StartDate = &start in.EndDate = &end })) if !errors.Is(err, domain.ErrValidation) { t.Fatalf("want ErrValidation, got %v", err) } } ``` (If no `newTestService`/`validRasterUpload` helper exists yet, add them next to the existing upload tests by extracting the setup the existing `TestDatasetService_Upload_Success` uses; the helpers take a mutator `func(*UploadInput)`.) - [ ] **Step 8: Run tests** — `go test ./internal/domain/ ./internal/service/ ./internal/transport/... ./internal/repository/...` → PASS; `go vet ./internal/...` clean. - [ ] **Step 9: OpenAPI + README** `api/openapi.yaml`: `YearColumn` (`start_date` required, `end_date`), `Observation` (`start_date`, `end_date`), `MappingInput` example, `/datasets/{id}/mapping` description; `Dataset` and `DatasetSummary` gain `dg_id` (uuid, nullable), `start_date`, `end_date` (`format: date`, nullable); `DatasetPut`/`DatasetPatch` gain the same; `POST /datasets` form gains `dg_id`, `start_date`, `end_date`; `GET /datasets` parameters `dg_id`, `start_date_from`, `start_date_to`, `end_date_from`, `end_date_to`. `README.md`: mapping example uses `start_date`; add a "Dataset periods" paragraph describing the fields and filters. - [ ] **Step 10: Commit** (lane A only; skip if the orchestrator commits) ```bash git add migrations/00010_dataset_groups_and_dates.sql internal/domain/dataset.go internal/domain/dataset_test.go internal/repository/postgres/dataset.go internal/repository/postgres/district.go internal/service/dataset.go internal/service/dataset_test.go internal/transport/http/dataset_handler.go api/openapi.yaml README.md git commit -m "feat(datasets): periods (start_date/end_date), dg_id column, date filters; rename observation dates" ``` --- ### Task 2: Dataset groups — CRUD, latest, group SLD, dataset SLD fallback, `CreateRasterFromFile` **Files:** - Create: `internal/domain/dataset_group.go`, `internal/repository/postgres/dataset_group.go`, `internal/service/dataset_group.go`, `internal/service/dataset_group_test.go`, `internal/transport/http/dataset_group_handler.go` - Modify: `internal/service/dataset.go`, `internal/service/dataset_test.go`, `internal/transport/http/router.go`, `internal/app/app.go`, `api/openapi.yaml`, `README.md` **Interfaces:** - Consumes: Task 1 (`DatasetGroupID`, `LatestInGroup`, `DatasetFields`). - Produces (used by Task 7): - `domain.DatasetGroup` - `postgres.DatasetGroupRepository` with `GetByCode(ctx, code string) (domain.DatasetGroup, error)`, `GetByID`. - `service.DatasetGroupRepository` interface (below), `service.NewDatasetGroupService(repo DatasetGroupRepository, datasets DatasetRepository, store ObjectStore)`. - `service.DatasetService.CreateRasterFromFile(ctx, in CreateRasterInput) (domain.Dataset, error)` and `service.CreateRasterInput{CategoryID uuid.UUID; DatasetGroupID *uuid.UUID; Name, Filename, Path string; StartDate, EndDate *string; Meta json.RawMessage; Automated bool}`. - `service.NewDatasetService(repo, store, categories, groups groupReader, jobs, parseColumns, parseRows, converter)` — **new 4th parameter** `groups groupReader` where `type groupReader interface{ GetByID(ctx context.Context, id uuid.UUID) (domain.DatasetGroup, error) }`. - [ ] **Step 1: Domain** (`internal/domain/dataset_group.go`) ```go package domain import ( "time" "github.com/google/uuid" ) // DatasetGroup groups "одноимённые" datasets (the same indicator delivered for // different periods). The freshest dataset of a group is the one with the // greatest end_date. A group may carry a shared SLD style used as a fallback by // datasets that have none of their own. type DatasetGroup struct { ID uuid.UUID `json:"id"` Code string `json:"code"` Name string `json:"name"` Description string `json:"description"` SldStorageKey *string `json:"sld_storage_key"` SldFilename *string `json:"sld_filename"` SldContentType *string `json:"sld_content_type"` CreatedAt time.Time `json:"created_at"` UpdatedAt time.Time `json:"updated_at"` } ``` - [ ] **Step 2: Repository** (`internal/repository/postgres/dataset_group.go`) ```go package postgres import ( "context" "fmt" "strings" "gis/internal/domain" "github.com/google/uuid" "github.com/jackc/pgx/v5" "github.com/jackc/pgx/v5/pgxpool" ) // DatasetGroupRepository persists dataset groups in Postgres. type DatasetGroupRepository struct{ pool *pgxpool.Pool } // NewDatasetGroupRepository returns a DatasetGroupRepository backed by pool. func NewDatasetGroupRepository(pool *pgxpool.Pool) *DatasetGroupRepository { return &DatasetGroupRepository{pool: pool} } const datasetGroupColumns = `id, code, name, description, sld_storage_key, sld_filename, sld_content_type, created_at, updated_at` func scanDatasetGroup(row pgx.Row) (domain.DatasetGroup, error) { var g domain.DatasetGroup err := row.Scan(&g.ID, &g.Code, &g.Name, &g.Description, &g.SldStorageKey, &g.SldFilename, &g.SldContentType, &g.CreatedAt, &g.UpdatedAt) return g, err } func (r *DatasetGroupRepository) Create(ctx context.Context, g domain.DatasetGroup) (domain.DatasetGroup, error) { row := r.pool.QueryRow(ctx, `INSERT INTO dataset_groups (code, name, description) VALUES ($1, $2, $3) RETURNING `+datasetGroupColumns, g.Code, g.Name, g.Description) out, err := scanDatasetGroup(row) return out, mapError(err) } func (r *DatasetGroupRepository) GetByID(ctx context.Context, id uuid.UUID) (domain.DatasetGroup, error) { out, err := scanDatasetGroup(r.pool.QueryRow(ctx, `SELECT `+datasetGroupColumns+` FROM dataset_groups WHERE id = $1`, id)) return out, mapError(err) } func (r *DatasetGroupRepository) GetByCode(ctx context.Context, code string) (domain.DatasetGroup, error) { out, err := scanDatasetGroup(r.pool.QueryRow(ctx, `SELECT `+datasetGroupColumns+` FROM dataset_groups WHERE code = $1`, code)) return out, mapError(err) } // List returns a page of groups ordered by code; code (exact) is optional. func (r *DatasetGroupRepository) List(ctx context.Context, code *string, limit, offset int) ([]domain.DatasetGroup, error) { where, args := groupWhere(code) args = append(args, limit, offset) rows, err := r.pool.Query(ctx, fmt.Sprintf(`SELECT %s FROM dataset_groups%s ORDER BY code LIMIT $%d OFFSET $%d`, datasetGroupColumns, where, len(args)-1, len(args)), args...) if err != nil { return nil, mapError(err) } defer rows.Close() out := make([]domain.DatasetGroup, 0) for rows.Next() { g, err := scanDatasetGroup(rows) if err != nil { return nil, mapError(err) } out = append(out, g) } return out, mapError(rows.Err()) } func (r *DatasetGroupRepository) Count(ctx context.Context, code *string) (int, error) { where, args := groupWhere(code) var n int err := r.pool.QueryRow(ctx, `SELECT count(*) FROM dataset_groups`+where, args...).Scan(&n) return n, mapError(err) } func groupWhere(code *string) (string, []any) { if code == nil { return "", nil } return " WHERE code = $1", []any{*code} } func (r *DatasetGroupRepository) Update(ctx context.Context, g domain.DatasetGroup) (domain.DatasetGroup, error) { row := r.pool.QueryRow(ctx, `UPDATE dataset_groups SET code = $2, name = $3, description = $4, updated_at = now() WHERE id = $1 RETURNING `+datasetGroupColumns, g.ID, g.Code, g.Name, g.Description) out, err := scanDatasetGroup(row) return out, mapError(err) } // Delete removes a group. Returns domain.ErrNotFound if absent and // domain.ErrConflict (via the FK) if datasets still reference it. func (r *DatasetGroupRepository) Delete(ctx context.Context, id uuid.UUID) error { tag, err := r.pool.Exec(ctx, `DELETE FROM dataset_groups WHERE id = $1`, id) if err != nil { return mapError(err) } if tag.RowsAffected() == 0 { return domain.ErrNotFound } return nil } func (r *DatasetGroupRepository) SetSLD(ctx context.Context, id uuid.UUID, key, filename, contentType string) (domain.DatasetGroup, error) { row := r.pool.QueryRow(ctx, `UPDATE dataset_groups SET sld_storage_key = $2, sld_filename = $3, sld_content_type = $4, updated_at = now() WHERE id = $1 RETURNING `+datasetGroupColumns, id, key, filename, contentType) out, err := scanDatasetGroup(row) return out, mapError(err) } var _ = strings.TrimSpace // keep import if unused after edits; remove if not needed ``` (Remove the trailing `strings` line and import if you do not use `strings`.) - [ ] **Step 3: Service test first** (`internal/service/dataset_group_test.go`) ```go package service import ( "context" "errors" "strings" "testing" "gis/internal/domain" "github.com/google/uuid" ) type stubGroupRepo struct { store map[uuid.UUID]domain.DatasetGroup } func newStubGroupRepo() *stubGroupRepo { return &stubGroupRepo{store: map[uuid.UUID]domain.DatasetGroup{}} } func (r *stubGroupRepo) Create(_ context.Context, g domain.DatasetGroup) (domain.DatasetGroup, error) { for _, e := range r.store { if e.Code == g.Code { return domain.DatasetGroup{}, domain.ErrConflict } } g.ID = uuid.New() r.store[g.ID] = g return g, nil } func (r *stubGroupRepo) GetByID(_ context.Context, id uuid.UUID) (domain.DatasetGroup, error) { g, ok := r.store[id] if !ok { return domain.DatasetGroup{}, domain.ErrNotFound } return g, nil } func (r *stubGroupRepo) GetByCode(_ context.Context, code string) (domain.DatasetGroup, error) { for _, g := range r.store { if g.Code == code { return g, nil } } return domain.DatasetGroup{}, domain.ErrNotFound } func (r *stubGroupRepo) List(_ context.Context, code *string, limit, offset int) ([]domain.DatasetGroup, error) { out := []domain.DatasetGroup{} for _, g := range r.store { if code == nil || g.Code == *code { out = append(out, g) } } return out, nil } func (r *stubGroupRepo) Count(_ context.Context, code *string) (int, error) { l, _ := r.List(context.Background(), code, 0, 0) return len(l), nil } func (r *stubGroupRepo) Update(_ context.Context, g domain.DatasetGroup) (domain.DatasetGroup, error) { if _, ok := r.store[g.ID]; !ok { return domain.DatasetGroup{}, domain.ErrNotFound } r.store[g.ID] = g return g, nil } func (r *stubGroupRepo) Delete(_ context.Context, id uuid.UUID) error { if _, ok := r.store[id]; !ok { return domain.ErrNotFound } delete(r.store, id) return nil } func (r *stubGroupRepo) SetSLD(_ context.Context, id uuid.UUID, key, filename, contentType string) (domain.DatasetGroup, error) { g, ok := r.store[id] if !ok { return domain.DatasetGroup{}, domain.ErrNotFound } g.SldStorageKey, g.SldFilename, g.SldContentType = &key, &filename, &contentType r.store[id] = g return g, nil } func TestDatasetGroupService_Create_RequiresSlug(t *testing.T) { svc := NewDatasetGroupService(newStubGroupRepo(), newStubDatasetRepo(), newStubStore()) _, err := svc.Create(context.Background(), DatasetGroupInput{Code: "Bad Code", Name: "x"}) if !errors.Is(err, domain.ErrValidation) { t.Fatalf("want ErrValidation, got %v", err) } } func TestDatasetGroupService_Latest_PrefersGreatestEndDateThenCreatedAt(t *testing.T) { groups := newStubGroupRepo() datasets := newStubDatasetRepo() svc := NewDatasetGroupService(groups, datasets, newStubStore()) g, _ := svc.Create(context.Background(), DatasetGroupInput{Code: "pop", Name: "Population"}) s := func(v string) *string { return &v } old := domain.Dataset{ID: uuid.New(), DatasetGroupID: &g.ID, EndDate: s("2019-12-31"), Status: domain.DatasetStatusReady} fresh := domain.Dataset{ID: uuid.New(), DatasetGroupID: &g.ID, EndDate: s("2020-12-31"), Status: domain.DatasetStatusReady} undated := domain.Dataset{ID: uuid.New(), DatasetGroupID: &g.ID, Status: domain.DatasetStatusReady} for _, d := range []domain.Dataset{old, fresh, undated} { datasets.store[d.ID] = d } got, err := svc.Latest(context.Background(), g.ID, nil, nil) if err != nil { t.Fatal(err) } if got.ID != fresh.ID { t.Fatalf("want %s, got %s", fresh.ID, got.ID) } } func TestDatasetGroupService_Latest_NotFoundWhenEmpty(t *testing.T) { groups := newStubGroupRepo() svc := NewDatasetGroupService(groups, newStubDatasetRepo(), newStubStore()) g, _ := svc.Create(context.Background(), DatasetGroupInput{Code: "pop", Name: "Population"}) if _, err := svc.Latest(context.Background(), g.ID, nil, nil); !errors.Is(err, domain.ErrNotFound) { t.Fatalf("want ErrNotFound, got %v", err) } } func TestDatasetGroupService_UploadSLD_RejectsBadExtension(t *testing.T) { groups := newStubGroupRepo() svc := NewDatasetGroupService(groups, newStubDatasetRepo(), newStubStore()) g, _ := svc.Create(context.Background(), DatasetGroupInput{Code: "pop", Name: "Population"}) _, err := svc.UploadSLD(context.Background(), g.ID, UploadSLDInput{Filename: "style.txt", ContentType: "text/plain", Size: 3, Reader: strings.NewReader("")}) if !errors.Is(err, domain.ErrValidation) { t.Fatalf("want ErrValidation, got %v", err) } } ``` `newStubStore()` — the in-memory ObjectStore stub already used in `dataset_test.go` (reuse its constructor name; if it is named differently, use that name). - [ ] **Step 4: Run** `go test ./internal/service/ -run DatasetGroupService` → FAIL (undefined). - [ ] **Step 5: Service** (`internal/service/dataset_group.go`) ```go package service import ( "bytes" "context" "fmt" "io" "path" "path/filepath" "strings" "gis/internal/domain" "github.com/google/uuid" ) // DatasetGroupRepository is the persistence behaviour DatasetGroupService needs. type DatasetGroupRepository interface { Create(ctx context.Context, g domain.DatasetGroup) (domain.DatasetGroup, error) GetByID(ctx context.Context, id uuid.UUID) (domain.DatasetGroup, error) GetByCode(ctx context.Context, code string) (domain.DatasetGroup, error) List(ctx context.Context, code *string, limit, offset int) ([]domain.DatasetGroup, error) Count(ctx context.Context, code *string) (int, error) Update(ctx context.Context, g domain.DatasetGroup) (domain.DatasetGroup, error) Delete(ctx context.Context, id uuid.UUID) error SetSLD(ctx context.Context, id uuid.UUID, key, filename, contentType string) (domain.DatasetGroup, error) } // DatasetGroupInput carries the mutable fields of a group. type DatasetGroupInput struct { Code string Name string Description string } // DatasetGroupPage is a page of groups. type DatasetGroupPage struct { Items []domain.DatasetGroup Page int PageSize int Total int } // DatasetGroupService implements dataset-group business rules. type DatasetGroupService struct { repo DatasetGroupRepository datasets DatasetRepository store ObjectStore } // NewDatasetGroupService wires the group repository, the dataset repository // (for the "latest" lookup) and the object store (for the group SLD). func NewDatasetGroupService(repo DatasetGroupRepository, datasets DatasetRepository, store ObjectStore) *DatasetGroupService { return &DatasetGroupService{repo: repo, datasets: datasets, store: store} } func (s *DatasetGroupService) Create(ctx context.Context, in DatasetGroupInput) (domain.DatasetGroup, error) { if !domain.ValidSlug(in.Code) { return domain.DatasetGroup{}, fmt.Errorf("%w: code must be a slug (lowercase latin letters, digits, and dashes)", domain.ErrValidation) } return s.repo.Create(ctx, domain.DatasetGroup{Code: in.Code, Name: in.Name, Description: in.Description}) } func (s *DatasetGroupService) Get(ctx context.Context, id uuid.UUID) (domain.DatasetGroup, error) { return s.repo.GetByID(ctx, id) } // List returns a page of groups; page/pageSize are clamped like datasets. func (s *DatasetGroupService) List(ctx context.Context, code *string, page, pageSize int) (DatasetGroupPage, error) { if page < 1 { page = 1 } if pageSize < 1 { pageSize = DefaultPageSize } if pageSize > MaxPageSize { pageSize = MaxPageSize } items, err := s.repo.List(ctx, code, pageSize, (page-1)*pageSize) if err != nil { return DatasetGroupPage{}, err } total, err := s.repo.Count(ctx, code) if err != nil { return DatasetGroupPage{}, err } return DatasetGroupPage{Items: items, Page: page, PageSize: pageSize, Total: total}, nil } func (s *DatasetGroupService) Update(ctx context.Context, id uuid.UUID, in DatasetGroupInput) (domain.DatasetGroup, error) { if !domain.ValidSlug(in.Code) { return domain.DatasetGroup{}, fmt.Errorf("%w: code must be a slug (lowercase latin letters, digits, and dashes)", domain.ErrValidation) } return s.repo.Update(ctx, domain.DatasetGroup{ID: id, Code: in.Code, Name: in.Name, Description: in.Description}) } // Delete removes a group (409 via ErrConflict while datasets reference it) and // then best-effort removes its SLD object. func (s *DatasetGroupService) Delete(ctx context.Context, id uuid.UUID) error { g, err := s.repo.GetByID(ctx, id) if err != nil { return err } if err := s.repo.Delete(ctx, id); err != nil { return err } if g.SldStorageKey != nil { _ = s.store.Remove(ctx, *g.SldStorageKey) } return nil } // Latest returns the freshest dataset of the group (greatest end_date, NULLs // last, then newest), optionally restricted by status and file type. func (s *DatasetGroupService) Latest(ctx context.Context, id uuid.UUID, status *string, fileType *domain.FileType) (domain.Dataset, error) { if _, err := s.repo.GetByID(ctx, id); err != nil { return domain.Dataset{}, err } return s.datasets.LatestInGroup(ctx, id, status, fileType) } // UploadSLD validates and stores the group's shared SLD, replacing any previous one. func (s *DatasetGroupService) UploadSLD(ctx context.Context, id uuid.UUID, in UploadSLDInput) (domain.DatasetGroup, error) { g, err := s.repo.GetByID(ctx, id) if err != nil { return domain.DatasetGroup{}, err } ext := strings.ToLower(filepath.Ext(in.Filename)) if !domain.ExtensionAllowedForSLD(ext) { return domain.DatasetGroup{}, fmt.Errorf("%w: sld file must have one of these extensions: %s", domain.ErrValidation, strings.Join(domain.AllowedSLDExtensions, ", ")) } head := make([]byte, 512) n, err := io.ReadFull(in.Reader, head) if err != nil && err != io.EOF && err != io.ErrUnexpectedEOF { return domain.DatasetGroup{}, fmt.Errorf("read sld upload: %w", err) } head = head[:n] if err := domain.ValidateFileContent(ext, head); err != nil { return domain.DatasetGroup{}, fmt.Errorf("%w: %s", domain.ErrValidation, err) } content := io.MultiReader(bytes.NewReader(head), in.Reader) key := path.Join("groups", g.ID.String(), "sld", in.Filename) if err := s.store.Put(ctx, key, content, in.Size, in.ContentType); err != nil { return domain.DatasetGroup{}, err } updated, err := s.repo.SetSLD(ctx, id, key, in.Filename, in.ContentType) if err != nil { _ = s.store.Remove(ctx, key) return domain.DatasetGroup{}, err } if g.SldStorageKey != nil && *g.SldStorageKey != key { _ = s.store.Remove(ctx, *g.SldStorageKey) } return updated, nil } // DownloadSLD returns the group and a reader for its SLD; ErrSLDNotFound when none. func (s *DatasetGroupService) DownloadSLD(ctx context.Context, id uuid.UUID) (domain.DatasetGroup, io.ReadCloser, error) { g, err := s.repo.GetByID(ctx, id) if err != nil { return domain.DatasetGroup{}, nil, err } if g.SldStorageKey == nil { return domain.DatasetGroup{}, nil, ErrSLDNotFound } obj, err := s.store.Get(ctx, *g.SldStorageKey) if err != nil { return domain.DatasetGroup{}, nil, err } return g, obj, nil } ``` - [ ] **Step 6: DatasetService changes** (`internal/service/dataset.go`) 1. Add `groupReader` interface and the `groups groupReader` field + **4th constructor parameter** (update every `NewDatasetService(` call: `internal/app/app.go`, tests). 2. `ensureGroupExists(ctx, id *uuid.UUID) error` — nil → nil; not found → `fmt.Errorf("%w: dataset group %s not found", domain.ErrValidation, id)`. Call it in `Upload`, `Update`, and `Patch` (when the group changes). 3. `DownloadSLD` fallback: when `dataset.SldStorageKey == nil` and `dataset.DatasetGroupID != nil`, load the group; if it has an SLD, return a dataset copy whose `SldStorageKey/SldFilename/SldContentType` are the group's, and stream the group's object. Otherwise `ErrSLDNotFound`. 4. `CreateRasterFromFile`: ```go // CreateRasterInput describes a raster file already on local disk that should // become a dataset (used by the Landscape Program to publish results). type CreateRasterInput struct { CategoryID uuid.UUID DatasetGroupID *uuid.UUID Name string Filename string // stored filename, e.g. "lp-z-integrated.tif" Path string // local path to read from StartDate *string EndDate *string Meta json.RawMessage Automated bool } // CreateRasterFromFile stores a local GeoTIFF as a new raster dataset and // enqueues the standard COG conversion, exactly like an uploaded raster. func (s *DatasetService) CreateRasterFromFile(ctx context.Context, in CreateRasterInput) (domain.Dataset, error) { f, err := os.Open(in.Path) if err != nil { return domain.Dataset{}, err } defer f.Close() info, err := f.Stat() if err != nil { return domain.Dataset{}, err } return s.Upload(ctx, UploadInput{ CategoryID: in.CategoryID, DatasetGroupID: in.DatasetGroupID, Name: in.Name, Meta: in.Meta, Automated: in.Automated, Filename: in.Filename, FileType: domain.FileTypeRaster, ContentType: "image/tiff", Size: info.Size(), Reader: f, StartDate: in.StartDate, EndDate: in.EndDate, }) } ``` Add tests in `dataset_test.go`: `TestDatasetService_DownloadSLD_FallsBackToGroup` (dataset without SLD, group with SLD in stub store → reader returns group content) and `TestDatasetService_CreateRasterFromFile` (temp file with `II*\x00` header → dataset created with `file_type raster`, group id, convert job enqueued). The stub group reader is `stubGroupRepo` from Step 3. - [ ] **Step 7: Handler** (`internal/transport/http/dataset_group_handler.go`) ```go package http import ( "io" "net/http" "strings" "gis/internal/domain" "gis/internal/service" "gis/pkg/httputil" "github.com/go-chi/chi/v5" "github.com/go-playground/validator/v10" ) // DatasetGroupHandler serves the /dataset-groups routes. type DatasetGroupHandler struct { svc *service.DatasetGroupService validate *validator.Validate } func NewDatasetGroupHandler(svc *service.DatasetGroupService, validate *validator.Validate) *DatasetGroupHandler { return &DatasetGroupHandler{svc: svc, validate: validate} } func (h *DatasetGroupHandler) Register(r chi.Router) { r.Get("/", h.list) r.Post("/", h.create) r.Get("/{id}", h.get) r.Put("/{id}", h.update) r.Delete("/{id}", h.delete) r.Get("/{id}/latest", h.latest) r.Post("/{id}/sld", h.uploadSLD) r.Get("/{id}/sld", h.downloadSLD) } type datasetGroupRequest struct { Code string `json:"code" validate:"required,max=255,slug"` Name string `json:"name" validate:"required,max=255"` Description string `json:"description" validate:"max=2000"` } ``` Implement `create` (201), `list` (`?code=`, `page`, `page_size` via `parsePositiveIntQuery`, `newPaginated`), `get`, `update`, `delete` (204), `latest` (`?status=` validated with `domain.ValidDatasetStatus`, `?file_type=` validated with `FileType.Valid()`; 404 via `respondDomainError`), `uploadSLD`/`downloadSLD` mirroring `DatasetHandler.uploadSLD/downloadSLD` (multipart `file`, default content type `application/xml`, `Content-Disposition: attachment`). Follow `category_handler.go` for decode/validate/error flow. - [ ] **Step 8: Wiring** — `router.go`: add `DatasetGroups *DatasetGroupHandler` to `RouterDeps` and `r.Route("/dataset-groups", deps.DatasetGroups.Register)`. `app.go`: `groupRepo := postgres.NewDatasetGroupRepository(pool)`, pass `groupRepo` as the 4th arg to `NewDatasetService`, add field `groups *service.DatasetGroupService` = `service.NewDatasetGroupService(groupRepo, datasetRepo, store)`, and `DatasetGroups: transporthttp.NewDatasetGroupHandler(a.groups, validate)`. - [ ] **Step 9: Run** `go test ./internal/service/ ./internal/transport/... && go build ./...` → PASS. - [ ] **Step 10: OpenAPI + README** — add tag `Dataset Groups`, paths `/dataset-groups`, `/dataset-groups/{id}`, `/dataset-groups/{id}/latest`, `/dataset-groups/{id}/sld`, schemas `DatasetGroup`, `DatasetGroupInput`, `DatasetGroupPage`; note the SLD fallback on `GET /datasets/{id}/sld`. README: "Dataset groups" section (what a group is, freshness rule, SLD fallback). - [ ] **Step 11: Commit** ```bash git add internal/domain/dataset_group.go internal/repository/postgres/dataset_group.go internal/service/dataset_group.go internal/service/dataset_group_test.go internal/service/dataset.go internal/service/dataset_test.go internal/transport/http/dataset_group_handler.go internal/transport/http/router.go internal/app/app.go api/openapi.yaml README.md git commit -m "feat(dataset-groups): CRUD, latest-in-group, group SLD with dataset fallback, CreateRasterFromFile" ``` --- ### Task 3: Raster I/O abstraction + godal implementation + Dockerfile **Files:** - Create: `internal/raster/io.go`, `internal/raster/godal.go`, `internal/raster/godal_test.go` - Modify: `go.mod`, `go.sum` (via `go get github.com/airbusgeo/godal@v0.0.18`), `build/package/Dockerfile`, `.golangci.yml` only if cgo needs a setting (usually not) **Interfaces:** - Produces (used by Task 6): ```go package raster type Grid struct { Width, Height int; GeoTransform [6]float64; ProjectionWKT string } type Window struct { X, Y, Width, Height int } type Reader interface { Grid() Grid; NoData() (float64, bool); ReadFloat64(w Window, dst []float64) error; Close() error } type WriterOptions struct { NoData int16; TileSize int; Compression string } type Writer interface { WriteInt16(w Window, src []int16) error; Close() error } type IO interface { Open(ctx context.Context, path string) (Reader, error) Create(ctx context.Context, path string, g Grid, o WriterOptions) (Writer, error) SameGrid(a, b Grid) bool } func DefaultWriterOptions() WriterOptions // {NoData: -9999, TileSize: 512, Compression: "DEFLATE"} func NewGodalIO() *GodalIO // implements IO ``` - [ ] **Step 1: Interfaces** (`internal/raster/io.go`) — write the block above with doc comments. Add: ```go // Windows tiles a w×h raster into row-major windows of at most size×size. func Windows(w, h, size int) []Window { var out []Window for y := 0; y < h; y += size { wh := min(size, h-y) for x := 0; x < w; x += size { out = append(out, Window{X: x, Y: y, Width: min(size, w-x), Height: wh}) } } return out } ``` Test (`internal/raster/io_test.go`): `Windows(10, 10, 4)` → 9 windows, last is `{8,8,2,2}`; `Windows(5, 3, 8)` → one window `{0,0,5,3}`. - [ ] **Step 2: Add dependency** — `go get github.com/airbusgeo/godal@v0.0.18`. Verify `CGO_ENABLED=1 go build ./internal/raster/` compiles (Homebrew GDAL provides `gdal.pc` in `/opt/homebrew/lib/pkgconfig`; if `pkg-config` is missing run `brew install pkgconf`). - [ ] **Step 3: Failing round-trip test** (`internal/raster/godal_test.go`) ```go package raster import ( "context" "path/filepath" "testing" ) func TestGodalIO_RoundTrip(t *testing.T) { rio := NewGodalIO() ctx := context.Background() path := filepath.Join(t.TempDir(), "out.tif") grid := Grid{Width: 10, Height: 7, GeoTransform: [6]float64{100, 30, 0, 200, 0, -30}, ProjectionWKT: "EPSG:32642"} w, err := rio.Create(ctx, path, grid, DefaultWriterOptions()) if err != nil { t.Fatal(err) } full := make([]int16, 10*7) for i := range full { full[i] = int16(i%6) - 1 // -1..4 (-1 stands in for garbage, not nodata) } full[0] = -9999 if err := w.WriteInt16(Window{0, 0, 10, 7}, full); err != nil { t.Fatal(err) } if err := w.Close(); err != nil { t.Fatal(err) } r, err := rio.Open(ctx, path) if err != nil { t.Fatal(err) } defer r.Close() g := r.Grid() if g.Width != 10 || g.Height != 7 || g.GeoTransform != grid.GeoTransform || g.ProjectionWKT == "" { t.Fatalf("grid mismatch: %+v", g) } if nd, ok := r.NoData(); !ok || nd != -9999 { t.Fatalf("nodata = %v %v", nd, ok) } dst := make([]float64, 3*2) if err := r.ReadFloat64(Window{X: 8, Y: 5, Width: 2, Height: 2}, dst[:4]); err != nil { t.Fatal(err) } // pixel (8,5) has index 58 -> 58%6-1 = 3 ; (9,5)=59 -> 4 ; (8,6)=68 -> 1 ; (9,6)=69 -> 2 want := []float64{3, 4, 1, 2} for i, v := range want { if dst[i] != v { t.Fatalf("pixel %d = %v want %v", i, dst[i], v) } } if !rio.SameGrid(g, grid) { t.Fatal("SameGrid should be true for identical grids") } other := grid other.GeoTransform[0] += 1 if rio.SameGrid(g, other) { t.Fatal("SameGrid should be false for shifted origin") } } ``` Run: `go test ./internal/raster/ -run TestGodalIO -v` → FAIL (undefined `NewGodalIO`). - [ ] **Step 4: Implementation** (`internal/raster/godal.go`) ```go package raster import ( "context" "fmt" "math" "sync" "github.com/airbusgeo/godal" ) var registerOnce sync.Once // GodalIO implements IO with the GDAL bindings (github.com/airbusgeo/godal). // GDAL datasets are not goroutine-safe: callers must not share a Reader or // Writer between goroutines without external synchronisation. type GodalIO struct{} // NewGodalIO registers GDAL drivers (once) and returns a GodalIO. func NewGodalIO() *GodalIO { registerOnce.Do(godal.RegisterAll) return &GodalIO{} } // DefaultWriterOptions returns the Landscape Program output profile. func DefaultWriterOptions() WriterOptions { return WriterOptions{NoData: -9999, TileSize: 512, Compression: "DEFLATE"} } type godalReader struct { ds *godal.Dataset grid Grid } func (g *GodalIO) Open(_ context.Context, path string) (Reader, error) { ds, err := godal.Open(path) if err != nil { return nil, fmt.Errorf("open raster %s: %w", path, err) } st := ds.Structure() if st.NBands < 1 { ds.Close() return nil, fmt.Errorf("open raster %s: no bands", path) } gt, err := ds.GeoTransform() if err != nil { ds.Close() return nil, fmt.Errorf("open raster %s: geotransform: %w", path, err) } return &godalReader{ds: ds, grid: Grid{Width: st.SizeX, Height: st.SizeY, GeoTransform: gt, ProjectionWKT: ds.Projection()}}, nil } func (r *godalReader) Grid() Grid { return r.grid } func (r *godalReader) NoData() (float64, bool) { return r.ds.Bands()[0].NoData() } func (r *godalReader) ReadFloat64(w Window, dst []float64) error { if len(dst) < w.Width*w.Height { return fmt.Errorf("read window: dst has %d elements, need %d", len(dst), w.Width*w.Height) } return r.ds.Bands()[0].Read(w.X, w.Y, dst[:w.Width*w.Height], w.Width, w.Height) } func (r *godalReader) Close() error { return r.ds.Close() } type godalWriter struct{ ds *godal.Dataset } func (g *GodalIO) Create(_ context.Context, path string, grid Grid, o WriterOptions) (Writer, error) { tile := fmt.Sprint(o.TileSize) ds, err := godal.Create(godal.GTiff, path, 1, godal.Int16, grid.Width, grid.Height, godal.CreationOption("TILED=YES", "BLOCKXSIZE="+tile, "BLOCKYSIZE="+tile, "COMPRESS="+o.Compression, "PREDICTOR=2", "BIGTIFF=IF_SAFER")) if err != nil { return nil, fmt.Errorf("create raster %s: %w", path, err) } if err := ds.SetGeoTransform(grid.GeoTransform); err != nil { ds.Close() return nil, fmt.Errorf("create raster %s: geotransform: %w", path, err) } if grid.ProjectionWKT != "" { if err := ds.SetProjection(grid.ProjectionWKT); err != nil { ds.Close() return nil, fmt.Errorf("create raster %s: projection: %w", path, err) } } if err := ds.Bands()[0].SetNoData(float64(o.NoData)); err != nil { ds.Close() return nil, fmt.Errorf("create raster %s: nodata: %w", path, err) } return &godalWriter{ds: ds}, nil } func (w *godalWriter) WriteInt16(win Window, src []int16) error { if len(src) < win.Width*win.Height { return fmt.Errorf("write window: src has %d elements, need %d", len(src), win.Width*win.Height) } return w.ds.Bands()[0].Write(win.X, win.Y, src[:win.Width*win.Height], win.Width, win.Height) } func (w *godalWriter) Close() error { return w.ds.Close() } // SameGrid reports whether two grids describe the same pixels: equal size, // geotransform within 1e-9 (absolute), and the same CRS (GDAL IsSame; an // empty projection only matches an empty projection). func (g *GodalIO) SameGrid(a, b Grid) bool { if a.Width != b.Width || a.Height != b.Height { return false } for i := range a.GeoTransform { if math.Abs(a.GeoTransform[i]-b.GeoTransform[i]) > 1e-9 { return false } } if a.ProjectionWKT == "" || b.ProjectionWKT == "" { return a.ProjectionWKT == b.ProjectionWKT } sa, err := godal.NewSpatialRefFromWKT(a.ProjectionWKT) if err != nil { return false } defer sa.Close() sb, err := godal.NewSpatialRefFromWKT(b.ProjectionWKT) if err != nil { return false } defer sb.Close() return sa.IsSame(sb) } ``` Check godal's actual API names while implementing (`ds.Structure()`, `Bands()`, `Band.Read/Write` generic over slice types, `NewSpatialRefFromWKT`, `SpatialRef.IsSame`, `godal.CreationOption`); adjust if a name differs in v0.0.18 (`go doc github.com/airbusgeo/godal Dataset`). - [ ] **Step 5: Run** `go test ./internal/raster/ -v` → PASS. - [ ] **Step 6: Dockerfile** (`build/package/Dockerfile`) ```dockerfile # Build context is the repo root: docker build -f build/package/Dockerfile . FROM golang:1.26-alpine AS builder # godal (GDAL bindings) needs cgo, a C toolchain and the GDAL headers. RUN apk add --no-cache gdal-dev gcc musl-dev pkgconfig WORKDIR /src COPY go.mod go.sum ./ RUN go mod download COPY . . RUN CGO_ENABLED=1 GOOS=linux go build -trimpath -o /out/gis ./cmd/gis FROM alpine:3.22 # gdal: shared library for godal (raster math in the worker). # gdal-tools: gdal_translate / gdalinfo (COG conversion) and ogr2ogr. RUN apk add --no-cache ca-certificates tzdata gdal gdal-tools \ && adduser -D -u 10001 app WORKDIR /app COPY --from=builder /out/gis /usr/local/bin/gis USER app EXPOSE 8080 ENTRYPOINT ["gis"] CMD ["serve"] ``` If Docker is available, verify with `docker build -f build/package/Dockerfile -t gis:lp .`; otherwise note in the report that the image build was not verified. - [ ] **Step 7: Commit** ```bash git add internal/raster/io.go internal/raster/io_test.go internal/raster/godal.go internal/raster/godal_test.go go.mod go.sum build/package/Dockerfile git commit -m "feat(raster): IO abstraction with godal implementation; cgo Dockerfile" ``` --- ### Task 4: Landscape graph, params and pixel kernels (pure Go) **Files:** - Create: `internal/landscape/graph.go`, `internal/landscape/params.go`, `internal/landscape/kernels.go`, `internal/landscape/kernels_test.go`, `internal/landscape/graph_test.go` **Interfaces (produced, used by Tasks 5–8):** ```go package landscape const ( NoData int16 = -9999 CategoryCode = "landscape" // dataset category for every LP dataset ) type Slot struct { Code, Title string } // Code is the dataset_groups.code type Output struct { Code, Title string } type Node struct { Code, Name string; Stage int; Inputs []string; Outputs []string; Kernel Kernel } // Kernel computes one window: in[i] follows Node.Inputs order, out[j] follows Node.Outputs order; // all slices have len == window pixels. Every out value must be set (NoData or class). type Kernel func(in [][]float64, out [][]int16, p Params) var Slots []Slot; var Outputs []Output; var Nodes []Node func NodeByCode(code string) (Node, bool) func Stages() [][]Node // nodes grouped by Stage ascending, each group ordered by Code func OutputByCode(code string) (Output, bool) func SlotByCode(code string) (Slot, bool) func NodeForOutput(group string) (Node, bool) // which node produces a group type Params struct { WindowSize int `json:"window_size"` TiePriority []int `json:"tie_priority"` V1MinSE12 int `json:"v1_min_s_e12"` V1MinSE3 int `json:"v1_min_s_e3"` V2MinSE3 int `json:"v2_min_s_e3"` V2MinSE4 int `json:"v2_min_s_e4"` } func DefaultParams() Params func (p Params) WithDefaults() Params // zero fields -> defaults func (p Params) Validate() error // window 256..8192; thresholds 1..5; tie_priority permutation of 1..5 func (p Params) ForNode(code string) Params // only fields the node uses (window_size never included); used in fingerprints ``` - [ ] **Step 1: graph.go** ```go // Package landscape defines the Landscape Program (ЛП) computation graph — the // per-pixel formulas that turn ecological (Z), vulnerability (U) and // suitability (P) class rasters into constraints (E), conflicts (C) and // functional zoning (V1–V3) — together with a window-based executor. package landscape import "sort" const ( // NoData is the output nodata value of every LP raster. NoData int16 = -9999 // CategoryCode is the dataset category holding every LP dataset. CategoryCode = "landscape" ) // Group codes. Slots are leaf inputs users upload; the rest are node outputs. const ( SlotZBiodiversity = "lp-z-biodiversity" SlotZLCR = "lp-z-lcr" SlotZSoilProtection = "lp-z-soil-protection" SlotZWR = "lp-z-wr" SlotUSurfWater = "lp-u-surf-water" SlotUSoil = "lp-u-soil" SlotUCrop = "lp-u-crop" SlotUVeg = "lp-u-veg" SlotPIrrigation = "lp-p-irrigation" SlotPSnow = "lp-p-snow" SlotPCrop = "lp-p-crop" SlotPPasture = "lp-p-pasture" SlotPTourism = "lp-p-tourism" OutZIntegrated = "lp-z-integrated" OutUIntegrated = "lp-u-integrated" OutZMax = "lp-z-max" OutUMax = "lp-u-max" OutE = "lp-e" OutCIrrigation = "lp-c-irrigation" OutCSnow = "lp-c-snow" OutCCrop = "lp-c-crop" OutCPasture = "lp-c-pasture" OutCTourism = "lp-c-tourism" OutPMax = "lp-p-max" OutV1 = "lp-v1-conservative" OutV2 = "lp-v2-balanced" OutV3 = "lp-v3-development" OutCMax = "lp-c-max" ) type Slot struct{ Code, Title string } type Output struct{ Code, Title string } // Kernel computes one window. in[i] follows Node.Inputs order and out[j] // follows Node.Outputs order; every slice has exactly window-pixel length. type Kernel func(in [][]float64, out [][]int16, p Params) // Node is one function of the graph. Stage = first digit of Code. type Node struct { Code, Name string Stage int Inputs []string Outputs []string Kernel Kernel } var Slots = []Slot{ {SlotZBiodiversity, "Z Biodiversity"}, {SlotZLCR, "Z LCR"}, {SlotZSoilProtection, "Z Soil Protection"}, {SlotZWR, "Z Water Resources"}, {SlotUSurfWater, "U Surface Water"}, {SlotUSoil, "U Soil"}, {SlotUCrop, "U Crop"}, {SlotUVeg, "U Vegetation"}, {SlotPIrrigation, "P Irrigation"}, {SlotPSnow, "P Snow"}, {SlotPCrop, "P Crop"}, {SlotPPasture, "P Pasture"}, {SlotPTourism, "P Tourism"}, } var Outputs = []Output{ {OutZIntegrated, "Z Integrated"}, {OutUIntegrated, "U Integrated"}, {OutZMax, "Z Max"}, {OutUMax, "U Max"}, {OutE, "E Ecological Constraint"}, {OutCIrrigation, "C Irrigation"}, {OutCSnow, "C Snow"}, {OutCCrop, "C Crop"}, {OutCPasture, "C Pasture"}, {OutCTourism, "C Tourism"}, {OutPMax, "P Max"}, {OutV1, "V1 Conservative"}, {OutV2, "V2 Balanced"}, {OutV3, "V3 Development"}, {OutCMax, "C Max"}, } var zInputs = []string{SlotZBiodiversity, SlotZLCR, SlotZSoilProtection, SlotZWR} var uInputs = []string{SlotUSurfWater, SlotUSoil, SlotUCrop, SlotUVeg} // order matters: water, soil, crop, veg var pInputs = []string{SlotPIrrigation, SlotPSnow, SlotPCrop, SlotPPasture, SlotPTourism} // sector order 1..5 var cOutputs = []string{OutCIrrigation, OutCSnow, OutCCrop, OutCPasture, OutCTourism} var Nodes = []Node{ {Code: "00", Name: "z_integrated", Stage: 0, Inputs: zInputs, Outputs: []string{OutZIntegrated}, Kernel: kernelZIntegrated}, {Code: "01", Name: "u_integrated", Stage: 0, Inputs: uInputs, Outputs: []string{OutUIntegrated}, Kernel: kernelUIntegrated}, {Code: "02", Name: "z_max", Stage: 0, Inputs: zInputs, Outputs: []string{OutZMax}, Kernel: kernelZMax}, {Code: "03", Name: "u_max", Stage: 0, Inputs: uInputs, Outputs: []string{OutUMax}, Kernel: kernelUMax}, {Code: "10", Name: "e_ecological_constraint", Stage: 1, Inputs: []string{OutZIntegrated, OutUIntegrated}, Outputs: []string{OutE}, Kernel: kernelE}, {Code: "20", Name: "c_conflicts", Stage: 2, Inputs: append(append([]string{}, pInputs...), OutE), Outputs: cOutputs, Kernel: kernelCConflicts}, {Code: "21", Name: "p_max", Stage: 2, Inputs: pInputs, Outputs: []string{OutPMax}, Kernel: kernelMaxAnyValid}, {Code: "30", Name: "func_zoning", Stage: 3, Inputs: append([]string{OutE}, cOutputs...), Outputs: []string{OutV1, OutV2, OutV3}, Kernel: kernelFuncZoning}, {Code: "31", Name: "c_max", Stage: 3, Inputs: cOutputs, Outputs: []string{OutCMax}, Kernel: kernelMaxAnyValid}, } func NodeByCode(code string) (Node, bool) { for _, n := range Nodes { if n.Code == code { return n, true } } return Node{}, false } func SlotByCode(code string) (Slot, bool) { for _, s := range Slots { if s.Code == code { return s, true } } return Slot{}, false } func OutputByCode(code string) (Output, bool) { for _, o := range Outputs { if o.Code == code { return o, true } } return Output{}, false } // NodeForOutput returns the node producing the given output group. func NodeForOutput(group string) (Node, bool) { for _, n := range Nodes { for _, o := range n.Outputs { if o == group { return n, true } } } return Node{}, false } // Stages groups nodes by stage, ascending; within a stage nodes are sorted by code. func Stages() [][]Node { byStage := map[int][]Node{} for _, n := range Nodes { byStage[n.Stage] = append(byStage[n.Stage], n) } keys := make([]int, 0, len(byStage)) for k := range byStage { keys = append(keys, k) } sort.Ints(keys) out := make([][]Node, 0, len(keys)) for _, k := range keys { ns := byStage[k] sort.Slice(ns, func(i, j int) bool { return ns[i].Code < ns[j].Code }) out = append(out, ns) } return out } ``` - [ ] **Step 2: graph_test.go** — assert: every node input is either a slot or an output of a node with a *lower* stage; every output has exactly one producer; `Stages()` returns 4 stages with codes `[00 01 02 03] [10] [20 21] [30 31]`; all group codes are valid slugs (`^[a-z0-9]+(?:-[a-z0-9]+)*$`). Run → PASS once graph.go compiles (kernels may be stubs until Step 4; add empty kernel funcs first so it compiles). - [ ] **Step 3: params.go** ```go package landscape import ( "errors" "fmt" ) // Params are the run-time knobs of the graph (TZ §6): window size, tie // priority for functional zoning, and the V1/V2 suitability thresholds. type Params struct { WindowSize int `json:"window_size"` TiePriority []int `json:"tie_priority"` V1MinSE12 int `json:"v1_min_s_e12"` V1MinSE3 int `json:"v1_min_s_e3"` V2MinSE3 int `json:"v2_min_s_e3"` V2MinSE4 int `json:"v2_min_s_e4"` } // DefaultParams returns the TZ defaults. func DefaultParams() Params { return Params{WindowSize: 2048, TiePriority: []int{1, 2, 3, 4, 5}, V1MinSE12: 3, V1MinSE3: 4, V2MinSE3: 3, V2MinSE4: 5} } // WithDefaults fills zero-valued fields from DefaultParams. func (p Params) WithDefaults() Params { d := DefaultParams() if p.WindowSize == 0 { p.WindowSize = d.WindowSize } if len(p.TiePriority) == 0 { p.TiePriority = d.TiePriority } if p.V1MinSE12 == 0 { p.V1MinSE12 = d.V1MinSE12 } if p.V1MinSE3 == 0 { p.V1MinSE3 = d.V1MinSE3 } if p.V2MinSE3 == 0 { p.V2MinSE3 = d.V2MinSE3 } if p.V2MinSE4 == 0 { p.V2MinSE4 = d.V2MinSE4 } return p } // Validate checks ranges: window 256..8192, thresholds 1..5, tie_priority a // permutation of 1..5. func (p Params) Validate() error { var errs []error if p.WindowSize < 256 || p.WindowSize > 8192 { errs = append(errs, fmt.Errorf("window_size must be in 256..8192")) } for name, v := range map[string]int{"v1_min_s_e12": p.V1MinSE12, "v1_min_s_e3": p.V1MinSE3, "v2_min_s_e3": p.V2MinSE3, "v2_min_s_e4": p.V2MinSE4} { if v < 1 || v > 5 { errs = append(errs, fmt.Errorf("%s must be in 1..5", name)) } } seen := map[int]bool{} for _, k := range p.TiePriority { if k < 1 || k > 5 || seen[k] { errs = append(errs, fmt.Errorf("tie_priority must be a permutation of 1..5")) break } seen[k] = true } if len(p.TiePriority) != 5 && len(seen) == len(p.TiePriority) { errs = append(errs, fmt.Errorf("tie_priority must list all of 1..5")) } return errors.Join(errs...) } // ForNode keeps only the fields that influence the node's result (window size // never does), so fingerprints do not change when irrelevant knobs change. func (p Params) ForNode(code string) Params { if code == "30" { return Params{TiePriority: p.TiePriority, V1MinSE12: p.V1MinSE12, V1MinSE3: p.V1MinSE3, V2MinSE3: p.V2MinSE3, V2MinSE4: p.V2MinSE4} } return Params{} } ``` Tests (`params_test.go`): defaults validate; `Params{}.WithDefaults()` equals `DefaultParams()`; `TiePriority: {1,1,2,3,4}` invalid; `WindowSize: 100` invalid; `ForNode("00")` is zero; `ForNode("30")` has no WindowSize. - [ ] **Step 4: Failing kernel tests** (`kernels_test.go`) — write these before the kernels: ```go package landscape import ( "reflect" "testing" ) func run(k Kernel, nOut int, in ...[]float64) [][]int16 { out := make([][]int16, nOut) for j := range out { out[j] = make([]int16, len(in[0])) } k(in, out, DefaultParams()) return out } func TestClassify(t *testing.T) { cases := map[float32]int16{1.0: 1, 1.8: 1, 1.81: 2, 2.6: 2, 2.75: 3, 3.4: 3, 3.5: 4, 4.2: 4, 4.25: 5, 5.0: 5} for x, want := range cases { if got := classify(x); got != want { t.Errorf("classify(%v)=%d want %d", x, got, want) } } if classify(0.9) != NoData || classify(5.1) != NoData { t.Error("out of range must be NoData") } } func TestKernelZIntegrated(t *testing.T) { // pixel0: all valid (1,2,3,4) -> 2.5 -> class 2 ; pixel1: one nodata -> NoData ; pixel2: all 5 -> 5 -> class 5 got := run(kernelZIntegrated, 1, []float64{1, -9999, 5}, []float64{2, 2, 5}, []float64{3, 3, 5}, []float64{4, 4, 5}) if want := []int16{2, NoData, 5}; !reflect.DeepEqual(got[0], want) { t.Fatalf("got %v want %v", got[0], want) } } func TestKernelUIntegrated_Applicability(t *testing.T) { // order: water, soil, crop, veg // p0: all valid 1,1,5,5 -> (0.25+0.25+1.25+1.25)/1.0 = 3.0 -> class 3 // p1: crop invalid: 1,1,-,5 -> (0.25+0.25+1.25)/0.75 = 2.333 -> class 2 // p2: crop and veg invalid -> NoData // p3: water invalid -> NoData got := run(kernelUIntegrated, 1, []float64{1, 1, 1, -9999}, []float64{1, 1, 1, 1}, []float64{5, 0, 0, 5}, []float64{5, 5, 0, 5}) if want := []int16{3, 2, NoData, NoData}; !reflect.DeepEqual(got[0], want) { t.Fatalf("got %v want %v", got[0], want) } } func TestKernelZMax_RequiresAllValid(t *testing.T) { got := run(kernelZMax, 1, []float64{1, 1}, []float64{4, 4}, []float64{2, -9999}, []float64{3, 3}) if want := []int16{4, NoData}; !reflect.DeepEqual(got[0], want) { t.Fatalf("got %v want %v", got[0], want) } } func TestKernelUMax_IgnoresInvalidCropVeg(t *testing.T) { // water, soil, crop, veg ; p0: 1,2,-,5 -> 5 ; p1: 3,2,0,0 -> NoData (needs crop or veg) ; p2: 3,2,1,0 -> 3 got := run(kernelUMax, 1, []float64{1, 3, 3}, []float64{2, 2, 2}, []float64{0, 0, 1}, []float64{5, 0, 0}) if want := []int16{5, NoData, 3}; !reflect.DeepEqual(got[0], want) { t.Fatalf("got %v want %v", got[0], want) } } func TestKernelE(t *testing.T) { got := run(kernelE, 1, []float64{2, 5, -9999}, []float64{4, 1, 3}) if want := []int16{4, 5, NoData}; !reflect.DeepEqual(got[0], want) { t.Fatalf("got %v want %v", got[0], want) } } func TestKernelCConflicts(t *testing.T) { // 5 P then E; p0: E=3 -> min(P,3); p1: E invalid -> all NoData; p2: P2 invalid e := []float64{3, 0, 4} got := run(kernelCConflicts, 5, []float64{5, 5, 5}, []float64{1, 1, 7}, []float64{3, 3, 3}, []float64{4, 4, 4}, []float64{2, 2, 2}, e) want := [][]int16{{3, NoData, 4}, {1, NoData, NoData}, {3, NoData, 3}, {3, NoData, 4}, {2, NoData, 2}} if !reflect.DeepEqual(got, want) { t.Fatalf("got %v want %v", got, want) } } func TestKernelMaxAnyValid(t *testing.T) { got := run(kernelMaxAnyValid, 1, []float64{1, 0, 0}, []float64{4, 0, 2}, []float64{-9999, 0, 0}) if want := []int16{4, NoData, 2}; !reflect.DeepEqual(got[0], want) { t.Fatalf("got %v want %v", got[0], want) } } // TZ §7 edge cases. Inputs: E, then C1..C5 (irrigation, snow, crop, pasture, tourism). func TestKernelFuncZoning_TZEdgeCases(t *testing.T) { e := []float64{0, 3, 5, 4, 2, 2, 3} s1 := []float64{3, 0, 3, 5, 3, 2, 4} s2 := []float64{3, 0, 3, 5, 3, 2, 4} s3 := []float64{3, 0, 3, 2, 3, 2, 5} s4 := []float64{3, 0, 3, 2, 3, 2, 1} s5 := []float64{3, 0, 3, 2, 3, 2, 1} got := run(kernelFuncZoning, 3, e, s1, s2, s3, s4, s5) // pixel: 0 E invalid -> NoData ; 1 all S invalid -> NoData ; 2 E=5 -> regulated(1) everywhere ; // 3 E=4, S=5 for sectors 1,2 -> V1 regulated, V2/V3 sector 1 -> class 2 ; // 4 all S equal (3), E=2 -> irrigation (2) in all ; 5 E=2, S=2 -> V1 regulated (S<3), V2 best=irrigation (2), V3 irrigation (2) // 6 E=3: S=[4,4,5,1,1] -> V1 (S>=4): winners 1,2,3 max 5 -> crop (4); V2 (S>=3) -> crop (4); V3 -> crop (4) want := [][]int16{ {NoData, NoData, 1, 1, 2, 1, 4}, // V1 {NoData, NoData, 1, 2, 2, 2, 4}, // V2 {NoData, NoData, 1, 2, 2, 2, 4}, // V3 } if !reflect.DeepEqual(got, want) { t.Fatalf("got %v want %v", got, want) } } func TestKernelFuncZoning_TiePriorityIsConfigurable(t *testing.T) { in := [][]float64{{2}, {3}, {3}, {3}, {3}, {3}} out := [][]int16{{0}, {0}, {0}} p := DefaultParams() p.TiePriority = []int{5, 4, 3, 2, 1} kernelFuncZoning(in, out, p) if out[0][0] != 6 || out[1][0] != 6 || out[2][0] != 6 { t.Fatalf("expected tourism (6) everywhere, got %v", out) } } ``` Run: `go test ./internal/landscape/ -run Kernel -v` → FAIL (undefined). - [ ] **Step 5: kernels.go** ```go package landscape // valid reports whether a class value is in the analysis domain (1..5). Any // other value, including nodata and NaN, is invalid. func valid(v float64) bool { return v >= 1 && v <= 5 } // classify maps a float32 score onto classes 1..5 with the fixed intervals // used by the reference notebook (comparisons in float32, like numpy on a // float32 array). Scores outside [1, 5] yield NoData. func classify(x float32) int16 { switch { case x < 1.0: return NoData case x <= 1.8: return 1 case x <= 2.6: return 2 case x <= 3.4: return 3 case x <= 4.2: return 4 case x <= 5.0: return 5 default: return NoData } } // kernelZIntegrated: all four Z valid -> classify(0.25*Z1 + 0.25*Z2 + 0.25*Z3 + 0.25*Z4). func kernelZIntegrated(in [][]float64, out [][]int16, _ Params) { for i := range out[0] { z1, z2, z3, z4 := in[0][i], in[1][i], in[2][i], in[3][i] if !(valid(z1) && valid(z2) && valid(z3) && valid(z4)) { out[0][i] = NoData continue } score := float32(0.25)*float32(z1) + float32(0.25)*float32(z2) + float32(0.25)*float32(z3) + float32(0.25)*float32(z4) out[0][i] = classify(score) } } // kernelUIntegrated: water and soil required, at least one of crop/veg; // U = (0.25*water + 0.25*soil + [0.25*crop] + [0.25*veg]) / (0.5 + [0.25] + [0.25]). func kernelUIntegrated(in [][]float64, out [][]int16, _ Params) { for i := range out[0] { water, soil, crop, veg := in[0][i], in[1][i], in[2][i], in[3][i] cropOK, vegOK := valid(crop), valid(veg) if !(valid(water) && valid(soil) && (cropOK || vegOK)) { out[0][i] = NoData continue } num := float32(0.25)*float32(water) + float32(0.25)*float32(soil) den := float32(0.5) if cropOK { num += float32(0.25) * float32(crop) den += 0.25 } if vegOK { num += float32(0.25) * float32(veg) den += 0.25 } out[0][i] = classify(num / den) } } // kernelZMax: all four valid -> max. func kernelZMax(in [][]float64, out [][]int16, _ Params) { for i := range out[0] { m, ok := int16(0), true for _, layer := range in { v := layer[i] if !valid(v) { ok = false break } if int16(v) > m { m = int16(v) } } if !ok { m = NoData } out[0][i] = m } } // kernelUMax: water and soil required plus one of crop/veg; max over valid layers. func kernelUMax(in [][]float64, out [][]int16, _ Params) { for i := range out[0] { water, soil, crop, veg := in[0][i], in[1][i], in[2][i], in[3][i] cropOK, vegOK := valid(crop), valid(veg) if !(valid(water) && valid(soil) && (cropOK || vegOK)) { out[0][i] = NoData continue } m := max(int16(water), int16(soil)) if cropOK { m = max(m, int16(crop)) } if vegOK { m = max(m, int16(veg)) } out[0][i] = m } } // kernelE: Z and U valid -> max(Z, U). func kernelE(in [][]float64, out [][]int16, _ Params) { for i := range out[0] { z, u := in[0][i], in[1][i] if !(valid(z) && valid(u)) { out[0][i] = NoData continue } out[0][i] = max(int16(z), int16(u)) } } // kernelCConflicts: inputs P1..P5, E; output j = min(Pj, E) where both valid. func kernelCConflicts(in [][]float64, out [][]int16, _ Params) { e := in[5] for j := 0; j < 5; j++ { p := in[j] for i := range out[j] { if !(valid(p[i]) && valid(e[i])) { out[j][i] = NoData continue } out[j][i] = min(int16(p[i]), int16(e[i])) } } } // kernelMaxAnyValid: max over the valid inputs; NoData when none is valid. // Used for P_max and C_max. func kernelMaxAnyValid(in [][]float64, out [][]int16, _ Params) { for i := range out[0] { m, any := int16(0), false for _, layer := range in { if v := layer[i]; valid(v) { any = true m = max(m, int16(v)) } } if !any { m = NoData } out[0][i] = m } } // kernelFuncZoning implements TZ_functional_zoning_go.md §4. Inputs: E, then // S1..S5 (irrigation, snow, crop, pasture, tourism). Outputs: V1, V2, V3. func kernelFuncZoning(in [][]float64, out [][]int16, p Params) { e := in[0] rules := [3]func(e, s int16) bool{ func(e, s int16) bool { return (e <= 2 && s >= int16(p.V1MinSE12)) || (e == 3 && s >= int16(p.V1MinSE3)) }, func(e, s int16) bool { return e <= 2 || (e == 3 && s >= int16(p.V2MinSE3)) || (e == 4 && s >= int16(p.V2MinSE4)) }, func(e, s int16) bool { return e <= 4 }, } var s [5]int16 var base [5]bool for i := range out[0] { if !valid(e[i]) { out[0][i], out[1][i], out[2][i] = NoData, NoData, NoData continue } ev := int16(e[i]) anyBase := false for k := 0; k < 5; k++ { base[k] = valid(in[k+1][i]) if base[k] { s[k] = int16(in[k+1][i]) anyBase = true } } if !anyBase { out[0][i], out[1][i], out[2][i] = NoData, NoData, NoData continue } for v := 0; v < 3; v++ { var cand [5]bool maxS, anyCand := int16(0), false for k := 0; k < 5; k++ { cand[k] = base[k] && rules[v](ev, s[k]) if cand[k] { anyCand = true if s[k] > maxS { maxS = s[k] } } } if !anyCand { out[v][i] = 1 // regulated continue } sector := int16(0) for _, code := range p.TiePriority { k := code - 1 if cand[k] && s[k] == maxS { sector = int16(code) break } } out[v][i] = sector + 1 } } } ``` - [ ] **Step 6: Run** `go test ./internal/landscape/ -v` → PASS. `go vet ./internal/landscape/` clean. - [ ] **Step 7: Commit** ```bash git add internal/landscape/graph.go internal/landscape/graph_test.go internal/landscape/params.go internal/landscape/params_test.go internal/landscape/kernels.go internal/landscape/kernels_test.go git commit -m "feat(landscape): computation graph, params and per-pixel kernels" ``` --- ### Task 5: Landscape domain, migration 00011 (seed + run tables), repository **Files:** - Create: `internal/domain/landscape.go`, `migrations/00011_landscape.sql`, `internal/repository/postgres/landscape.go` **Interfaces (produced, used by Task 7):** ```go package domain type LPRunStatus string // "queued" "running" "succeeded" "failed" type LPNodeStatus string // "pending" "running" "skipped" "succeeded" "failed" type LPDatasetRef struct { Group string `json:"group"`; DatasetID uuid.UUID `json:"dataset_id"` } type LPRun struct { ID uuid.UUID; Status LPRunStatus; FromStage *int; Params json.RawMessage; Overrides map[string]uuid.UUID; Error *string; CreatedAt time.Time; StartedAt, FinishedAt *time.Time; Nodes []LPRunNode } type LPRunNode struct { ID, RunID uuid.UUID; NodeCode string; Stage int; Status LPNodeStatus; Fingerprint string; Inputs []LPDatasetRef; Outputs []LPDatasetRef; Error *string; StartedAt, FinishedAt *time.Time } package postgres type LandscapeRepository — methods: CreateRun(ctx, run domain.LPRun) (domain.LPRun, error) // inserts run + its Nodes (status pending) in one tx; returns with ids GetRun(ctx, id uuid.UUID) (domain.LPRun, error) // with nodes ordered by node_code ListRuns(ctx, limit, offset int) ([]domain.LPRun, error) // without nodes, newest first CountRuns(ctx) (int, error) SetRunStatus(ctx, id uuid.UUID, status domain.LPRunStatus, errMsg *string) error // running -> started_at=now(); succeeded/failed -> finished_at=now() SetNodeStarted(ctx, runID uuid.UUID, code string, fingerprint string, inputs []domain.LPDatasetRef) error // status running, started_at SetNodeFinished(ctx, runID uuid.UUID, code string, status domain.LPNodeStatus, outputs []domain.LPDatasetRef, errMsg *string) error // finished_at FindReusableOutputs(ctx, code, fingerprint string, excludeRun uuid.UUID) ([]domain.LPDatasetRef, error) // latest succeeded node with same code+fingerprint; ErrNotFound if none ``` - [ ] **Step 1: Domain** (`internal/domain/landscape.go`) — write the types above with JSON tags: `id, status, from_stage, params, overrides, error, created_at, started_at, finished_at, nodes` and `id, run_id, node_code, stage, status, fingerprint, inputs, outputs, error, started_at, finished_at`. Add constants `LPRunQueued/Running/Succeeded/Failed` and `LPNodePending/Running/Skipped/Succeeded/Failed`. - [ ] **Step 2: Migration** (`migrations/00011_landscape.sql`) ```sql -- +goose Up -- Landscape Program: seed the category and dataset groups (input slots and -- node outputs; codes must match internal/landscape/graph.go), and store runs. INSERT INTO categories (code, name, description) VALUES ('landscape', 'Ландшафтная программа', 'Входные и результирующие растры Ландшафтной программы') ON CONFLICT (code) DO NOTHING; INSERT INTO dataset_groups (code, name) VALUES ('lp-z-biodiversity', 'Z Biodiversity'), ('lp-z-lcr', 'Z LCR'), ('lp-z-soil-protection', 'Z Soil Protection'), ('lp-z-wr', 'Z Water Resources'), ('lp-u-surf-water', 'U Surface Water'), ('lp-u-soil', 'U Soil'), ('lp-u-crop', 'U Crop'), ('lp-u-veg', 'U Vegetation'), ('lp-p-irrigation', 'P Irrigation'), ('lp-p-snow', 'P Snow'), ('lp-p-crop', 'P Crop'), ('lp-p-pasture', 'P Pasture'), ('lp-p-tourism', 'P Tourism'), ('lp-z-integrated', 'Z Integrated'), ('lp-u-integrated', 'U Integrated'), ('lp-z-max', 'Z Max'), ('lp-u-max', 'U Max'), ('lp-e', 'E Ecological Constraint'), ('lp-c-irrigation', 'C Irrigation'), ('lp-c-snow', 'C Snow'), ('lp-c-crop', 'C Crop'), ('lp-c-pasture', 'C Pasture'), ('lp-c-tourism', 'C Tourism'), ('lp-p-max', 'P Max'), ('lp-v1-conservative', 'V1 Conservative'), ('lp-v2-balanced', 'V2 Balanced'), ('lp-v3-development', 'V3 Development'), ('lp-c-max', 'C Max') ON CONFLICT (code) DO NOTHING; CREATE TABLE lp_runs ( id UUID PRIMARY KEY DEFAULT gen_random_uuid(), status VARCHAR(32) NOT NULL, from_stage SMALLINT, params JSONB NOT NULL, overrides JSONB NOT NULL DEFAULT '{}'::jsonb, error TEXT, created_at TIMESTAMPTZ NOT NULL DEFAULT now(), started_at TIMESTAMPTZ, finished_at TIMESTAMPTZ ); CREATE INDEX idx_lp_runs_created_at ON lp_runs (created_at DESC); CREATE TABLE lp_run_nodes ( id UUID PRIMARY KEY DEFAULT gen_random_uuid(), run_id UUID NOT NULL REFERENCES lp_runs (id) ON DELETE CASCADE, node_code VARCHAR(8) NOT NULL, stage SMALLINT NOT NULL, status VARCHAR(32) NOT NULL, fingerprint TEXT NOT NULL DEFAULT '', input_datasets JSONB NOT NULL DEFAULT '[]'::jsonb, output_datasets JSONB, error TEXT, started_at TIMESTAMPTZ, finished_at TIMESTAMPTZ, UNIQUE (run_id, node_code) ); CREATE INDEX idx_lp_run_nodes_lookup ON lp_run_nodes (node_code, fingerprint, status, finished_at DESC); -- +goose Down DROP TABLE lp_run_nodes; DROP TABLE lp_runs; -- Seeded category/groups are left in place (datasets may reference them). ``` - [ ] **Step 3: Repository** (`internal/repository/postgres/landscape.go`) — implement the methods listed in Interfaces. Key SQL: ```sql -- CreateRun (tx): INSERT INTO lp_runs (status, from_stage, params, overrides) VALUES ($1,$2,$3,$4) RETURNING INSERT INTO lp_run_nodes (run_id, node_code, stage, status) VALUES ($1,$2,$3,'pending') RETURNING -- per node -- GetRun: SELECT FROM lp_runs WHERE id=$1 ; SELECT FROM lp_run_nodes WHERE run_id=$1 ORDER BY node_code -- ListRuns: ... ORDER BY created_at DESC LIMIT $1 OFFSET $2 -- SetRunStatus: UPDATE lp_runs SET status=$2, error=$3, started_at = CASE WHEN $2 = 'running' THEN now() ELSE started_at END, finished_at = CASE WHEN $2 IN ('succeeded','failed') THEN now() ELSE finished_at END WHERE id=$1 -- SetNodeStarted: UPDATE lp_run_nodes SET status='running', fingerprint=$3, input_datasets=$4, started_at=now(), error=NULL WHERE run_id=$1 AND node_code=$2 -- SetNodeFinished: UPDATE lp_run_nodes SET status=$3, output_datasets=$4, error=$5, finished_at=now() WHERE run_id=$1 AND node_code=$2 -- FindReusableOutputs: SELECT output_datasets FROM lp_run_nodes WHERE node_code=$1 AND fingerprint=$2 AND status='succeeded' AND run_id <> $3 AND output_datasets IS NOT NULL ORDER BY finished_at DESC LIMIT 1 ``` `runColumns = id, status, from_stage, params, overrides, error, created_at, started_at, finished_at`; scan `overrides` into `map[string]uuid.UUID` via a `[]byte` + `json.Unmarshal` (empty → empty map). `nodeColumns = id, run_id, node_code, stage, status, fingerprint, input_datasets, output_datasets, error, started_at, finished_at`; JSON arrays scanned via pgx into `[]domain.LPDatasetRef` (pgx handles `jsonb` → struct slice; `output_datasets` NULL → nil). Every `Exec` with `RowsAffected()==0` returns `domain.ErrNotFound`; wrap errors with `mapError`. - [ ] **Step 4: Compile** — `go build ./internal/domain/ ./internal/repository/postgres/`; `go vet` both. (No DB in CI; SQL is exercised by the integration task.) - [ ] **Step 5: Commit** ```bash git add internal/domain/landscape.go migrations/00011_landscape.sql internal/repository/postgres/landscape.go git commit -m "feat(landscape): run/node domain, seed migration, repository" ``` --- ### Task 6: `Compute` — window executor over `raster.IO` **Files:** - Create: `internal/landscape/compute.go`, `internal/landscape/compute_test.go`, `internal/landscape/fakeio_test.go` **Interfaces:** - Consumes: Task 3 `raster.IO/Reader/Writer/Grid/Window/Windows/DefaultWriterOptions`; Task 4 `Node`, `Params`. - Produces (used by Task 7 via `service.ComputeFunc`): ```go // Compute runs one node. inputs maps every Node.Inputs group code to a local // file; outputs are written to filepath.Join(outDir, group+".tif") and returned // keyed by group code. Grids of all inputs must be identical (raster.IO.SameGrid) // or an error is returned before any output is created. func Compute(ctx context.Context, rio raster.IO, node Node, inputs map[string]string, outDir string, p Params) (map[string]string, error) ``` - [ ] **Step 1: Fake IO for tests** (`fakeio_test.go`) ```go package landscape import ( "context" "fmt" "sync" "gis/internal/raster" ) // fakeIO keeps rasters in memory keyed by path. type fakeIO struct { mu sync.Mutex files map[string]*fakeRaster } type fakeRaster struct { grid raster.Grid data []float64 // row-major } func newFakeIO() *fakeIO { return &fakeIO{files: map[string]*fakeRaster{}} } func (f *fakeIO) put(path string, g raster.Grid, data []float64) { f.files[path] = &fakeRaster{grid: g, data: data} } type fakeReader struct{ r *fakeRaster } func (r *fakeReader) Grid() raster.Grid { return r.r.grid } func (r *fakeReader) NoData() (float64, bool) { return -9999, true } func (r *fakeReader) Close() error { return nil } func (r *fakeReader) ReadFloat64(w raster.Window, dst []float64) error { for y := 0; y < w.Height; y++ { copy(dst[y*w.Width:(y+1)*w.Width], r.r.data[(w.Y+y)*r.r.grid.Width+w.X:(w.Y+y)*r.r.grid.Width+w.X+w.Width]) } return nil } type fakeWriter struct { f *fakeIO r *fakeRaster } func (w *fakeWriter) WriteInt16(win raster.Window, src []int16) error { w.f.mu.Lock() defer w.f.mu.Unlock() for y := 0; y < win.Height; y++ { for x := 0; x < win.Width; x++ { w.r.data[(win.Y+y)*w.r.grid.Width+win.X+x] = float64(src[y*win.Width+x]) } } return nil } func (w *fakeWriter) Close() error { return nil } func (f *fakeIO) Open(_ context.Context, path string) (raster.Reader, error) { f.mu.Lock() defer f.mu.Unlock() r, ok := f.files[path] if !ok { return nil, fmt.Errorf("fake: %s not found", path) } return &fakeReader{r: r}, nil } func (f *fakeIO) Create(_ context.Context, path string, g raster.Grid, _ raster.WriterOptions) (raster.Writer, error) { f.mu.Lock() defer f.mu.Unlock() r := &fakeRaster{grid: g, data: make([]float64, g.Width*g.Height)} f.files[path] = r return &fakeWriter{f: f, r: r}, nil } func (f *fakeIO) SameGrid(a, b raster.Grid) bool { return a == b } ``` - [ ] **Step 2: Failing tests** (`compute_test.go`) ```go package landscape import ( "context" "path/filepath" "testing" "gis/internal/raster" ) func grid(w, h int) raster.Grid { return raster.Grid{Width: w, Height: h, GeoTransform: [6]float64{0, 30, 0, 0, 0, -30}, ProjectionWKT: "EPSG:32642"} } func constRaster(w, h int, v float64) []float64 { d := make([]float64, w*h) for i := range d { d[i] = v } return d } func TestCompute_E_TilesWindowsAndWritesEveryPixel(t *testing.T) { fio := newFakeIO() g := grid(10, 7) z := constRaster(10, 7, 2) u := constRaster(10, 7, 4) z[0] = -9999 // pixel 0 invalid fio.put("z.tif", g, z) fio.put("u.tif", g, u) node, _ := NodeByCode("10") p := DefaultParams() p.WindowSize = 4 // forces partial windows (10x7 with 4x4 tiles) out, err := Compute(context.Background(), fio, node, map[string]string{OutZIntegrated: "z.tif", OutUIntegrated: "u.tif"}, "/out", p) if err != nil { t.Fatal(err) } path := out[OutE] if path != filepath.Join("/out", OutE+".tif") { t.Fatalf("unexpected output path %q", path) } res := fio.files[path] if res.grid != g { t.Fatal("output grid must copy the first input's grid") } for i, v := range res.data { want := 4.0 if i == 0 { want = -9999 } if v != want { t.Fatalf("pixel %d = %v want %v", i, v, want) } } } func TestCompute_FailsOnGridMismatch(t *testing.T) { fio := newFakeIO() fio.put("z.tif", grid(10, 7), constRaster(10, 7, 2)) fio.put("u.tif", grid(9, 7), constRaster(9, 7, 4)) node, _ := NodeByCode("10") _, err := Compute(context.Background(), fio, node, map[string]string{OutZIntegrated: "z.tif", OutUIntegrated: "u.tif"}, "/out", DefaultParams()) if err == nil { t.Fatal("expected grid mismatch error") } if _, created := fio.files[filepath.Join("/out", OutE+".tif")]; created { t.Fatal("no output must be created on mismatch") } } func TestCompute_FailsOnMissingInput(t *testing.T) { fio := newFakeIO() fio.put("z.tif", grid(2, 2), constRaster(2, 2, 2)) node, _ := NodeByCode("10") if _, err := Compute(context.Background(), fio, node, map[string]string{OutZIntegrated: "z.tif"}, "/out", DefaultParams()); err == nil { t.Fatal("expected error for missing input group") } } func TestCompute_MultiOutputNode(t *testing.T) { fio := newFakeIO() g := grid(3, 1) paths := map[string]string{} for i, s := range []string{SlotPIrrigation, SlotPSnow, SlotPCrop, SlotPPasture, SlotPTourism} { paths[s] = s + ".tif" fio.put(paths[s], g, constRaster(3, 1, float64(i+1))) } paths[OutE] = "e.tif" fio.put("e.tif", g, constRaster(3, 1, 3)) node, _ := NodeByCode("20") out, err := Compute(context.Background(), fio, node, paths, "/o", DefaultParams()) if err != nil { t.Fatal(err) } if len(out) != 5 { t.Fatalf("want 5 outputs, got %d", len(out)) } if fio.files[out[OutCTourism]].data[0] != 3 || fio.files[out[OutCIrrigation]].data[0] != 1 { t.Fatal("min(P, E) not applied per output") } } ``` Run: `go test ./internal/landscape/ -run Compute -v` → FAIL. - [ ] **Step 3: compute.go** ```go package landscape import ( "context" "fmt" "path/filepath" "runtime" "sync" "gis/internal/raster" "golang.org/x/sync/errgroup" ) // Compute runs one node over its input rasters. See the package doc for the // concurrency model: windows are processed by a worker pool; each worker owns // its own Reader handles (GDAL datasets are not goroutine-safe) and writes are // serialised per output. Window order does not affect the result. func Compute(ctx context.Context, rio raster.IO, node Node, inputs map[string]string, outDir string, p Params) (map[string]string, error) { p = p.WithDefaults() paths := make([]string, len(node.Inputs)) for i, g := range node.Inputs { path, ok := inputs[g] if !ok || path == "" { return nil, fmt.Errorf("node %s (%s): missing input %q", node.Code, node.Name, g) } paths[i] = path } // Validate grids with one set of handles before creating anything. grid, err := checkGrids(ctx, rio, node, paths) if err != nil { return nil, err } outPaths := make(map[string]string, len(node.Outputs)) writers := make([]raster.Writer, len(node.Outputs)) for j, g := range node.Outputs { path := filepath.Join(outDir, g+".tif") w, err := rio.Create(ctx, path, grid, raster.DefaultWriterOptions()) if err != nil { closeWriters(writers[:j]) return nil, fmt.Errorf("node %s: create output %s: %w", node.Code, g, err) } writers[j] = w outPaths[g] = path } windows := raster.Windows(grid.Width, grid.Height, p.WindowSize) jobs := make(chan raster.Window) var writeMu sync.Mutex eg, gctx := errgroup.WithContext(ctx) workers := min(runtime.GOMAXPROCS(0), len(windows)) for w := 0; w < workers; w++ { eg.Go(func() error { readers, err := openAll(gctx, rio, paths) if err != nil { return err } defer closeReaders(readers) size := p.WindowSize * p.WindowSize in := make([][]float64, len(paths)) for i := range in { in[i] = make([]float64, size) } out := make([][]int16, len(writers)) for j := range out { out[j] = make([]int16, size) } for win := range jobs { n := win.Width * win.Height inView := make([][]float64, len(in)) for i, r := range readers { if err := r.ReadFloat64(win, in[i][:n]); err != nil { return fmt.Errorf("node %s: read %s window %+v: %w", node.Code, node.Inputs[i], win, err) } inView[i] = in[i][:n] } outView := make([][]int16, len(out)) for j := range out { outView[j] = out[j][:n] } node.Kernel(inView, outView, p) writeMu.Lock() for j, wr := range writers { if err := wr.WriteInt16(win, outView[j]); err != nil { writeMu.Unlock() return fmt.Errorf("node %s: write %s window %+v: %w", node.Code, node.Outputs[j], win, err) } } writeMu.Unlock() } return nil }) } eg.Go(func() error { defer close(jobs) for _, win := range windows { select { case jobs <- win: case <-gctx.Done(): return gctx.Err() } } return nil }) runErr := eg.Wait() if err := closeWriters(writers); err != nil && runErr == nil { runErr = err } if runErr != nil { return nil, runErr } return outPaths, nil } // checkGrids opens every input once, verifies all grids match the first, and // returns that grid. func checkGrids(ctx context.Context, rio raster.IO, node Node, paths []string) (raster.Grid, error) { readers, err := openAll(ctx, rio, paths) if err != nil { return raster.Grid{}, err } defer closeReaders(readers) ref := readers[0].Grid() for i := 1; i < len(readers); i++ { if g := readers[i].Grid(); !rio.SameGrid(ref, g) { return raster.Grid{}, fmt.Errorf("node %s: input %s grid (%dx%d, %v) differs from %s (%dx%d, %v); inputs must be aligned", node.Code, node.Inputs[i], g.Width, g.Height, g.GeoTransform, node.Inputs[0], ref.Width, ref.Height, ref.GeoTransform) } } return ref, nil } func openAll(ctx context.Context, rio raster.IO, paths []string) ([]raster.Reader, error) { readers := make([]raster.Reader, 0, len(paths)) for _, p := range paths { r, err := rio.Open(ctx, p) if err != nil { closeReaders(readers) return nil, err } readers = append(readers, r) } return readers, nil } func closeReaders(rs []raster.Reader) { for _, r := range rs { if r != nil { _ = r.Close() } } } func closeWriters(ws []raster.Writer) error { var first error for _, w := range ws { if w == nil { continue } if err := w.Close(); err != nil && first == nil { first = err } } return first } ``` - [ ] **Step 4: Run** `go test ./internal/landscape/ -race -v` → PASS (race detector must be clean). - [ ] **Step 5: Commit** ```bash git add internal/landscape/compute.go internal/landscape/compute_test.go internal/landscape/fakeio_test.go git commit -m "feat(landscape): window executor Compute with worker pool and grid checks" ``` --- ### Task 7: LandscapeService (runs, planning, input resolution, execution) + RabbitMQ job **Files:** - Create: `internal/service/landscape.go`, `internal/service/landscape_test.go`, `internal/messaging/rabbitmq/landscape_consumer.go` - Modify: none outside these (wiring is Task 8) **Interfaces:** - Consumes: Task 2 (`DatasetRepository.LatestInGroup`, `DatasetGroupRepository.GetByCode`, `DatasetService.Upload/CreateRasterFromFile`, `ObjectStore`), Task 4 (`landscape.Nodes/Stages/Slots/Outputs/Params/…`), Task 5 (`domain.LPRun*`, repo methods), `raster.IO` (Task 3). - Produces (used by Task 8): ```go package service type ComputeFunc func(ctx context.Context, rio raster.IO, node landscape.Node, inputs map[string]string, outDir string, p landscape.Params) (map[string]string, error) type LandscapeRepository interface { /* the 8 methods of postgres.LandscapeRepository, same signatures */ } type LandscapeJobEnqueuer interface { EnqueueRun(ctx context.Context, runID uuid.UUID) error } type datasetPublisher interface { Upload(ctx context.Context, in UploadInput) (domain.Dataset, error) CreateRasterFromFile(ctx context.Context, in CreateRasterInput) (domain.Dataset, error) } func NewLandscapeService(runs LandscapeRepository, datasets DatasetRepository, groups DatasetGroupRepository, categories categoryReader, publisher datasetPublisher, store ObjectStore, jobs LandscapeJobEnqueuer, rio raster.IO, compute ComputeFunc, log *slog.Logger) *LandscapeService type SlotInfo struct { Code string `json:"code"`; Title string `json:"title"`; GroupID uuid.UUID `json:"group_id"`; Latest *domain.DatasetSummary `json:"latest"` } type ResultInfo struct { Group string `json:"group"`; Title string `json:"title"`; Node string `json:"node"`; GroupID uuid.UUID `json:"group_id"`; Latest *domain.DatasetSummary `json:"latest"`; CogURL *string `json:"cog_url"`; SldURL *string `json:"sld_url"` } type SlotUploadInput struct { Filename, ContentType string; Size int64; Reader io.Reader; Name string; StartDate, EndDate *string } type RunInput struct { FromStage *int; Overrides map[string]uuid.UUID; Params landscape.Params } type LPRunPage struct { Items []domain.LPRun; Page, PageSize, Total int } func (s *LandscapeService) Slots(ctx) ([]SlotInfo, error) func (s *LandscapeService) UploadToSlot(ctx, code string, in SlotUploadInput) (domain.Dataset, error) // ErrNotFound for unknown slot func (s *LandscapeService) CreateRun(ctx, in RunInput) (domain.LPRun, error) func (s *LandscapeService) GetRun(ctx, id uuid.UUID) (domain.LPRun, error) func (s *LandscapeService) ListRuns(ctx, page, pageSize int) (LPRunPage, error) func (s *LandscapeService) Results(ctx) ([]ResultInfo, error) func (s *LandscapeService) Execute(ctx, runID uuid.UUID) error // worker entry point func (s *LandscapeService) MarkRunFailed(ctx, runID uuid.UUID, reason string) error package rabbitmq const LandscapeRunRoutingKey = "landscape.run"; const LandscapeRunQueue = "gis.landscape.run" type LandscapeJob struct { RunID uuid.UUID `json:"run_id"` } type LandscapeJobPublisher struct{ pub *Publisher }; func NewLandscapeJobPublisher(pub *Publisher) *LandscapeJobPublisher; func (p *LandscapeJobPublisher) EnqueueRun(ctx, runID uuid.UUID) error type LandscapeProcessor interface { Execute(ctx, runID uuid.UUID) error; MarkRunFailed(ctx, runID uuid.UUID, reason string) error } func NewLandscapeRunHandler(p LandscapeProcessor, log *slog.Logger) Handler ``` - [ ] **Step 1: Failing service tests** (`internal/service/landscape_test.go`). Build stubs: `stubLPRepo` (in-memory runs/nodes, records `FindReusableOutputs` hits), reuse `stubDatasetRepo`, `stubGroupRepo` (seed all `landscape.Slots`/`Outputs` codes as groups), `stubCategoryRepo` (from `category_test.go`, seed code `landscape`), a `stubPublisher` implementing `datasetPublisher` that records `CreateRasterFromFile` calls and inserts a ready raster dataset with the given group into `stubDatasetRepo`, `stubStore` with `Get` returning bytes, `stubLPJobs`, `fakeIO`-free approach: `compute` is a `ComputeFunc` stub that writes an empty file per output and records the `inputs` map it received. Tests: ```go func TestLandscapeService_CreateRun_ValidatesParamsAndOverrides(t *testing.T) // invalid tie priority -> ErrValidation; override for unknown group -> ErrValidation; // override dataset not ready/not raster -> ErrValidation; valid -> run queued with 9 pending nodes and job enqueued. func TestLandscapeService_Execute_ResolvesInputsAndPublishesOutputs(t *testing.T) // seed ready raster datasets for all 13 slots; Execute; expect: run succeeded, 9 nodes succeeded, // compute stub called 9 times, node "10" received inputs produced by nodes "00" and "01" in this run // (dataset ids equal to the publisher's created ids), each published dataset has Automated=true, // DatasetGroupID = its output group, Meta contains "lp_run_id". func TestLandscapeService_Execute_ReusesFingerprintMatches(t *testing.T) // run once (all computed); run again with unchanged inputs -> every node skipped, compute stub not called, // outputs equal to first run's outputs. func TestLandscapeService_Execute_RecomputesDownstreamOfChangedInput(t *testing.T) // after first run, upload a newer dataset to lp-p-crop (later end_date); second run: nodes 00,01,02,03,10 skipped; // 20,21 computed (inputs changed); 30,31 computed (their inputs are new C datasets). func TestLandscapeService_Execute_FromStageForcesRecompute(t *testing.T) // second run with FromStage=2 and unchanged inputs -> stages 0,1 skipped, stages 2,3 computed. func TestLandscapeService_Execute_MissingSlotFailsRun(t *testing.T) // no dataset in lp-z-wr -> node 00 and 02 failed with message containing "lp-z-wr"; run failed; stage 1 never started. func TestLandscapeService_UploadToSlot_UnknownSlot(t *testing.T) // ErrNotFound func TestLandscapeService_Results_ListsLatestPerOutput(t *testing.T) // cog_url = "/datasets/.cog"; sld_url only when group has SLD ``` Write the test bodies fully (arrange/act/assert) following `dataset_test.go` style. Run → FAIL (undefined). - [ ] **Step 2: Service implementation** (`internal/service/landscape.go`) — implement per spec §3.5–3.7. Core pieces: ```go // fingerprint is a stable hash of the node, its resolved inputs and the // node-relevant params. func fingerprint(node landscape.Node, inputs []domain.LPDatasetRef, p landscape.Params) string { sorted := append([]domain.LPDatasetRef(nil), inputs...) sort.Slice(sorted, func(i, j int) bool { return sorted[i].Group < sorted[j].Group }) payload, _ := json.Marshal(struct { Node string `json:"node"` Inputs []domain.LPDatasetRef `json:"inputs"` Params landscape.Params `json:"params"` }{node.Code, sorted, p.ForNode(node.Code)}) sum := sha256.Sum256(payload) return hex.EncodeToString(sum[:]) } ``` `Execute(ctx, runID)`: 1. `run := runs.GetRun`; parse `run.Params` into `landscape.Params` (`WithDefaults`); `runs.SetRunStatus(running)`. 2. `produced := map[string]uuid.UUID{}` guarded by a mutex (outputs created in this run). 3. For each stage from `landscape.Stages()`: `errgroup` over nodes → `s.runNode(ctx, run, node, params, produced)`; on error: `SetRunStatus(failed, err)` and `return err`. 4. `SetRunStatus(succeeded, nil)`. `runNode`: 1. Resolve inputs: for each group `g` in `node.Inputs`: `run.Overrides[g]` → else `produced[g]` → else `datasets.LatestInGroup(ctx, groupID(g), &ready, &raster)`; `groupID` via `groups.GetByCode`. Missing → `fmt.Errorf("node %s: no ready raster dataset in group %q", …)`. 2. `fp := fingerprint(node, refs, params)`; `runs.SetNodeStarted(run.ID, node.Code, fp, refs)`. 3. If `run.FromStage == nil || node.Stage < *run.FromStage`: `outs, err := runs.FindReusableOutputs(ctx, node.Code, fp, run.ID)`; if found and every `outs[i].DatasetID` exists with `Status == ready` (`datasets.GetByID`) → record into `produced`, `SetNodeFinished(skipped, outs, nil)`, return. 4. Temp dir `os.MkdirTemp("", "lp-"+node.Code+"-*")` (deferred `RemoveAll`); download each input object (`store.Get(dataset.StorageKey)` → file `/in-.tif`). 5. `outPaths, err := s.compute(ctx, s.rio, node, inputPaths, tmp, params)`. 6. For each output group: `publisher.CreateRasterFromFile(ctx, CreateRasterInput{CategoryID: landscapeCategory.ID, DatasetGroupID: &gid, Name: title + " " + run.StartedAt(or now).Format("2006-01-02"), Filename: group + ".tif", Path: outPaths[group], StartDate: minStart(inputs), EndDate: maxEnd(inputs), Meta: {"lp_run_id", "lp_node", "inputs"}, Automated: true})`; collect refs; `produced[group] = ds.ID`. 7. `SetNodeFinished(succeeded, outRefs, nil)`. On any error: `SetNodeFinished(failed, nil, &msg)` and return the error. `CreateRun`: `params := in.Params.WithDefaults(); params.Validate()` → wrap into `ErrValidation`; `FromStage` in 0..3; each override: group code must be a known group (`landscape.SlotByCode` or `OutputByCode`) and the dataset must exist, be `raster` and `ready` (else `ErrValidation`); marshal params; build `domain.LPRun{Status: queued, FromStage, Params, Overrides, Nodes: one pending node per landscape.Nodes}`; `runs.CreateRun`; `jobs.EnqueueRun(run.ID)` (on failure `SetRunStatus(failed, "failed to enqueue: …")` and return error). `UploadToSlot`: `landscape.SlotByCode` (else `ErrNotFound`); `groups.GetByCode(code)`; `categories.GetByCode(landscape.CategoryCode)`; `publisher.Upload(UploadInput{CategoryID, DatasetGroupID: &g.ID, Name (default filename), Filename, FileType: raster, ContentType, Size, Reader, StartDate, EndDate})`. `Slots` / `Results`: iterate `landscape.Slots` / `landscape.Outputs`, look up group by code, `LatestInGroup` (`Slots`: no status filter; `Results`: status ready) → `DatasetSummary` built from the `Dataset` (write a small `summaryOf(d domain.Dataset) domain.DatasetSummary` helper), `cog_url = "/datasets/" + id + ".cog"`, `sld_url = "/dataset-groups/" + gid + "/sld"` when the group has `SldStorageKey`. `MarkRunFailed` → `runs.SetRunStatus(failed, &reason)`. - [ ] **Step 3: RabbitMQ** (`internal/messaging/rabbitmq/landscape_consumer.go`) — mirror `parse_consumer.go`: constants, `LandscapeJob`, `LandscapeJobPublisher.EnqueueRun` (marshal `{run_id}` and `pub.Publish(ctx, LandscapeRunRoutingKey, body)`), `LandscapeProcessor`, `landscapeHandler.Handle` (decode, log, call `Execute`; on error call `MarkRunFailed` best-effort and return the error so the delivery is nacked), `NewLandscapeRunHandler`. - [ ] **Step 4: Run** `go test ./internal/service/ ./internal/messaging/... -race` → PASS. - [ ] **Step 5: Commit** ```bash git add internal/service/landscape.go internal/service/landscape_test.go internal/messaging/rabbitmq/landscape_consumer.go git commit -m "feat(landscape): run planning/execution service with fingerprint reuse; RabbitMQ job" ``` --- ### Task 8: HTTP API, wiring, OpenAPI, README, integration check **Files:** - Create: `internal/transport/http/landscape_handler.go`, `internal/transport/http/landscape_handler_test.go` - Modify: `internal/transport/http/router.go`, `internal/app/app.go`, `api/openapi.yaml`, `README.md`, `deployments/docker-compose.yml` (only if the worker needs an extra env; likely none) **Interfaces:** consumes Task 7's `LandscapeService` API and Task 3's `raster.NewGodalIO()`; Task 6's `landscape.Compute` is passed as `service.ComputeFunc`. - [ ] **Step 1: Handler** (`internal/transport/http/landscape_handler.go`) ```go // LandscapeHandler serves the /landscape routes. type LandscapeHandler struct { svc *service.LandscapeService validate *validator.Validate } func NewLandscapeHandler(svc *service.LandscapeService, validate *validator.Validate) *LandscapeHandler func (h *LandscapeHandler) Register(r chi.Router) { r.Get("/slots", h.slots) r.Post("/slots/{code}", h.uploadSlot) r.Get("/runs", h.listRuns) r.Post("/runs", h.createRun) r.Get("/runs/{id}", h.getRun) r.Get("/results", h.results) } type runRequest struct { FromStage *int `json:"from_stage" validate:"omitempty,gte=0,lte=3"` Overrides map[string]string `json:"overrides" validate:"omitempty,dive,uuid"` Params *landscape.Params `json:"params"` } ``` `uploadSlot`: `r.ParseMultipartForm(maxUploadBytes)`, `FormFile("file")`, optional `name`, `start_date`, `end_date`; content type default `image/tiff`; 201 with dataset; unknown slot → 404 via `respondDomainError`. `createRun`: decode, validate, parse override UUIDs, `Params` nil → `landscape.Params{}`; 202 with run. `getRun`/`listRuns` (`page`, `page_size`, `newPaginated`)/`slots`/`results` → 200 JSON. Handler tests (`httptest` + a `LandscapeService` built on the stubs from `landscape_test.go` is in another package — instead test only request parsing paths that fail before the service is needed: invalid JSON → 400, `from_stage: 7` → 422, override not a UUID → 422; construct the handler with a nil service since those paths return early). - [ ] **Step 2: Wiring** `router.go`: `Landscape *LandscapeHandler` in `RouterDeps`; `r.Route("/landscape", deps.Landscape.Register)`. `app.go`: ```go lpRepo := postgres.NewLandscapeRepository(pool) lpJobs := rabbitmq.NewLandscapeJobPublisher(publisher) rio := raster.NewGodalIO() landscapeSvc := service.NewLandscapeService(lpRepo, datasetRepo, groupRepo, categoryRepo, datasetsSvc, store, lpJobs, rio, landscape.Compute, log) ``` Add `landscape *service.LandscapeService` to `App`; `Landscape: transporthttp.NewLandscapeHandler(a.landscape, validate)`; add `LandscapeConsumer()` returning `rabbitmq.NewConsumer(a.rabbit, rabbitmq.LandscapeRunQueue, rabbitmq.LandscapeRunRoutingKey, "gis-landscape-runner", rabbitmq.NewLandscapeRunHandler(a.landscape, a.Log), a.Log)` and include it in `Consumers()`. Update `workerCmd.Short` to mention landscape runs. - [ ] **Step 3: OpenAPI + README** — tag `Landscape`; paths `/landscape/slots`, `/landscape/slots/{code}`, `/landscape/runs`, `/landscape/runs/{id}`, `/landscape/results`; schemas `LandscapeSlot`, `LandscapeRunInput`, `LandscapeParams`, `LandscapeRun`, `LandscapeRunNode`, `LandscapeDatasetRef`, `LandscapeResult`, `LandscapeRunPage`. README: new section "Landscape Program (ЛП)" — graph table (copy from spec §3.1), pixel rules summary, how to upload inputs, start a run, incremental recompute (`from_stage`), where results live, build requirements (cgo + GDAL: `brew install gdal` locally; Dockerfile handles the image). - [ ] **Step 4: Full verification** ```bash go build ./... && go vet ./... && go test ./... -race ``` Expected: all PASS. If Docker is available: `docker compose -f deployments/docker-compose.yml build` and `docker compose -f deployments/docker-compose.yml up -d postgres` + `go run ./cmd/gis migrate up` against it to prove migrations 00010/00011 apply (then `migrate status`). Report exactly what was and was not verified. - [ ] **Step 5: Commit** ```bash git add internal/transport/http/landscape_handler.go internal/transport/http/landscape_handler_test.go internal/transport/http/router.go internal/app/app.go internal/cli/worker.go api/openapi.yaml README.md git commit -m "feat(landscape): HTTP API, worker consumer, wiring, docs" ``` --- ## Self-review notes - Spec §2.1–2.4 → Tasks 1–2. §3.1–3.2 → Task 4. §3.3 → Task 3. §3.4 → Task 6. §3.5–3.6 → Tasks 5, 7 (+ `CreateRasterFromFile` in Task 2). §3.7 → Task 8. §3.8 error handling is embedded in Tasks 6–7. §3.9 tests are spread across tasks. - Name consistency: `LatestInGroup(ctx, groupID, status *string, fileType *domain.FileType)` (Tasks 1, 2, 7); `CreateRasterInput` / `CreateRasterFromFile` (Tasks 2, 7); `raster.Windows`, `DefaultWriterOptions` (Tasks 3, 6); `landscape.Stages`, `NodeByCode`, `SlotByCode`, `OutputByCode`, `Params.WithDefaults/Validate/ForNode` (Tasks 4, 6, 7); `domain.LPDatasetRef` (Tasks 5, 7).