23 Commits

Author SHA1 Message Date
d06d67436b test(landscape): make node-failure test deterministic under sibling cancellation
Scope the stub publisher failure to node 00's output; when every stage-0
publish failed, the first failing sibling cancelled node 00 and its
recorded error was "context canceled" instead of the publish error.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-05 16:01:33 +05:00
4051506b95 fix(landscape): guard redelivery, detach terminal writes, bound workers
F2: Execute had no status guard, so a redelivered or stale message reran a
finished run. Runs that are neither queued nor running are logged and acked
without work; "running" still executes so a worker that crashed mid-run
recovers.

F4: failRun, finishNodeFailed and MarkRunFailed wrote the terminal status on
the caller's context, which is exactly the context that is already cancelled
when the run failed because of a shutdown or a sibling node — the write was
refused and the run stayed stuck in "running". They now write on a
context.WithoutCancel copy with a 10s timeout; the consumer's best-effort
MarkRunFailed does the same.

F5: peak memory scaled with the host's core count. LP_MAX_WORKERS (default 8)
is now the total window-worker budget of a run and LP_MAX_NODES (default 2) the
node fan-out; each node gets max(1, workers/nodes) through the new
landscape.ComputeWithWorkers. Compute and service.ComputeFunc keep their
signatures: Compute delegates with MaxWorkersPerNode, and a legacy ComputeFunc
handed to NewLandscapeService is wrapped.

F6: the stage loop used a plain errgroup, so a failing node let its siblings
run to completion. It now uses errgroup.WithContext and passes the group
context into runNode, cancelling siblings mid-compute.

F7: wrap every float32 product in an explicit conversion so the compiler
cannot fuse it into an FMA, keeping the kernels bit-identical to numpy's
per-op rounding.

F8: README's builder package list now matches the Dockerfile (adds g++), and
the Landscape section documents the memory model and the two new knobs.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-05 14:44:34 +05:00
bcda76003c fix(datasets): accept BigTIFF headers and sanitise object keys
F1: ValidateFileContent rejected the pipeline's own output. GDAL writes a
BigTIFF header ("II+\0" / "MM\0+") whenever BIGTIFF=IF_SAFER kicks in for a
large raster, so accept both alongside the classic TIFF magic and reword the
error to "file is not a valid TIFF/BigTIFF GeoTIFF". GodalIO.Create now
delegates to an unexported createGTiff so a test can force BIGTIFF=YES and
assert the produced header passes the check.

F3: a user-supplied filename went straight into the S3 object key for dataset
uploads and for both dataset and dataset-group SLD uploads, so "../../x.sld"
escaped the dataset's own prefix. Reduce every such name to its base with a
shared safeObjectName helper, rejecting names with no usable base.

F9: rename the shadowing inner "raw" in the PATCH dg_id case to "s".

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-05 14:44:21 +05:00
d74277552f fix(landscape): bound node fan-out, cover override resolution, test the run consumer
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-05 13:49:27 +05:00
5fba77b59c feat(landscape): run planning/execution service with fingerprint reuse; RabbitMQ job
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-05 13:40:25 +05:00
bc524f3d98 feat(dataset-groups): CRUD, latest-in-group, group SLD with dataset fallback, CreateRasterFromFile
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-05 13:29:01 +05:00
922812b765 feat(datasets): periods (start_date/end_date), dg_id column, date filters; rename observation dates
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-05 13:16:07 +05:00
1e0844ca82 feat: Add SLD endpoints, add date_to to vector_with_kato 2026-08-01 22:09:59 +05:00
00a0cd1b8e feat: Add PUT and PATCH datasets endpoints 2026-07-21 00:19:52 +05:00
47b2e8653d fix: Fix processing for vector attributres table 2026-07-10 23:34:19 +05:00
017f869380 feat: Remove geometry and properties from GET /datasets/{id} endpoint, new endpoint for download cog file 2026-07-02 16:27:36 +05:00
d5c291fb9b feat: cache geojson data for vector files 2026-07-01 17:40:25 +05:00
ec8a014e82 feat: Remove redundant GeoJSON formats and return only mapped result for vector_with_kato 2026-07-01 17:29:22 +05:00
6113202704 fix: Mark dataset as failed for PostGIS errors 2026-07-01 16:58:51 +05:00
f74253140f fix: Change geojson properties structure 2026-07-01 00:48:02 +05:00
8d9e11db0c feat: Filters for datasets enpoint 2026-07-01 00:28:07 +05:00
27a8d8b4f5 feat: Districts table and geojson stub 2026-07-01 00:23:22 +05:00
f364ce4c6e fix: Fix vector with KATO geometry 2026-06-28 18:48:24 +05:00
30f7ae1e07 feat: reprocess all command 2026-06-28 00:40:50 +05:00
1e71b94fdb fix: Fix geometry parsing 2026-06-28 00:37:29 +05:00
7c469a524b feat: Add reprocess command 2026-06-27 23:30:54 +05:00
148d1fb841 fix: Code for category 2026-06-27 19:35:53 +05:00
9cf791b789 Second checkpoint 2026-06-25 01:17:42 +05:00