fix(landscape): guard redelivery, detach terminal writes, bound workers
F2: Execute had no status guard, so a redelivered or stale message reran a finished run. Runs that are neither queued nor running are logged and acked without work; "running" still executes so a worker that crashed mid-run recovers. F4: failRun, finishNodeFailed and MarkRunFailed wrote the terminal status on the caller's context, which is exactly the context that is already cancelled when the run failed because of a shutdown or a sibling node — the write was refused and the run stayed stuck in "running". They now write on a context.WithoutCancel copy with a 10s timeout; the consumer's best-effort MarkRunFailed does the same. F5: peak memory scaled with the host's core count. LP_MAX_WORKERS (default 8) is now the total window-worker budget of a run and LP_MAX_NODES (default 2) the node fan-out; each node gets max(1, workers/nodes) through the new landscape.ComputeWithWorkers. Compute and service.ComputeFunc keep their signatures: Compute delegates with MaxWorkersPerNode, and a legacy ComputeFunc handed to NewLandscapeService is wrapped. F6: the stage loop used a plain errgroup, so a failing node let its siblings run to completion. It now uses errgroup.WithContext and passes the group context into runNode, cancelling siblings mid-compute. F7: wrap every float32 product in an explicit conversion so the compiler cannot fuse it into an FMA, keeping the kernels bit-identical to numpy's per-op rounding. F8: README's builder package list now matches the Dockerfile (adds g++), and the Landscape section documents the memory model and the two new knobs. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
27
README.md
27
README.md
@@ -232,6 +232,31 @@ one, an `sld_url` (`/dataset-groups/{id}/sld`). A node failure fails its stage;
|
||||
later stages are not started and the run ends `failed` with the message. Jobs are
|
||||
nacked without requeue, so there is no automatic retry — start a new run.
|
||||
|
||||
### Memory and parallelism
|
||||
|
||||
A run has two nested levels of parallelism: nodes of a stage run concurrently,
|
||||
and each node processes its windows with its own worker pool. Every worker holds
|
||||
one window buffer per input **and** per output, so the worker's peak resident set
|
||||
is roughly
|
||||
|
||||
```
|
||||
nodes × workers-per-node × window_size² × (inputs + outputs) × 8 bytes
|
||||
```
|
||||
|
||||
(the widest node, `20 c_conflicts`, has 6 inputs and 5 outputs). Two environment
|
||||
variables bound that product, independently of how many cores the host has:
|
||||
|
||||
| Variable | Default | Meaning |
|
||||
| --- | --- | --- |
|
||||
| `LP_MAX_WORKERS` | `8` | Total window-worker budget shared by all nodes of a run. |
|
||||
| `LP_MAX_NODES` | `2` | How many graph nodes of one stage run at the same time. |
|
||||
|
||||
Each node gets `max(1, LP_MAX_WORKERS / LP_MAX_NODES)` workers, further capped by
|
||||
`GOMAXPROCS` and by the number of windows, so the total stays at
|
||||
`LP_MAX_WORKERS`. At the defaults and `window_size: 2048` that is about
|
||||
8 × 2048² × 11 × 8 B ≈ 2.9 GiB worst case — lower `LP_MAX_WORKERS` (or
|
||||
`window_size`) on a small host, raise it on a large one.
|
||||
|
||||
### Build requirements
|
||||
|
||||
The raster math uses [godal](https://github.com/airbusgeo/godal), a **cgo**
|
||||
@@ -244,7 +269,7 @@ go build ./... # cgo is on by default for a native build
|
||||
```
|
||||
|
||||
`build/package/Dockerfile` already handles the image: the builder stage adds
|
||||
`gdal-dev gcc musl-dev pkgconfig` and builds with `CGO_ENABLED=1`, and the
|
||||
`gdal-dev gcc g++ musl-dev pkgconfig` and builds with `CGO_ENABLED=1`, and the
|
||||
runtime stage installs `gdal` (the shared library godal needs) plus `gdal-tools`
|
||||
(`gdal_translate`/`gdalinfo` for the COG conversion).
|
||||
|
||||
|
||||
Reference in New Issue
Block a user