fix(landscape): guard redelivery, detach terminal writes, bound workers

F2: Execute had no status guard, so a redelivered or stale message reran a
finished run. Runs that are neither queued nor running are logged and acked
without work; "running" still executes so a worker that crashed mid-run
recovers.

F4: failRun, finishNodeFailed and MarkRunFailed wrote the terminal status on
the caller's context, which is exactly the context that is already cancelled
when the run failed because of a shutdown or a sibling node — the write was
refused and the run stayed stuck in "running". They now write on a
context.WithoutCancel copy with a 10s timeout; the consumer's best-effort
MarkRunFailed does the same.

F5: peak memory scaled with the host's core count. LP_MAX_WORKERS (default 8)
is now the total window-worker budget of a run and LP_MAX_NODES (default 2) the
node fan-out; each node gets max(1, workers/nodes) through the new
landscape.ComputeWithWorkers. Compute and service.ComputeFunc keep their
signatures: Compute delegates with MaxWorkersPerNode, and a legacy ComputeFunc
handed to NewLandscapeService is wrapped.

F6: the stage loop used a plain errgroup, so a failing node let its siblings
run to completion. It now uses errgroup.WithContext and passes the group
context into runNode, cancelling siblings mid-compute.

F7: wrap every float32 product in an explicit conversion so the compiler
cannot fuse it into an FMA, keeping the kernels bit-identical to numpy's
per-op rounding.

F8: README's builder package list now matches the Dockerfile (adds g++), and
the Landscape section documents the memory model and the two new knobs.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
2026-09-05 14:44:34 +05:00
parent bcda76003c
commit 4051506b95
12 changed files with 563 additions and 69 deletions

View File

@@ -5,6 +5,7 @@ import (
"encoding/json"
"fmt"
"log/slog"
"time"
"github.com/google/uuid"
amqp "github.com/rabbitmq/amqp091-go"
@@ -15,6 +16,9 @@ const (
LandscapeRunRoutingKey = "landscape.run"
// LandscapeRunQueue receives Landscape Program run jobs.
LandscapeRunQueue = "gis.landscape.run"
// markRunFailedTimeout bounds the best-effort terminal status write made
// after Execute failed, on a context detached from the delivery's.
markRunFailedTimeout = 10 * time.Second
)
// LandscapeJob is the message body for a Landscape Program run job.
@@ -72,9 +76,14 @@ func (h landscapeHandler) Handle(ctx context.Context, d amqp.Delivery) error {
// status write), since Execute already marks the run failed for every
// node failure. Best-effort: log if marking fails, but still return the
// original error so the delivery is nacked.
if markErr := h.proc.MarkRunFailed(ctx, job.RunID, err.Error()); markErr != nil {
// Detached from ctx: Execute may well have failed *because* ctx was
// cancelled (shutdown), and the write must still land or the run stays
// "running" forever.
mctx, cancel := context.WithTimeout(context.WithoutCancel(ctx), markRunFailedTimeout)
if markErr := h.proc.MarkRunFailed(mctx, job.RunID, err.Error()); markErr != nil {
h.log.Error("mark landscape run failed", "run_id", job.RunID, "error", markErr)
}
cancel()
}
return err
}