fix(landscape): guard redelivery, detach terminal writes, bound workers
F2: Execute had no status guard, so a redelivered or stale message reran a finished run. Runs that are neither queued nor running are logged and acked without work; "running" still executes so a worker that crashed mid-run recovers. F4: failRun, finishNodeFailed and MarkRunFailed wrote the terminal status on the caller's context, which is exactly the context that is already cancelled when the run failed because of a shutdown or a sibling node — the write was refused and the run stayed stuck in "running". They now write on a context.WithoutCancel copy with a 10s timeout; the consumer's best-effort MarkRunFailed does the same. F5: peak memory scaled with the host's core count. LP_MAX_WORKERS (default 8) is now the total window-worker budget of a run and LP_MAX_NODES (default 2) the node fan-out; each node gets max(1, workers/nodes) through the new landscape.ComputeWithWorkers. Compute and service.ComputeFunc keep their signatures: Compute delegates with MaxWorkersPerNode, and a legacy ComputeFunc handed to NewLandscapeService is wrapped. F6: the stage loop used a plain errgroup, so a failing node let its siblings run to completion. It now uses errgroup.WithContext and passes the group context into runNode, cancelling siblings mid-compute. F7: wrap every float32 product in an explicit conversion so the compiler cannot fuse it into an FMA, keeping the kernels bit-identical to numpy's per-op rounding. F8: README's builder package list now matches the Dockerfile (adds g++), and the Landscape section documents the memory model and the two new knobs. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
@@ -217,3 +217,36 @@ func TestRemoveOutputs_DeletesFiles(t *testing.T) {
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// TestComputeWithWorkers_RespectsCap checks the caller's worker budget bounds
|
||||
// the pool: each pool worker opens one reader per input and holds a buffer per
|
||||
// input and output, so the open count bounds peak memory. checkGrids opens one
|
||||
// extra set of handles before the pool starts, hence the (workers+1) bound.
|
||||
func TestComputeWithWorkers_RespectsCap(t *testing.T) {
|
||||
node, _ := NodeByCode("10")
|
||||
inputs := len(node.Inputs)
|
||||
|
||||
for _, workers := range []int{1, 2, 4, 0, -3} {
|
||||
t.Run(fmt.Sprintf("workers=%d", workers), func(t *testing.T) {
|
||||
fio := newFakeIO()
|
||||
g := grid(64, 64)
|
||||
fio.put("z.tif", g, constRaster(64, 64, 2))
|
||||
fio.put("u.tif", g, constRaster(64, 64, 4))
|
||||
p := DefaultParams()
|
||||
p.WindowSize = 8 // 64 windows, so the pool is never window-bound
|
||||
|
||||
if _, err := ComputeWithWorkers(context.Background(), fio, node,
|
||||
map[string]string{OutZIntegrated: "z.tif", OutUIntegrated: "u.tif"}, "/out", p, workers); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
effective := max(1, workers)
|
||||
if got, limit := fio.openCount(), (effective+1)*inputs; got > limit {
|
||||
t.Fatalf("opened %d readers with workers=%d, want at most %d", got, workers, limit)
|
||||
}
|
||||
if fio.openCount() < inputs {
|
||||
t.Fatalf("opened %d readers, want at least one set of %d", fio.openCount(), inputs)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user