cgpadwick/saage

This is the one big bet: a typed primitive set is the only thing that keeps graph semantics coherent as team and flow complexity grow 10x; ad-hoc loops fork the mental model.

Cost Requires committing to PocketFlow (or equivalent) as the graph substrate and enforcing the primitive API at code-review time, teams will resist the constraint.

At 10x run volume, zombie 'running' states compound into double-billing, duplicate submissions, and silent data corruption, this is the class of bug that causes production post-mortems.

Cost One extra CAS or transactional write at run exit; low code cost but requires discipline to never update status separately from checkpoint.

Without a hard ceiling, a single model regression or adversarial prompt can pin a worker indefinitely and drain cloud budget; this is the cheapest safety primitive with the highest expected-value payoff.

Cost One integer config per agent definition; the only friction is tuning budgets high enough not to break legitimate long runs.

File-tree shipping is O(repo size) and breaks on large datasets or generated artifacts already in-tree; git refs are O(1) and give you replay, diff, and provenance for free.

Cost Requires all work-in-progress to be committed before handoff, changes the development loop and fails silently if teams use untracked files.

The asymmetry (poll failure = retry, wall-clock expiry = terminate) is the correct invariant; inverting it is the common mistake that creates runaway cloud spend at scale.

Cost Requires a watchdog with reliable wall-clock access outside the main process, adds infrastructure complexity but is non-negotiable for cost safety.

Mocking the full stack hides integration failures; mocking only LLM responses gives fast, deterministic CI that catches the real failure modes (bad prompts, wrong file paths, subprocess contract drift).

Cost Requires a testkit with a scriptable LLM double and careful discipline not to mock side-effecting infrastructure, upfront investment but pays off immediately.

Without a stable contract, every model upgrade can silently change generated script shape and break the harness; this is the interface boundary that makes LLM-generated code composable with deterministic infrastructure.

Cost Must be injected into every code-generation prompt and enforced by a schema validator at runtime, prompt engineering discipline required across all flows.

Long GPU runs ($$$) that die mid-flight force full reruns; checkpoint+resume directly converts a total loss into a partial one, which is the #1 reason teams abandon agentic automation for serious workloads

Cost Requires persistent writable state path and node-level hooks; retrofitting onto an existing PocketFlow graph means touching every node exit, medium implementation lift, high ops discipline to not break resume semantics

Without a hard ceiling, one bad model response (looping tool calls, confused state) burns unbounded tokens and blocks the pipeline, this is the single change that makes agentic systems safe to hand to non-engineers

Cost Near-zero: a counter and a check; the only cost is agreeing on what 'step' means and surfacing the limit as a first-class config knob so users don't hit it silently

Cloud GPU billing starts the moment an instance is allocated; a process that crashes mid-poll and abandons the wait leaves a running instance with no owner, this is a silent cost bleed that users only notice on their bill

Cost Requires distinguishing transient poll errors from genuine launch failures and holding wall-clock state across retries; subtle to get right, but the logic is isolated to the provisioning layer

Without a pinned interface, the harness can't reliably parse LLM-generated code outputs, this contract is what makes autonomous ML research (propose → implement → evaluate) deterministic enough to trust scores

Cost Low for new flows; retrofitting existing flows requires auditing every generated script boundary and adding validation, teams resist because it constrains what the LLM can produce

Without this, the headline score from an autonomous hill-climb is optimistically biased by test-set selection, results look better than they are, and the team ships a model that underperforms on real data

Cost Zero code cost; purely a flow design constraint, but it requires flow authors to provision a held-out split upfront and resist the temptation to peek, the hard part is enforcement, not implementation

Users who route reasoning models through NVIDIA NIM get no chain-of-thought and no cost visibility; they believe the system is working correctly when it is silently omitting the most valuable model output

Cost Fixing requires surfacing the gap at runtime (a warning or explicit error) rather than silent omission, low code cost, but requires a product decision to either block the feature or degrade gracefully with a loud warning

Multi-machine remote runs fail silently when key name collision overwrites authorized_keys, the error surfaces only when the SSH connection is attempted, long after provisioning, making it extremely hard to diagnose

Cost Isolated to the Lambda provisioning layer; the fix is a content hash comparison before registration, low lift, but requires reading and comparing key files rather than trusting names, which is a subtle API behavior change

Enables fast, zero-cost CI for flows that would otherwise require live model calls, the saage_testkit pattern already demonstrates this scales to full integration coverage without network dependency.

Cost Medium: requires building and maintaining a deterministic LLM double per provider interface, and discipline to keep scripted turns in sync with real prompt changes.

Keeps CI fast and green by default while still validating contrib wiring, new engineers get a clear model of what is tested vs what is only structurally checked.

Cost Low: purely organizational; requires a documented contrib/README convention and a hydrate-only CI job, both already present here.

Eliminates the class of runaway-agent incidents that burn compute budget and block downstream CI jobs, a missing safeguard that turns a model regression into an on-call page.

Cost Low: a single integer bound threaded into the loop entry point; no architectural change needed.

Library forks are invisible to dependency update bots and reviewers; without explicit documentation and a pinned version, a routine dep bump silently breaks the override and surfaces only at runtime.

Cost Low: a comment block plus a pinned version constraint in pyproject.toml; the discipline cost is higher than the mechanical cost.

Without a stable contract, each LLM-generated script is a unique interface the harness must parse ad hoc, this is the difference between a reproducible pipeline and a flaky one that breaks on model output variation.

Cost Medium: requires authoring and enforcing a schema in every skill.md prompt; existing flows must be audited for compliance.

Ghost 'running' states force manual intervention to clear and waste resume budget re-executing expensive nodes, exactly the failure mode that erodes trust in automated pipelines.

Cost Low: a single atomic write replacing two sequential writes in the checkpoint flush path; no schema change required.

Silent omissions in observability (cost) and correctness (reasoning trace) are the hardest class of bug for engineers to diagnose, they show up as anomalous downstream results, not errors.

Cost Medium: requires either completing the provider implementation or adding explicit runtime warnings when reasoning_content is dropped and cost is None, so the gap is visible in logs.

Regex-based shell command filtering is the canonical bypassable control: encoding tricks, $IFS substitution, process substitution, and here-strings all evade pattern matching; an LLM agent with tool-use can probe the denylist and find a bypass in its own reasoning loop.

Cost High, requires replacing the denylist with a restricted execution model (allowlist of discrete verbs + argument schemas, or a seccomp/container boundary) rather than patching the regex.

The contract governs the interface, not the body, LLM-generated script internals are arbitrary code execution with whatever privileges the process inherits; a poisoned prompt or jailbreak produces a weaponized script that the harness will faithfully execute and trust its output JSON.

Cost High, sandboxing generated code (gVisor, nsjail, or at minimum a dedicate low-privilege uid + network-off namespace) requires infrastructure changes, not just interface conventions.

Deferred auth means the full prompt, tool outputs, and conversation context are serialized and transmitted to an external endpoint before the credential failure is detected; if the endpoint is misconfigured or the model id resolves to a different provider, prompt content leaves the trust boundary unauthenticated.

Cost Low, fail-fast on missing keys at client construction; the placeholder pattern adds complexity that a simple precondition check eliminates.

Any process that can set an environment variable or write to a user-writable path can silently zero out all cost estimates, enabling a supply-chain or insider attack that runs arbitrarily expensive workloads while reporting $0 cost to any upstream budget gate.

Cost Medium, restrict the override path to a config-owned location (not arbitrary env-var-pointed path), validate schema strictly, and require the override file to be owned by the effective uid.

Unconditionally removing the prior host key before connecting disables TOFU: an adversary who can ARP-spoof or BGP-hijack the IP during provisioning gets a clean MITM window with no StrictHostKeyChecking rejection, and the agent will then upload credentials and git refs over the attacker-controlled channel.

Cost Medium, instead of blind removal, capture the new instance's host key via the cloud API's console output before first SSH and pin it explicitly, or use a CA-signed host certificate.

Security patches in the upstream orchestration layer (auth header handling, timeout enforcement, deserialization) will never reach the fork; pinning the version prevents the patch from landing even when a maintainer explicitly upgrades for security reasons.

Cost Low to medium, the idea correctly names the risk; the missing step is adding a CI check that diffs the fork against the upstream symbol on each dependency bump, making drift visible rather than advisory.

Silently dropping audit log writes means a storage quota exhaustion, permission change, or targeted deletion by malware produces a forensically empty run record; incident response has no ground truth for what the agent executed or what data it accessed.

Cost Low, emit a structured warning to stderr and increment a metric on write failure rather than swallowing; the run can still complete while the failure is observable.

Runaway agent loops are the top unplanned incident type in agentic systems; a hard ceiling converts an unbounded failure mode into a deterministic, testable one.

Cost Add a step counter and a single guard raise at loop entry; zero new dependencies, one test case.

Stale 'running' state causes double-execution of the final node on resume, corrupting ledgers and billing; atomicity eliminates the race with no protocol overhead.

Cost Merge two sequential writes into one dict write; requires identifying every exit path in the primitives layer.

Without per-node checkpoints, any crash in a multi-hour ML run restarts from zero; this is the single feature that makes remote cloud runs economically viable.

Cost Requires a persistent key-value store per run and step-index tracking wired into every loop primitive's post-exec hook.

Aborting a poll-wait is the exact action that leaks a metered GPU instance; distinguishing transient errors from timeouts is the only correct policy.

Cost Implement two error classes (transient vs. terminal) in the poll loop and a background watchdog thread; moderate complexity, high cloud-cost payoff.

Without a fixed interface, every LLM-generated script requires ad-hoc parsing; a contract lets the harness stay dumb and makes generated code unit-testable.

Cost Write a one-page skill.md spec and add a contract-validation step to CI; LLM prompts must include the contract verbatim.

All major LLM APIs have transient 429/5xx rates that silently abort long flows without retry; backoff+jitter prevents thundering-herd pile-ons.

Cost Wrap all LLM call sites in a single retry decorator; jitter requires a random seed choice, document it for reproducibility.

Without holdout discipline, a hill-climb's headline number reflects test-set selection bias and overstates real generalization, making the run's output scientifically invalid.

Cost Add one eval step at flow tail and enforce split isolation in data-prep; requires competition datasets to ship with an explicit split key.