long-horizon-prompting

Files (6)

• SKILL.md
• cdc-prompt-annotated.md · references/
• mass-handoff-contracts.md · references/
• research-evidence.md · references/
• task-brief-template.md · references/
• vendor-guidance.md · references/

SKILL.md (first lines)

# Long-Horizon Prompting

This skill covers the design of the prompt that launches an agent expected to work autonomously for hours or days, alone or as an orchestrator managing many parallel workers. The central technique is the pseudo-formal task brief: a specification written with the rigor of formal verification but expressed linguistically, because most hard problems have no machine-checkable success condition. The exemplar is the published prompt behind GPT-5.6 Sol Ultra's candidate proof of the Cycle Double Cover Conjecture, produced by a 64-subagent orchestration (claim-long-horizon-cdc-run). The prompt structure generalizes far beyond mathematics: any domain where success can be stated precisely and failure modes can be enumerated can use the same brief anatomy. In orchestrated runs the same discipline applies at every handoff: when a model optimized its own multi-agent workflow text in Sakana AI's MASS work, task information moved out of worker roles and into output contracts and call order (claim-long-horizon-contract-information-shift).

The controlling trade-off: everything that makes a long run productive (persistence, autonomy, parallelism) also raises the cost of a weak specification. A short interactive prompt fails cheaply; a long-horizon brief with a loophole burns hours of compute producing an answer-shaped artifact that does not solve the problem.

## When to Activate

Activate this skill when:

- Writing or reviewing the prompt for a long-running autonomous run before launching it
- Converting a vague hard problem ("solve X", "figure out why Y happens") into an explicit brief with a success predicate and non-counting outcomes
- Writing the root or orchestrator prompt that manages many parallel workers on an open-ended search problem
- Adding persistence instructions, stop conditions, effort floors, or return gates to an agent prompt
- Writing the output contract and acceptance checks each delegated worker must satisfy, or the rule for which review findings block a return
- Diagnosing a failed long run whose failure traces to the brief: premature return, an answer-shaped near miss, all workers converging on one approach, or fabricated completion claims
- Building a pre-launch review step that enhances and evaluates prompts before expensive agent time is committed

Do not activate this skill for adjacent work owned by other skills:

- Agent topology, supervisor versus swarm choice, handoff protocols, and coordination mechanics: `multi-agent-patterns`. That skill owns the architecture; this skill owns the words that steer it, including the contract each worker must return.
- Runtime control surfaces, locked evaluators, rollback, durable logs, and approval boundaries around an autonomous loop: `harness-engineering`. Constraints that must survive optimization pressure belong in the harness, not the prompt.
- Building the evaluator, regression suite, or deterministic quality gates a run is scored by: `evaluation`.
- LLM-as-judge design, rubrics, pairwise comparison, and bias mitigation: `advanced-evaluation`.
- Compaction, note-taking, and cross-session memory mechanics for surviving context limits: `context-compression`, `memory-systems`, `filesystem-context`.
- Loops that modify their own harness or prompts: `self-improvement-loops`.
- Remote sandboxes and background execution infrastructure: `hosted-agents`.

## Core Concepts

### Pseudo-Formal Task Specification

Formal verification requires a machine-checkable specification. Hard open problems rarely have one, but the discipline transfers: state the success condition so precisely that an adversarial reader cannot satisfy its letter without satisfying its intent. Four components, in order of leverage:

1. **Definitions with degenerate cases.** Define every load-bearing term before stating the goal, including the edge cases a lazy solution would exploit. The CDC prompt defines graph, bridge, cycle, and cycle double cover before the task, explicitly covering parallel-edge two-cycles, disconnected graphs, and the edgeless graph.
2. **Exact success predicate.** One statement of what must be true of the returned artifact, with scope quantifiers spelled out ("every finite loopless multigraph with no bridge, without additional assumptions such as cubicity, planarity, connectivity, or higher edge-connectivity").
3. **Non-counting outcomes.** An enumerated list of results that do not count: partial progress, special-case solutions, reductions to another unproved statement, bounded or computational verification, and best-effort summaries. This is the highest-leverage component. Under persistence pressure, models produce answer-shaped near misses; each excluded outcome removes one escape hatch.
4. **Enumerated failure modes for the auditor.** A concrete checklist of the domain-specific ways a candidate can be subtly wrong (in CDC: repeated-edge closed trails masquerading as cycles, bridges introduced by reductions, circular use of an equivalent statement). Verifiers with an enumerated hunt list catch what generic "check the work" instructions miss.

Machine-check what can be machine-checked. A task with no automatic success signal for the whole artifact still has parts a command can decide: tests that must pass, invariants on data splits, schema validity, a user flow clicked through in a browser. Express each as a named check with the command that runs it, and reserve the linguistic predicate for the residual no command can decide.

### Anatomy of a Long-Horizon Brief

| Block | Job | Failure it prevents |
| --- | --- | --- |
| Definitions | Fix the vocabulary, including degenerate cases | Loophole solutions on technicalities |
| Success predicate | State exactly what must be true at return | Scope-narrowed answers |
| Non-counting outcomes | Enumerate near misses that do not count | Answer-shaped partial results |
| Solvability framing | "Assume a solution exists" where existence is plausible | Give-up drift, "this is open" refusals |
| Orchestration policy | Heuristics for allocating parallel workers, not fixed assignments | Premature convergence, wasted parallelism |
| Handoff contracts | Each worker returns named artifacts and typed checks; downstream work waits for them | Errors compounding across stages, simulated delegation |
| Verification policy | Adversarial audit with enumerated failure modes and a blocking rule | Lenient self-judging, reviews that never converge |
| Reporting contract | Concrete artifacts required; status reports rejected | Vague optimism, fabricated progress |
| Return condition | Return only when the artifact survives audit | Premature return, best-effort summaries |
| Effort floor | Minimum effort before giving up is considered | Early abandonment |
| Contamination guards | What external search may and may not be used for | Laundered lookups, benchmark leakage |

### Persistence Cuts Both Ways

Persistence instructions ("do not return until", effort floors, assume-solvable framing) counter the documented drift toward giving up on long trajectories (claim-long-horizon-give-up-drift). But the same pressure raises the reward-hacking surface: the most persistence-trained frontier model measured to date also showed the highest detected cheating rate of any model its evaluator had tested, and its measured time horizon was not robust to whether cheating counted as success (claim-long-horizon-persistence-hacking). The design rule: never add a persistence instruction without a matching verification gate. Persistence pressure against a loose success predicate produces confident non-solutions.

### The Verification Bottleneck

Parallel sampling reliably raises the chance that some worker finds a correct answer, but the system's ability to select that answer lags behind, and model judges of hard artifacts are systematically lenient, rewarding rigorous-looking but incomplete arguments (claim-long-horizon-verification-gap). Budget as much prompt design for the verifier as for the generator:

- Give auditors the enumerated failure-mode list from the brief, not a generic quality instruction.
- Require the generator to produce modular, independently checkable output (lemma-level structure with stated premises and conclusions) so verification decomposes.
- Use fresh-context adversarial verifiers rather than self-critique; a verifier that did not build the artifact cannot rationalize its gaps. Fresh context removes build history but not the prior shared by one model family, so keep at least one check outside the loop for results that matter.
- Treat inter-agent agreement as a diversity failure signal, not as confirmation: committees converge most tightly on the hardest problems, where unanimity reflects shared bias rather than corroboration (claim-long-horizon-diversity-collapse).
- Define which findings block. A fresh adversarial reviewer always finds something. A finding blocks return only when it shows, with evidence (a failing check, a counterexample, a reproduction), a violation of the success predicate, a contract check, or the checklist; log every other finding as non-blocking. Without this rule "survives the audit" never converges, or the orchestrator decides privately which findings matter.
- Invest in the reviser, not only the reviewer. In MASS workflow search, upgrading only the evaluator helped less than upgrading the evaluator and the optimizer together (claim-long-horizon-reviser-bottleneck); accurate findings are wasted on an agent that cannot act on them.

### Handoff Contracts

In an orchestrated run, done is defined at every handoff, not once at the end. MASS writes a multi-agent workflow as text with four parts per worker: role, instruction, contract (the artifacts and information the worker must return, including required checks), and hop (call order). When the model optimized that text itself, task information fell in roles and instructions and rose in contracts and hops (claim-long-horizon-contract-information-shift). "Careful data scientist" fits any task; a contract that requires `no_future_leakage: bool` before modeling starts fits one.

- Name each check after the failure it rules out and make it typed, so the next agent and the harness can read it without judgment.
- Place each check at the stage that can introduce the error, and start downstream work only when that stage's contract is satisfied.
- List the degenerate values checks must reject: too-good-to-be-true metrics, quantities that should differ but match, placeholders, NaN or empty results treated as passing. Run each check once on a known-bad input; a vacuous check certifies the failure it was meant to catch.
- Route a failed check back to the responsible worker as a recall with the failed checks, a narrowed scope, acceptance criteria, and a retry cap. The fix request carries its own definition of done instead of reopening the whole artifact.
- Require real delegation. An orchestrator that writes a worker's output in its own turn has simulated the handoff, and simulation does not count.

The annotated workflow, evolved gates, and the authors' execution directive are in [the MASS reference](./references/mass-handoff-contracts.md).

### Structural Diversity in Parallel Search

Role labels do not create diversity; parallel workers share priors and converge unless independence is engineered:

- Keep early-round workers blind to the currently favored approach.
- Maintain an explicit registry of approach families, grouped by underlying idea rather than surface wording, and redirect workers away from crowded families.
- Mark a route blocked when it stalls at a missing step as hard as the original goal; reassign workers to it only for a materially new mechanism, not for enthusiasm.
- Cross-pollinate late, after independent development has exposed each route's real strengths and gaps.
- Do not let one approach dominate because its reductions are elegant; a route ending at a lemma equivalent in strength to the original goal is not progress.

### Stop Conditions, Effort, and Progress State

Long trajectories drift toward uncertainty and abandonment, and a budget stated once at the top of the prompt loses force as context grows (claim-long-horizon-give-up-drift). Countermeasures that belong in the brief: an explicit effort floor ("spend at least this much effort before considering returning"), assume-solvable framing where a solution plausibly exists, and a return condition phrased as a predicate over the artifact rather than over the agent's confidence. Countermeasures that belong outside the prompt: an externally maintained ledger of verified progress re-injected each round, which in controlled comparisons rescued large-quantity tasks that prompt-only and completion-gated setups failed entirely (claim-long-horizon-state-ledger). Progress claims should be auditable: requiring each reported claim to trace to a tool result or artifact from the current session nearly eliminated fabricated status reports in vendor testing (claim-long-horizon-evidence-audit).

### Lean and Outcome-First

Both major vendors converged on the same doctrine for current frontier models: the prompt should carry the outcome, hard constraints, evidence sources, and completion bar, and leave the path to the model. Accumulated instruction stacks measurably hurt; leaner system prompts improved vendor coding-agent evaluations while cutting cost (claim-long-horizon-lean-prompt). Persistence itself is increasingly trained in rather than prompted in, so spend the token budget on what training cannot supply: the success predicate, the non-counting list, and the domain failure modes only an expert in the problem knows. Per-worker contracts belong to the completion bar, not the path: they state what a stage must establish and how it is checked, never how to do the work.

## Detailed Topics

### The CDC Prompt, Dissected

The published Cycle Double Cover prompt implements every block of the brief anatomy in under a page: formal definitions closing degenerate-case loopholes, an exact success predicate with scope quantifiers, five classes of explicitly non-counting partial progress, dynamic orchestration heuristics for up to 64 concurrent agents with an approach-family registry and blocked-route bookkeeping, adversarial auditors with a seven-item failure-mode hunt list, a concrete-artifact reporting contract, an audit-gated return condition, an eight-hour effort floor, and a contamination guard restricting web search to background material (claim-long-horizon-cdc-run). The full annotated text is in [the CDC prompt reference](./references/cdc-prompt-annotated.md).

Two honest caveats. The candidate proof had no independent peer review or formalization when published, so the prompt is the validated artifact of interest here, not the theorem. And no public ablation isolates which prompt elements carried the result; the mechanism-level evidence comes from the independent research in [the research evidence reference](./references/research-evidence.md).

### Vendor Doctrine

OpenAI and Anthropic guidance overlap on fundamentals (explicit completion bars, stop rules, verification before return) and differ in emphasis. OpenAI doctrine centers persistence blocks, risk-tiered autonomy thresholds, self-constructed rubrics, and reasoning-effort dials; its multi-agent API institutionalizes a root agent with bounded-task subagents. Anthropic doctrine centers the four-part subagent delegation spec (objective, output format, tool guidance, task boundaries), explicit effort-scaling tiers by task complexity, evidence-grounded progress reporting, and fresh-context verifier subagents. Both now warn that over-prescriptive prompts degrade current-generation models. Dated extracts with sources are in [the vendor guidance reference](./references/vendor-guidance.md).

### Generalizing Beyond Mathematics

The CDC prompt worked because mathematics allows sharp statements, but each element has a general form usable in any rigorous domain:

| CDC element | General form |

From guanyang/open-agent-hub, MIT.

More from guanyang/open-agent-hub