A long-trajectory reference
Claude Opus 4.8 took 42 turns, 41 tool calls and 2.68 million cumulative prompt tokens on this task — a good sample of long-horizon retrieval and hypothesis revision.
Anomaly Discovery at XX Corp
EDGAR-US-V6-0003-S1 · Stage 1 · SEC periodic-report audit reasoning
The same audit reasoning task over an offline corpus of 15 SEC filings (about 950,000 characters); the reference trajectory took 42 agent turns and 2.68 million prompt tokens.
SEC EDGARAnomaly discoveryLong trajectoryEnglish
1515 SEC filings
945Kcharacters of material
5result dimensions
42agent turns in the longest reference rollout
2.68Mcumulative prompt tokens in one rollout
01
Highlights
Same tools, different company
Each registrant has its own corpus and retrieval service, and train/eval splits follow grouping_key so no company leaks across sets.
A suggested audit workflow
The prompt suggests a workflow — inventory periods, screen anomaly shapes, fix account and direction, classify, gather evidence — without naming any specific account.
A 20-type fraud taxonomy
Each finding maps to one of 20 subtypes across six families (revenue, expenses, assets, liabilities, disclosure, other), describing a direction rather than an accusation.
02
Instruction excerpt
An excerpt of the instruction the agent actually receives, showing the task setup and constraints.
instruction.md · 15,514 chars
You are a financial statement auditor. You have been given the periodic filings of XX CORP and asked to perform an audit reasoning task.
Suggested audit-reasoning workflow
- Inventory the available periods and identify comparable filings before searching individual facts.
- Screen for observable anomaly shapes: restatement deltas, inconsistent disclosures, unusual account trajectories, policy-versus-result conflicts, omissions, and relationships that fail to reconcile.
- For each candidate, identify the affected account, assertion, transaction class, or disclosure and state the anomaly's direction and period.
- Classify the red flag under the taxonomy. Classification describes the accounting/disclosure consequence the filing pattern is consistent with; it is not a conclusion that fraud occurred.
- Retrieve and cite the filing facts that establish both sides of the pattern. Separate what the filings show from what you infer.
Available material
The offline case library contains 10 periodic reports, 3 supplemental disclosures, and 2 selected exhibits (945,318 characters in total).
NOTEExcerpt from the instruction; the taxonomy table, constraints and output structure are omitted.
03
Retrieval corpus
Everything visible inside the agent container, with statistics generated from the task package. Original files ship with the full sample package.
10periodic reports (10-K / 10-Q)
3supplemental disclosures (8-K, amendments, DEF 14A)
2selected exhibits (EX-99 / EX-10 etc.)
945Kcharacters in the corpus
| Main document type (excluding exhibits) | Count |
|---|---|
| 10-Q | 7 |
| 8-K | 3 |
| 10-K | 3 |
| DEF 14A | 2 |
The corpus is served by a separate read-only retrieval service; inside the agent container it is accessed through search → fetch → read commands, with no raw files mounted.
04
Scoring design
5result dimensions × execution factor
Full criteria text, weights and reference answers ship with the complete data package.
5 result dimensions matched one-to-one against labelled events, multiplied by an execution factor.
- Result dimensions: account, description, fraud type, period, filing evidence
- The execution factor covers data hallucination, interference resistance, tool failures and cost
- Hard negatives take a separate branch: a clean “no finding” earns full credit; reported findings are judged on their evidence chain
- Labels and scoring configuration live only in tests/ and are invisible to the agent
05
Model results
Trajectory statistics from frontier models in real agent harnesses. One turn is one model call; the context curve shows how state accumulates over a long-horizon task.
| Model | Agent turns | Tool calls | Cumulative prompt tokens | Peak context | Time | Result |
|---|---|---|---|---|---|---|
| Claude Opus 4.8Claude Code · Harbor, offline | 41 | 2.68M | 92K | 17m 58s | Completed |
Every task in this dataset ships with a full Claude Opus 4.8 reference trajectory (all tool calls and reasoning); trajectories contain no reference answers or scores.
06
Specification
| Task ID | EDGAR-US-V6-0003-S1 |
|---|---|
| Task format | Harbor task.toml |
| Agent time limit | 2 h |
| Agent container | 2 vCPU · 4 GB memory |
| Network | Fully offline (empty allowlist) |
| Instruction length | 15,514 characters |
| Dataset | EDGAR-US v6 · 609 tasks |
| Task type | Stage 1 · anomaly discovery |
Contact
Want to run this task on your own platform?
Leave your details and we will send the full sample package for this task (instruction, attachments, verifier and reference trajectories) and walk you through volume and customisation for similar tasks.
- Full task package and verifier
- More tasks in this subdomain
- Integration and re-run support