2.63 million characters — too much to read
20 10-K/10-Q filings, 38 supplemental disclosures and 30 exhibits. The model must use full-text search to pick accounts and periods, then read only the passages that matter.
Anomaly Discovery at XX Industries
EDGAR-US-V6-0052-S1 · Stage 1 · SEC periodic-report audit reasoning
Search an offline corpus of 88 SEC filings (2.63 million characters), identify anomalies observable in the financial statements, and support every finding with evidence from the filings.
SEC EDGARAnomaly discoveryRetrieval toolsEnglish
8888 SEC filings
2.6Mcharacters of material
5result dimensions
18agent turns in the longest reference rollout
880Kcumulative prompt tokens in one rollout
01
Highlights
Ground truth from SEC enforcement
Labels come from SEC litigation releases and AAERs; the corpus stops before the matters were made public, so the answer cannot be found in the text.
“No finding” is a valid answer
The dataset includes companies the SEC never charged. A model that flags every company is penalized on precision; only genuine audit reasoning scores.
A tool-based environment
The corpus is served by a separate retrieval service (search → fetch → read); the rollout runs fully offline and the model may write Python for multi-period analysis.
02
Instruction excerpt
An excerpt of the instruction the agent actually receives, showing the task setup and constraints.
instruction.md · 15,527 chars
You are a financial statement auditor. You have been given the periodic filings of XX INDUSTRIES INC and asked to perform an audit reasoning task.
Objective
Identify observable anomalous patterns (red flags) in the financial statements and disclosures. For each red flag, state the affected subject, the shape and direction of the anomaly, the filings it touches, and the filing-based evidence that establishes it.
Two outcomes are possible, and both are legitimate answers. Some companies under review have no red flag that is observable in the supplied periodic filings. On a company with observable red flags, abstention scores zero. Unsupported findings lower precision. Decide from the evidence.
Temporal boundary
The corpus stops at a cutoff date that precedes any public confirmation of the matters at issue. Filings made after that date are not in the corpus, and neither is any enforcement document, restatement announcement, or news report that would settle the question for you.
Available material
The offline case library contains 20 periodic reports, 38 supplemental disclosures, and 30 selected exhibits (2,631,525 characters in total). This is far more text than you can read end to end. Choose accounts and periods before retrieval, use FTS and --grep to locate evidence, and read only the chunks that matter.
NOTEExcerpt from the instruction; the taxonomy table, constraints and output structure are omitted.
03
Retrieval corpus
Everything visible inside the agent container, with statistics generated from the task package. Original files ship with the full sample package.
20periodic reports (10-K / 10-Q)
38supplemental disclosures (8-K, amendments, DEF 14A)
30selected exhibits (EX-99 / EX-10 etc.)
2.6Mcharacters in the corpus
| Main document type (excluding exhibits) | Count |
|---|---|
| 8-K | 46 |
| 10-Q | 15 |
| 8-K/A | 10 |
| 10-K/A | 7 |
| 10-K | 6 |
| DEF 14A | 4 |
The corpus is served by a separate read-only retrieval service; inside the agent container it is accessed through search → fetch → read commands, with no raw files mounted.
04
Scoring design
5result dimensions × execution factor
Full criteria text, weights and reference answers ship with the complete data package.
5 result dimensions matched one-to-one against labelled events, multiplied by an execution factor.
- Result dimensions: account, description, fraud type, period, filing evidence
- The execution factor covers data hallucination, interference resistance, tool failures and cost
- Hard negatives take a separate branch: a clean “no finding” earns full credit; reported findings are judged on their evidence chain
- Labels and scoring configuration live only in tests/ and are invisible to the agent
05
Model results
Trajectory statistics from frontier models in real agent harnesses. One turn is one model call; the context curve shows how state accumulates over a long-horizon task.
| Model | Agent turns | Tool calls | Cumulative prompt tokens | Peak context | Time | Result |
|---|---|---|---|---|---|---|
| Claude Opus 4.8Claude Code · Harbor, offline | 17 | 880K | 69K | 6m 46s | Completed |
Every task in this dataset ships with a full Claude Opus 4.8 reference trajectory (all tool calls and reasoning); trajectories contain no reference answers or scores.
06
Specification
| Task ID | EDGAR-US-V6-0052-S1 |
|---|---|
| Task format | Harbor task.toml |
| Agent time limit | 2 h |
| Agent container | 2 vCPU · 4 GB memory |
| Network | Fully offline (empty allowlist) |
| Instruction length | 15,527 characters |
| Dataset | EDGAR-US v6 · 609 tasks |
| Task type | Stage 1 · anomaly discovery |
Contact
Want to run this task on your own platform?
Leave your details and we will send the full sample package for this task (instruction, attachments, verifier and reference trajectories) and walk you through volume and customisation for similar tasks.
- Full task package and verifier
- More tasks in this subdomain
- Integration and re-run support