RDB-1: Regulated Document Benchmark

RDB-1 is a benchmark for one question: do you have to trade data security for quality? It puts models that run inside an aviation company's own perimeter next to cloud models and a general agent tool, on one task: a technical report a domain expert would sign. It measures that on a consolidated data set from a completed regulatory assessment, the same files, the same brief and the same template for every run, scored against the sources. In this first edition the best local model on Wingman360 Teammate scored 74 out of 100, level with the best cloud model on the same harness and above a frontier model on a general agent tool at 66. The 100 mark is a reference report corrected through three review iterations with a domain expert; no first output reached it.

1. The data set

One regulatory framework was screened requirement by requirement for applicability to a new aircraft design, reviewed line by line by domain experts, and consolidated with a full change log and threaded comments. The data set adds the technical baseline documents behind the assessment, the scope document that defines the assignment, and the regulation library needed to interpret it. The target of the task is the technical end report: what the assessment found, what is still open, what has to be decided, on a controlled document template under a fixed set of rules.

Nothing is pre-digested. The Excel workbooks are too large to read without a script, the decisive standard is 258 pages long, the planning documents contradict each other in places, and the brief deliberately contains one wrong statement about the assignment structure that the scope document corrects.

2. Rules and protocol

Rules in the brief

• No recommendation, no effort sizing. State the gaps in plain language and leave the decision to the reader.
• Authority frame. A specific authority frame, named in the brief, must be applied and not merely mentioned.
• House style. Fixed typographic and document-control conventions. The report covers the whole assignment, not one work package.
• Process. Plan the structure first and discuss it before writing.

Protocol

• One run per configuration on the unchanged data set. A run found to have used benchmark material or an earlier report is excluded.
• The brief is given verbatim. Planning questions are answered minimally and consistently. Harness, model, reasoning setting, date and tool failures are recorded. The .docx the run produces is the only artefact scored.
• The reviewer is an AI system with full source access, working under a human domain expert who challenges every finding. The reviewer's own first draft is part of the field as a reference, scored with the same rubric.

3. Scoring

Each output file goes through an automated pass and a manual read. The checker validates the file against the Office Open XML schema, counts hyperlinks, heading styles, template placeholders, over-long paragraphs and spreadsheet cell addresses in prose, and pattern-matches seventeen stated facts against a ground-truth file computed from the sources. The manual read verifies every further number and claim against the workbooks and PDFs and scores the judgement dimensions. A stated value that contradicts the sources counts against the run; a value the report does not state does not.

What 100 means. Seven dimensions are scored 0 to 5 and weighted to 70 points, then rescaled to 100. The 100 mark is the benchmark reference: the same task completed by an AI teammate and then corrected through three review iterations with a human domain expert. A run at 100 would need no correction.

Hallucination taxonomy

Fabricated or misread data: a stated figure the sources contradict. Knowledge error: a false statement from model memory, such as a wrong acronym expansion. Omission: relevant source content not used. The first two reduce the data and knowledge scores; omissions reduce depth.

4. Results

Four local models and one cloud model on Wingman360 Teammate, plus one reference on a general-purpose agent tool, all first outputs before any human correction, scored 6 September 2026. The 100 mark is the reference after three review iterations.

Total score out of 100

• GPT 6 AstraWingman360 Teammate · cloud model
  74
• DeepSeek V4 Flash (local)Wingman360 Teammate
  74
• GLM 5.3 Flash, high reasoning (local)Wingman360 Teammate
  71
• Qwen 3.8 Flash (local)Wingman360 Teammate
  66
• Claude Fable 5.1Reference · general agent tool
  66
• GLM 5.3 Flash, low reasoning (local)Wingman360 Teammate
  40

Wingman360 TeammateReference: general agent tool

Scores by dimension, 0 to 5

Automated signals and verified error counts

"Facts correct" counts ground-truth values stated correctly; unstated values are not penalised. "Facts wrong" and "Knowledge errors" are the counts confirmed in the manual read. "Coverage items" is out of thirteen required content items. Schema errors are Office Open XML validation findings; one table-border defect from the document tool appears in every file built with it.

5. What separates the runs

• Fabricated data is the discriminator. Two runs stated no figure the sources contradict, three stated exactly one, and the weakest stated four: an assumption count that does not exist, transition counts never computed, line counts that contradict themselves within the same document. Raw counts copied from a sheet were right everywhere; interpreted figures were where a model went wrong.
• Reasoning effort is worth thirty points. The same model in the same harness moved from 40 to 71 when reasoning was turned up. The extra effort removed the invented figures and the missed workbook column; it did not add domain knowledge.
• Knowledge errors are model signatures. Wrong acronym and authority expansions came from model memory, not from the files, and each model had its own: one mis-expanded the same safety-level acronym in two independent runs; another mis-expanded two terms a domain reader meets on the first page. These are the errors an expert notices first.
• Depth and polish rarely came together. The cloud model produced the deepest analysis and a file with 336 schema errors and 142 spreadsheet coordinates in prose. The 15,600-word local run read the workbook most thoroughly and made a finding about a gap in the assessment that nobody else made, and produced 27 paragraphs over 100 words with 27 schema errors. The cleanest file came with a one-page analysis.
• Reading the standard separated the field. Three runs, two of them local, found the two decisive clauses in the 258-page standard and paraphrased them accurately. None found the means-of-compliance issue that sets the quantitative safety target, so every run left the largest technical difference unquantified.
• The brief's trap caught one run. Five of six read the scope document and corrected the wrong assignment structure planted in the brief; the lowest-scoring run copied it into its output.
• Security did not cost quality. A frontier model on a general agent tool scored 66. On Wingman360 Teammate the best local model, running inside the perimeter, scored 74, level with the best cloud model on the same harness. The 100 mark is the reference report after three review iterations with a domain expert.

Limits of this edition

RDB-1 v1.0 is a single-domain benchmark with one run per configuration. A score therefore reflects one output of one model on one day, not a distribution over repeated runs. The manual dimensions carry reviewer judgement even with the anchors given in section 3. The reviewer is an AI system working under a human domain expert, and the reference run is that system's own first output on a general agent tool; both facts are disclosed and the reference is scored with the same rubric as every other run. The brief contains one wrong statement by design. Reasoning settings and token budgets were left at each configuration's default unless stated. Scores will move as configurations are added; the version and date are on this page.

Lessons learned from this study

Local models with the right harness perform at the level of frontier cloud models on this task. The best local run on Wingman360 Teammate scored 74, level with the cloud model on the same harness and above a frontier model on a general agent tool, with no data leaving the company's perimeter. Data security and sovereignty did not cost quality.

The harness carries much of that result. Built-in aviation skills and an aviation-optimised data ingestion and retrieval strategy give a local model the regulatory context, the workbook structure and the document rules it cannot hold on its own. That is what lets a model running inside the perimeter match a frontier cloud model on a task that no configuration completed without correction.