# Benchmark method

This report preserves three original complete cohorts: five virtual-data apps,
ten SharePoint-connected apps and five complex multi-prompt apps. A separate
cost cross-check adds four reviewed virtual observations, five SharePoint
observations and three complex-app observations. Every accepted-app measurement has an
observed final cost and an accepted final assessment; the methods and sample
sizes remain distinct.
Test 4 adds two accepted capability-stress observations under a separate
four-stage specification and separate core/advanced rubrics. It is not pooled
with the preceding cohorts.
Test 5 separately recreates a private Annual Leave Manager HTML reference.
All five original settings are finalized; only accepted completed-app costs
are published, with one adaptive observation per selected setting.

## Same-tenant Test 1 pair: 28 September 2026

Exactly two fresh GPT 5.5 / Medium builds were registered in one demo tenant,
with identical initial business prompts and the external-only twenty-group
procurement rubric. No replacement builds, paid self-review, controller source
edits or injected business data were allowed. Native owner and model/effort
controls were confirmed for each start. The second build began only after the
first was accepted and costed.

Existing preview-origin stores were read and protected before app execution.
Each first output contained the same existing-key collision, sealed before
guidance; both received the same targeted isolation correction. No actual
cross-app overwrite is claimed. Corrected workflows were exercised through
native controls, including reload/reopen and desktop/mobile interaction.

Acceptance remains at least 85/100 with every critical check, not mandatory
perfection. Build 2 retains a failed responsive group because its mobile
dashboard overflowed; primary mobile request and approval worked. Its 95-point
accepted result is not labelled fully responsive. Build 1 received 100 points.
The final owner previews were reviewed, not separately verified republished.

Only one final cumulative `/cost` followed each completed review and actual
final-image inspection: 281 and 221 credits. Every submitted authoring prompt,
visible output, final app/cost image and review result is retained privately.
The separately reported 251-credit mean and 60-credit range describe two
observations, not a price guarantee. Original Test 1 included paid native review
and a different persistence-scoring scope, so its average is not recalculated.

## New model version: Opus 5.5 / Medium

Six fresh apps were originally registered across the five scopes: Test 5 first,
then Test 1, two Test 2 SharePoint builds, Test 3 and Test 4. Medium was the
observed native default. Frozen business prompts, reference bytes, fixtures
and scope-specific criteria are preserved; independent review and bounded
adaptive corrections stay in the same app. Only one final cumulative cost
is taken after completion. This update reports the five completed builds across
Tests 1, 2, 3 and 5. The unfinished new-version Test 4 run is omitted at the
user's request; its exclusion changes reporting only, not the frozen execution
registration or any historical measurement. No cost, partial prompt total or
capability result is inferred for the omitted run.

This is a new-version cohort, not a replacement for or repeat of Opus 5.
Neither version nor different test scopes are pooled. Prompt totals include
all authoring guidance but not the cost command or external review. Detailed
capability verdicts, rather than a universal quality percentage, accompany
each completed result. New-version raw evidence and screenshots remain private.

The first new-version SharePoint app required one build, one read-error
correction and one clarification to deliver existing schema evidence as files.
All three are included in its sole 610-credit completed-app reading. The
repair also changed create/readback guards, so review made one documented
supplemental native save while preserving all earlier rows and inventory.
Both created test records were removed with exact ETags only after final
evidence and cost; all seed fields and ETags stayed unchanged.
Actual downloaded schema bodies were checked against the live binding and
server evidence. Their field summaries do not themselves contain nested
Choice properties or site/list IDs; those details require corroboration.
Read-failure/Retry coverage is not a permission, throttling or ambiguous-commit
test, and adds no retroactive verdict to historical untested cases.

The second SharePoint build received the identical initial prompt and reached
the same ten-group target for 419 credits with two authoring prompts. Its one
correction addressed the same demonstrated failed-read/false-empty defect and
also requested actual downloadable schema files, avoiding a separate
evidence-delivery turn. A changed save/readback path again justified exactly
one guarded supplemental native save without resetting the existing fixture.
Its failure scenario covered request-list reads; do not claim all connector
failure modes were exercised. Both runs retain their original 9/10 outcomes
and explicit presentation limitations. The completed pair's 514.5-credit and
2.5-prompt means are descriptive n=2 statistics, not a stable price, a causal
prompt-cost estimate or an Opus 5 repeat.

The new Test 3 lifecycle app completed after its four unchanged planned prompts:
one build and three planned changes, with no corrections or clarifications.
Stage transitions required independently retained same-app/data/profile
evidence, not interim cost commands. Its only cost reading was **920 credits**
after final functional review and inspection of 70 native screenshots across
desktop/mobile and light/dark scenes. The original draft and old-policy request
survived the changes before their registered final review actions.
Real own-profile loading, read failure, Retry and reopen were exercised.
All optional profile values were present; genuinely absent-value handling was
not exercised or fabricated. One controller-created low-stock request used a
hyphenated justification instead of the frozen unhyphenated wording; device,
quantity, price, date, persona, workflow and resulting stock all matched.
This review-input variation is retained, not presented as byte-identical input.
The four authoring prompts themselves were unchanged. The final owner preview
was reviewed, not republished; the exact initial published resource was deleted
once after evidence/cost, with read-only verification that other apps remained.
No cross-test suite average is published.

Test 5 reused the supplied reference screens and business logic; one correction
fixed inherited theme styling. This observed strategy was not separately
randomized, so its cost difference cannot establish a model-only or reuse-only
saving. One observation per scope, or two SharePoint observations when
complete, remains exploratory.

## Test 5: matched adaptive repeat

A separately registered repeat uses the same original model version, effort,
prompt, attachment and target in a fresh task. The completed Sonnet 5 / High,
GPT 5.5 / Medium and Sonnet 5 / Medium pairs contain both actual costs and
prompt counts; their arithmetic means are descriptive, not stable prices.
On 24 September, the user authorized resuming
the GPT 5.5 / Medium and Sonnet 5 / Medium repeats. An explicit order amendment
skips the unavailable Opus 5 slots, retaining GPT before Sonnet and preserving
the original run IDs, inputs, review criteria and completed-app-only costs.
Each accepted repeat is published independently; the other settings need not
finish first. Opus 5.5 observations never fill an Opus 5 repeat slot.
Guided replay is a third, distinct design and remains unstarted.

For the Sonnet Medium repeat, all 30 groups were adjudicated and eight final
app-only images inspected before its sole cumulative cost reading. It met the
minimum at 18/20 functional and 8/10 visual groups, so optional polishing
stopped despite the disclosed print, search/shortcut, typography and density
failures. Seven real CSV downloads matched the full reference schemas and
values; a visible Print button did not substitute for a working print layout.
Owner-preview acceptance does not establish published-revision parity.
The exact published resource was independently bound through the native
preview's Open in browser action and verified in the owner library before
authorized deletion. Its successful deletion was never repeated when the
first readback did not settle; later read-only verification preserved all
other observed entries. Raw repeat evidence stays private.

Until all five matched pairs are complete, the overview retains the original
five completed costs and their original-cohort average. The detailed results
show each completed pair's two costs, mean, range and capability outcomes.
There is no mixed average combining single observations with two-build means,
and an unfinished repeat never hides or invalidates its completed original.

## Test 5: reference fidelity and completed-app cost

Five fresh original tasks use the same frozen initial prompt and byte-identical
HTML attachment, one per selected model/reasoning setting. Repairs continue in
the same app until the frozen joint target: at least 17 of 20 functional groups
and 8 of 10 visual groups, including every critical group. All 30 groups must
be adjudicated; a pass requires every subcase. Meeting this minimum is not
complete parity or an overall app-quality percentage.

Review is independent and native: reference and app at 1440px/390px in both
themes, all areas and settled overlays, role-specific workflows, actual CSV
downloads, print layout, reset/reload behavior and isolation from prior apps.
Only native UI actions change business state. Read-only DOM, source, state and
actual exports corroborate the result; the reviewer does not patch generated
code, invoke app functions or inject fixtures or clocks. No real HR transactions,
communications or physical print jobs are performed.
Zero document overflow alone does not prove containment: native scrolling must
reach hidden columns/rows, and nested legends/values must fit their own cards
without sibling overlap. Final images are inspected before the cost command;
a missed critical defect withdraws provisional acceptance and triggers another
bounded repair in the same app.

Every corrective prompt is at most 180 whitespace-delimited words and covers
at most three demonstrated issue groups. First-output evidence is sealed
before guidance. There is no arbitrary authoring-time or credit cutoff; optional
repairs stop once joint acceptance passes. There is no paid Cowork self-review.

After complete review and final app screenshots, one final cumulative `/cost`
reading supplies the completed-app cost, including all authoring and guidance.
No interim or repair-only cost is published. Authoring counts include the
initial build, corrections and clarifications; free cost commands and external
review are excluded. Approximate authoring time excludes review gaps and is not
pure model compute time.

Final repairs are verified in the owner preview; they are not claimed
republished. Exact initial published resources are removed only after final
evidence, with unrelated resources verified unchanged. Successful exact prompt
sequences are sealed for future use. The separately authorized
[Test 5 showcase](test-05-showcase.html) publishes the byte-identical supplied
HTML reference, its fictional fixture and selected final app-only dashboard
screenshots. It supports desktop/light and mobile/dark comparison by setting.
The embedded reference is a sandboxed standalone demo, not a generated Cowork
deployment. Image hashes and capture dates are in the public asset manifest.
Transcripts, generated app source, account/task/resource identifiers, cost
screenshots and other raw evidence remain private. The measurement export
retains frozen prompt/reference hashes and allowlisted capability verdicts.

The original adaptive cohort and any later guided replay remain separate.
Guided replay has not started; donor selection follows closure of all five
original slots. One observation per setting does not establish repeatability,
internal worker identity, causal model pricing or guaranteed future savings.

## Scope and attribution

Testing was conducted by **Ryan Bowen**, with automation assistance from
**GitHub Copilot**.

This is a small demonstration benchmark using synthetic apps in demo
tenant(s), not a production-cost study. Unique
prompts, use cases, app complexity, tenant setup, model/reasoning settings and
follow-up work can produce substantially different costs. These results offer
only a brief indication of app-building consumption, not a quote, guaranteed
price or complete running cost.

The managed app builder is available in Copilot Cowork and Copilot Studio.
This report measures Cowork `/cost` credits only; Copilot Studio costs were
not measured separately. The top-of-page banner retains this distinction.
The overview capability-summary cards focus on Test 3; Tests 1 and 2 retain
their cost results and detailed capability evidence.

## Comparing app complexity

Complexity is described qualitatively across three separate dimensions:
screen/form behavior, business workflow and live integration. It is not a
measured complexity index, and build cost is not used as a proxy for complexity.

Test 1 has five core views, single-device requests, a single approval step,
stock revalidation and confirmed reset over virtual/local synthetic data.
Test 2 has four views and a narrower submit-and-track workflow, but must
actually read and write two SharePoint lists, independently read back the
saved record and retain it after reopening.

Test 3 has seven screens and seven related virtual business tables: Devices,
DemoPeople, Requests, RequestLines, Activity, ApprovalHistory and Assets.
Multi-line forms, manager/finance transitions, approval-policy changes,
revisions, cancellation, atomic fulfilment, per-unit assets and conditional
returns interact with saved records and history. Light/dark themes and local
preferences add UI state. Four planned stages evolve the same app without
resetting its data; this is more than a one-prompt screen-generation exercise.

The live integration in Test 3 is only a read-only current-user profile.
Business data, approval actors and inventory remain simulated. Thus Test 3
is a more complex workflow prototype, not a stronger demonstration of live
business-backend integration than Test 2, nor an enterprise-production
readiness claim. Real authorization, concurrency and scale are not established.

Test 4 adds versioned policy snapshots, integer-pence discounts/VAT, budget
reservations, penny-conserving partial fulfilment, stale-view conflicts and
atomic price imports across eight areas. It deliberately omits Test 3's live
profile connector and asset returns. It is a different stress workload, not
an all-features superset or a comparable scoring scale.

The comparison describes target scope. Actual per-run verdicts retain
unverified or incomplete capabilities; acceptance does not make every target
feature a pass. Apply these distinctions to original and additional cohorts
without altering their recorded costs, scores or review-method differences.

## Comparing cost with delivered capability

Overview cards report how many apps met that test's acceptance criteria,
not a shared quality score. A score of 95 in one test is not evidence of a
better app than a score of 90 in a different test: their checks and scope differ.
Within a matched cohort, all models and reasoning settings use the same
checklist; it is the per-app outcome, not the scoring standard, that can vary.
Recorded per-run scores and verified/unverified capabilities remain in each
test's detailed results.

Use independently verified checks, not spending or a completion message, to
identify capability differences. Compare models and Medium/High reasoning
within the same cohort. Keep original/additional observations and methods
separate; show every reading when presenting a configuration mean.

Equal scores do not necessarily mean equal features: two apps can miss different
checks, or differ within a partially verified check. Identify the actual
feature gain, trade-off or unresolved coverage rather than ranking by score
alone. Do not interpret Test 1's changed persistence scoring as improvement.

Additional builds are fresh apps, not successive versions of an app. Their
cost/coverage differences are descriptive, with changed publication permission,
not causal price effects or evidence that a model improved over time.
An accepted app's final corrective work can demonstrate a concrete usability
improvement, but its cumulative cost difference is not a general repair price.

These conclusions concern the completed apps shown. They do not estimate a
model's success rate or prove equal aesthetics, code quality, maintainability,
security, scalability or production reliability. No credits-per-quality-point
ratio is used: rubric points are not a universal value scale.

### Interpreting observed credit differences

The model-selection examples compare settings within the same original cohort.
Every observation contributing to either setting must have the same verified
check-ID set; equal point totals alone are insufficient. Do not combine original
and additional cohorts or compare different Test 3/Test 4 rubrics as equivalent
quality.

For a lower-cost setting and a higher-cost comparison, the reported difference
is `higher mean final credits - lower mean final credits`; the percentage is
`difference / higher mean final credits * 100`, rounded to one decimal place.
For one observation per setting, these are individual readings, not repeated
means. Show sample sizes and correction counts alongside the difference.

Final task credits include corrective guidance, but do not price external
review labor. An observed credit difference is not a monetary saving, a
guaranteed future price reduction or an isolated model tariff. The measured
check set also cannot establish equivalent aesthetics or overall app quality.

All registered Test 4 settings shared the complete-app target, acceptance
criteria and planned time/correction limits. There was no credit-price stopping
rule or exemption for a higher-cost setting; shared planned controls do not
claim that every registered run received every stage. A missing accepted result
is not a zero score or proof of incapability. Platform and observation limits
must not be relabeled as model-quality findings. Test 4's original GPT and
follow-up Opus Medium results support a specific observed cost comparison for
the same verified core target. They are not matched repeats, a five-way ranking,
equal advanced coverage or a success-rate estimate.

## Controls shared within each test

Test 4's later follow-up registration adds one fresh app for each of the four
settings without an accepted original result: Sonnet Medium, Opus High, Opus
Medium and Sonnet High. The original GPT result and historical outcomes are
preserved. Follow-ups use the identical four prompts, fixtures, core/advanced
rubrics and authoring/correction bounds, and are explicitly labelled rather
than presented as original first attempts.

All five settings now have actual final readings in the chart and outcome
table; all four registered follow-ups are closed. Final spend for an
unaccepted outcome is labelled spend-only and
excluded from the accepted-app average and accepted capability matrix. This
distinguishes testing coverage from accepted-result coverage without inventing
zero costs, zero capability scores or a model success rate. Sonnet Medium,
Opus High and Sonnet High did not deliver an independently observed complete
target within the unchanged authoring bounds. Their partial review evidence
and post-stop previews are not promoted to completed-target scores. Their
final credits are actual free `/cost` observations, not estimates or partial
stage readings. The report retains 34 accepted observations, including two
Test 4 apps with a mean of 2,255.5 credits.

Start a fresh Cowork task and managed app for each run. Select and record the
model and reasoning setting before submitting the same authoring specification.
Keep the starting fixture, requirements and acceptance checks consistent within
the cohort. Record any clarification or corrective prompts separately from
planned changes.

Exercise the actual app independently, including its forms, data operations,
reopening and desktop/mobile layout. A completion message or compile alone is
not acceptance. Evaluate the delivered app against its rubric: at least 85/100
and every required critical check. Unverified checks earn no points.

## Test 1: virtual-data procurement

Five builds: Sonnet 5 Medium twice, and Sonnet 5 High, Opus 5 Medium and Opus 5
High once each. Virtual/local business data is allowed. Managed-store
persistence, AC14 in the original rubric, is not a required acceptance gate for
this test. Recorded 95/100 point totals are retained without adding storage
points.

Each task receives one identical native self-review prompt in addition to
independent external checking. Its final `/cost` therefore includes build and
native review. Do not treat this as the external-only review method of Tests 2
and 3.

Materials: [build prompt](benchmark/build-prompt.txt),
[native review prompt](benchmark/review-prompt.txt),
[original rubric](benchmark/acceptance.json). Apply the virtual-data scope above;
the effective final check requirements are also recorded in `data.json`.

## Test 2: SharePoint-connected procurement

Five configurations are each tested twice, in reverse-order blocks:
Sonnet 5 Medium, Opus 5 High, Opus 5 Medium, Sonnet 5 High and GPT 5.5 Medium.
The app contains Dashboard, Catalogue, New request and My requests.
Approval/rejection, inventory editing and app reset are outside this scope.

All ten builds use the same established demo-tenant SharePoint connection and
two synthetic lists. The starting fixture contains six device models, 25
available units and four requests, including three Pending with GBP 3,800
pending value. Restore and independently verify that fixture between runs.

Acceptance uses ten checks worth ten points each. Each app must read the live
lists, submit a complete Pending request, support independent server read-back,
and retain the request after reopening. The controller reviews the app and
backend externally; no native self-review prompt is submitted. Clarification
and corrective prompts are each limited to two per run and are counted.

Materials: [build prompt template](benchmark/test-02-sharepoint-clean/build-prompt-template.txt),
[acceptance rubric](benchmark/test-02-sharepoint-clean/acceptance.json).
The public template uses placeholders rather than private connection values.

## Test 3: complex multi-prompt virtual-data app

Five demo apps, with one observation per configuration:
Sonnet 5 Medium, Opus 5 High, Opus 5 Medium, Sonnet 5 High and GPT 5.5 Medium.
Every app receives four identical ordered prompts:

1. [Build the foundation](benchmark/test-03/stage-01-build.txt): related virtual
   business tables, multiple screens and forms.
2. [Connect the existing app](benchmark/test-03/stage-02-connect-profile.txt):
   real read-only Office 365 Users profile and a profile page.
3. [Extend workflows](benchmark/test-03/stage-03-change.txt): virtual approvals,
   asset lifecycle, themes and preferences.
4. [Change requirements](benchmark/test-03/stage-04-change.txt): approval policy,
   revisions, cost centres and responsive refinement.

Business data and approval actors remain synthetic and virtual/local.
Only the signed-in user's profile is live; synthetic personas do not replace
that identity. Independently verify the profile response and displayed fields
inside the app, including after reopening.

Use the same twenty-check rubric, with five points per check. Final scoring
covers the complete four-stage target. Verify data continuity across the planned
changes, approval thresholds, required comments, stock-constrained fulfilment,
returns, themes, preferences, desktop/mobile layouts and sampled contrast.

Review is external-only. Three planned follow-up prompts per app are not
corrections or self-reviews. Count any additional clarification or corrective
guidance separately and retain it in the final task cost.

Materials: [protocol](benchmark/test-03/protocol.json),
[rubric](benchmark/test-03/acceptance.json),
[policy carryover case](benchmark/test-03/policy-carryover-case.json),
[low-stock crossing case](benchmark/test-03/low-stock-crossing-case.json).

## Additional external-review cost cross-check

The additional virtual sample contains one observation each for Sonnet Medium,
Opus High, Sonnet High and GPT Medium. The connected sample contains those four
settings plus Opus Medium. All nine use independent external review, zero native
self-review prompts, and at most two narrowly evidenced corrections or two
clarifications. No configuration mean is represented as a repeatability estimate.

The virtual business specification includes approvals/rejections, stock
transitions, oversell prevention, confirmed reset and explicitly allowed
browser-local persistence. The same twenty checks are applied within this
sample, including actual reload/reopen persistence. Its recorded 100/100 scores
do not retrospectively increase the original Test 1 scores.

The connected business scope is the same four-view app as original Test 2.
Restore the exact six-device/four-request fixture before each build, save one
complete Pending request, independently read it back and reopen it. Invalid
submissions must leave every backend row unchanged. Desktop and actual 390px
mobile controls and contained table scrolling are exercised. BC09 remains
unverified and receives no points.

The virtual observations span two frozen prompt versions: `virtual-v1` for
Sonnet Medium/Opus High and `virtual-v2` for Sonnet High/GPT Medium. Their business
requirements are identical; version 2 explicitly permits private owner-only
publication. All five additional connected builds use the same
`sharepoint-v2` prompt with that permission. Current completed-result acceptance
allows private publication, not sharing. Preserve original prompt identities
and historical evidence rather than rewriting the earlier protocol.

Final review, actual final cost and the conversation archive are retained before
user-authorized deletion of an exact published app. Only the controller's owned
synthetic SharePoint request is then removed; original fixture rows, lists and
connections are preserved. Cleanup is not a billed Cowork prompt.

Raw additional means are **1,530 / 4 = 382.5** virtual and
**3,612 / 5 = 722.4** SharePoint credits. The equal-model-mix comparison uses one
observation for each of the four common settings: **382.5 versus 692.5**.
It excludes the SharePoint-only Opus Medium value from the matched mean, not
from its published results. Different business scope and publication permissions
still prevent treating the difference as an isolated connector price.

## Additional Test 3 observations

Three completed four-stage observations add GPT Medium, Sonnet High and Opus
High to the original complex-app results, in that relative order. Use the
same account and tenant, exact synthetic fixture, policy-carryover
and low-stock scenarios, twenty checks and critical gates. The foundation
[prompt](benchmark/test-03-repeat/stage-01-build.txt) changes only permission
for private owner-only publication; the three subsequent prompts are identical
to the originals. Do not silently rewrite the original protocol or pool its
five-setting mean with a differently composed sample.

Each result has one fresh task/app, three planned follow-ups, at most two
evidenced corrective prompts and zero native self-review prompts. Review is
external-only. Preserve the first complete-target assessment before any
subsequent correction, and retain final observed cost, profile reread,
saved-state reopening, screenshots and conversation archive before deleting
only the exact authorized published app.

Preference coverage distinguishes Save/Cancel and persistence from actual
consumption. A density change requires a measured row-height or spacing
difference of at least 0.5 CSS pixels; floating-point noise is not a pass.
Default department must demonstrably affect a new-request department or a
department filter, rather than just retain a selected value. Report the
specific surface actually exercised.

The additional mean is **5,567 / 3 = 1,855.7** credits (rounded). The matched
original mean is **6,306 / 3 = 2,102** for those same settings. Their descriptive
six-observation mean is **11,873 / 6 = 1,978.8**. Each paired setting has two
observations, with labelled publication permissions; these are neither
reliability estimates nor isolated model-pricing effects.

## Test 4: capability stress

The Device Allocation and Policy Hub uses four frozen prompts on one app:

1. [Foundation](benchmark/test-04/stage-01-build.txt): eight areas, related
   virtual tables, app-isolated persistence and stable multi-line editing.
2. [Workflows](benchmark/test-04/stage-02-workflows.txt): versioned approvals,
   separation of duties, budget reservation, fulfilment and revisions.
3. [Policy and pricing](benchmark/test-04/stage-03-policy-change.txt): V2 routing,
   exact-pence discount/VAT arithmetic, proportional costs and both themes,
   while preserving existing V1 requests and history.
4. [Resilience](benchmark/test-04/stage-04-resilience.txt): competing stale views,
   at-most-once actions, all-or-nothing price imports and accessible interaction.

Use only synthetic actors and virtual/local business data, with no live
business or profile connector. Four device models start with 31 units and
two draft requests worth 60,000 pence. Operations and Engineering have separate
300,000/150,000-pence budgets. V1 requires finance strictly above 100,000;
V2 strictly above 75,000. Existing submissions retain their captured policy.

The controller reviews the native owner preview, not a model self-review.
Before any reset, exercise the original V1 pending and approved carryovers
created before the policy change. Subsequent independent cases use the
confirmed in-app reset only; never inject missing history or patch app source.
Competing-view cases use the same browser context and native task, leaving
the second view intentionally stale before its attempted action.
If the app automatically refreshes that form first, record the actual behavior
and the missing stale prerequisite. Do not disable its listeners, inject records
or award a full pass by substituting a refreshed form. A held confirmation can
still exercise a separate stale-operation subcase without satisfying every
other subcase.

The [core rubric](benchmark/test-04/acceptance.json) has ten ten-point checks.
Acceptance requires at least 85 and every critical check: effectively at least
90 with this rubric. The separate
[advanced rubric](benchmark/test-04/advanced-acceptance.json) also has ten
ten-point checks, but is not an acceptance gate. Every frozen subcase must be
verified for a pass; failed, blocked and untested checks each earn zero.
Do not combine these into a universal quality or credits-per-point score.

Freeze first-complete-target cost and both scorecards before later guidance.
At most two corrective prompts and two clarifications are allowed per app.
Early corrections count toward first-target cost; later incremental cost is
final minus first-target credits. Preserve earlier unaffected evidence and
identify targeted post-correction rechecks, rather than implying that all
historical scenarios were recreated.

The final interaction sample covers actual 1440px and 390px widths in both
themes, including visible controls, field errors, contrast, focus and bounded
table scrolling. The exact advanced keyboard scenario asks for request search
and a rejection-reason dialog, while the authoring prompts permit other
interfaces. If that exact scenario cannot be exercised, retain a coverage gap
instead of automatically treating a different interface as a requested-feature
defect or awarding partial points.

Two accepted observations are shown: original GPT Medium at 1,573 and follow-up
Opus Medium at 2,938 credits. Both have core 100; their advanced scores are 90 and 70.
The observed 1,365-credit difference is 46.5% of 2,938. It includes guidance and
compares the same frozen core target, but does not imply equal advanced
coverage or matched repeatability. Opus's exact stale-form gaps must remain
distinct from its measured light-error contrast defect. The selected Cowork
setting does not establish every internal builder worker's model.
Controlled local stale-state simulations and sampled accessibility are not
production concurrency, authorization or conformance certification.

Final task cost, final app/cost images, structured checks and the conversation
archive are retained before owner-authorized deletion of the exact generated
resource. Later edits were reviewed in the same native owner preview and are
not claimed republished.

Materials: [protocol summary](benchmark/test-04/protocol.json),
[exact scenarios](benchmark/test-04/scenarios.json), and the frozen prompts
and rubrics linked above. This protocol summary is a sanitized projection;
the prompt, rubric and scenario files retain their frozen bytes.

## Cost and timing

Record the actual native `/cost` card after authoring checkpoints and at the
end of each task. The reported unit is credits and the confirmed scope is the
Cowork task. Preserve the observed numbers rather than estimating token prices
or substituting another product's billing.

All stage readings are cumulative. The **final reading is the task total**;
do not add earlier readings to it. Final costs include any guidance and, for
original Test 1, native review. Shared setup and app runtime are not included in these
build-task totals. Do not infer a currency conversion or a complete tenant bill.

The headline **average final credits per app** is the arithmetic mean of all
final task readings in the selected original cohort: 3,219 / 5 = 643.8 for Test 1,
4,580 / 10 = 458 for Test 2, and 9,990 / 5 = 1,998 for Test 3. Each build receives
equal weight, including repeated settings. This describes the observed
model/reasoning mix, not a single model's price. Per-configuration averages use
only that configuration's builds; individual readings remain visible.
The separate Test 4 mean is **(1,573 + 2,938) / 2 = 2,255.5**. Its denominator
is two accepted observations, not the five registered settings. Spend-only and
unfinished outcomes are excluded; this mean is not comparable to Test 3's
different five-setting workload.

Reported **build time per app** sums the recorded authoring turns: initial build,
planned updates, clarifications and corrections. Match each prompt submission
to its completion observation by prompt identifier. Use the first observed idle
timestamp, before the monitor's additional idle-confirmation wait. Where a last
busy sample exists, it supplies the lower bound; the displayed time uses the
first-idle upper endpoint, rounded to seconds.

This excludes external review, screenshot-taking gaps, time between prompts,
`/cost` commands and native self-review. Both an initial clarification question
and the response to its answer are included, but time waiting for the answer
between turns is not. Waits within an authoring turn remain included, so this is
an approximate observed authoring duration, not pure model compute time.

All authoring prompts must have completion observations to report a build total.
Missing or partial coverage remains unavailable, never zero or a substitute
test-session elapsed time. Original Test 1 lacks build-completion timing.
Original Tests 2 and 3 and the additional observations have complete authoring
coverage; the bounds and counts are in
`data.json` under each run's `timing.build`.

## Interpretation and evidence

Compare configurations within the same test. Different scope, backend and
review method prevent treating cross-test differences as the isolated cost of
a model or connector. Original Test 2 has two observations per setting and one
additional observation per setting under amended publication permission;
the original Test 3 cohort has one per setting, with a second published
observation for GPT Medium, Sonnet High and Opus High.
These sample sizes do not establish reliable variance, success rates or future
prices. Order, caching, service load and internal builder routing may influence
the observations.

Acceptance is not full coverage or production readiness. The report retains
the exact final scores and capability gaps, including unverified profile
fault/Retry coverage in Test 3 and connection-error/no-fallback coverage in
Test 2. Synthetic personas and sequential stock checks do not establish real
authorization or concurrent safety.

Exact prompts, visible outputs, screenshots, checks and cost observations are
retained privately. Public exports contain completed measurements and safe
benchmark materials, not credentials, account-bearing screenshots, tenant or
connection identifiers, or private app/task links.