{ "schemaVersion": 1, "builtAt": "2026-10-19T11:06:40.870Z", "harness": "1.49.0", "title": "A3 Aggregator: release confidence", "score": { "subject": "Main to RC forecast", "candidate": "sha-2bff81a", "baseline": "17.0.0", "ranAt": "2026-10-19T10:46:23.293Z", "report": "main-preview/reports/2026-10-19T10-46-23Z-release/report.html", "runUrl": "https://github.com/tutors-sdk/tutors-release-harness/actions/runs/39236680101", "gate": "FAIL", "why": "FAIL because 8 differences have no claim and no PR declared them (#501's CHANGELOG line says (image-hardening, image-manifest, metrics); startup moved; #481's CHANGELOG line says (network); #505's CHANGELOG line says (dom, focus); persistence moved). The other 84 are declared in its PR's CHANGELOG line (#505: 82, #501: 2): nothing is owed on main, and the release author claims them from those lines when the release is cut. A CHANGELOG line that names what moved, a Rule that claims it, or a fix on main turns this to PASS; the gap below says where each came from.", "rcs": null, "band": null, "note": "No RCS: Gate FAIL. The Gate wins: no number talks a FAIL back on. The dimensions are shown for the 5 Whys, not for a decision.", "dimensions": [ { "id": "claim-coverage", "name": "Claim coverage", "weight": 20, "score": 0, "status": "measured", "lost": 20 }, { "id": "noise-health", "name": "Noise health", "weight": 15, "score": 80, "status": "measured", "lost": 3 }, { "id": "statistical-margin", "name": "Statistical margin", "weight": 10, "score": 100, "status": "measured", "lost": 0 }, { "id": "rehearsals", "name": "Rehearsals", "weight": 10, "score": 100, "status": "measured", "lost": 0 }, { "id": "test-signal", "name": "Test signal", "weight": 15, "score": 10, "status": "measured", "lost": 13.5 }, { "id": "traceability", "name": "Requirements traceability", "weight": 10, "score": null, "status": "not measured", "lost": null, "reason": "needs the changelog, the EARS files and the claims side by side: pass --traceability ; changes.json already reports 13 orphan changes between the tags (a changelog entry with no diff, or a diff with no entry), a floor breach once it is measured" }, { "id": "change-risk", "name": "Change risk", "weight": 15, "score": 0, "status": "measured", "lost": 15 }, { "id": "post-deploy", "name": "Post-deploy history", "weight": 5, "score": null, "status": "not measured", "lost": null, "reason": "needs the last release's post-deploy record: pass --post-deploy " } ], "scorecard": { "score": 40, "grade": "D" }, "glance": 2, "lastRelease": { "candidate": "17.0.0-rc.1", "baseline": "16.2.2", "verdict": "WARN", "ranAt": "2026-10-15T12:02:12.207Z", "report": "release-records/reports/2026-10-15T12-02-12Z-release/report.html" }, "aa": { "verdict": "PASS", "ranAt": "2026-10-19T09:47:56.867Z", "report": "noise/reports/2026-10-19T09-47-56Z-noise/report.html" }, "postDeploy": { "red": 0, "runs": 32, "since": "2026-10-15T09:07:50Z", "lastConclusion": "success", "deployedAt": "2026-10-15T09:07:50Z", "release": "17.0.0", "latest": "https://github.com/tutors-sdk/tutors-release-harness/actions/runs/39212328998", "issue": { "number": 131, "url": "https://github.com/tutors-sdk/tutors-release-harness/issues/131", "openedAt": "2026-10-17T11:59:23Z" } }, "openCountermeasures": 2 }, "background": [ "Tutors ships four apps (reader, catalogue, live, time) as signed images. The release harness runs each candidate beside production, diffs everything a student can observe, and stops the line unless every difference is claimed by a Rule or a changelog entry.", "This A3 reads every run the harness has kept: 54 runs from 2026-09-26 to 2026-10-19 (Release candidate 5, Main to RC forecast 35, Nightly A/A 14), the kaizen register (3 filled 5 Whys), and the workflows' own history on GitHub. It is rebuilt with the site, so it always describes the newest evidence.", "Why it matters: production is 17.0.0; main (sha-2bff81a) is what the next release would ship. Every unclaimed difference is a claim or a fix that release will need, and every red check after a deploy is a question about what students are running." ], "current": { "facts": [ "Comparing main at sha-2bff81a (Main to RC forecast) against production 17.0.0, run 2026-10-19 10:46 UTC. 103 differences: claims explain 11; #505 declared 82 in its CHANGELOG line; #501 declared 2 in its CHANGELOG line; #501 contributed 4 (#501's CHANGELOG line says (image-hardening, image-manifest, metrics); startup moved); #478 most likely contributed 4 (#481's CHANGELOG line says (network); #505's CHANGELOG line says (dom, focus); persistence moved); 0 unexplained.", "Gate FAIL on the Main to RC forecast sha-2bff81a beside 17.0.0, so no RCS.", "Last release candidate kept: 17.0.0-rc.1 beside 16.2.2, WARN on 2026-10-15.", "Post-deploy since 17.0.0 was deployed: green, rollback issue #131 open.", "A/A: PASS on 2026-10-19.", "Quality: Speed look, Metrics look, Tests look.", "Real courses: 1 of 5 real courses load on main sha-2bff81a, 1 on production 17.0.0; none is broken by main alone." ], "paretos": [ { "id": "unclaimed", "title": "Unclaimed differences, by cause", "unit": "differences", "total": 92, "bars": [ { "label": "dom: semantic DOM differs", "value": 41, "cumulative": 0.44565217391304346, "vital": true }, { "label": "focus: keyboard order changed", "value": 41, "cumulative": 0.8913043478260869, "vital": true }, { "label": "startup: ready slower on b", "value": 4, "cumulative": 0.9347826086956522, "vital": false, "note": "1 distinct, once in each of 4 images" }, { "label": "persistence: RPC realtime_claim_live_publish — N row(s) on a…", "value": 3, "cumulative": 0.967391304347826, "vital": false }, { "label": "metrics: new series on b", "value": 2, "cumulative": 0.9891304347826086, "vital": false }, { "label": "persistence: RPC realtime_watch — N row(s) on a, N on b", "value": 1, "cumulative": 1, "vital": false } ], "vitalFew": 2, "vitalShare": 0.8913043478260869, "source": [ { "label": "Main to RC forecast sha-2bff81a: report.json", "href": "main-preview/reports/2026-10-19T10-46-23Z-release/report.html" } ], "caption": "2 causes of 6 carry 89% of the 92 unclaimed differences. dom: semantic DOM differs alone is 45%." }, { "id": "line-stops", "title": "Why the line stops, every red run about the monorepo", "unit": "stops", "total": 65, "bars": [ { "label": "Main to RC forecast: unclaimed differences", "value": 33, "cumulative": 0.5076923076923077, "vital": true }, { "label": "Post-deploy: production differs from the recorded release", "value": 18, "cumulative": 0.7846153846153846, "vital": true }, { "label": "Main to RC forecast: stale claims", "value": 8, "cumulative": 0.9076923076923077, "vital": true }, { "label": "Release candidate: unclaimed differences", "value": 4, "cumulative": 0.9692307692307692, "vital": false }, { "label": "Release candidate: stale claims", "value": 2, "cumulative": 1, "vital": false } ], "vitalFew": 3, "vitalShare": 0.9076923076923077, "source": [ { "label": "40 kept runs of the monorepo (index.json of the release and forecast streams; the harness's own A/A and failures left out)" }, { "label": "18 failed post-deploy runs since the deploy of 2026-10-15 (GitHub Actions)", "href": "https://github.com/tutors-sdk/tutors-release-harness/actions/workflows/post-deploy.yml" } ], "caption": "65 stops over the kept runs and the post-deploy runs since the current release was deployed, about the monorepo and production only: the harness's own A/A and its failures are not counted. 3 causes carry 91%: Main to RC forecast: unclaimed differences; Post-deploy: production differs from the recorded release; Main to RC forecast: stale claims." }, { "id": "confidence", "title": "Where confidence is lost, points of 100", "unit": "points", "total": 66.5, "bars": [ { "label": "Claim coverage", "value": 20, "cumulative": 0.3007518796992481, "vital": true, "note": "5 deductions, e.g. unclaimed: dom /: reader:home: semantic DOM differs (+1 −1 lines at line 10)" }, { "label": "Change risk", "value": 15, "cumulative": 0.5263157894736842, "vital": true, "note": "19 deductions, e.g. PR #476 changed Reader, Development but no changelog entry under v2bff81a779424d44aab12c4e4074a75c4457eac8 names it" }, { "label": "Test signal", "value": 13.5, "cumulative": 0.7293233082706767, "vital": true, "note": "3 deductions, e.g. mutation score 72.73% on @tutors/i18n, below 80%" }, { "label": "Requirements traceability (not measured)", "value": 10, "cumulative": 0.8796992481203008, "vital": true, "note": "up to 10: needs the changelog, the EARS files and the claims side by side: pass --traceability ; changes.json already reports 13 orphan changes between the tags (a changelog entry with no diff, or a diff with no entry), a floor breach once it is measured", "unknown": true }, { "label": "Post-deploy history (not measured)", "value": 5, "cumulative": 0.9548872180451128, "vital": false, "note": "up to 5: needs the last release's post-deploy record: pass --post-deploy ", "unknown": true }, { "label": "Noise health", "value": 3, "cumulative": 1, "vital": false, "note": "4 deductions, e.g. mask realtime-rest-fallback-warning fired nothing in this run (a mask that never fires is a mask to delete)" } ], "vitalFew": 4, "vitalShare": 0.8796992481203008, "source": [ { "label": "confidence.json", "href": "main-preview/reports/2026-10-19T10-46-23Z-release/confidence.json" } ], "caption": "52 points lost on the dimensions measured, and 15 more are not measured on this run (hatched: a ceiling, never scored). The vital few: Claim coverage, Change risk, Test signal, Requirements traceability (not measured)." }, { "id": "change-risk", "title": "Change risk findings, by rule", "unit": "findings", "total": 19, "bars": [ { "label": "a diff and its changelog entry disagree", "value": 13, "cumulative": 0.6842105263157895, "vital": true, "note": "130 points" }, { "label": "a hotspot file touched", "value": 3, "cumulative": 0.8421052631578947, "vital": true, "note": "9 points" }, { "label": "churn above the app's median", "value": 1, "cumulative": 0.8947368421052632, "vital": false, "note": "5 points" }, { "label": "dependency", "value": 1, "cumulative": 0.9473684210526315, "vital": false, "note": "5 points" }, { "label": "production lines with too few test lines", "value": 1, "cumulative": 1, "vital": false, "note": "10 points" } ], "vitalFew": 2, "vitalShare": 0.8421052631578947, "source": [ { "label": "changes.json (v17.0.0..2bff81a)", "href": "main-preview/reports/2026-10-19T10-46-23Z-release/changes.json" } ], "caption": "19 findings over 16 PRs; a diff and its changelog entry disagree: 13 of 16." } ], "valueStream": { "stages": [ { "id": "merge", "name": "Merge to main", "where": "monorepo pull requests", "processMs": null, "n": 0, "note": "review and merge: people's time, not measured here" }, { "id": "build", "name": "Build and sign images", "where": "monorepo image-build.yml", "processMs": 225000, "n": 85, "firstPass": { "green": 85, "runs": 87 }, "note": "four images to Quay, signed with cosign, SBOM attested" }, { "id": "aa", "name": "Nightly A/A", "where": "nightly-noise.yml", "processMs": 674000, "n": 40, "firstPass": { "green": 40, "runs": 48 }, "note": "production against itself: the noise floor" }, { "id": "forecast", "name": "Main to RC forecast", "where": "main-preview.yml", "processMs": 776000, "n": 43, "firstPass": { "green": 43, "runs": 44 }, "note": "main judged as if it were a candidate" }, { "id": "gate", "name": "Release gate (Jidoka)", "where": "release.yml", "processMs": 737000, "n": 2, "firstPass": { "green": 2, "runs": 5 }, "note": "stops the line on any unclaimed difference" }, { "id": "deploy", "name": "Promote and deploy", "where": "monorepo deploy.yml, and today Netlify", "processMs": null, "n": 0, "note": "1 deploy(s) announced to the harness" }, { "id": "verify", "name": "Post-deploy verify", "where": "post-deploy.yml", "processMs": 71000, "n": 75, "firstPass": { "green": 75, "runs": 100 }, "note": "production against the recorded release" } ], "waits": [ { "before": 1, "ms": 3000, "label": "merge to image build" }, { "before": 3, "ms": 8182293, "label": "images ready to forecast" }, { "before": 4, "ms": 342268663, "label": "since the last release gate" }, { "before": 5, "ms": 431057002, "label": "production 17.0.0 unchanged" } ], "inventory": [ { "before": 0, "count": 16, "label": "PRs merged since v17.0.0, not in production" }, { "before": 4, "count": 92, "label": "unclaimed differences the next release owes" }, { "before": 6, "count": 2, "label": "open countermeasures" } ], "leadMs": 358459870, "processMs": 1738000, "flowEfficiency": 0.004848520421546769, "notMeasured": [] } }, "goal": [ { "metric": "Gate on the newest forecast", "now": "FAIL", "target": "PASS", "met": false, "source": "confidence.json" }, { "metric": "Unclaimed differences", "now": "92", "target": "0", "met": false, "source": "report.json" }, { "metric": "Claims: known and gaps", "now": "11 claimed, 92 unclaimed (10% covered)", "target": "every difference claimed or fixed", "met": false, "source": "report.json" }, { "metric": "Real courses load on main (course corpus)", "now": "1 of 5 courses, 155 of 161 pages (production: 1 of 5)", "target": "every corpus course", "met": false, "source": "course-capture.yml, course-check.json" }, { "metric": "Nothing a student uses is worse on main (real courses)", "now": "2 page(s) and 0 journey(s) worse, 0 page(s) better", "target": "0 worse", "met": false, "source": "course-check.json, both sides (src/course/compare.ts)" }, { "metric": "Release Confidence Score", "now": "none (Gate wins)", "target": "75 or more (Amber), on the way to 90 (Green)", "met": false, "source": "docs/lean.md, bands" }, { "metric": "Post-deploy", "now": "green", "target": "green on a confirmed build", "met": true, "source": "post-deploy.yml" }, { "metric": "Releases scored (C0 evidence gate)", "now": "1", "target": "3", "met": false, "source": "scoreboard branch" }, { "metric": "Countermeasures verified closed (C4 evidence gate)", "now": "1 of 3", "target": "every one, in a release", "met": false, "source": "kaizen/README.md" } ], "rca": [ { "id": "gate", "question": "Why does main (sha-2bff81a) stop the line?", "answer": "92 unclaimed differences. 2 causes of 6 carry 89% of the 92 unclaimed differences. dom: semantic DOM differs alone is 45%.", "evidence": [ { "label": "report.html", "href": "main-preview/reports/2026-10-19T10-46-23Z-release/report.html" }, { "label": "the run", "href": "https://github.com/tutors-sdk/tutors-release-harness/actions/runs/39236680101" } ], "depth": { "kind": "5 whys", "file": "2026-09-29-sha-185e87f-gate.md", "title": "sha-185e87f, Gate FAIL on the Main to RC forecast" }, "pareto": "unclaimed" }, { "id": "post-deploy", "question": "Why is post-deploy red?", "answer": "Post-deploy is green on its latest run (32 runs since 17.0.0 was deployed (2026-10-15 09:07 UTC)).", "evidence": [ { "label": "newest post-deploy run", "href": "https://github.com/tutors-sdk/tutors-release-harness/actions/runs/39212328998" }, { "label": "post-deploy runs", "href": "https://github.com/tutors-sdk/tutors-release-harness/actions/workflows/post-deploy.yml" }, { "label": "issue #131", "href": "https://github.com/tutors-sdk/tutors-release-harness/issues/131" } ], "depth": { "kind": "5 whys", "file": "2026-09-29-16.2.2-rollback.md", "title": "16.2.2, post-deploy red on every run" } }, { "id": "decisions", "question": "Which differences were decisions, not failures, and why were they made?", "answer": "2 fixes on b across 1 page (console). A fix never gates: it is a decision, and a claim is where its why is written. None has a claim saying why yet. Fixed with no note yet: catalogue:home (Cannot read properties of undefined (reading 'from')). That is good news; a claim is an optional note that the fix was meant, and it switches the glance item off.", "evidence": [ { "label": "report.html, differences", "href": "main-preview/reports/2026-10-19T10-46-23Z-release/report.html#differences" }, { "label": "src/claims/decisions.ts", "href": "https://github.com/tutors-sdk/tutors-release-harness/blob/main/src/claims/decisions.ts" } ], "depth": { "kind": "5 whys", "file": "2026-09-29-sha-185e87f-fixed-on-b.md", "title": "sha-185e87f, claimed fixes read as unclaimed and their claims as stale" } }, { "id": "confidence", "question": "Where is confidence lost?", "answer": "52 points lost on the dimensions measured, and 15 more are not measured on this run (hatched: a ceiling, never scored). The vital few: Claim coverage, Change risk, Test signal, Requirements traceability (not measured). The Gate is FAIL, so none of this is a score: it is where the next release has to earn its points.", "evidence": [ { "label": "confidence.json", "href": "main-preview/reports/2026-10-19T10-46-23Z-release/confidence.json" }, { "label": "docs/lean.md, visual management", "href": "https://github.com/tutors-sdk/tutors-release-harness/blob/main/docs/lean.md#visual-management-one-number-broken-down" } ], "depth": { "kind": "evidence stops", "why": "2 dimensions not measured on this run: Requirements traceability (needs the changelog, the EARS files and the claims side by side: pass --traceability ; changes.json already reports 13 orphan changes between the tags (a changelog entry with no diff, or a diff with no entry), a floor breach once it is measured); Post-deploy history (needs the last release's post-deploy record: pass --post-deploy )" }, "pareto": "confidence" }, { "id": "change-risk", "question": "Why is change risk 0 of 100?", "answer": "19 findings over 16 PRs. 0 of 16 were merged with no approving review, and any one of them breaches the floor, so change risk cannot rise above it until review coverage does. The points themselves go to a diff and its changelog entry disagree (13), a hotspot file touched (3), churn above the app's median (1), dependency (1), production lines with too few test lines (1).", "evidence": [ { "label": "changes.json", "href": "main-preview/reports/2026-10-19T10-46-23Z-release/changes.json" } ], "depth": { "kind": "evidence stops", "why": "the next why is the monorepo's branch rules and review load, which the kept reports do not carry; open a 5 Whys with harness why --finding two-of-three-below-75:change-risk once the scoreboard fires it" }, "pareto": "change-risk" }, { "id": "aa", "question": "Can the A/A be trusted to say the harness sees clearly?", "answer": "20 of 37 A/A nights were clean, 9 noisy (2 on 2026-09-22, 13 on 2026-09-23, 1 on 2026-09-24, 1 on 2026-09-27, 1 on 2026-10-01, 2 on 2026-10-03, 1 on 2026-10-16, 1 on 2026-10-19, 1 on 2026-10-19) and 8 degraded. 2 clean nights ran on a harness older than 1.4.4, which could not tell a clean night from a blind one (a journey failing on both sides), so their clean cannot be counted as seeing. 4 of 28 masks fired nothing on the latest night: each is a blind spot to review.", "evidence": [ { "label": "noise-history.json", "href": "noise/noise-history.json" }, { "label": "PR #22", "href": "https://github.com/tutors-sdk/tutors-release-harness/pull/22" } ], "depth": { "kind": "evidence stops", "why": "the detection that answers it shipped in 1.4.4; every night since says degraded when it is" } }, { "id": "flow", "question": "How long does a change wait to reach production, and where?", "answer": "The oldest merge to main not yet in production has waited 4.1d, while the machine time of the whole stream is 29m: flow efficiency 0.48%. The wait is inventory: 16 PRs merged since v17.0.0, not in production; 92 unclaimed differences the next release owes; 2 open countermeasures.", "evidence": [ { "label": "the value stream map below" } ], "depth": { "kind": "evidence stops", "why": "release cadence is a decision, not a defect: the map shows the batch, the team chooses its size" } } ], "fiveWhys": [ { "file": "2026-09-29-sha-185e87f-gate.md", "title": "sha-185e87f, Gate FAIL on the Main to RC forecast", "trigger": "Gate FAIL", "release": "sha-185e87f", "whys": [ { "n": 1, "question": "Why did the Gate FAIL on sha-185e87f?", "answer": "The harness's own trace (facts, each one a link or a file):\n\n- **Gate:** FAIL (sha-185e87f beside 16.2.2): 893 unclaimed diff(s); 4 claim(s) matched nothing and should be removed from the changelog\n- **Report:** [report.html](https://tutors-sdk.github.io/tutors-release-harness/main-preview/reports/2026-09-28T08-59-53Z-release/report.html), `report.json` `compare.unclaimed`\n- **Unclaimed differences:** 893 of 986 hunks (93 more are info: fixes, never gating)\n- **The same Gate on the two forecasts before it:** 892 unclaimed (2026-09-28T03:45Z, sha-017ceb6), 865 (2026-09-27T09:14Z, sha-54a8f83): the line has stopped on every forecast since the Main to RC stream began." }, { "n": 2, "question": "Why are there 893 unclaimed differences?", "answer": "Because one kind of difference is most of them. By artefact and change (the A3 Aggregator's Pareto, from\n`report.json`): 696 are `sbom` \"package removed\" and 12 are `sbom` \"package bumped\", so 708 of 893 (79%) are the\nimages' software bill of materials. The 696 removals are 174 distinct packages (`npm`, `corepack`, `yarn`,\n`node-gyp`, `@npmcli/*`, `@sigstore/*`, `tar` 7.5.11 and the rest of npm's own dependency tree) counted once in each\nof the four images. The 12 bumps are `node` 22.23.2 to 22.23.3, `tar` and `tzdata`, again once per image. The next\nbars are the Paper UI rebuild (`network` 85, `dom` 33, `screenshot` 16, `focus` 15, `logs` 15): 185 in all." }, { "n": 3, "question": "Why did 174 packages leave every image?", "answer": "The monorepo's runtime image stopped shipping a package manager. Commit\n[4070315](https://github.com/tutors-sdk/tutors-mono-repo/commit/4070315d6b4f7658f072f2e3fc16cc8071910d4a) \"build:\npatch Debian and drop package managers from the runtime image\" removes `npm`, `npx`, `corepack` and `yarn` from the\nruntime stage of the `Dockerfile` (the Trivy gate failed on npm's bundled `tar` and `brace-expansion`). It reached\nmain with [#143](https://github.com/tutors-sdk/tutors-mono-repo/pull/143) on 2026-09-19 and is not in v16.2.2\n(`caa53d0`, `git merge-base --is-ancestor` says no). It is an intended, security-positive change: the same run\nreports 72 advisories gone on b (18 per image, `vulns` info hunks), and the Debian patch is the 12 bumps." }, { "n": 4, "question": "Why is an intended change like that unclaimed?", "answer": "Because nothing tells the release how to claim a removed tool. `release/claims.yaml` on main holds 4 claims, all\nfor the accessibility fixes of #313 (mono-repo #364), and a claim is written in the release PR (SOP step 4), so the\nforecast shows the claims the next release still owes. For this change the owed claim is awkward to write:\n`claims/README.md` shows how to claim one bumped package and a framework bump by glob (`*/@sveltejs/*`), but a\nremoved tool takes its whole dependency tree with it, 174 packages with unrelated names, one hunk per package per\nimage. The precise claim is not obvious and the easy one (`*/**`) is a broad claim that needs an approver." } ], "chainEndsAt": 4, "endsIn": "process", "kind": "claim guidance", "countermeasure": "claims/README.md gains \"A tool removed from an image\": one narrow `sbom` claim per removed tree, its scope a brace list of the removed packages generated from report.json, citing the CHANGELOG entry, so the next release claims all 708 SBOM hunks without a broad claim.", "owner": "Leigh Griffin", "due": "2026-10-06", "verifiedBy": "", "inPlace": "claims/README.md", "check": "the release gate's SBOM differences are all claimed by narrow claims: none unclaimed, and claim hygiene flags no sbom claim. 17.0.0 did not show it (release gate 2026-10-15 12:02): 212 SBOM differences unclaimed, and one sbom claim covers 696.", "status": "in place", "answers": [ "The harness's own trace (facts, each one a link or a file):\n\n- **Gate:** FAIL (sha-185e87f beside 16.2.2): 893 unclaimed diff(s); 4 claim(s) matched nothing and should be removed from the changelog\n- **Report:** [report.html](https://tutors-sdk.github.io/tutors-release-harness/main-preview/reports/2026-09-28T08-59-53Z-release/report.html), `report.json` `compare.unclaimed`\n- **Unclaimed differences:** 893 of 986 hunks (93 more are info: fixes, never gating)\n- **The same Gate on the two forecasts before it:** 892 unclaimed (2026-09-28T03:45Z, sha-017ceb6), 865 (2026-09-27T09:14Z, sha-54a8f83): the line has stopped on every forecast since the Main to RC stream began.", "Because one kind of difference is most of them. By artefact and change (the A3 Aggregator's Pareto, from\n`report.json`): 696 are `sbom` \"package removed\" and 12 are `sbom` \"package bumped\", so 708 of 893 (79%) are the\nimages' software bill of materials. The 696 removals are 174 distinct packages (`npm`, `corepack`, `yarn`,\n`node-gyp`, `@npmcli/*`, `@sigstore/*`, `tar` 7.5.11 and the rest of npm's own dependency tree) counted once in each\nof the four images. The 12 bumps are `node` 22.23.2 to 22.23.3, `tar` and `tzdata`, again once per image. The next\nbars are the Paper UI rebuild (`network` 85, `dom` 33, `screenshot` 16, `focus` 15, `logs` 15): 185 in all.", "The monorepo's runtime image stopped shipping a package manager. Commit\n[4070315](https://github.com/tutors-sdk/tutors-mono-repo/commit/4070315d6b4f7658f072f2e3fc16cc8071910d4a) \"build:\npatch Debian and drop package managers from the runtime image\" removes `npm`, `npx`, `corepack` and `yarn` from the\nruntime stage of the `Dockerfile` (the Trivy gate failed on npm's bundled `tar` and `brace-expansion`). It reached\nmain with [#143](https://github.com/tutors-sdk/tutors-mono-repo/pull/143) on 2026-09-19 and is not in v16.2.2\n(`caa53d0`, `git merge-base --is-ancestor` says no). It is an intended, security-positive change: the same run\nreports 72 advisories gone on b (18 per image, `vulns` info hunks), and the Debian patch is the 12 bumps.", "Because nothing tells the release how to claim a removed tool. `release/claims.yaml` on main holds 4 claims, all\nfor the accessibility fixes of #313 (mono-repo #364), and a claim is written in the release PR (SOP step 4), so the\nforecast shows the claims the next release still owes. For this change the owed claim is awkward to write:\n`claims/README.md` shows how to claim one bumped package and a framework bump by glob (`*/@sveltejs/*`), but a\nremoved tool takes its whole dependency tree with it, 174 packages with unrelated names, one hunk per package per\nimage. The precise claim is not obvious and the easy one (`*/**`) is a broad claim that needs an approver." ] }, { "file": "2026-09-29-16.2.2-rollback.md", "title": "16.2.2, post-deploy red on every run", "trigger": "rollback", "release": "16.2.2", "whys": [ { "n": 1, "question": "Why did post-deploy FAIL against 16.2.2?", "answer": "The harness's own trace (facts, each one a link or a file):\n\n- **Verdict:** FAIL: 5 new difference(s) between production and the recorded candidate; open a rollback issue\n- **The differences:** `network` GET {{course}}/course.png on reference:lab; `console` new messages on reference:course, reference:lab and reference:note; `focus` keyboard order changed on reference:course ([#38](https://github.com/tutors-sdk/tutors-release-harness/issues/38))\n- **Production build:** NOT CONFIRMED: /version did not answer; production was built at 2026-09-18T08:06:22Z from an unnamed build ([run log](https://github.com/tutors-sdk/tutors-release-harness/actions/runs/36505701173))\n- **History:** `post-deploy.yml` concluded failure on all 64 of its runs from 2026-09-17 to 2026-09-29: 36 found no recorded release run to compare with (before the first release gate passed on 2026-09-27), 17 could not start on a push (2026-09-17 to 2026-09-21), and 11 found production different, each adding to the one open rollback issue." }, { "n": 2, "question": "Why does production differ from the recorded 16.2.2?", "answer": "Because production is not the build the harness recorded. The recording is the signed image\n`quay.io/tutors-sdk/tutors-reader:16.2.2` (digest `sha256:7567d59…`, created 2026-09-19T18:07Z from `caa53d0`).\ntutors.dev was built at 2026-09-18T08:06Z, three minutes after the release PR\n[#264](https://github.com/tutors-sdk/tutors-mono-repo/pull/264) merged `caa53d0` (08:03Z), by Netlify from source\n(`apps/reader/netlify.toml`: `pnpm --filter tutors-reader... build`). Same commit, a different build and a\ndifferent host, so the two can differ without any change to the code." }, { "n": 3, "question": "Why is production built outside the image the harness gates?", "answer": "Because the deploy path that ships the gated image is built but does not serve tutors.dev yet. The monorepo's\n`deploy.yml` pins signed digests into `deploy/k8s/overlays` and `release/deployed.json` (16.2.2, deployedAt\n2026-09-21T10:57Z) for OpenShift, and the OpenShift and AWS migration that would make those overlays production is\nplanned, not done. So the \"deployed\" dispatch that tells the harness what production serves has never been sent:\nnone of the 64 runs was a `repository_dispatch`, every judged one was the 15-minute schedule." }, { "n": 4, "question": "Why has a red check not stopped anything for twelve days?", "answer": "Because the release SOP has no step that says which build production serves before post-deploy is trusted. The\nmonitor finds the same five differences every quarter hour, the rollback issue deduplicates them by signature, and\na standing difference with no confirmed build reads as a standing alarm rather than a stop. The Jidoka rule holds\nfor the gate before a release; after it, nothing in the standard work says what to do when production is not the\ntested artefact." } ], "chainEndsAt": 4, "endsIn": "process", "kind": "SOP change", "countermeasure": "release/SOP.md's deploy step records the build production serves (the pinned digest, or while Netlify serves tutors.dev a /version naming the commit) and post-deploy judges only a confirmed build, reporting any other as \"not the tested build\" instead of a rollback.", "owner": "Leigh Griffin", "due": "2026-10-13", "verifiedBy": "17.0.0", "inPlace": "src/modes/production-build.ts", "check": "post-deploy names the build production serves and judges only that one: its report reads \"production build: match\" for the deployed release. 17.0.0 showed it: post-deploy run 38791747525 (2026-10-15 19:11) read \"production build: match (production serves the recorded candidate's commit 432d46864626)\".", "status": "closed", "answers": [ "The harness's own trace (facts, each one a link or a file):\n\n- **Verdict:** FAIL: 5 new difference(s) between production and the recorded candidate; open a rollback issue\n- **The differences:** `network` GET {{course}}/course.png on reference:lab; `console` new messages on reference:course, reference:lab and reference:note; `focus` keyboard order changed on reference:course ([#38](https://github.com/tutors-sdk/tutors-release-harness/issues/38))\n- **Production build:** NOT CONFIRMED: /version did not answer; production was built at 2026-09-18T08:06:22Z from an unnamed build ([run log](https://github.com/tutors-sdk/tutors-release-harness/actions/runs/36505701173))\n- **History:** `post-deploy.yml` concluded failure on all 64 of its runs from 2026-09-17 to 2026-09-29: 36 found no recorded release run to compare with (before the first release gate passed on 2026-09-27), 17 could not start on a push (2026-09-17 to 2026-09-21), and 11 found production different, each adding to the one open rollback issue.", "Because production is not the build the harness recorded. The recording is the signed image\n`quay.io/tutors-sdk/tutors-reader:16.2.2` (digest `sha256:7567d59…`, created 2026-09-19T18:07Z from `caa53d0`).\ntutors.dev was built at 2026-09-18T08:06Z, three minutes after the release PR\n[#264](https://github.com/tutors-sdk/tutors-mono-repo/pull/264) merged `caa53d0` (08:03Z), by Netlify from source\n(`apps/reader/netlify.toml`: `pnpm --filter tutors-reader... build`). Same commit, a different build and a\ndifferent host, so the two can differ without any change to the code.", "Because the deploy path that ships the gated image is built but does not serve tutors.dev yet. The monorepo's\n`deploy.yml` pins signed digests into `deploy/k8s/overlays` and `release/deployed.json` (16.2.2, deployedAt\n2026-09-21T10:57Z) for OpenShift, and the OpenShift and AWS migration that would make those overlays production is\nplanned, not done. So the \"deployed\" dispatch that tells the harness what production serves has never been sent:\nnone of the 64 runs was a `repository_dispatch`, every judged one was the 15-minute schedule.", "Because the release SOP has no step that says which build production serves before post-deploy is trusted. The\nmonitor finds the same five differences every quarter hour, the rollback issue deduplicates them by signature, and\na standing difference with no confirmed build reads as a standing alarm rather than a stop. The Jidoka rule holds\nfor the gate before a release; after it, nothing in the standard work says what to do when production is not the\ntested artefact." ] }, { "file": "2026-09-29-sha-185e87f-fixed-on-b.md", "title": "sha-185e87f, claimed fixes read as unclaimed and their claims as stale", "trigger": "finding", "release": "sha-185e87f", "whys": [ { "n": 1, "question": "Why is this a place to look: reader-auth:sign-in: 2 axe violations fixed on b, unclaimed?", "answer": "The harness's own trace (facts, each one a link or a file):\n\n- **Glance item:** rank 2 of 7 (fixed on b; novelty 1.00 × exposure 0.17): reader-auth:sign-in: 2 axe violations fixed on b, unclaimed; a fix nobody claimed can be a behaviour change\n- **Hunks:** `axe:reader-auth:sign-in:158` color-contrast (serious) and `axe:reader-auth:sign-in:159` document-title (serious), both severity info\n- **Nearest claim:** `axe reader-auth:sign-in`: Rule 0216: The reader shall show the sign-in page with no critical or serious WCAG 2.1 AA violations; why it did not cover: it is info-only (the claim matched nothing in this run: stale)\n- **The same run's Gate reasons:** \"4 claim(s) matched nothing and should be removed from the changelog\"; glance items 2 to 5 say the same of reference:course, reference:note and reference:topic" }, { "n": 2, "question": "Why do a fix and the claim written for it both read as unmatched?", "answer": "Because the matcher never offers an info hunk to a claim. The four claims in the monorepo's `release/claims.yaml`\n(Rules 0216, 0051 twice and 0036, added by [#364](https://github.com/tutors-sdk/tutors-mono-repo/pull/364)) name\nexactly the five `axe` hunks, and all five are severity info (\"axe violation fixed on b\"). `src/claims/matcher.ts`\nassigns only failing hunks (\"Info hunks are reported but never need a claim\"), so the four claims are never used\nand `staleClaims` lists them, and `compare.matches` carries no claim for the fixes." }, { "n": 3, "question": "Why does the glance repeat it?", "answer": "Because the glance reads the same matches. `fixedOnB` in `src/glance/rank.ts` looks for the claim of each fixed\nhunk in `compare.matches`, finds none, and words the item \"unclaimed; a fix nobody claimed can be a behaviour\nchange\". The tool asks for a claim on a fix, then calls the claim stale: the Contributor who follows the SOP is\ntold by the Gate to remove it and by the glance to add it." } ], "chainEndsAt": 3, "endsIn": "tool", "kind": "glance rule", "countermeasure": "a claim that names an info hunk is recorded as matching it, without gating it, so the glance's fixed-on-b rule reads \"claimed\" and the claim is no longer reported stale.", "owner": "Leigh Griffin", "due": "2026-10-06", "verifiedBy": "", "inPlace": "src/claims/matcher.ts", "check": "a release with a claim that names a fix on b (an info hunk) reports that claim as matching, not stale, and the glance's fixed-on-b item reads \"claimed\". Not yet shown: 17.0.0 claimed no fix on b.", "status": "in place", "answers": [ "The harness's own trace (facts, each one a link or a file):\n\n- **Glance item:** rank 2 of 7 (fixed on b; novelty 1.00 × exposure 0.17): reader-auth:sign-in: 2 axe violations fixed on b, unclaimed; a fix nobody claimed can be a behaviour change\n- **Hunks:** `axe:reader-auth:sign-in:158` color-contrast (serious) and `axe:reader-auth:sign-in:159` document-title (serious), both severity info\n- **Nearest claim:** `axe reader-auth:sign-in`: Rule 0216: The reader shall show the sign-in page with no critical or serious WCAG 2.1 AA violations; why it did not cover: it is info-only (the claim matched nothing in this run: stale)\n- **The same run's Gate reasons:** \"4 claim(s) matched nothing and should be removed from the changelog\"; glance items 2 to 5 say the same of reference:course, reference:note and reference:topic", "Because the matcher never offers an info hunk to a claim. The four claims in the monorepo's `release/claims.yaml`\n(Rules 0216, 0051 twice and 0036, added by [#364](https://github.com/tutors-sdk/tutors-mono-repo/pull/364)) name\nexactly the five `axe` hunks, and all five are severity info (\"axe violation fixed on b\"). `src/claims/matcher.ts`\nassigns only failing hunks (\"Info hunks are reported but never need a claim\"), so the four claims are never used\nand `staleClaims` lists them, and `compare.matches` carries no claim for the fixes.", "Because the glance reads the same matches. `fixedOnB` in `src/glance/rank.ts` looks for the claim of each fixed\nhunk in `compare.matches`, finds none, and words the item \"unclaimed; a fix nobody claimed can be a behaviour\nchange\". The tool asks for a claim on a fix, then calls the claim stale: the Contributor who follows the SOP is\ntold by the Gate to remove it and by the glance to add it." ] } ], "decisions": { "fixes": 2, "decided": 0, "pages": 1, "byArtefact": [ { "artefact": "console", "fixes": 2, "decided": 0 } ], "rows": [ { "artefact": "console", "scope": "catalogue:home", "what": [ "Cannot read properties of undefined (reading 'from')" ], "hunk": "main-preview/reports/2026-10-19T10-46-23Z-release/report.html#hunk-console:catalogue:home:52", "state": "undecided" } ] }, "quality": { "lights": [ { "id": "speed", "name": "Speed", "state": "look", "reading": "p95 2.03 ms on production, 2.01 ms on main; 0 of 600 requests failed; 4 slowdowns to look at, advice only: catalogue: ready slower on b: median 2366 ms → 5471 ms (+131%), p=0.012, n=5/5; live: ready slower on b: median 2362 ms → 5342 ms (+126%), p=0.012, n=5/5; reader: ready slower on b: median 2370 ms → 5392 ms (+128%), p=0.012, n=5/5; and 1 more", "rule": "No journey, page or startup significantly slower (advice: timing never fails the Gate); k6 failure rate 0 on main", "checks": [ { "id": "slower", "label": "No journey or startup significantly slower, or outside production's control limits", "state": "look", "reading": "4 slowdowns (advice, not a failure): catalogue: ready slower on b: median 2366 ms → 5471 ms (+131%), p=0.012, n=5/5; live: ready slower on b: median 2362 ms → 5342 ms (+126%), p=0.012, n=5/5; reader: ready slower on b: median 2370 ms → 5392 ms (+128%), p=0.012, n=5/5; and 1 more; control chart: every journey's median on main inside production's limits (18)", "evidence": [ { "label": "startup:catalogue/ready:105", "href": "main-preview/reports/2026-10-19T10-46-23Z-release/report.html#hunk-startup:catalogue/ready:105" }, { "label": "startup:live/ready:106", "href": "main-preview/reports/2026-10-19T10-46-23Z-release/report.html#hunk-startup:live/ready:106" }, { "label": "startup:reader/ready:107", "href": "main-preview/reports/2026-10-19T10-46-23Z-release/report.html#hunk-startup:reader/ready:107" } ] }, { "id": "load", "label": "k6 failure rate 0 on main", "state": "within reason", "reading": "p95 2.03 ms on production, 2.01 ms on main; 0 of 600 requests failed", "evidence": [ { "label": "the load", "href": "main-preview/reports/2026-10-19T10-46-23Z-release/report.html#load" } ] } ], "evidence": [ { "label": "startup:catalogue/ready:105", "href": "main-preview/reports/2026-10-19T10-46-23Z-release/report.html#hunk-startup:catalogue/ready:105" }, { "label": "startup:live/ready:106", "href": "main-preview/reports/2026-10-19T10-46-23Z-release/report.html#hunk-startup:live/ready:106" }, { "label": "startup:reader/ready:107", "href": "main-preview/reports/2026-10-19T10-46-23Z-release/report.html#hunk-startup:reader/ready:107" }, { "label": "the load", "href": "main-preview/reports/2026-10-19T10-46-23Z-release/report.html#load" } ] }, { "id": "metrics", "name": "Metrics", "state": "look", "reading": "2 unclaimed metrics hunks: time: new series on b: http_request_duration_seconds_count; time: new series on b: http_requests_total", "rule": "No unclaimed metrics hunk", "checks": [ { "id": "metrics", "label": "No unclaimed metrics hunk", "state": "look", "reading": "2 unclaimed metrics hunks: time: new series on b: http_request_duration_seconds_count; time: new series on b: http_requests_total", "evidence": [ { "label": "metrics:http_request_duration_seconds_count:54", "href": "main-preview/reports/2026-10-19T10-46-23Z-release/report.html#hunk-metrics:http_request_duration_seconds_count:54" }, { "label": "metrics:http_requests_total:55", "href": "main-preview/reports/2026-10-19T10-46-23Z-release/report.html#hunk-metrics:http_requests_total:55" } ] } ], "evidence": [ { "label": "metrics:http_request_duration_seconds_count:54", "href": "main-preview/reports/2026-10-19T10-46-23Z-release/report.html#hunk-metrics:http_request_duration_seconds_count:54" }, { "label": "metrics:http_requests_total:55", "href": "main-preview/reports/2026-10-19T10-46-23Z-release/report.html#hunk-metrics:http_requests_total:55" } ] }, { "id": "tests", "name": "Tests", "state": "look", "reading": "Harness: within reason. Monorepo (record for the judged commit, 68m old): 3 changed packages below 80%: @tutors/ui-navigators 55.11%, @tutors/i18n 72.73%, @tutors/metrics 75.93%; the Nightly within reason.", "rule": "Harness: every journey completed on both sides, the A/A clean and at most 2 days old, every mutant caught. Monorepo: the Nightly green on the commit, no changed package below 80% mutation score", "checks": [ { "id": "journeys", "half": "harness", "label": "Every journey completed on both sides", "state": "within reason", "reading": "every journey completed on both sides", "evidence": [] }, { "id": "aa", "half": "harness", "label": "The A/A it relied on clean, at most 2 days old", "state": "within reason", "reading": "A/A clean (0 differences), 58m before the run", "evidence": [ { "label": "the A/A", "href": "main-preview/reports/2026-10-19T10-46-23Z-release/report.html#noise" } ] }, { "id": "mutants", "half": "harness", "label": "The newest weekly mutants self-test caught every mutant", "state": "within reason", "reading": "16 of 16 mutants caught, 2026-10-19", "evidence": [ { "label": "self-test of 2026-10-19", "href": "https://github.com/tutors-sdk/tutors-release-harness/actions/runs/39229012328" } ] }, { "id": "nightly", "half": "monorepo", "label": "The Nightly green on the commit", "state": "within reason", "reading": "Nightly green on 2bff81a", "evidence": [ { "label": "quality.json", "href": "main-preview/reports/2026-10-19T10-46-23Z-release/quality.json" }, { "label": "the Nightly run", "href": "https://github.com/tutors-sdk/tutors-mono-repo/actions/runs/39224727587" } ] }, { "id": "mutation", "half": "monorepo", "label": "No changed package below 80% mutation score", "state": "look", "reading": "3 changed packages below 80%: @tutors/ui-navigators 55.11%, @tutors/i18n 72.73%, @tutors/metrics 75.93%", "evidence": [ { "label": "quality.json", "href": "main-preview/reports/2026-10-19T10-46-23Z-release/quality.json" }, { "label": "the Nightly run", "href": "https://github.com/tutors-sdk/tutors-mono-repo/actions/runs/39224727587" } ] } ], "halves": { "harness": "within reason", "monorepo": "look" }, "evidence": [ { "label": "the A/A", "href": "main-preview/reports/2026-10-19T10-46-23Z-release/report.html#noise" }, { "label": "self-test of 2026-10-19", "href": "https://github.com/tutors-sdk/tutors-release-harness/actions/runs/39229012328" }, { "label": "quality.json", "href": "main-preview/reports/2026-10-19T10-46-23Z-release/quality.json" }, { "label": "the Nightly run", "href": "https://github.com/tutors-sdk/tutors-mono-repo/actions/runs/39224727587" } ] } ], "record": { "commit": "2bff81a779424d44aab12c4e4074a75c4457eac8", "judged": true, "generatedAt": "2026-10-19T09:38:43Z", "ageMs": 4060293, "href": "main-preview/reports/2026-10-19T10-46-23Z-release/quality.json", "evidence": "https://github.com/tutors-sdk/tutors-mono-repo/actions/runs/39224727587" }, "note": "Tests read the monorepo's quality record for the judged commit, generated 2026-10-19 (68m before the forecast). The strip is a reading aid: it never changes the Gate, the score or an exit code." }, "courses": { "production": "17.0.0", "candidate": "sha-2bff81a", "sha": "2bff81a779424d44aab12c4e4074a75c4457eac8", "checkedAt": "2026-10-19T11:06:05Z", "runUrl": "https://github.com/tutors-sdk/tutors-release-harness/actions/runs/39239950622", "total": 5, "loadOnProduction": 1, "loadOnMain": 1, "pagesOnMain": { "ok": 155, "sampled": 161 }, "worsePages": 2, "betterPages": 0, "worseJourneys": 0, "tutorsJson": { "checked": 5, "conforming": 1, "schema": "55f3aff" }, "rows": [ { "id": "databases-2024", "title": "Database", "a": { "loaded": false, "files": { "checked": 405, "problems": 0 }, "pages": { "sampled": 40, "ok": 39, "failed": 1, "medianMs": 691, "maxMs": 891 }, "failures": [ "/lab/localhost:8194/topic-01-Conceptual/unit-1-Introduction/book-a/01: title \"Download Visual Paradigm\" not shown: locator.waitFor: Timeout 30000ms exceeded." ] }, "b": { "loaded": false, "files": { "checked": 405, "problems": 0 }, "pages": { "sampled": 40, "ok": 39, "failed": 1, "medianMs": 586, "maxMs": 764 }, "failures": [ "/lab/localhost:8194/topic-01-Conceptual/unit-1-Introduction/book-a/01: title \"Download Visual Paradigm\" not shown: locator.waitFor: Timeout 30000ms exceeded." ] }, "state": "fails on both", "medianDeltaMs": -105, "compare": { "compared": 40, "worse": [ { "path": "/video/localhost:8194/topic-00-Assignments/unit-2-assignment-2/talk-1/vNzjCvOApCA?start=5244", "type": "talk", "title": "Assignment Two Video", "reasons": [ "new accessibility violation(s): button-name" ] } ], "better": [], "journeys": { "worse": [], "better": [], "compared": 2 } }, "tutorsJson": { "schema": "55f3aff", "conforms": false, "problems": 168, "sample": [ "/ must have required property 'authLevel'", "/los/[topic-00-Assignments] must have required property 'authLevel'", "/los/[topic-00-Assignments]/los/[unit-0-repeat-ca] must have required property 'authLevel'", "/los/[topic-00-Assignments]/los/[unit-0-repeat-ca]/los/[note-1] must have required property 'authLevel'", "/los/[topic-00-Assignments]/los/[unit-0-repeat-ca]/los/[web-1] must have required property 'authLevel'", "/los/[topic-00-Assignments]/los/[unit-1-assignment-1] must have required property 'authLevel'", "/los/[topic-00-Assignments]/los/[unit-1-assignment-1]/los/[book-00-Assignment] must have required property 'authLevel'", "/los/[topic-00-Assignments]/los/[unit-2-assignment-2] must have required property 'authLevel'", "/los/[topic-00-Assignments]/los/[unit-2-assignment-2]/los/[talk-1] must have required property 'authLevel'", "/los/[topic-00-Assignments]/los/[unit-2-assignment-2]/los/[book-1] must have required property 'authLevel'" ] } }, { "id": "setu-hdip-comp-sci-2024-comp-sys", "title": "Computer Systems & Networks", "a": { "loaded": false, "files": { "checked": 493, "problems": 0 }, "pages": { "sampled": 40, "ok": 39, "failed": 1, "medianMs": 691, "maxMs": 1340 }, "failures": [ "/lab/localhost:8193/topic-08-week8/unit-1/book-3/01: title \"Raspberry Pi OS\" not shown: locator.waitFor: Timeout 30000ms exceeded." ] }, "b": { "loaded": false, "files": { "checked": 493, "problems": 0 }, "pages": { "sampled": 40, "ok": 39, "failed": 1, "medianMs": 622, "maxMs": 1251 }, "failures": [ "/lab/localhost:8193/topic-08-week8/unit-1/book-3/01: title \"Raspberry Pi OS\" not shown: locator.waitFor: Timeout 30000ms exceeded." ] }, "state": "fails on both", "medianDeltaMs": -69, "compare": { "compared": 40, "worse": [], "better": [], "journeys": { "worse": [], "better": [], "compared": 2 } }, "tutorsJson": { "schema": "55f3aff", "conforms": false, "problems": 164, "sample": [ "/ must have required property 'authLevel'", "/los/[topic-00-Assessment_Details] must have required property 'authLevel'", "/los/[topic-00-Assessment_Details]/los/[talk-1] must have required property 'authLevel'", "/los/[topic-00-Assessment_Details]/los/[talk-2] must have required property 'authLevel'", "/los/[topic-00-Assessment_Details]/los/[book-1] must have required property 'authLevel'", "/los/[topic-00-Assessment_Details-2] must have required property 'authLevel'", "/los/[topic-00-Assessment_Details-2]/los/[talk-1] must have required property 'authLevel'", "/los/[topic-00-Assessment_Details-2]/los/[archive] must have required property 'authLevel'", "/los/[topic-00-Assessment_Details-2]/los/[archive-2] must have required property 'authLevel'", "/los/[topic-000-overview] must have required property 'authLevel'" ] } }, { "id": "wit-hdip-comp-sci-2024", "title": "Higher Diploma in Computer Science 2024", "a": { "loaded": true, "files": { "checked": 6, "problems": 0 }, "pages": { "sampled": 1, "ok": 1, "failed": 0, "medianMs": 551, "maxMs": 551 }, "failures": [] }, "b": { "loaded": true, "files": { "checked": 6, "problems": 0 }, "pages": { "sampled": 1, "ok": 1, "failed": 0, "medianMs": 744, "maxMs": 744 }, "failures": [] }, "state": "loads on both", "medianDeltaMs": 193, "compare": { "compared": 1, "worse": [], "better": [], "journeys": { "worse": [], "better": [], "compared": 0 } }, "tutorsJson": { "schema": "55f3aff", "conforms": false, "problems": 28, "sample": [ "/ must have required property 'authLevel'", "/los/[unit-1] must have required property 'authLevel'", "/los/[unit-1]/los/[web-0-workshop-4] must have required property 'authLevel'", "/los/[unit-1]/los/[web-2-mobile-app-dev] must have required property 'authLevel'", "/los/[unit-1]/los/[web-3-security] must have required property 'authLevel'", "/los/[unit-1]/los/[web-4-schedules] must have required property 'authLevel'", "/los/[unit-1]/los/[web-5-placement] must have required property 'authLevel'", "/los/[unit-1]/los/[web-6-workshop-1-3] must have required property 'authLevel'", "/los/[unit-1]/los/[web-7-cert] must have required property 'authLevel'", "/los/[unit-2] must have required property 'authLevel'" ] } }, { "id": "wit-hdip-comp-sci-2024-full-stack-1", "title": "Full Stack Web Development", "a": { "loaded": false, "files": { "checked": 616, "problems": 0 }, "pages": { "sampled": 40, "ok": 38, "failed": 2, "medianMs": 843, "maxMs": 1353 }, "failures": [ "/lab/localhost:8192/unit-1-node/topic-02-hapi/unit-2/book-1-playtime-0-2-0/04: title \"Sessions\" not shown: locator.waitFor: Timeout 30000ms exceeded.", "/lab/localhost:8192/unit-6-sveltekit/topic-19-svelte-routing/side-unit/book-2-donation-svelte-1/05: title \"Signup\" not shown: locator.waitFor: Timeout 30000ms exceeded." ] }, "b": { "loaded": false, "files": { "checked": 616, "problems": 0 }, "pages": { "sampled": 40, "ok": 38, "failed": 2, "medianMs": 847, "maxMs": 1367 }, "failures": [ "/lab/localhost:8192/unit-1-node/topic-02-hapi/unit-2/book-1-playtime-0-2-0/04: title \"Sessions\" not shown: locator.waitFor: Timeout 30000ms exceeded.", "/lab/localhost:8192/unit-6-sveltekit/topic-19-svelte-routing/side-unit/book-2-donation-svelte-1/05: title \"Signup\" not shown: locator.waitFor: Timeout 30000ms exceeded." ] }, "state": "fails on both", "medianDeltaMs": 4, "compare": { "compared": 40, "worse": [ { "path": "/topic/localhost:8192/unit-1-node/topic-01-node", "type": "topic", "title": "01: Node", "reasons": [ "new console error: Permissions policy violation: compute-pressure is not allowed in this document." ] } ], "better": [], "journeys": { "worse": [], "better": [], "compared": 2 } }, "tutorsJson": { "schema": "55f3aff", "conforms": true, "problems": 0, "sample": [] } }, { "id": "wit-hdip-comp-sci-2024-programming", "title": "Programming Fundamentals", "a": { "loaded": false, "files": { "checked": 578, "problems": 0 }, "pages": { "sampled": 40, "ok": 38, "failed": 2, "medianMs": 590, "maxMs": 969 }, "failures": [ "/topic/localhost:8191/topic-00-assignments/hold-unit-2/previous assignments/2022: title \"Constraints\" not shown: locator.waitFor: Timeout 30000ms exceeded.", "/lab/localhost:8191/topic-02-selection_and_iteration/unit-02a-if-stmt-and-events/book/05: title \"Mouse events\" not shown: locator.waitFor: Timeout 30000ms exceeded." ] }, "b": { "loaded": false, "files": { "checked": 578, "problems": 0 }, "pages": { "sampled": 40, "ok": 38, "failed": 2, "medianMs": 666, "maxMs": 879 }, "failures": [ "/topic/localhost:8191/topic-00-assignments/hold-unit-2/previous assignments/2022: title \"Constraints\" not shown: locator.waitFor: Timeout 30000ms exceeded.", "/lab/localhost:8191/topic-02-selection_and_iteration/unit-02a-if-stmt-and-events/book/05: title \"Mouse events\" not shown: locator.waitFor: Timeout 30000ms exceeded." ] }, "state": "fails on both", "medianDeltaMs": 76, "compare": { "compared": 40, "worse": [], "better": [], "journeys": { "worse": [], "better": [], "compared": 2 } }, "tutorsJson": { "schema": "55f3aff", "conforms": false, "problems": 265, "sample": [ "/ must have required property 'authLevel'", "/los/[topic-00-assignments] must have required property 'authLevel'", "/los/[topic-00-assignments]/los/[unit-1] must have required property 'authLevel'", "/los/[topic-00-assignments]/los/[unit-1]/los/[talk-1-submissions-a1images2023] must have required property 'authLevel'", "/los/[topic-00-assignments]/los/[unit-1]/los/[book-01assignment-1] must have required property 'authLevel'", "/los/[topic-00-assignments]/los/[unit-1]/los/[panelvideo] must have required property 'authLevel'", "/los/[topic-00-assignments]/los/[unit-2] must have required property 'authLevel'", "/los/[topic-00-assignments]/los/[unit-2]/los/[book-02assesments] must have required property 'authLevel'", "/los/[topic-00-assignments]/los/[unit-2]/los/[panelvideo] must have required property 'authLevel'", "/los/[topic-00-assignments]/los/[unit-3] must have required property 'authLevel'" ] } } ], "summary": "1 of 5 real courses load on main sha-2bff81a, 1 on production 17.0.0; none is broken by main alone." }, "gap": { "comparing": "main at sha-2bff81a (Main to RC forecast) against production 17.0.0, run 2026-10-19 10:46 UTC", "sentence": "103 differences: claims explain 11; #505 declared 82 in its CHANGELOG line; #501 declared 2 in its CHANGELOG line; #501 contributed 4 (#501's CHANGELOG line says (image-hardening, image-manifest, metrics); startup moved); #478 most likely contributed 4 (#481's CHANGELOG line says (network); #505's CHANGELOG line says (dom, focus); persistence moved); 0 unexplained.", "explanation": { "total": 103, "claimed": { "hunks": 11, "rows": [ { "label": "Rule 0330: When a student opens a learning object, the reader shall send the learning event to the reader's server without publishing it over Realtime from the browser. — Rule 0330: the reader sends each learning event to its own server (POST /api/live/activity) instead of broadcasting it from the browser (#481)", "rule": "0330", "reason": "Rule 0330: the reader sends each learning event to its own server (POST /api/live/activity) instead of broadcasting it from the browser (#481)", "hunks": 8 }, { "label": "Rule 0330: When a student opens a learning object, the reader shall send the learning event to the reader's server without publishing it over Realtime from the browser. — Rule 0330: moving learning events from browser Realtime to the reader's server adds one immutable chunk on reference:course and reference:note (#478, #481)", "rule": "0330", "reason": "Rule 0330: moving learning events from browser Realtime to the reader's server adds one immutable chunk on reference:course and reference:note (#478, #481)", "hunks": 2 }, { "label": "Rule 0331: When the reader's server receives a learning event, the reader shall publish it only to the activity feeds a live dashboard has watched in the last 90 seconds. — Rule 0331: the reader's server claims the topics to publish to, and charges the day's Live budget, through realtime_claim_live_publish (#481)", "rule": "0331", "reason": "Rule 0331: the reader's server claims the topics to publish to, and charges the day's Live budget, through realtime_claim_live_publish (#481)", "hunks": 1 } ] }, "contributed": { "hunks": 8, "rows": [ { "prs": [ { "pr": 501, "sha": "a22cc9adf34f2b1ccef57dcfd59e7546e38e86db", "title": "feat(image): declare a HEALTHCHECK and expose request SLIs from start (image-hardening, image-manifest, metrics)", "url": "https://github.com/tutors-sdk/tutors-mono-repo/pull/501", "files": 1, "direct": 1, "declares": { "artefacts": [ "image-hardening", "image-manifest", "metrics" ], "entry": "feat(image): declare a HEALTHCHECK and expose request SLIs from start", "source": "title" } } ], "hunks": 4, "causes": [ { "key": "startup: ready slower on b", "hunks": 4 } ], "firstSeen": [ { "id": "2026-10-19T08-41-29Z-release", "ranAt": "2026-10-19T08:41:29.517Z" } ], "missed": [ { "pr": 501, "sha": "a22cc9adf34f2b1ccef57dcfd59e7546e38e86db", "title": "feat(image): declare a HEALTHCHECK and expose request SLIs from start (image-hardening, image-manifest, metrics)", "url": "https://github.com/tutors-sdk/tutors-mono-repo/pull/501", "files": 1, "direct": 1, "declares": { "artefacts": [ "image-hardening", "image-manifest", "metrics" ], "entry": "feat(image): declare a HEALTHCHECK and expose request SLIs from start", "source": "title" } } ] }, { "prs": [ { "pr": 478, "sha": "7c4028ade19fde26ea11b236e2b16dff572f1a5e", "title": "Route every Realtime channel through a budgeted port", "url": "https://github.com/tutors-sdk/tutors-mono-repo/pull/478", "files": 13, "direct": 1 }, { "pr": 481, "sha": "3613ee3644e9aa2a10c9fa8f0b404d0cd3e372f0", "title": "Tutors Live: publish from the reader's server within a daily Realtime budget", "url": "https://github.com/tutors-sdk/tutors-mono-repo/pull/481", "files": 6, "direct": 1, "declares": { "artefacts": [ "network" ], "entry": "Tutors Live activity is published by the reader's server, only to watched feeds, within a daily Realtime budget", "source": "body" } }, { "pr": 476, "sha": "90152846e9fea43cfe37ec0489a038ce9f2bfa8f", "title": "Give each whiteboard room a Realtime budget", "url": "https://github.com/tutors-sdk/tutors-mono-repo/pull/476", "files": 1, "direct": 1 }, { "pr": 483, "sha": "1279669f9afe1db619edf837270137b2c6c0ba54", "title": "Release 17.0.0 to main and pin production to it", "url": "https://github.com/tutors-sdk/tutors-mono-repo/pull/483", "files": 1, "direct": 1 }, { "pr": 505, "sha": "83d4ccf7f5ca2440bbc94224e40c5bfef3e410d9", "title": "fix(a11y): button names contain their visible text", "url": "https://github.com/tutors-sdk/tutors-mono-repo/pull/505", "files": 8, "direct": 0, "declares": { "artefacts": [ "dom", "focus" ], "entry": "Screen readers announce the Preferences button as \"Preferences\", and the search and close buttons announce their keyboard shortcuts", "source": "body" } }, { "pr": 480, "sha": "9bf447dee21d5d88653e92ba52ae7a603412ab42", "title": "Tutors Time: read only the chosen days of a course's calendar", "url": "https://github.com/tutors-sdk/tutors-mono-repo/pull/480", "files": 1, "direct": 0 }, { "pr": 487, "sha": "d3d72277634afc7370a4432eb3fdd64d57e5bf9a", "title": "fix(community): show an empty catalogue without Supabase instead of logging errors", "url": "https://github.com/tutors-sdk/tutors-mono-repo/pull/487", "files": 1, "direct": 0 } ], "hunks": 4, "causes": [ { "key": "persistence: RPC realtime_claim_live_publish — # row(s) on a, # on b", "hunks": 3 }, { "key": "persistence: RPC realtime_watch — # row(s) on a, # on b", "hunks": 1 } ], "firstSeen": [ { "id": "2026-10-18T02-06-17Z-release", "ranAt": "2026-10-18T02:06:17.110Z" }, { "id": "2026-10-18T07-39-15Z-release", "ranAt": "2026-10-18T07:39:15.422Z" } ], "missed": [ { "pr": 481, "sha": "3613ee3644e9aa2a10c9fa8f0b404d0cd3e372f0", "title": "Tutors Live: publish from the reader's server within a daily Realtime budget", "url": "https://github.com/tutors-sdk/tutors-mono-repo/pull/481", "files": 6, "direct": 1, "declares": { "artefacts": [ "network" ], "entry": "Tutors Live activity is published by the reader's server, only to watched feeds, within a daily Realtime budget", "source": "body" } }, { "pr": 505, "sha": "83d4ccf7f5ca2440bbc94224e40c5bfef3e410d9", "title": "fix(a11y): button names contain their visible text", "url": "https://github.com/tutors-sdk/tutors-mono-repo/pull/505", "files": 8, "direct": 0, "declares": { "artefacts": [ "dom", "focus" ], "entry": "Screen readers announce the Preferences button as \"Preferences\", and the search and close buttons announce their keyboard shortcuts", "source": "body" } } ] } ] }, "declared": { "hunks": 84, "rows": [ { "pr": { "pr": 505, "sha": "83d4ccf7f5ca2440bbc94224e40c5bfef3e410d9", "title": "fix(a11y): button names contain their visible text", "url": "https://github.com/tutors-sdk/tutors-mono-repo/pull/505", "files": 8, "direct": 0, "declares": { "artefacts": [ "dom", "focus" ], "entry": "Screen readers announce the Preferences button as \"Preferences\", and the search and close buttons announce their keyboard shortcuts", "source": "body" } }, "hunks": 82, "causes": [ { "key": "dom: semantic DOM differs (+# −# lines at line #)", "hunks": 41 }, { "key": "focus: keyboard order changed (# stops on a, # on b)", "hunks": 41 } ], "firstSeen": [ { "id": "2026-10-17T07-12-45Z-release", "ranAt": "2026-10-17T07:12:45.685Z" } ] }, { "pr": { "pr": 501, "sha": "a22cc9adf34f2b1ccef57dcfd59e7546e38e86db", "title": "feat(image): declare a HEALTHCHECK and expose request SLIs from start (image-hardening, image-manifest, metrics)", "url": "https://github.com/tutors-sdk/tutors-mono-repo/pull/501", "files": 1, "direct": 0, "declares": { "artefacts": [ "image-hardening", "image-manifest", "metrics" ], "entry": "feat(image): declare a HEALTHCHECK and expose request SLIs from start", "source": "title" } }, "hunks": 2, "causes": [ { "key": "metrics: new series on b", "hunks": 2 } ], "firstSeen": [ { "id": "2026-10-19T08-41-29Z-release", "ranAt": "2026-10-19T08:41:29.517Z" } ] } ] }, "unexplained": { "hunks": 0, "rows": [] }, "rules": [ { "rule": "0582", "journey": "rule-0582-signed-in-lab", "exercises": [ "0011", "0250", "0252", "0260", "0303", "0330" ], "hunks": 9, "artefacts": [ "dom", "focus", "persistence" ] }, { "rule": "0591", "journey": "rule-0591-time-labs", "exercises": [ "0319" ], "hunks": 8, "artefacts": [ "dom", "focus" ] }, { "rule": "0584", "journey": "rule-0584-quiz-notebook-talk", "exercises": [ "0047", "0049", "0050" ], "hunks": 6, "artefacts": [ "dom", "focus" ] }, { "rule": "0588", "journey": "rule-0588-content-lock", "exercises": [ "0052", "0053", "0054" ], "hunks": 6, "artefacts": [ "dom", "focus" ] }, { "rule": "0589", "journey": "rule-0589-live", "exercises": [ "0330", "0335" ], "hunks": 6, "artefacts": [ "dom", "focus", "persistence" ] }, { "rule": "0590", "journey": "rule-0590-time-calendar", "exercises": [ "0319" ], "hunks": 6, "artefacts": [ "dom", "focus" ] }, { "rule": "0585", "journey": "rule-0585-preferences", "exercises": [ "0037", "0038", "0039" ], "hunks": 4, "artefacts": [ "dom", "focus" ] }, { "rule": "0586", "journey": "rule-0586-private-course", "exercises": [ "0011", "0267" ], "hunks": 4, "artefacts": [ "dom", "focus" ] }, { "rule": "0583", "journey": "rule-0583-presence", "exercises": [ "0046", "0064", "0280" ], "hunks": 3, "artefacts": [ "dom", "focus", "persistence" ] }, { "rule": "0587", "journey": "rule-0587-calendar-llm", "exercises": [ "0042", "0284" ], "hunks": 2, "artefacts": [ "dom", "focus" ] }, { "rule": "0592", "journey": "rule-0592-whiteboard", "exercises": [ "0275", "0276" ], "hunks": 2, "artefacts": [ "dom", "focus" ] } ], "history": { "runs": 15, "first": "2026-10-15T18:04:41.578Z", "last": "2026-10-19T10:14:10.750Z" } } }, "timing": { "journeys": [ { "journey": "anonymous-student-searches", "nights": [ { "ranAt": "2026-10-15T21:21:41.791Z", "production": 2829, "main": 2614 }, { "ranAt": "2026-10-16T07:11:14.011Z", "production": 2999, "main": 2843 }, { "ranAt": "2026-10-17T07:12:45.685Z", "production": 2898, "main": 2839 }, { "ranAt": "2026-10-17T09:04:10.937Z", "production": 2620, "main": 2681 }, { "ranAt": "2026-10-17T09:25:47.556Z", "production": 2843, "main": 2744 }, { "ranAt": "2026-10-17T09:47:54.314Z", "production": 2814, "main": 2870 }, { "ranAt": "2026-10-17T10:09:15.863Z", "production": 2688, "main": 2862 }, { "ranAt": "2026-10-18T02:06:17.110Z", "production": 2695, "main": 2526 }, { "ranAt": "2026-10-18T07:39:15.422Z", "production": 2949, "main": 2963 }, { "ranAt": "2026-10-18T12:16:24.694Z", "production": 2991, "main": 2957 }, { "ranAt": "2026-10-19T02:24:07.366Z", "production": 2981, "main": 3094 }, { "ranAt": "2026-10-19T08:41:29.517Z", "production": 2882, "main": 2925 }, { "ranAt": "2026-10-19T10:14:10.750Z", "production": 2232, "main": 2107 }, { "ranAt": "2026-10-19T10:46:23.293Z", "production": 2276, "main": 2313 } ], "limits": { "n": 14, "centre": 2764.07, "mrBar": 156.38, "ucl": 3180.05, "lcl": 2348.09, "provisional": false }, "from": "nights", "state": "below LCL", "words": "main 2313ms is below the limits from production's last 14 nights (2348–3180ms): a special cause to investigate, not a failure" }, { "journey": "anonymous-student-reads-course", "nights": [ { "ranAt": "2026-10-15T21:21:41.791Z", "production": 4902, "main": 4895 }, { "ranAt": "2026-10-16T07:11:14.011Z", "production": 5101, "main": 5133 }, { "ranAt": "2026-10-17T07:12:45.685Z", "production": 4956, "main": 5032 }, { "ranAt": "2026-10-17T09:04:10.937Z", "production": 4776, "main": 4781 }, { "ranAt": "2026-10-17T09:25:47.556Z", "production": 4896, "main": 4850 }, { "ranAt": "2026-10-17T09:47:54.314Z", "production": 4925, "main": 4894 }, { "ranAt": "2026-10-17T10:09:15.863Z", "production": 4894, "main": 4873 }, { "ranAt": "2026-10-18T02:06:17.110Z", "production": 4730, "main": 4720 }, { "ranAt": "2026-10-18T07:39:15.422Z", "production": 5435, "main": 5079 }, { "ranAt": "2026-10-18T12:16:24.694Z", "production": 5050, "main": 5299 }, { "ranAt": "2026-10-19T02:24:07.366Z", "production": 5247, "main": 5257 }, { "ranAt": "2026-10-19T08:41:29.517Z", "production": 5356, "main": 5141 }, { "ranAt": "2026-10-19T10:14:10.750Z", "production": 4125, "main": 4096 }, { "ranAt": "2026-10-19T10:46:23.293Z", "production": 4463, "main": 4447 } ], "limits": { "n": 14, "centre": 4918.29, "mrBar": 294.85, "ucl": 5702.58, "lcl": 4133.99, "provisional": false }, "from": "nights", "state": "inside", "words": "main 4447ms is inside the limits from production's last 14 nights (4134–5703ms): common cause, nothing to investigate" }, { "journey": "catalogue-loads", "nights": [ { "ranAt": "2026-10-15T21:21:41.791Z", "production": 1564, "main": 1310 }, { "ranAt": "2026-10-16T07:11:14.011Z", "production": 1526, "main": 1626 }, { "ranAt": "2026-10-17T07:12:45.685Z", "production": 1564, "main": 1411 }, { "ranAt": "2026-10-17T09:04:10.937Z", "production": 1426, "main": 1434 }, { "ranAt": "2026-10-17T09:25:47.556Z", "production": 1415, "main": 1343 }, { "ranAt": "2026-10-17T09:47:54.314Z", "production": 1521, "main": 1457 }, { "ranAt": "2026-10-17T10:09:15.863Z", "production": 1386, "main": 1410 }, { "ranAt": "2026-10-18T02:06:17.110Z", "production": 1359, "main": 1360 }, { "ranAt": "2026-10-18T07:39:15.422Z", "production": 1577, "main": 1568 }, { "ranAt": "2026-10-18T12:16:24.694Z", "production": 1514, "main": 1474 }, { "ranAt": "2026-10-19T02:24:07.366Z", "production": 1537, "main": 1429 }, { "ranAt": "2026-10-19T08:41:29.517Z", "production": 1557, "main": 1486 }, { "ranAt": "2026-10-19T10:14:10.750Z", "production": 1154, "main": 1063 }, { "ranAt": "2026-10-19T10:46:23.293Z", "production": 1227, "main": 1227 } ], "limits": { "n": 14, "centre": 1451.93, "mrBar": 99.46, "ucl": 1716.5, "lcl": 1187.36, "provisional": false }, "from": "nights", "state": "inside", "words": "main 1227ms is inside the limits from production's last 14 nights (1187–1717ms): common cause, nothing to investigate" }, { "journey": "live-loads", "nights": [ { "ranAt": "2026-10-15T21:21:41.791Z", "production": 1535, "main": 1520 }, { "ranAt": "2026-10-16T07:11:14.011Z", "production": 1397, "main": 1608 }, { "ranAt": "2026-10-17T07:12:45.685Z", "production": 1501, "main": 1658 }, { "ranAt": "2026-10-17T09:04:10.937Z", "production": 1415, "main": 1536 }, { "ranAt": "2026-10-17T09:25:47.556Z", "production": 1535, "main": 1522 }, { "ranAt": "2026-10-17T09:47:54.314Z", "production": 1524, "main": 1482 }, { "ranAt": "2026-10-17T10:09:15.863Z", "production": 1405, "main": 1562 }, { "ranAt": "2026-10-18T02:06:17.110Z", "production": 1420, "main": 1465 }, { "ranAt": "2026-10-18T07:39:15.422Z", "production": 1657, "main": 1634 }, { "ranAt": "2026-10-18T12:16:24.694Z", "production": 1668, "main": 1688 }, { "ranAt": "2026-10-19T02:24:07.366Z", "production": 1681, "main": 1749 }, { "ranAt": "2026-10-19T08:41:29.517Z", "production": 1580, "main": 1748 }, { "ranAt": "2026-10-19T10:14:10.750Z", "production": 1150, "main": 1223 }, { "ranAt": "2026-10-19T10:46:23.293Z", "production": 1295, "main": 1317 } ], "limits": { "n": 14, "centre": 1483.07, "mrBar": 117.69, "ucl": 1796.13, "lcl": 1170.01, "provisional": false }, "from": "nights", "state": "inside", "words": "main 1317ms is inside the limits from production's last 14 nights (1170–1796ms): common cause, nothing to investigate" }, { "journey": "reference-course-reads", "nights": [ { "ranAt": "2026-10-15T21:21:41.791Z", "production": 7069, "main": 7065 }, { "ranAt": "2026-10-16T07:11:14.011Z", "production": 7828, "main": 7789 }, { "ranAt": "2026-10-17T07:12:45.685Z", "production": 7452, "main": 7461 }, { "ranAt": "2026-10-17T09:04:10.937Z", "production": 7624, "main": 7087 }, { "ranAt": "2026-10-17T09:25:47.556Z", "production": 7344, "main": 7285 }, { "ranAt": "2026-10-17T09:47:54.314Z", "production": 7711, "main": 7759 }, { "ranAt": "2026-10-17T10:09:15.863Z", "production": 6857, "main": 7319 }, { "ranAt": "2026-10-18T02:06:17.110Z", "production": 7264, "main": 6742 }, { "ranAt": "2026-10-18T07:39:15.422Z", "production": 8238, "main": 8017 }, { "ranAt": "2026-10-18T12:16:24.694Z", "production": 7685, "main": 7732 }, { "ranAt": "2026-10-19T02:24:07.366Z", "production": 8019, "main": 7688 }, { "ranAt": "2026-10-19T08:41:29.517Z", "production": 7809, "main": 7802 }, { "ranAt": "2026-10-19T10:14:10.750Z", "production": 6191, "main": 6028 }, { "ranAt": "2026-10-19T10:46:23.293Z", "production": 5762, "main": 6157 } ], "limits": { "n": 14, "centre": 7346.64, "mrBar": 564.08, "ucl": 8847.09, "lcl": 5846.2, "provisional": false }, "from": "nights", "state": "inside", "words": "main 6157ms is inside the limits from production's last 14 nights (5846–8847ms): common cause, nothing to investigate" }, { "journey": "replay-course-urls", "nights": [ { "ranAt": "2026-10-15T21:21:41.791Z", "production": 23572, "main": 14999 }, { "ranAt": "2026-10-16T07:11:14.011Z", "production": 19943, "main": 23220 }, { "ranAt": "2026-10-17T07:12:45.685Z", "production": 8332, "main": 7692 }, { "ranAt": "2026-10-17T09:04:10.937Z", "production": 11765, "main": 15080 }, { "ranAt": "2026-10-17T09:25:47.556Z", "production": 20270, "main": 26997 }, { "ranAt": "2026-10-17T09:47:54.314Z", "production": 26907, "main": 8017 }, { "ranAt": "2026-10-17T10:09:15.863Z", "production": 7592, "main": 18903 }, { "ranAt": "2026-10-18T02:06:17.110Z", "production": 26998, "main": 18903 }, { "ranAt": "2026-10-18T07:39:15.422Z", "production": 9712, "main": 27400 }, { "ranAt": "2026-10-18T12:16:24.694Z", "production": 23230, "main": 27077 }, { "ranAt": "2026-10-19T02:24:07.366Z", "production": 27596, "main": 22750 }, { "ranAt": "2026-10-19T08:41:29.517Z", "production": 14835, "main": 26962 }, { "ranAt": "2026-10-19T10:14:10.750Z", "production": 7325, "main": 6581 }, { "ranAt": "2026-10-19T10:46:23.293Z", "production": 7804, "main": 7250 } ], "limits": { "n": 14, "centre": 16848.64, "mrBar": 9881.23, "ucl": 43132.72, "lcl": 0, "provisional": false }, "from": "nights", "state": "inside", "words": "main 7250ms is inside the limits from production's last 14 nights (0–43133ms): common cause, nothing to investigate" }, { "journey": "rule-0582-signed-in-lab", "nights": [ { "ranAt": "2026-10-18T02:06:17.110Z", "production": 4531, "main": 4615 }, { "ranAt": "2026-10-18T07:39:15.422Z", "production": 5354, "main": 5057 }, { "ranAt": "2026-10-18T12:16:24.694Z", "production": 4945, "main": 4981 }, { "ranAt": "2026-10-19T02:24:07.366Z", "production": 5106, "main": 5074 }, { "ranAt": "2026-10-19T08:41:29.517Z", "production": 5027, "main": 4907 }, { "ranAt": "2026-10-19T10:14:10.750Z", "production": 3922, "main": 3878 }, { "ranAt": "2026-10-19T10:46:23.293Z", "production": 4083, "main": 4192 } ], "limits": { "n": 7, "centre": 4709.71, "mrBar": 456.33, "ucl": 5923.56, "lcl": 3495.87, "provisional": true }, "from": "nights", "state": "inside", "words": "main 4192ms is inside the limits from production's last 7 nights (3496–5924ms): common cause, nothing to investigate; provisional limits, 7 nights" }, { "journey": "rule-0583-presence", "nights": [ { "ranAt": "2026-10-18T02:06:17.110Z", "production": 11131, "main": 10716 }, { "ranAt": "2026-10-18T07:39:15.422Z", "production": 4701, "main": 4702 }, { "ranAt": "2026-10-18T12:16:24.694Z", "production": 4487, "main": 4501 }, { "ranAt": "2026-10-19T02:24:07.366Z", "production": 4586, "main": 4446 }, { "ranAt": "2026-10-19T08:41:29.517Z", "production": 4465, "main": 4351 }, { "ranAt": "2026-10-19T10:14:10.750Z", "production": 3028, "main": 3120 }, { "ranAt": "2026-10-19T10:46:23.293Z", "production": 3237, "main": 3269 } ], "limits": { "n": 7, "centre": 5090.71, "mrBar": 1418.33, "ucl": 8863.48, "lcl": 1317.95, "provisional": true }, "from": "nights", "state": "inside", "words": "main 3269ms is inside the limits from production's last 7 nights (1318–8863ms): common cause, nothing to investigate; provisional limits, 7 nights" }, { "journey": "rule-0584-quiz-notebook-talk", "nights": [ { "ranAt": "2026-10-18T02:06:17.110Z", "production": 4917, "main": 5187 }, { "ranAt": "2026-10-18T07:39:15.422Z", "production": 13039, "main": 12944 }, { "ranAt": "2026-10-18T12:16:24.694Z", "production": 13067, "main": 12996 }, { "ranAt": "2026-10-19T02:24:07.366Z", "production": 13209, "main": 13088 }, { "ranAt": "2026-10-19T08:41:29.517Z", "production": 13162, "main": 13123 }, { "ranAt": "2026-10-19T10:14:10.750Z", "production": 12642, "main": 12684 }, { "ranAt": "2026-10-19T10:46:23.293Z", "production": 12967, "main": 12880 } ], "limits": { "n": 7, "centre": 11857.57, "mrBar": 1530.67, "ucl": 15929.14, "lcl": 7786, "provisional": true }, "from": "nights", "state": "inside", "words": "main 12880ms is inside the limits from production's last 7 nights (7786–15929ms): common cause, nothing to investigate; provisional limits, 7 nights" }, { "journey": "rule-0585-preferences", "nights": [ { "ranAt": "2026-10-18T02:06:17.110Z", "production": 12060, "main": 12024 }, { "ranAt": "2026-10-18T07:39:15.422Z", "production": 6879, "main": 6497 }, { "ranAt": "2026-10-18T12:16:24.694Z", "production": 6789, "main": 6264 }, { "ranAt": "2026-10-19T02:24:07.366Z", "production": 6628, "main": 6695 }, { "ranAt": "2026-10-19T08:41:29.517Z", "production": 6139, "main": 6237 }, { "ranAt": "2026-10-19T10:14:10.750Z", "production": 4250, "main": 4006 }, { "ranAt": "2026-10-19T10:46:23.293Z", "production": 4804, "main": 4606 } ], "limits": { "n": 7, "centre": 6792.71, "mrBar": 1394, "ucl": 10500.75, "lcl": 3084.67, "provisional": true }, "from": "nights", "state": "inside", "words": "main 4606ms is inside the limits from production's last 7 nights (3085–10501ms): common cause, nothing to investigate; provisional limits, 7 nights" }, { "journey": "rule-0586-private-course", "nights": [ { "ranAt": "2026-10-18T02:06:17.110Z", "production": 4199, "main": 4141 }, { "ranAt": "2026-10-18T07:39:15.422Z", "production": 10173, "main": 10196 }, { "ranAt": "2026-10-18T12:16:24.694Z", "production": 10279, "main": 10182 }, { "ranAt": "2026-10-19T02:24:07.366Z", "production": 10090, "main": 10537 }, { "ranAt": "2026-10-19T08:41:29.517Z", "production": 10846, "main": 10329 }, { "ranAt": "2026-10-19T10:14:10.750Z", "production": 10753, "main": 10917 }, { "ranAt": "2026-10-19T10:46:23.293Z", "production": 10679, "main": 10604 } ], "limits": { "n": 7, "centre": 9574.14, "mrBar": 1198.67, "ucl": 12762.6, "lcl": 6385.69, "provisional": true }, "from": "nights", "state": "inside", "words": "main 10604ms is inside the limits from production's last 7 nights (6386–12763ms): common cause, nothing to investigate; provisional limits, 7 nights" }, { "journey": "rule-0587-calendar-llm", "nights": [ { "ranAt": "2026-10-18T02:06:17.110Z", "production": 12177, "main": 12255 }, { "ranAt": "2026-10-18T07:39:15.422Z", "production": 4832, "main": 4847 }, { "ranAt": "2026-10-18T12:16:24.694Z", "production": 4671, "main": 4642 }, { "ranAt": "2026-10-19T02:24:07.366Z", "production": 4584, "main": 4859 }, { "ranAt": "2026-10-19T08:41:29.517Z", "production": 4630, "main": 4439 }, { "ranAt": "2026-10-19T10:14:10.750Z", "production": 3191, "main": 3018 }, { "ranAt": "2026-10-19T10:46:23.293Z", "production": 3362, "main": 3382 } ], "limits": { "n": 7, "centre": 5349.57, "mrBar": 1541.5, "ucl": 9449.96, "lcl": 1249.18, "provisional": true }, "from": "nights", "state": "inside", "words": "main 3382ms is inside the limits from production's last 7 nights (1249–9450ms): common cause, nothing to investigate; provisional limits, 7 nights" }, { "journey": "rule-0588-content-lock", "nights": [ { "ranAt": "2026-10-18T07:39:15.422Z", "production": 10192, "main": 13784 }, { "ranAt": "2026-10-18T12:16:24.694Z", "production": 17255, "main": 10106 }, { "ranAt": "2026-10-19T02:24:07.366Z", "production": 13263, "main": 17099 }, { "ranAt": "2026-10-19T08:41:29.517Z", "production": 17023, "main": 16939 }, { "ranAt": "2026-10-19T10:14:10.750Z", "production": 15132, "main": 15162 }, { "ranAt": "2026-10-19T10:46:23.293Z", "production": 15687, "main": 15667 } ], "limits": { "n": 6, "centre": 14758.67, "mrBar": 3452.2, "ucl": 23941.52, "lcl": 5575.81, "provisional": true }, "from": "nights", "state": "inside", "words": "main 15667ms is inside the limits from production's last 6 nights (5576–23942ms): common cause, nothing to investigate; provisional limits, 6 nights" }, { "journey": "rule-0589-live", "nights": [ { "ranAt": "2026-10-18T07:39:15.422Z", "production": 12509, "main": 12631 }, { "ranAt": "2026-10-18T12:16:24.694Z", "production": 6426, "main": 12812 }, { "ranAt": "2026-10-19T02:24:07.366Z", "production": 12584, "main": 6697 }, { "ranAt": "2026-10-19T08:41:29.517Z", "production": 6607, "main": 6149 }, { "ranAt": "2026-10-19T10:14:10.750Z", "production": 4154, "main": 4008 }, { "ranAt": "2026-10-19T10:46:23.293Z", "production": 4691, "main": 4717 } ], "limits": { "n": 6, "centre": 7828.5, "mrBar": 4241.6, "ucl": 19111.16, "lcl": 0, "provisional": true }, "from": "nights", "state": "inside", "words": "main 4717ms is inside the limits from production's last 6 nights (0–19111ms): common cause, nothing to investigate; provisional limits, 6 nights" }, { "journey": "rule-0590-time-calendar", "nights": [ { "ranAt": "2026-10-18T07:39:15.422Z", "production": 7787, "main": 7198 }, { "ranAt": "2026-10-18T12:16:24.694Z", "production": 7127, "main": 7605 }, { "ranAt": "2026-10-19T02:24:07.366Z", "production": 7263, "main": 7510 }, { "ranAt": "2026-10-19T08:41:29.517Z", "production": 7129, "main": 6910 }, { "ranAt": "2026-10-19T10:14:10.750Z", "production": 4705, "main": 4741 }, { "ranAt": "2026-10-19T10:46:23.293Z", "production": 5237, "main": 5482 } ], "limits": { "n": 6, "centre": 6541.33, "mrBar": 777.2, "ucl": 8608.69, "lcl": 4473.98, "provisional": true }, "from": "nights", "state": "inside", "words": "main 5482ms is inside the limits from production's last 6 nights (4474–8609ms): common cause, nothing to investigate; provisional limits, 6 nights" }, { "journey": "rule-0591-time-labs", "nights": [ { "ranAt": "2026-10-18T07:39:15.422Z", "production": 9095, "main": 8555 }, { "ranAt": "2026-10-18T12:16:24.694Z", "production": 8538, "main": 8434 }, { "ranAt": "2026-10-19T02:24:07.366Z", "production": 8587, "main": 8482 }, { "ranAt": "2026-10-19T08:41:29.517Z", "production": 8782, "main": 8348 }, { "ranAt": "2026-10-19T10:14:10.750Z", "production": 5885, "main": 5465 }, { "ranAt": "2026-10-19T10:46:23.293Z", "production": 6602, "main": 6245 } ], "limits": { "n": 6, "centre": 7914.83, "mrBar": 883, "ucl": 10263.61, "lcl": 5566.05, "provisional": true }, "from": "nights", "state": "inside", "words": "main 6245ms is inside the limits from production's last 6 nights (5566–10264ms): common cause, nothing to investigate; provisional limits, 6 nights" }, { "journey": "rule-0592-whiteboard", "nights": [ { "ranAt": "2026-10-18T12:16:24.694Z", "production": 13837, "main": 13985 }, { "ranAt": "2026-10-19T02:24:07.366Z", "production": 13658, "main": 13918 }, { "ranAt": "2026-10-19T08:41:29.517Z", "production": 13907, "main": 13746 }, { "ranAt": "2026-10-19T10:14:10.750Z", "production": 13668, "main": 13643 }, { "ranAt": "2026-10-19T10:46:23.293Z", "production": 13863, "main": 13925 } ], "limits": { "n": 5, "centre": 13786.6, "mrBar": 215.5, "ucl": 14359.83, "lcl": 13213.37, "provisional": true }, "from": "nights", "state": "inside", "words": "main 13925ms is inside the limits from production's last 5 nights (13213–14360ms): common cause, nothing to investigate; provisional limits, 5 nights" }, { "journey": "student-signs-in", "nights": [ { "ranAt": "2026-10-15T21:21:41.791Z", "production": 4023, "main": 3934 }, { "ranAt": "2026-10-16T07:11:14.011Z", "production": 4173, "main": 4290 }, { "ranAt": "2026-10-17T07:12:45.685Z", "production": 4001, "main": 4002 }, { "ranAt": "2026-10-17T09:04:10.937Z", "production": 3915, "main": 3739 }, { "ranAt": "2026-10-17T09:25:47.556Z", "production": 3930, "main": 4131 }, { "ranAt": "2026-10-17T09:47:54.314Z", "production": 4005, "main": 4174 }, { "ranAt": "2026-10-17T10:09:15.863Z", "production": 4106, "main": 3885 }, { "ranAt": "2026-10-18T02:06:17.110Z", "production": 3789, "main": 3795 }, { "ranAt": "2026-10-18T07:39:15.422Z", "production": 4359, "main": 4225 }, { "ranAt": "2026-10-18T12:16:24.694Z", "production": 4335, "main": 4466 }, { "ranAt": "2026-10-19T02:24:07.366Z", "production": 4243, "main": 4186 }, { "ranAt": "2026-10-19T08:41:29.517Z", "production": 4418, "main": 4242 }, { "ranAt": "2026-10-19T10:14:10.750Z", "production": 3205, "main": 3078 }, { "ranAt": "2026-10-19T10:46:23.293Z", "production": 3253, "main": 3381 } ], "limits": { "n": 14, "centre": 3982.5, "mrBar": 233.69, "ucl": 4604.12, "lcl": 3360.88, "provisional": false }, "from": "nights", "state": "inside", "words": "main 3381ms is inside the limits from production's last 14 nights (3361–4604ms): common cause, nothing to investigate" } ] }, "countermeasures": [ { "kind": "claim guidance", "what": "claims/README.md gains \"A tool removed from an image\": one narrow `sbom` claim per removed tree, its scope a brace list of the removed packages generated from report.json, citing the CHANGELOG entry, so the next release claims all 708 SBOM hunks without a broad claim.", "owner": "Leigh Griffin", "due": "2026-10-06", "status": "in place", "from": "2026-09-29-sha-185e87f-gate.md", "rca": [ "gate" ], "inPlace": "claims/README.md", "check": "the release gate's SBOM differences are all claimed by narrow claims: none unclaimed, and claim hygiene flags no sbom claim. 17.0.0 did not show it (release gate 2026-10-15 12:02): 212 SBOM differences unclaimed, and one sbom claim covers 696.", "verifiedBy": "" }, { "kind": "SOP change", "what": "release/SOP.md's deploy step records the build production serves (the pinned digest, or while Netlify serves tutors.dev a /version naming the commit) and post-deploy judges only a confirmed build, reporting any other as \"not the tested build\" instead of a rollback.", "owner": "Leigh Griffin", "due": "2026-10-13", "status": "closed", "from": "2026-09-29-16.2.2-rollback.md", "rca": [ "post-deploy" ], "inPlace": "src/modes/production-build.ts", "check": "post-deploy names the build production serves and judges only that one: its report reads \"production build: match\" for the deployed release. 17.0.0 showed it: post-deploy run 38791747525 (2026-10-15 19:11) read \"production build: match (production serves the recorded candidate's commit 432d46864626)\".", "verifiedBy": "17.0.0" }, { "kind": "glance rule", "what": "a claim that names an info hunk is recorded as matching it, without gating it, so the glance's fixed-on-b rule reads \"claimed\" and the claim is no longer reported stale.", "owner": "Leigh Griffin", "due": "2026-10-06", "status": "in place", "from": "2026-09-29-sha-185e87f-fixed-on-b.md", "rca": [ "decisions" ], "inPlace": "src/claims/matcher.ts", "check": "a release with a claim that names a fix on b (an info hunk) reports that claim as matching, not stale, and the glance's fixed-on-b item reads \"claimed\". Not yet shown: 17.0.0 claimed no fix on b.", "verifiedBy": "" } ], "plan": [ { "what": "claims/README.md gains \"A tool removed from an image\": one narrow `sbom` claim per removed tree, its scope a brace list of the removed packages generated from report.json, citing the CHANGELOG entry, so the next release claims all 708 SBOM hunks without a broad claim.", "who": "Leigh Griffin", "when": "2026-10-06", "status": "in place" }, { "what": "a claim that names an info hunk is recorded as matching it, without gating it, so the glance's fixed-on-b rule reads \"claimed\" and the claim is no longer reported stale.", "who": "Leigh Griffin", "when": "2026-10-06", "status": "in place" }, { "what": "Score 3 real releases so the weights can be judged against the team's view (C0)", "who": "the release captain", "when": "the next three releases", "status": "1 of 3" }, { "what": "Record 2 reviewer's glances (C3)", "who": "the reviewer", "when": "the next two releases", "status": "0 of 2" } ], "followUp": [ { "check": "Every countermeasure closed in a release (Verified by)", "state": "1 of 3 closed", "met": false }, { "check": "Open countermeasures not rising (run rule)", "state": "2 open, 0 overdue", "met": true }, { "check": "The next forecast's Pareto: the top bar shrinks", "state": "dom: semantic DOM differs: 41", "met": false }, { "check": "Post-deploy green on a build it can confirm", "state": "green", "met": true } ], "sources": { "runs": 54, "streams": { "release-records": 5, "main-preview": 35, "noise": 14 }, "first": "2026-09-26T12:38:31.986Z", "last": "2026-10-19T10:46:23.293Z", "github": "read 2026-10-19T11:06:40.870Z", "kaizen": 3 } }