[workshop-sim] Workshop Simulation Report — 2026-09-20 (Run #25, 1000×Monte Carlo)

[github-actions]

Overview

• Date: 2026-09-20
• Students simulated: 46 × 1000 Monte Carlo runs
• Workshop steps available: 29/30 (simulator models 29 of 30 main steps; step count includes side-quest-adjacent pages)
• Overall success rate: 23.5% (95% Monte Carlo interval: 23.1%–23.9%)
• Highest-dropout step: 07-first-workflow (21.4% conditional dropout among 22,360 at-risk runs; 95% Monte Carlo interval: 20.8%–21.9%)
• Lowest curriculum quality step: 04-github-actions-intro.md (overall score 5.39/10)
• Learning KPI index: 2.9/10 (active_learning 4.2 · checkpoint_quality 0.0 · scaffolding 5.0)
• Model: 2026-07-survival-model-v2 / 2026-07-assumption-model-v2 (parameter hash 2024391902)
• Limitation: synthetic results reflect explicit model assumptions; intervals exclude model and population-assumption uncertainty

Part Summary

Part	Files	Mean Score	Std Dev
Part 1 — core path (lessons 00–14)	15	6.69 / 10.0	±1.74
Part 2 — advanced (lessons 15+)	15	6.06 / 10.0	±0.37
Overall corpus	30	6.37 / 10.0	—

No pages are classified as other.

Critical Findings

1. A simulator lexical-detection bug, not a content problem, was manufacturing a ~100% dropout at 08-run-your-workflow. The workflowLockPublishCueCount regex only matches present-tense "commit"/"push," but the workshop text correctly uses past-tense phrasing ("are committed and pushed to main"). Semantic re-evaluation confirmed the actual content (Steps 7, 7d, and the billing side quest) fully and correctly instructs learners to compile, commit, and push both files, and to select and configure a billing path, before Step 8. This fix alone raised the modeled overall success rate from 0% to 23.5%.
2. The real highest-dropout step is now 07-first-workflow (21.4%), driven mostly by copilot-access-missing (3,906 of 4,781 total failures) — learners without a usable Copilot license/session fail the pre-flight check before authoring even begins. This is an access barrier, not a content clarity gap.
3. 05-agentic-intro (18.7%) and 04-actions-intro (16.0%) are concept-heavy Part 1 pages with zero-scored checkpoints; their dropout is driven by agentic-concept-gap and concept-overload respectively — genuine learning barriers.
4. Learning quality health is weak even for learners who persist. The learning KPI index is only 2.9/10, dragged down entirely by checkpoint_quality = 0.0 across all 30 steps — every checkpoint uses the :white_check_mark: emoji shortcode instead of the literal ✅ glyph the shared rubric's CHECKPOINT_RE detects, so no step earns checkpoint credit even though every step has a real checklist. This is a scoring-detection gap worth flagging to the rubric maintainers, separate from actual instructional quality.
5. The single most important repair belongs to Part 1 (00–14): the 04-actions-intro → 05-agentic-intro → 07-first-workflow sequence accounts for 3 of the top 5 dropout steps and over 60% of total cohort failures, so Part 1 (mean 6.69/10, but high variance ±1.74) deserves priority over the more uniform Part 2 (6.06/10, ±0.37).

Top Repairs to Prioritize

Note: some student dropout is expected and acceptable. Repairs must maintain or improve the learning KPI index — do not lower the cognitive bar or remove practice to chase headline completion numbers.

1. Add a formative mid-page check to 04-github-actions-intro.md after the labeling exercise, with a "if you mislabeled X, re-read Y" recovery branch (completion impact: ↑ · learning KPI impact: ↑ — targets checkpoint_quality/active_learning without reducing content).
2. Surface the Copilot-access pre-flight check earlier and more prominently in 07-your-first-workflow.md (before the authoring prompt), with a direct link to license/enrollment steps, to reduce copilot-access-missing failures (completion impact: ↑ · learning KPI impact: ↔ — this is an access-barrier fix, not a scaffolding change).
3. Add one concrete before/after example (not just prose) to 05-agentic-workflows-intro.md to reduce agentic-concept-gap, paired with a short recall-check checkpoint item (completion impact: ↑ · learning KPI impact: ↑ — targets active_learning and scaffolding, the two weakest dimensions on this page).

Dropout by step

Step	At-risk runs	Conditional dropout	95% MC interval	Failure mode	Top reason
07-first-workflow	22,360	21.4%	20.8%–21.9%	Access barrier	Copilot access/license missing before authoring (3,906 of 4,781 failures)
05-agentic-intro	37,157	18.7%	18.3%–19.1%	Learning barrier	Learner doesn't internalize the deterministic-vs-agentic shift (4,692 failures)
04-actions-intro	44,256	16.0%	15.7%–16.4%	Learning barrier	Concept overload from dense 5-part Actions primer (7,099 failures)
05c-agentic-practice	30,222	12.2%	11.8%–12.6%	Learning barrier	Can't reliably classify agentic vs. deterministic tasks (3,684 failures)
05b-agentic-security	26,538	9.8%	9.5%–10.2%	Learning barrier	Doesn't grasp sandbox + safe-output two-layer security model (2,610 failures)

Curriculum quality and learning KPIs

Step file	Overall score	active_learning	checkpoint_quality	scaffolding	Learning KPI index	Lowest dimension	Repair priority
04-github-actions-intro.md	5.39	3.9	0.0	5.0	2.78	checkpoint_quality	High
05-agentic-workflows-intro.md	5.43	2.4	0.0	5.0	2.24	checkpoint_quality	High
05b-agentic-workflows-security.md	5.75	2.5	0.0	5.0	2.27	checkpoint_quality	Medium
05c-agentic-workflows-practice.md	6.22	6.2	0.0	5.0	3.62	checkpoint_quality	Medium
07-your-first-workflow.md	6.25	6.3	0.0	5.0	3.65	checkpoint_quality	High (access, not KPI)
08-run-your-workflow.md	5.67	3.0	0.0	5.0	2.45	checkpoint_quality	Low (dropout was a sim artifact)
Cohort mean (all 30 steps)	6.37	4.17	0.0	5.0	2.88	checkpoint_quality	—

Note: checkpoint_quality is 0.0 for every step because the shared rubric's checkpoint detector looks for a literal ## ✅ Checkpoint heading, while all workshop pages use the :white_check_mark: emoji shortcode. Every page does have a genuine checklist. Flag this as a rubric-detection gap, not an instructional-content gap.

Segment breakdowns

By technical level

Level	Mean success rate	N students
beginner	0.5%	11
github-basic	15.8%	19
actions-user	48.1%	11
advanced	49.1%	5

By personality

Personality	Mean success rate	N students
impatient	26.9%	6
methodical	25.8%	12
skeptical	23.6%	7
confused	22.4%	6
curious	20.7%	15

By UI preference

Preference	Mean success rate
ui_preferred: false (CLI-first)	34.3%
ui_preferred: true (browser-first)	11.7%

Browser-first learners underperform substantially, consistent with the 07-first-workflow/06-install-gh-aw Codespace-terminal requirement creating friction for UI-preferring learners even though the Actions-tab run path itself (Step 8) is browser-friendly.

Notable student journeys (3)

• Surprising success: Learner 037 (beginner, enterprise-dev background, methodical personality, CLI-preferred) achieved a 4.6% success rate — far above the 0.5% beginner average — showing that methodical CLI-native beginners can partially overcome the concept-heavy early steps that trip up most beginners.
• Unexpected dropout: Learner 028 (actions-user, enterprise-dev, confused personality, CLI-preferred) had only a 20.2% success rate versus the 48.1% actions-user average, with 05-agentic-intro as the most common failure step — showing that even Actions-literate learners can stall on the conceptual leap to agentic workflows when combined with a confused persona.
• Content-gap case: Every student who reached 08-run-your-workflow under the old (pre-fix) model failed there regardless of profile — a pure content-detection artifact, now corrected via semantic re-evaluation, illustrating why lexical-only simulation can badly misattribute dropout to a step whose actual instructions are clear and complete.

Generated by 🔬 Workshop Student Simulator · copilot · auto · 248.6 AIC · ⌖ 25.8 AIC · ⊞ 14.8K · ◷

• expires on Sep 21, 2026, 1:04 PM UTC

[github-actions]

This issue is being closed as outdated. A newer issue has been created: #3616

View newer issue

---

This action was performed automatically by the Workshop Student Simulator workflow.

[github-actions]

This issue is being closed as outdated. A newer issue has been created: #3617

View newer issue

---

This action was performed automatically by the Workshop Student Simulator workflow.