Design the Practice AI Must Not Do

Judy Hanwen Shen and Alex Tamkin gave 52 Python-experienced programmers coding tasks in an unfamiliar asynchronous library, with or without a GPT-4o chat assistant. On an immediate quiz that prohibited AI, those with access showed statistically lower comprehension, and their average speed improvement was not statistically significant. 1

Starter tasks once delivered an output and a learning action together. AI can now deliver the output. Leaders must decide which learning action stays with the employee, and how to know the practice works.

Our recommendation is a design to test. Select a judgment the role still needs after automation. Have the learner attempt a diagnosis before any answer appears. Require a reason for accepting, rejecting or repairing the suggestion. Supply expert-checked feedback. Then use a different case with assistance stated, and check retention later.

No study has tested that combination. Classroom trials suggest the design of help can matter 24, and a customer-support study points to learning from suggestions on the job 3. Employers should pilot the design and compare costs with results.

These sources chiefly bear on immediate comprehension and near transfer. They do not establish delayed retention, far transfer or long-run expertise at work.

1. 1. AI access can lower comprehension during a short learning task. Shen and Tamkin's assisted programmers understood less on an AI-prohibited quiz, with no statistically significant average speed improvement. 1
2. 2. Interaction patterns give a design lead, short of a proven cure. Conceptual questions appeared in higher-learning patterns and heavy delegation in lower-learning ones, but neither was separately randomized. 1
3. 3. One classroom trial directly ties tutor design to avoiding an exam deficit. Ordinary chat left unaided exam grades 17% below control in relative terms, while teacher-designed tutoring was statistically indistinguishable from control. 2
4. 4. Workplace exposure may teach, which weakens the case for inevitable dependence. Support agents previously exposed to AI handled chats faster than their pre-AI baseline during outages, on noisy duration estimates. 3
5. 5. The practice design we propose is a hypothesis for employers to test. No cited study tested the combined steps, and both classroom treatments changed several features at once. 24

Two earlier Briefs set the boundary for this one. Brief 025, "AI Took the Starter Tasks. Who Teaches Judgment Now?", warned about lost starter repetitions and proposed supervised cases, shadowing and redos. It rested on employment and employer evidence and said it lacked direct evidence that judgment now forms more slowly. Brief 010, "The Proxy Learner Problem", asked what completion records reveal about who did the work and called for capability and transfer checks. This Brief examines the ground between them: what happens inside the interaction while someone works with AI. A supervised case can still fail if the assistant supplies the diagnosis before the learner has formed one. A capability check can expose that failure afterward but cannot prevent it. The sequence inside the task is the management decision here.

Ordinary assistance can finish the task and skip the learning

The Shen and Tamkin programmers knew Python but not the Trio library. That resembles a common moment at work, when a capable person meets an unfamiliar system. The quiz barred AI and tested concepts, code reading and debugging, the abilities someone would call on to check generated code. 1 The assisted group understood less, without a statistically significant average speed improvement. That leaves room for some speed effect, but lost comprehension did not clearly buy faster work here. 1

The more useful detail concerns how people used the assistant. Conceptual questions and explanation-seeking appeared in the higher-learning patterns, heavy delegation in the lower-learning ones. 1 A manager will be tempted to tell employees to ask for explanations. The study cannot support that rule, because those strategies were not separately randomized. People who already understood more may simply have asked better questions.

This is a vendor-affiliated preprint on a short task, with no employer apprenticeship and no measure of expertise over time. Its value is the problem it exposes: a learner can reach working code without forming a view of why it works, and the finished output hides the gap. The next question is whether that loss follows from AI help itself or from one way of giving it.

Classroom contrasts point to the design of help, taken as a bundle

The sharpest contrast comes from Hamsa Bastani and colleagues, who randomized classrooms of nearly 1,000 students at one Turkish high school across four 90-minute sessions. Classes had no AI, ordinary chat or a teacher-designed tutor. Ordinary chat improved assisted practice, yet grades on the closed-book exam that followed were 17% below control in relative terms. 2 A dashboard of assisted output would have shown only the first half.

The teacher-designed tutor largely removed the deficit, and its exam results were statistically indistinguishable from control. 2 That is no demonstrated gain over the no-AI classes. Nor can the study name the active ingredient, because teacher-provided correct solutions, common mistakes, hints and the answer policy changed together. 2

In a randomized crossover covering two lessons across two weeks, Greg Kestin and colleagues studied 194 eligible Harvard physics students. A custom tutor with sequenced questions, expert-prepared solutions and feedback produced stronger immediate post-test results than active classroom instruction. 4 Self-paced home use also differed from in-person teaching, so the credit goes to a bundle. Only the Turkish trial compares designed tutoring with ordinary chat.

One shared feature is usable. In both trials, someone with subject expertise decided in advance what the tool knew about the problem and how it would respond. 24 That makes expert-checked content and control over when answers appear reasonable things to test. Both results are short-run and neither reaches the workplace.

Workplace evidence narrows the design to judgments ordinary work will not teach

Employers could conclude that every assisted task needs a tutor. Erik Brynjolfsson, Danielle Li and Lindsey Raymond give reason to resist. In a staggered rollout across 5,172 customer-support agents, productivity rose with AI access. During tool outages, previously exposed workers handled chats faster than their pre-AI baseline, with gains clearer after greater exposure and among those who had followed suggestions more closely. 3

That fits learning from trying suggestions and observing customers, without proving lasting expertise or isolating a mechanism. Outages were rare and lasted minutes to hours, estimates were noisy, chat mix may have differed, and the analysis measured duration only. 3

We draw a narrowing inference. A support agent tries a suggestion and sees a customer respond. Other starter tasks give no such signal: someone who accepts a drafted finding may never learn whether it held up. So selection has two tests: the role still needs the judgment after automation, and ordinary work does not reveal whether the judgment was right.

Where both hold, we would use the full sequence: attempt, reason, expert-checked feedback. The employer learns whether that works from a different case with assistance stated, a later retention check and a comparison with plain assisted work. No source sets a quota, retest delay or effect size.

Autonoma forecast: Over the next 12 months, we expect employer claims about AI learning to rest mostly on assisted output and immediate quizzes, and the first credible workplace evidence to come from small pilots comparing attempt-first designs with ordinary assistance. This is our analytical forecast, not a source finding.

• Independent replication of the coding result inside employers, over longer learning periods. 1
• Trials that separately randomize attempt-first, explanation-seeking and delegation conditions. 1
• Delayed retests and far-transfer tasks for designed tutors, since classroom results so far stop at immediate tests. 24
• Workplace experiments that randomize learning design during assisted work and report quality alongside duration. 3
• Tutor studies that separate answer policy, feedback and pacing, and publish what the build cost. 24
• Local assignment check: For each automated starter task, a manager can name the judgment retained and the case where the employee attempts it first.
• Local readiness check: Before expansion, the pilot shows a different-case result with assistance stated, a later retention result and expert cost against a comparison group.

For CLOs and L&D: Name the judgment each pilot role must keep after automation, and write the attempt-then-reason step into that workflow. Report immediate comprehension, near transfer and delayed retention as separate results.

For CHROs: Decide who funds practice time and expert feedback before the pilot starts. If automation savings are booked in full, the design has no budget. Keep pilot data out of performance ratings while the measures are unproven.

For HR technology and procurement leads: Require configurable answer timing, a field for the employee's reason and an export of the assistance condition. Ask vendors to separate immediate from delayed results and state what experts must build. 4

For CIOs: Decide whether the assistant can hold its suggestion for selected case types without slowing every workflow, and log each suggestion and response. Treat unplanned outages as noisy evidence 3; do not stage them as training.

For legal and compliance: Reason logs are records of employee judgment. Set purpose, retention and access before collection, and decide whether they may inform discipline. Block any description of pilot results as validated competence.

Weight: Strong. If employees will always work with AI, unaided practice may test the wrong skill. The coding quiz and the Turkish exam both barred assistance. 12 Cost sharpens the objection: expert-checked feedback consumes the senior time automation was meant to free, the physics tutor took work to build 4, and support agents appeared to gain from ordinary use 3. For many tasks the sound answer may be to automate and move on.

We accept much of this. The attempt exists to give the employee a view to compare with the suggestion, and the tool follows at once. The later check should match how the role operates, with assistance stated. We prescribe neither manual drudgery nor a universal tool ban. Where a pilot shows no difference worth its cost, drop the practice.

Automation projects usually start from the output. Start one step earlier, at the point where a person must commit to a view.

Keep that commitment small and place it only where the role needs the judgment and the work hides the result.

Decision rule: Automate the output freely, but where the role still needs the judgment and the work will not reveal mistakes, keep the employee's attempt and reason ahead of the answer, and scale only if a comparison pilot shows gains on a different case and at a later retention check, at a cost the business will pay.