How AI Impacts Skill Formation

Studies of AI assistance at work usually measure output: tasks finished, code merged, tickets resolved, with the largest gains often going to less experienced workers. This paper asks what happens to the worker's own learning when a new skill is acquired with AI help. Its motivation is oversight: if people are to check and debug AI-written code, they need the skills to do so, and those skills are usually built on the job, the same place AI assistance is spreading fastest (Section 1). The authors test the question with a randomized experiment in which developers learn an unfamiliar Python library with or without an AI assistant and then take a quiz without one.

The work was done through the Anthropic Safety Fellows Program (Acknowledgments). The arXiv preprint is 2601.20245 (this note uses v2, 1 February 2026); Anthropic's summary post, "How AI assistance impacts the formation of coding skills", appeared on 29 January 2026, and the study was preregistered on OSF.

Terms used in this note

• Skill formation. Here, the knowledge of a new library gained by completing tasks with it, measured by a quiz taken afterwards without AI.
• Trio. A Python library for asynchronous concurrency. It was chosen because it is less familiar than Python's built-in asyncio, introduces new concepts such as structured concurrency, and is designed to be easy to learn (Section 4.1).
• Cognitive offloading. Handing mental work to a tool; the paper cites survey evidence that heavy AI use goes with more offloading and reduced effort (Section 2.1).
• Cohen's d. A difference in group means divided by the pooled standard deviation; 0.738 means the groups differ by about three quarters of a standard deviation.
• Treatment and control. Participants with the AI assistant (treatment) and without it (control); both could use web search and the task instructions (Section 7).

What the paper does

Task and assistant

Participants completed two short Trio tasks, designed to work like a self-guided tutorial: a timer that prints each second while other functions run (introducing nurseries, starting tasks and concurrency), and a record-retrieval function that handles missing-record errors (introducing error handling and memory channels) (Section 4.1). They worked in an online interview platform. In the treatment condition a chat assistant built on GPT-4o, prompted to be a coding assistant, could see the participant's code and could produce the full correct solution to both tasks if asked (Section 4.1). The assistant was therefore not Claude, and it was a chat sidebar rather than an agent; the summary post notes that agentic tools like Claude Code differ and that their effects on skill development are "likely to be more pronounced".

Evaluation

Drawing on a review of computer science assessments, the authors list four question types: debugging, code reading, code writing and conceptual (Section 4.2). The quiz covered seven Trio concepts with 14 questions worth 27 points, using debugging, code reading and conceptual questions; code writing was left out so that syntax slips, easily fixed with a search or a query, would not dominate. Five quiz versions were tested in pilots using item response theory, and the grading rubric was preregistered.

Study design and pilots

Each participant did a warm-up task without AI, then had up to 35 minutes for the two Trio tasks, then took the quiz and a survey, all without AI (Section 4.3). Everyone was told to work as fast as possible. Four pilots shaped the design (Section 5.1, Table 2). On the first crowdworking platform, 35% of control participants used AI anyway, and about 25% still did after explicit warnings, so the study moved to a second platform with screen recording. Pilots also showed participants using one quiz question to answer another (fixed by splitting the quiz across pages), and control participants stalled on unrelated Python syntax (fixed with syntax hints). In Pilot D (n = 20), the AI group was faster (d = 1.11, p = 0.03) and scored much lower (d = 1.7, p = 0.003); the main study was powered for a more conservative effect of d = 0.85.

Participants

The main study enrolled 52 participants, 26 per condition, recruited through a crowdworking platform and paid a flat 150 USD (Sections 4.3 and 5.2.1). All had more than a year of Python experience, used Python at least weekly, had tried AI coding assistants, and had never used Trio. Groups were balanced on coding experience, Python use, asyncio familiarity and a short async concept check (Table 1). Most held a bachelor's degree, were 25 to 35, and worked as freelance or professional developers (Figure 17). The summary post describes them as "mostly junior" software engineers.

What it finds

Quiz scores fall, speed does not significantly rise

The AI group scored 4.15 points lower on the 27-point quiz, which the paper reports as a 17% score difference or about two grade points (Cohen's d = 0.738, p = 0.010); controlling for warm-up time, d = 0.725, p = 0.016 (Section 5.2.2, Figure 6). The summary post gives the group means as 50% for the AI group and 67% for the hand-coding group. The AI group finished slightly faster on average, but the difference was not significant (p = 0.391, Figure 6). Every AI participant completed both tasks, while 4 of 26 controls did not finish the second within the time limit (Section 5.2.2). Control participants scored higher at every level of prior coding experience (Figure 7).

Debugging shows the largest gap

In exploratory analysis that was not preregistered, the largest score gap was on debugging questions and the smallest on code reading (Section 5.2.2, Figure 8). The authors relate this to errors: the median control participant hit three errors during the task against one for the AI group, and 10 of the 12 participants who finished without any error were in the AI group (Section 6.2, Table 4). Controls in particular met errors tied to Trio concepts, such as a coroutine that was never awaited, which the quiz then tested (Section 6.2, Figures 14 and 15).

Experience of the task

Controls reported learning more; both groups reported enjoying the task; the AI group found the task easier, but both found the quiz similarly hard (Figures 9 and 10). In free-text feedback, some AI participants said they felt "lazy" and that "there are still a lot of gaps in (their) understanding" (Section 6.4).

Why AI did not save more time

From the screen recordings (51 of 52 available), the authors find that some participants spent up to 11 minutes in total composing queries, and a handful spent up to six minutes on a single query, when the median AI-condition completion time was 19 minutes (Section 6.1, Figure 12). Explanation requests were the most common query type (79 queries; 21 of 25 treatment participants asked at least one), followed by code generation (51 queries, from 16 of 25 participants) and debugging (9) (Table 3). Participants who pasted AI code directly (n = 9) were fastest; those who retyped it by hand (n = 9) or mixed the two (n = 4) were about as fast as controls, and typing versus pasting made no notable difference to quiz scores (Section 6.1, Figure 13).

Six patterns of AI use

The authors grouped AI-condition participants into six interaction patterns by query types, number of queries and active coding time (Section 6, Figure 11). Three had average quiz scores below 40%:

• AI delegation (n = 4): asked only for code and pasted it in. This group was fastest, finishing in 19.5 minutes against 23 for the control group, but it was only about a fifth of the treatment group (Section 7).
• Progressive AI reliance (n = 4): asked a question or two on the first task, then delegated all code on the second, and missed the second task's concepts.
• Iterative AI debugging (n = 4): relied on the assistant to check and fix code, asking many questions but not to understand; this group was also slow.

Three had average scores of 65% or higher:

• Generation then comprehension (n = 2): generated code, then asked follow-up questions about it. Apart from the follow-up questions, this looked almost the same as delegation.
• Hybrid code and explanation (n = 3): asked for code together with an explanation of it.
• Conceptual inquiry (n = 7): asked only conceptual questions and fixed errors alone; the fastest high-scoring pattern and second fastest overall.

The Discussion summarizes the split as 65% to 86% for the high-scoring patterns against 24% to 39% for the low-scoring ones, and attributes it to cognitive engagement (Section 7). The authors stress that each cluster is small, and the summary post says the qualitative analysis does not establish a causal link between pattern and learning.

The prompts and instructions used

The study's behavior-shaping text is modest. The assistant was GPT-4o "prompted to be an intelligent coding assistant", and treatment participants were prompted to use it (Section 4.1); the paper does not print the system prompt. All participants were instructed to finish "as fast as they could" (Section 4.3), and compensation was a flat fee, which the authors note when discussing time spent on queries (Section 6.1, footnote 5). The summary post adds that participants were told a quiz would follow the task but were encouraged to work as quickly as possible. The paper's discussion points to products that change the assistant's instructions, such as learning or explanatory modes, as one way to keep users engaged, but it does not test any (Section 7.1).

Limitations and open questions

The authors list these (Section 7.1, and the summary post):

• One task, one interface. A single library and a chat assistant; agentic tools, which need even less human input, may show larger effects.
• Short horizon. Learning was measured within about an hour; whether immediate quiz performance predicts longer-term skill is not answered.
• Participant incentives. Crowdworkers had no job-related reason to learn Trio.
• Prompting skill unmeasured. Familiarity with AI tools was self-reported.
• Evaluation format. A quiz rather than a follow-up task.
• No human-help comparison. The study does not compare AI help with help from a person, as in pair programming or code review.

Three further points follow from the paper's design. The sample is small (26 per group; the summary post also calls it "relatively small"), and the six interaction patterns rest on groups of two to seven people. The assistant was GPT-4o (Section 4.1), so the study did not test any Claude model. The summary post also places the finding beside Anthropic's observational estimate that AI can cut some task times by 80% (Estimating AI productivity gains from Claude conversations), suggesting the two may coexist: speedups on tasks people already know how to do, and slower skill acquisition on new ones.

Research questions

1. Does the quiz gap persist? Learning was measured immediately (Section 7.1). Retesting participants days or weeks later, or asking them to complete a new Trio task without AI, would show whether the 4.15-point difference reflects lasting skill.
2. Is the effect larger with agentic tools? The authors expect it to be (Section 7.1). Repeating the design with an agent that edits files directly would measure it.
3. Can the assistant's instructions preserve learning? The high-scoring patterns involved explanation and conceptual questions (Figure 11). Randomizing participants between a default assistant and one instructed to explain and ask before giving code would test whether an assistant can induce those patterns rather than relying on users to choose them.
4. Do the interaction patterns cause the score differences? The patterns are observational and small (Section 6). Assigning participants to a usage pattern, for example conceptual questions only versus code generation only, would test causation.
5. Does AI help differ from human help? The authors name this gap (Section 7.1). Adding a condition with a human tutor or pair programmer would separate the effect of assistance from the effect of AI.
6. How does the effect vary with prior experience? Only four participants had 1 to 3 years of coding (Section 5.2.2). A larger sample of novices would show whether the junior engineers the authors worry about are affected more.

How it connects

• Productivity estimates. The summary post sets this result beside Estimating AI productivity gains from Claude conversations, and the paper cites the first Economic Index paper (The Anthropic Economic Index) and Clio (Clio: Privacy-Preserving Insights into Real-World AI Use) for evidence on how AI is used.
• Oversight. The paper cites Bowman et al. (2022), Measuring Progress on Scalable Oversight for Large Language Models, for the difficulty of supervising more capable systems when human abilities are weaker.
• Later work citing it. The Anthropic Education Report: The AI Fluency Index links this study when reporting that users who produce artifacts with Claude question its output less.

References

• Shen, J. H., and Tamkin, A. (2026). How AI Impacts Skill Formation. arXiv:2601.20245v2, 1 February 2026. https://arxiv.org/abs/2601.20245 (section, figure and table numbers in this note refer to v2). Preregistration: https://osf.io/w49e7. Annotated transcripts: https://github.com/safety-research/how-ai-impacts-skill-formation.
• Anthropic. (2026). How AI assistance impacts the formation of coding skills. 29 January 2026. https://www.anthropic.com/research/AI-assistance-coding-skills (source of the 50% and 67% group means and the "mostly junior" description).
• Works cited above only as the paper describes them (Bowman et al., 2022; Handa et al., 2025; Tamkin et al., 2024; Lee et al., 2025) are listed in the paper and were not reread for this note.