From ChatGPT to Classroom Practice: A Visualized Protocol for Generative AI-Supported Feedback, Critical Thinking, and EFL Writing Instruction

Summary

This study examined a visualized protocol for ChatGPT-assisted English as a foreign language writing instruction. The two-class comparison showed larger writing-score changes and higher post-intervention feedback-judgment accuracy in the protocol class. Class confounding and measurement limitations limit interpretation.

Abstract

This study examined associations between Visualized Protocol-guided ChatGPT Writing Instruction (VP-CGWIP), writing-score trajectories, critical-thinking dispositions, and post-intervention feedback judgment in an English as a foreign language (EFL) course. Two intact second-year university classes completed a ten-week intervention and four-week follow-up. The complete-case sample included 43 experimental and 42 control participants. Writing was assessed using the International English Language Testing System (IELTS) Task 2 criteria, and dispositions using the Critical Thinking Disposition Inventory–Chinese Version (CTDI-CV). The judgment task was administered only at the immediate (T1) and delayed (T2) posttests. The experimental-minus-control differences in overall-writing change were 0.48 points at T1 (pointwise 95% confidence interval [CI], 0.26–0.70) and 0.44 at T2 (0.22–0.66). The overall score and all four writing dimensions had interaction p values below 0.05 after post hoc Holm adjustment across 18 tests. Among seven disposition dimensions, only analyticity had an adjusted interaction p value below 0.05; the total score also met this threshold. Judgment accuracy was higher in the experimental class at both posttests, whereas the adjusted p values for perception–gain correlations exceeded 0.05. Log and journal summaries provided descriptive context. Because one class represented each condition, participant-level models cannot separate condition from class. These exploratory findings describe observed associations over the four-week follow-up; measurement limitations constrain their interpretation.

Introduction

Generative AI tools, such as ChatGPT, can generate feedback on student writing and have been studied in English as a foreign language (EFL) instruction^1,2. Their availability creates opportunities for revision and iterative improvement, but fluent feedback is not necessarily accurate or appropriate to a writer's intended meaning. Instruction must therefore address how students evaluate suggestions as well as how they obtain them.

AI-assisted composing places new demands on learners' decision-making³. A grammatically plausible suggestion may conflict with the surrounding argument, while a fluent reformulation may alter the writer's stance. Research on ChatGPT-assisted EFL writing and feedback describes both opportunities and challenges^4,5. A classroom protocol should make the grounds for accepting, modifying, or rejecting feedback explicit.

Existing work has examined course design, AI-assisted writing activities, and learners' thinking processes^6,7,8. Protocol-based instruction is not new: Ghafouri et al. described a three-phase ChatGPT-based writing instruction protocol that combined planning, instruction, and assessment over 10 weeks⁹. The present study builds on this direction by making feedback verification, Accept, Modify, or Reject decisions, and written rationales explicit within a seven-step cycle. The present protocol combines and evaluates these activities; it does not represent the first use of a structured ChatGPT writing protocol. Collaborative processing of ChatGPT feedback has also been studied as a distinct instructional arrangement¹⁰.

Critical-thinking disposition and performance on a specific judgment task are related but distinct constructs. A disposition inventory asks about a tendency to engage in critical thinking; a feedback judgment task samples decisions about particular suggestions. Research on the assessment of critical thinking through feedback provides a methodological basis for distinguishing these constructs¹¹. The present task focuses on judgments about AI-generated writing feedback.

VP-CGWIP organizes independent conceptualization, drafting, AI feedback, verification, judgment, revision, and reflection into a visible cycle. The study asks whether the two instructional conditions differ in writing-score trajectories, whether they differ in selected critical-thinking dispositions and post-intervention feedback judgments, and how participants in the experimental class describe their engagement with the protocol. Because the design includes only one class per condition, these questions concern exploratory associations. They do not identify a causal effect or establish that any single protocol component is responsible for the observed differences.

Protocol

Ethical approval

The study protocol was approved by the Ethics Committee of Guangzhou Vocational University of Science and Technology (approval number 2026120130404). All participants provided informed consent before enrollment and were informed that they could withdraw without academic consequence.

Study design

This study employed a quasi-experimental, two-group, three-time-point mixed-methods design to examine immediate and delayed between-group trajectory differences associated with VP-CGWIP in EFL learners' writing performance and critical thinking disposition. Learner perceptions of usefulness, ease of use, and behavioral intention (BI) were assessed descriptively in the experimental group at T1 rather than treated as intervention outcomes. The design followed a quantitatively driven, qualitatively embedded logic (QUAN(qual)): quantitative data formed the primary evidentiary chain for estimating group-by-time differences, while qualitative data were embedded throughout the intervention to illuminate the cognitive and behavioral processes accompanying those differences. The two strands were integrated at the interpretive stage through a joint display¹². Because one intact class represented each condition, class membership and instructional condition could not be statistically separated; accordingly, all between-group findings were interpreted as exploratory associations rather than definitive causal effects.

A quasi-experimental rather than a fully randomized design was adopted because the writing course was organized around two pre-existing intact classes and no student-level randomization was conducted. One intact class received VP-CGWIP and the other served as the control condition. Baseline comparisons were used to describe the comparability of the final analytic sample, but were not treated as proof of equivalence or as a means of eliminating class-level confounding. Three measurement points were established: a baseline pretest (T0, one week before the intervention), an immediate posttest (T1, during the final week of the ten-week intervention), and a delayed posttest (T2, four weeks after the intervention concluded). The control condition was matched to the experimental condition in AI access, writing-task load, and scheduled instructional hours. Because VP-CGWIP combined protocol visualization, structured Accept/Modify/Reject judgment, critical dialogue, adoption-log recording, and reflective activity, the between-group contrast estimated the association with the combined intervention rather than the isolated contribution of any single component.

The software, instruments, platforms, and study-specific materials used in this protocol are summarized in the Table of Materials.

Preregistration and analysis status

The study was not prospectively preregistered. The outcome hierarchy and multiplicity correction were specified retrospectively. All inferential analyses are exploratory.

Participants and sampling

From January to April 2026, participants were recruited from two intact classes of second-year university students who were not English majors enrolled in an EFL writing course at Guangzhou Vocational University of Science and Technology, Guangzhou, Guangdong, China. One class received VP-CGWIP and the other received the control instruction; no student-level randomization occurred. Eligibility required a College English Test Band 4 (CET-4) score of at least 425 and no previous systematic training in structured AI-assisted writing. Observed completer CET-4 scores ranged from 449 to 532. Each class initially included 50 students. Seven experimental and eight control participants were excluded for attendance or delayed-assessment reasons, leaving 43 and 42 participants, respectively; 15 of 100 participants (15.0%) were excluded. Figure 2 summarizes exclusions for missing at least two sessions (experimental, 4; control, 5) or not completing T2 (3 in each class). The complete-case sample contributed outcomes at T0, T1, and T2. Baseline tests concern completers only. Additional descriptive baseline summaries for the excluded students are reported in the Results; these summaries do not rule out selective attrition.

The VP-CGWIP intervention

The experimental class received the ten-week VP-CGWIP intervention shown in Figure 1. During weeks 1–2, the instructor presented the seven-step cycle and introduced prompt construction and checks of factual accuracy, logical consistency, and rhetorical appropriateness. During weeks 3–8, students submitted independently written drafts to ChatGPT, recorded individual suggestions in an AI Feedback Adoption Log, and classified each suggestion as Accept, Modify, or Reject with a written reason. Classroom dialogue addressed disagreements with AI feedback. During weeks 9–10, students continued judgment and logging activities and combined self-assessment, teacher feedback, and log entries in final revisions. Reflective journals followed the six elements of Gibbs' cycle: description, feelings, evaluation, analysis, conclusion, and action plan¹³. Students accessed ChatGPT through the official OpenAI web interface (chatgpt.com) using ChatGPT Plus accounts. Browsing, file-upload, and other auxiliary tool functions were disabled during the instructional activities. No temperature or other model-generation parameters were manually specified; the platform-default settings were retained.

Operational sequence and fidelity

Use an independent plan and first draft as the inputs to the cycle. For each AI suggestion, compare the proposal with the intended meaning, relevant language evidence, and the task requirements before assigning Accept, Modify, or Reject. Record the suggestion, the decision, the reason, and any resulting revision in the log; retain a reason for non-adoption. Discuss unresolved or competing judgments in class and revisit them during revision and reflection.

Control group design

The control class completed the same writing tasks and scheduled instructional hours, had access to ChatGPT, and received conventional teacher feedback. The experimental class additionally used the visualized cycle, structured judgment, and adoption logs, critical dialogue, and reflection activities. The same instructor taught both classes. Each class had two 90 min sessions per week for ten weeks, corresponding to 20 sessions and 30 scheduled instructional hours. The amount of teacher feedback and the actual time on task were not available for comparison. Equal scheduled hours do not establish equal instructional intensity. The contrast concerns the integrated package, and additional attention or practice may partly account for the observed class difference.

Study procedure and timeline

The intervention and follow-up covered 14 weeks: ten intervention weeks followed by four weeks without further intervention input. Baseline assessment (T0) occurred one week before the intervention; the interval from baseline to T2 was therefore approximately 15 weeks. T0 included timed writing, CTDI-CV, and background information. At T1 (intervention week 10), both classes completed writing, CTDI-CV, and the AI-feedback judgment task; only the experimental class completed the Technology Acceptance Model (TAM) questionnaire. At T2 (week 14), both classes repeated writing, CTDI-CV, and the judgment task to examine four-week maintenance. No baseline judgment task was administered. Adoption logs covered weeks 3–10, and reflective journals covered weeks 9–10. Figure 2 distinguishes the assessment schedule from participant flow.

Outcome measures and instruments

Table 1 summarizes the outcomes and measurement occasions. Overall writing score is the main outcome emphasized in this report; the four writing dimensions and the remaining outcomes are secondary or exploratory. Two IELTS-certified raters, blinded to group assignment, used the Task 2 dimensions Task Response (TR), Coherence and Cohesion (CC), Lexical Resource (LR), and Grammatical Range and Accuracy (GRA). The IELTS Task 2 band descriptors specify these four criteria¹⁴; published work on AI-assisted accuracy assessment provides additional context¹⁵. The administered CTDI-CV comprised 70 items, with ten items per dimension and a six-point response scale (1–6). It assessed seven self-reported disposition dimensions: truth-seeking, open-mindedness, analyticity, systematicity, critical-thinking self-confidence, inquisitiveness, and cognitive maturity. The feedback judgment task sampled responses to valid suggestions and deliberately flawed or contextually inappropriate suggestions at T1 and T2. It is treated as task-specific performance, not a validated general measure of critical thinking. The experimental-class adoption logs and journals describe classroom processes. At T1, an adapted TAM questionnaire measured perceived usefulness (PU), perceived ease of use (PEOU), and BI in the experimental class¹⁶. The TAM instrument contained 15 items rated on a five-point scale (1–5): six PU items, six PEOU items, and three BI items.

Data collection procedures

Writing assessments lasted 40 min and were completed in a classroom without AI assistance. Different prompts were used at T0, T1, and T2. Overall writing score was the arithmetic mean of the four criterion scores. Electronic questionnaires recorded response durations, and logs and journals were submitted through a writing platform. Process summaries concern the 43 experimental-class completers. Limitations of prompt comparability and measurement reliability are considered in the discussion.

Judgment task development

The judgment task comprised 20 items: ten trap-feedback items and ten valid-feedback items. Parallel forms were used at T1 and T2. Three EFL instructors reviewed the task, and it was piloted with 20 students who were excluded from the main study. Judgment scores are expressed as percentages. The task measures performance on the selected feedback items.

Quantitative analysis

All statistical analyses were conducted in R. The complete-case analysis included 85 participants and 255 repeated observations. A unique group-plus-participant identifier linked observations across time. Gaussian linear mixed-effects models were fitted using lme4¹⁷, with inferential tests implemented using lmerTest¹⁸: outcome ~ group * time + (1 | uid), where group and time are categorical, Ctrl and T0 are the reference levels, and uid uniquely identifies each participant. Restricted maximum likelihood (REML), Satterthwaite denominator degrees of freedom, independent constant-variance residuals, and participant random intercepts were used. This structure implies compound-symmetric marginal covariance. The two interaction coefficients quantify experimental-minus-control differences in change from T0 to T1 and T0 to T2; pointwise 95% t intervals are reported. All 13 outcome models converged without singular fits. A sensitivity analysis added grand-mean-centered CET-4 as a fixed covariate. Continuous baseline comparisons and judgment comparisons used two-sided Welch t tests; sex counts used Pearson’s chi-square test without continuity correction. Judgment Cohen’s d used pooled sample SD. TAM associations used Pearson correlations with T1-minus-T0 overall writing change and Fisher z CIs. Aggregate baseline characteristics of excluded participants were summarized descriptively. No missing-data imputation or individual-level attrition sensitivity analysis was performed. Participant-level models cannot separate the effects of class and condition.

Multiple testing

Post hoc Holm adjustment was applied to 18 nominal p values: five writing interactions, eight CTDI-CV interactions, two judgment comparisons, and three TAM correlations¹⁹. Baseline tests and CET-4 sensitivity models were not included in this family. Pointwise contrast and correlation intervals were not adjusted for multiplicity; adjusted omnibus p values do not test a particular T1 contrast. All analyses are exploratory.

Qualitative analysis and integration strategy

The analysis in NVivo 20 followed Braun and Clarke's thematic-analysis framework²⁰. Codes combined deductive categories from TAM, CTDI-CV, and content, process, and premise reflection with inductive categories arising from the logs and journals. Two coders worked independently, with disagreements resolved through discussion. All qualitative summaries concern the 43 experimental-class completers. The joint display places aggregate qualitative descriptions alongside quantitative results as contextual complements. Individual-level convergence and causal mechanisms were not tested; limitations of coding reliability and case selection are considered in the Discussion.

Results

Baseline comparability in the complete-case analytic sample

Table 2 summarizes baseline characteristics. The experimental and control completers had mean ages of 19.19 years ± 0.39 years and 19.19 years ± 0.40 years, CET-4 scores of 489.02 ± 29.04 and 491.00 ± 29.11, and weekly AI use of 2.45 h ± 0.90 h and 2.40 h ± 0.90 h. Female counts were 22/43 (51.2%) and 21/42 (50.0%). Continuous values are mean ± sample standard deviation (SD). All baseline-test p

Discussion

The analysis describes different trajectories in two intact EFL writing classes. Adjusted interaction p values were below 0.05 for all writing outcomes, analyticity, and the CTDI-CV total, but not the other six disposition dimensions. Judgment accuracy differed at both posttests, with no baseline judgment measurement. Logs and journals describe recorded decisions and reasons. Together these findings motivate further evaluation of the integrated protocol, while neither participant-level models nor aggregate quali...

Materials

List of materials used in this article

Name	Company	Catalog Number	Comments
Critical Thinking Disposition Inventory–Chinese Version (CTDI-CV)	Chinese-language study instrument / instrument authors	N/A	70-item inventory; 7 dimensions; 10 items per dimension; 6-point response scale (1–6); administered at T0, T1, and T2.
ChatGPT	OpenAI	N/A	Official OpenAI web interface (chatgpt.com), accessed using ChatGPT Plus accounts. Browsing, file-upload, and other auxiliary tools disabled during instruction. Platform-default generation settings retained; no manual temperature or other generation parameters.
R statistical software	R Foundation for Statistical Computing	N/A	R 4.6.0; used for quantitative statistical analyses.
lme4 package for R	CRAN / lme4 developers	N/A	Version 1.1-37; used to fit Gaussian linear mixed-effects models by restricted maximum likelihood (REML), with participant-specific random intercepts.
lmerTest package for R	CRAN / lmerTest developers	N/A	Version 3.1-3; used for inferential tests and Satterthwaite approximations for denominator degrees of freedom.
NVivo	Lumivero (formerly QSR International)	N/A	NVivo 20; used for thematic coding and qualitative analysis of adoption-log rationales and reflective journals.
IELTS Academic Writing Task 2 band descriptors	IELTS	N/A	Official Task 2 scoring criteria for Task Response, Coherence and Cohesion, Lexical Resource, and Grammatical Range and Accuracy; used by two IELTS-certified raters.
College English Test Band 4 (CET-4)	National College English Testing Committee, China	N/A	English-proficiency eligibility measure; inclusion threshold ≥425; participant scores also used in sensitivity analysis.
AI Feedback Adoption Log	Study team	N/A	Intervention process record used during weeks 3–10 to record each AI suggestion, Accept/Modify/Reject decision, rationale, and resulting revision.
AI-feedback judgment task	Study team	N/A	20-item task: 10 trap-feedback items and 10 valid-feedback items; parallel forms at T1 and T2; reviewed by three EFL instructors and piloted with 20 students excluded from the main study.
Reflective journal prompts	Study team; structured with Gibbs' reflective cycle	N/A	Used during weeks 9–10; prompts covered description, feelings, evaluation, analysis, conclusion, and action plan.
Technology Acceptance Model (TAM) questionnaire	Study team; adapted from Davis (1989)	N/A	15 items on a 5-point scale: 6 perceived usefulness (PU), 6 perceived ease of use (PEOU), and 3 behavioral intention (BI) items; experimental class at T1 only.
Electronic questionnaire / writing submission platform	Study institution	N/A	Used to administer CTDI-CV and TAM, record response duration, and collect AI Feedback Adoption Logs and reflective journals.
VP-CGWIP seven-step visualized protocol	Study team	N/A	Instructional scaffold: independent conceptualization, drafting, AI feedback, verification, judgment, revision, and reflection.