Expertise as elaboration: teachers’ reflections on a writing rubric integrated into an AI tool
[ad_1]
AI-based tools can signal the integration of technology into learning in profound ways; however, the long trajectory of edtech has yet to change the fundamental organizational structure between teacher and student. Teachers – with the vast majority of schools still organized as one teacher for every 15 to 35 students – mediate students’ experiences in the classroom in multiple ways. Although opportunities for students to work independently using instructional learning systems clearly exist in most settings, the frequency of their use, for what purposes, and for which students varies widely.
As an example, the Topeka Project featured an automated essay-grading tool that provided students in grades 6-8 with individualized line-level feedback on argumentative essays responding to six different prompts. Each prompt offered aligned sources of information, and teaching materials and other educational materials accompanied the tool. The Topeka Project rubric described students’ argumentative writing along four dimensions: affirmation and focus, support and evidence, organization, and language and style, at four levels of performance (emerging, developing, competent, advanced).
Drawing on our research on teachers’ approaches to using AI in the classroom and how teachers’ grading of argumentative papers differed from the automated essay-scoring tool, this companion piece illustrates the expertise teachers relied on to reveal their understanding of the writing rubric, how they used it, and the extent to which the rubric captured or missed what they see and expect from their students’ argumentative writing. Educators’ insights into the column underscore questions we need to continue to ask as edtech products incorporate and evolve logics that reduce, rather than increase, transparency in how technology facilitates student learning.
Over three waves of implementation (winter 2020, fall 2020, and school year 2021-22), nearly all teachers using the Topeka Project agreed that the dimensions scored by the AI tool were appropriate and agreed with the scores their students received. However, a majority also told us that students did not know how to respond to comments. Teachers needed to help students interpret and apply feedback and provide more holistic feedback. (See Exhibit 1.)
Exhibit 1: Teacher perceptions of automated Topeka Project essay grading
Rubric discussions (as part of a grading process for teachers to grade samples of student work) revealed the critical ways in which teachers used their expertise to emphasize key elements of the rubric and frame feedback to students. Below are highlights of teachers’ perspectives on three of the four dimensions of the rubric.
Claim and focus. Competent definition – “The essay introduces a clear assertion based on the topic or text(s). The essay maintains a primary focus on the purpose and task, but may not develop the claim consistently throughout the essay while meeting the requirements of the prompt.
While the AI tool seemed to provide feedback on whether students wrote specific sentences that made a single statement that they could then substantiate, teachers emphasized consistency throughout. Beyond looking for a statement at the beginning of an article, one teacher clarified: “[I] entered on ‘not uniformly developed’ [from rubric level] throughout – it’s not just the statement [claim] himself, but [it’s] referring to the consistency of the entire essay. So we shouldn’t just look at a specific statement [as the claim]but we have to look at the entire essay and determine whether or not the entire essay supports this claim.
Support and proof. Competent definition – “The essay uses clear and relevant evidence and explains how the evidence supports the claim. The essay demonstrates logical reasoning and an understanding of the subject matter or text(s). Counterclaims are acknowledged but may not be sufficiently explained and/or distinguished from the central claim of the essay.
Teachers emphasized the need for students to be able to identify and apply reliable evidence to their argument, particularly on whether the student could explain why the evidence they used supports the claim or addresses a potential counterclaim to their argument: [the evidence] say? Is the evidence reliable? Is it relevant? If yes, [students] it also needs to be explained. Don’t just summarize [of the evidence].” In other words, teachers wanted to see the student’s original writings that explained why they used the evidence they had chosen as the most important aspect of that dimension to grade.
Organization. Competent definition – “The essay incorporates an organizational structure with clear and consistent use of transitional words and phrases that show the relationship between and among ideas. The essay includes a progression of ideas from beginning to end, including an introduction and a statement or concluding section.
Teachers highlighted how organizing strengthened advocacy and focus as related dimensions. Especially with lower grades emphasizing how to write a well-crafted paragraph, students do not necessarily have enough practice in constructing multi-paragraph pieces. One teacher explained, “Students write well-structured paragraphs, but we want them to connect the paragraphs. The relationship – the connection – has to be there. You may be proficient at writing single paragraphs, but to be proficient at writing an essay, you need to jump from paragraph to paragraph.
This relationship is not sufficiently established with transition words, as many students are taught. Another teacher shared: “[W]We’re hooked on looking at transition words, but the rubric demands more. Ideas are moving but not coherently. If I take your paragraph in isolation, is it related to your request? This is how I see the organization. The relationship between and among ideas – how do you teach this? Essentially, teachers were looking for a logical sequence in the way students organized their arguments.
What teachers emphasized in their grading illustrates the weight they place on different aspects of the rubric as the most critical skills in argumentative writing. The point is not that what teachers are looking for differs from what the AI tool is looking for – that difference may be unavoidable, especially with machine learning, where decision rules change over time. The fact is, teachers have expertise and apply professional judgment that integrates knowledge of writing, teaching, students, relationships, and culture in tacit and subtle ways that are hard to capture – at least for now – by AI tools. We need education technology that is grounded in an understanding of how teacher expertise mediates and complements technology-enabled learning solution offerings, tools that reflect expert teachers’ cross-curricular knowledge of content and students and their expectations of what students are capable of achieving.
[ad_2]
Source link
