Can We Agree? Exploring the Shared Mental Model in Assessing Student Write-Ups Using the RIME Framework

As the discipline strives for equitable and unbiased assessments in education and moves towards competency and criterion-based assessments, one promising approach is the analysis of student write-ups as a measure of their clinical skills over time in clerkships. To ensure the effectiveness of any assessment method, reliability among different users is crucial. The reporter-interpreter-manager-educator (RIME) framework has been shown to better demonstrate student growth over time and was proposed to promote educational equity of clinical evaluations. The workshop presenters, spanning three different institutions, have reviewed and adjudicated nearly 100 student write-ups using the RIME framework. After collectively adjudicating just 10 notes, the team achieved rapid inter-rater reliability reaching a Kappa of 0.642. This workshop creates a safe space for faculty members to assess sample student write-ups and establish and practice scoring writeups with a shared mental model. Participants will receive a basic review of the RIME framework applied to the assessment of write-ups, then be asked to individually categorize a sample student write-up into RIME categories. In small groups, participants will discuss their assessment and explore unique perspectives. In the large group, team will share adjudication of the sample note and explore the small group conclusions. Participants will then independently review a second sample write-up. Responses will be recorded, and the large group will explore the results and evaluate together the ability to quickly create and possibly improve interrater reliability and how it might be applied to individual learning environments.