Development and evaluation of authentic rubric
- Author
- Pun, Wik
- Published
- [University Park, Pennsylvania] : Pennsylvania State University, 2017.
- Physical Description
- 1 electronic document
- Additional Creators
- Suen, Hoi K.
Access Online
- etda.libraries.psu.edu , Connect to this object online.
- Graduate Program
- Restrictions on Access
- Open Access.
- Summary
- Systematic evaluation is a key component of reliable and valid performance assessment. Scoring rubrics are often used to achieve this goal by promoting inter-rater agreement evaluating performances (Johnson, Penny, & Gordon, 2009). More recently, some scholars (e.g., Goffin, Gellatly, Paunonen, Jackson, & Meyer, 1996; Goffin & Olson, 2011; Pollitt, 2004, 2012) advocated for the use of comparative judgment, a scoring procedure adopted from the Thurstones (Thurstone, 1927a) Law of Comparative Judgment, to replace scoring rubrics in performance assessment. It was argued that the comparative judgment method holds multiple advantages over scoring rubrics. Empirically, some studies were able to show that evaluations elicited via comparative judgment methods had higher criterion-related validity evidence (Goffin et al., 1996; Goffin, Jelley, Powell, & Johnston, 2009; McMahon & Jones, 2014; Olson, Goffin, & Haynes, 2007; Shah, Bradley, Parekh, Wainwright, & Ramchandran, 2013). However, comparative judgment in its original implementation has major drawbacks including laborious evaluation process, not able to discern the absolute quality of the performance, and unable to communicate the evaluation standards to examinee effectively. In this study, a new implementation of the comparative judgment method termed authentic rubric was proposed. The authentic rubric replaced the scoring categories in the scoring rubric with expert evaluated performances and asked the raters to compare performances to be evaluated against these anchors.22 raters were recruited to evaluate 100 argumentative essays using either a holistic rubric or the proposed authentic rubric. The authentic rubrics were constructed by selecting essays evaluated by two professional raters. Five hypotheses related to user experience, psychometric properties of the evaluations, and efficiency of the evaluation process were proposed and tested. Among the five hypotheses, only one was confirmed and showed that raters who used authentic rubric found the evaluation experience to be more enjoyable. Nonetheless, examining the data showed that authentic rubric evaluations had marginally higher reliability and criterion-related validity. Post-hoc analysis revealed that a larger sample size may be needed to reach statistical significance conclusion. Implications of the study findings and future areas of research were addressed.
- Other Subject(s)
- Genre(s)
- Dissertation Note
- Ph.D. Pennsylvania State University, 2017.
- Reproduction Note
- Microfilm (positive). 1 reel ; 35 mm. (University Microfilms 13918145)
- Technical Details
- The full text of the dissertation is available as an Adobe Acrobat .pdf file ; Adobe Acrobat Reader required to view the file.
View MARC record | catkey: 20513636