Maintaining Reliable Assessors

Quality and accuracy are not taken lightly in our department. It is in our name after all! Our assessors go through rigorous and continuous training and reliability checks to limit rater drift, eliminate bias, and ensure a high-quality assessment each and every time. Our assessors are trained in a wide variety of assessments so we are able to utilize the tool or scale that best fits a project’s need. For the Environment Rating Scales, our department receives training directly from the Inter-Rater Reliability Team in Clayton’s Research and Evaluation Department. This team has been providing inter-rater reliability services for ratings in our state since 2004 and they are trained directly by the authors of the scales: the Environment Rating Scale Institute (ERSI). ERSI approves state anchors to go back to their state and provide reliability. After an assessor completes the initial trainings on the content of the scales, they then go out with a state anchor for three consecutive observations to obtain reliability. Once reliable, assessors complete reliability checks every 10th observation. Reliability checks consists of assessors observing in the same classroom at the same time and scoring independently. After scoring, scores are reviewed and discussed. Assessors must show their scores align within one point of the state anchor’s score. The reliability process is modeled after how ERSI conducts reliabilities for state anchors nationally. To learn more about the Environment Rating scales and the reliability process, visit ERSI’s website here: https://www.ersi.info/

For the CLASS® tools, each assessor is certified by Teachstone® through their required training. Assessors are then required to pair up to observe and code classrooms every 10th observation; this is called “Double Coding.” Similar to reliability, assessors code independently and discuss scores and the scores must be within one for each dimension. Additionally, Clayton goes through calibrations that are hosted by Teachstone® to measure coding accuracy. To learn more about CLASS® and Teachstone®, visit Teachstone’s website here: https://teachstone.com/

Assessors may not observe independently if they do not maintain reliability or accurate double codes. Throughout the year, our state anchors and project managers provide continuous training to ensure consistency and continuous professional development on the assessments we utilize. These rigorous reliability, double coding, and training processes are unique to Clayton Early Learning, as our organization values having the highest quality, most qualified staff across our organization.

connect