skip to main content skip to footer

Model Adequacy Checking for Applying Harmonic Regression to Assessment Quality Control RMSE

Author(s):
Qian, Jiahe; Li, Shuhong
Publication Year:
2021
Report Number:
RR-21-13
Source:
ETS Research Report
Document Type:
Report
Page Count:
26
Subject/Key Words:
Harmonic Regression, Jackknife Estimation, Quality Control, Model Fit, Seasonality, Cross-Validation, Prediction Errors, Root Mean Square Error (RMSE)

Abstract

In recent years, harmonic regression models have been applied to implement quality control for educational assessment data consisting of multiple administrations and displaying seasonality. As with other types of regression models, it is imperative that model adequacy checking and model fit be appropriately conducted. However, there has been no literature on how to perform a comprehensive model adequacy evaluation when applying harmonic regression models to sequential data with seasonality in the educational assessment field. This paper is intended to fill this gap with an illustration of real data from an English language assessment. Two types of cross-validation, leave-one-out and out-of-sample, were designed to measure prediction errors and check model validation. Three types of R-squared (R squared, R subscript adj squared, and R subscript pred squared) and various residual diagnostics were applied to check model adequacy and model fitting.

Read More