Abstract #9

# 9
Model evaluation.
E. Kebreab*1, 1University of California, Davis, Davis, CA.

Statistical measures of model performance commonly compare predictions with observations judged to be reliable. Model evaluation indicates the level of accuracy and precision of model predictions by assessing the credibility or reliability of a model in comparison to real-world observations. Quantitative statistical model evaluation methods can be classified into 3 types including (1) standard regression statistics, which determines strength of linear relationship, (2) error index, which quantifies deviation in observed units, and (3) relative model evaluation that are dimensionless. Within the first category, analysis of residuals involves regressing residuals against predicted or other model variables. In this method, the model is unbiased if residuals are not correlated with predictions and the slope is not significantly different from zero. Predicted values can also be centered making the slope and intercept estimates in the regression orthogonal and thus, independent. This allows for mean biases to be assessed using the intercepts of the regression equations, and the slopes to determine the presence of linear biases. Mean square error of prediction (MSEP) and its square root (RMSEP) are commonly used methods of evaluation. In general RMSEP values less than half of observed SD may be considered having a good performance. The MSEP can be decomposed into error due to 1) overall bias of prediction, 2) deviation of the regression slope from unity, and 3) disturbance. Examples of the third category include concordance correlation coefficient (CCC), and the Nash-Sutcliffe index (NSE). The CCC can be represented as a product of 2 components (range from 0 to 1 and 1 indicates perfect fit): a correlation coefficient estimate that measures precision and a bias correction factor that indicates how far the regression line deviates from the line of unity. The NSE is a normalized statistic that determines relative magnitude of residual variance compared with observed data variance. During model evaluation, a combination of the methods described above should be used to gain insight on model performance. The hands-on excercises include coding a function to calculate RMSEP, NSE and CCC for a set of data, which will be provided to participants.

Key Words: model performance, modeling, prediction accuracy