BAYESIAN DIAGNOSTICS FOR TEST DESIGN AND ANALYSIS

  • Rajitha Silva Department of Statistics, University of Sri Jayewardenepura, Sri Lanka
  • Yuping Guan Pelesys Learning Systems Inc, Canada
  • Tim Swartz Department of Statistics and Actuarial Science, Simon Fraser University, 8888 University Drive, Burnaby, V5A1S6, Canada

Abstract

This paper attempts to bridge the gap between classical test theory and item response theory. It is demonstrated that the familiar and popular statistics used in classical test theory can be translated into a Bayesian framework where all of the advantages of the Bayesian paradigm can be realized. In particular, prior opinion can be introduced and inferences can be obtained using posterior distributions. In addition, the use of the JAGS programming language facilitates extensions to more complex scenarios involving the assessment of tests and questionnaires.

References


  • An, X., Yung, Y.-F. (2014) ‘Item response theory: what it is and how you can use the IRT procedure to apply it’. SAS Institute Inc., Cary NC, Paper SAS364-2014. [online], Available:  https://pdfs.semanticscholar.org/d85a/7025441f5685b287b53234ce6456dcd40192.pdf [5 Jun 2017].

  • Brozova, H., Rydval J. (2014) ‘Analysis of exam results of the subject ”Applied Mathematics for It”’, Journal on Efficiency and Responsibility in Education and Science, vol. 7, no. 3-4, pp. 59-65. https://doi.org/10.7160/eriesj.2014.070303

  • Cai, L., Choi, K., Hansen, M., Harrell, L. (2016) ‘Item response theory’. Annual Review of Statistics and Its Application, vol. 3, pp. 297-321. https://doi.org/10.1146/annurev-statistics- 041715-033702

  • Choi, J. (2017) ‘A review of PROC IRT in SAS’, Journal of Educational and Behavioral Statistics, vol. 42, no. 2, pp. 195-205. https://doi.org/10.3102/1076998616664568

  • DeVellis, R.F. (2012) Scale Development: Theory and Applications, Third Edition, Applied Social Methods Research Series, Editors L. Bickman and D.J. Rog, Sage, Los Angeles.

  • Fan, X. (1998) ‘Item response theory and classical test theory: an empirical comparison of their item/person statistics’, Educational and Psychological Measurement, vol. 58, no. 3, pp. 357-381. https://doi.org/10.1177/0013164498058003001

  • Fox, J-P. (2010) Bayesian Item Response Modeling: Theory and Applications, Statistics for Social and Behavioral Sciences Series, Editors S.E. Fienberg and W.J. van der Linden, Springer, New York.

  • Guler, N., Uyanik, G.K., Teker, G.T. (2014) ‘Comparison of classical test theory and item response theory in terms of item parameters’, European Journal of Research on Education, vol. 2, no. 1, pp. 1-6. [Online], Available: http://iassr2.org/rs/020101.pdf [5 Jun 2017].

  • Hambleton, R.K., Jones, R.W. (1993) ‘Comparison of classical test theory and item response theory and their application to test development’, Educational Measurement: Issues and Practice, vol. 12, no. 3, pp. 38-47.  https://doi.org/10.1111/j.1745-3992.1993.tb00543.x

  • Jabrayilov, R., Emons, W.H.M., Sijtsma, K. (2016) ‘Comparison of classical test theory and item response theory in individual change assessment”, Applied Psychological Measurement, vol. 40, no. 8, pp. 559-572. https://doi.org/10.1177/0146621616664046

  • Kohli, N., Koran, J., Henn, L. (2015) ‘Relationships among classical test theory and item response theory frameworks via factor analytic models’, Educational and Psychological Measurement, vol. 75, no. 3, pp. 389-405. https://doi.org/10.1177/0013164414559071

  • Levy, R., Mislevy, R.J. (2016) Bayesian Psychometric Modeling, Chapman & Hall/CRC Statistics in the Social and Behavioral Science Series, Boca Raton.

  • Lunn, D., Jackson, C., Best, N., Thomas, A., Spiegelhalter, D. (2013) The BUGS Book: A Practical Introduction to Bayesian Analysis, Chapman & Hall/CRC Texts in Statistical Science Series, Boca Raton.

  • Plummer, M. (2015). JAGS Version 4.0 User Manual, [Online], Available: http://www.uvm.edu/bbeckage/Teaching/DataAnalysis/Manuals/manual.jags.pdf [5 Jun 2017].

  • Raykov, T., Marcoulides, G.A. (2016) ‘On the relationship between classical test theory and item response theory: from one to the other and back’, Educational and Psychological Measurement, vol. 76, no. 2, pp. 325-338. https://doi.org/10.1177/0013164415576958

  • Sijtsma, K. (2009) ‘On the use, misuse, and the very limited usefulness of Cronbach’s alpha’, Psy- chometrika, vol. 74, no. 1, pp. 107-120.  https://doi.org/10.1007/s11336-008-9101-0

  • Skoda, J., Doulik, P., Hajerova-Mullerova, L. (2006) ‘Zasady spravne tvorby, pouziti a hodnoceni didaktickych testu v pripave budoucich ucitelu’. [Online], Available: http://cvicebnice.ujep.cz/cvicebnice/FRVS1973F5d [5 Jun 2017]

  • Swartz, T.B. (2011) ‘Bayesian clustering with priors on partitions’, Statistica Neerlandica, vol. 65, no. 4, pp. 371-386. https://doi.org/10.1111/j.1467-9574.2011.00490.x

  • Yuan, W., Deng, C., Zhu, H., Li, J. (2012) ‘The statistical analysis and evaluation of exam- ination results of materials research methods course’, Creative Education, vol. 3, pp. 162-164. https://doi.org/10.4236/ce.2012.37B042

Published
2017-07-17
How to Cite
Silva, R.; Guan, Y.; Swartz, T. (2017) 'BAYESIAN DIAGNOSTICS FOR TEST DESIGN AND ANALYSIS', Journal on Efficiency and Responsibility in Education and Science, vol. 10, no. 2, pp. 44-50. https://doi.org/10.7160/eriesj.2017.100202.
Section
Research Paper

Keywords

classical test theory; empirical bayes; item response theory; markov chain monte carlo; JAGS