BAYESIAN DIAGNOSTICS FOR TEST DESIGN AND ANALYSIS

Authors

  • Rajitha Silva Department of Statistics, University of Sri Jayewardenepura, Sri Lanka
  • Yuping Guan Pelesys Learning Systems Inc, Canada
  • Tim Swartz Department of Statistics and Actuarial Science, Simon Fraser University, 8888 University Drive, Burnaby, V5A1S6, Canada

DOI:

https://doi.org/10.7160/eriesj.2017.100202

Keywords:

classical test theory, empirical bayes, item response theory, markov chain monte carlo, JAGS

Abstract

This paper attempts to bridge the gap between classical test theory and item response theory. It is demonstrated that the familiar and popular statistics used in classical test theory can be translated into a Bayesian framework where all of the advantages of the Bayesian paradigm can be realized. In particular, prior opinion can be introduced and inferences can be obtained using posterior distributions. In addition, the use of the JAGS programming language facilitates extensions to more complex scenarios involving the assessment of tests and questionnaires.

References

An, X., Yung, Y.-F. (2014) ‘Item response theory: what it is and how you can use the IRT procedure to apply it’. SAS Institute Inc., Cary NC, Paper SAS364-2014. [online], Available:  https://pdfs.semanticscholar.org/d85a/7025441f5685b287b53234ce6456dcd40192.pdf [5 Jun 2017].

Brozova, H., Rydval J. (2014) ‘Analysis of exam results of the subject ”Applied Mathematics for It”’, Journal on Efficiency and Responsibility in Education and Science, vol. 7, no. 3-4, pp. 59-65. https://doi.org/10.7160/eriesj.2014.070303

Cai, L., Choi, K., Hansen, M., Harrell, L. (2016) ‘Item response theory’. Annual Review of Statistics and Its Application, vol. 3, pp. 297-321. https://doi.org/10.1146/annurev-statistics- 041715-033702

Choi, J. (2017) ‘A review of PROC IRT in SAS’, Journal of Educational and Behavioral Statistics, vol. 42, no. 2, pp. 195-205. https://doi.org/10.3102/1076998616664568

DeVellis, R.F. (2012) Scale Development: Theory and Applications, Third Edition, Applied Social Methods Research Series, Editors L. Bickman and D.J. Rog, Sage, Los Angeles.

Fan, X. (1998) ‘Item response theory and classical test theory: an empirical comparison of their item/person statistics’, Educational and Psychological Measurement, vol. 58, no. 3, pp. 357-381. https://doi.org/10.1177/0013164498058003001

Fox, J-P. (2010) Bayesian Item Response Modeling: Theory and Applications, Statistics for Social and Behavioral Sciences Series, Editors S.E. Fienberg and W.J. van der Linden, Springer, New York.

Guler, N., Uyanik, G.K., Teker, G.T. (2014) ‘Comparison of classical test theory and item response theory in terms of item parameters’, European Journal of Research on Education, vol. 2, no. 1, pp. 1-6. [Online], Available: http://iassr2.org/rs/020101.pdf [5 Jun 2017].

Hambleton, R.K., Jones, R.W. (1993) ‘Comparison of classical test theory and item response theory and their application to test development’, Educational Measurement: Issues and Practice, vol. 12, no. 3, pp. 38-47.  https://doi.org/10.1111/j.1745-3992.1993.tb00543.x

Jabrayilov, R., Emons, W.H.M., Sijtsma, K. (2016) ‘Comparison of classical test theory and item response theory in individual change assessment”, Applied Psychological Measurement, vol. 40, no. 8, pp. 559-572. https://doi.org/10.1177/0146621616664046

Kohli, N., Koran, J., Henn, L. (2015) ‘Relationships among classical test theory and item response theory frameworks via factor analytic models’, Educational and Psychological Measurement, vol. 75, no. 3, pp. 389-405. https://doi.org/10.1177/0013164414559071

Levy, R., Mislevy, R.J. (2016) Bayesian Psychometric Modeling, Chapman & Hall/CRC Statistics in the Social and Behavioral Science Series, Boca Raton.

Lunn, D., Jackson, C., Best, N., Thomas, A., Spiegelhalter, D. (2013) The BUGS Book: A Practical Introduction to Bayesian Analysis, Chapman & Hall/CRC Texts in Statistical Science Series, Boca Raton.

Plummer, M. (2015). JAGS Version 4.0 User Manual, [Online], Available: http://www.uvm.edu/bbeckage/Teaching/DataAnalysis/Manuals/manual.jags.pdf [5 Jun 2017].

Raykov, T., Marcoulides, G.A. (2016) ‘On the relationship between classical test theory and item response theory: from one to the other and back’, Educational and Psychological Measurement, vol. 76, no. 2, pp. 325-338. https://doi.org/10.1177/0013164415576958

Sijtsma, K. (2009) ‘On the use, misuse, and the very limited usefulness of Cronbach’s alpha’, Psy- chometrika, vol. 74, no. 1, pp. 107-120.  https://doi.org/10.1007/s11336-008-9101-0

Skoda, J., Doulik, P., Hajerova-Mullerova, L. (2006) ‘Zasady spravne tvorby, pouziti a hodnoceni didaktickych testu v pripave budoucich ucitelu’. [Online], Available: http://cvicebnice.ujep.cz/cvicebnice/FRVS1973F5d [5 Jun 2017]

Swartz, T.B. (2011) ‘Bayesian clustering with priors on partitions’, Statistica Neerlandica, vol. 65, no. 4, pp. 371-386. https://doi.org/10.1111/j.1467-9574.2011.00490.x

Yuan, W., Deng, C., Zhu, H., Li, J. (2012) ‘The statistical analysis and evaluation of exam- ination results of materials research methods course’, Creative Education, vol. 3, pp. 162-164. https://doi.org/10.4236/ce.2012.37B042

Additional Files

Published

2017-07-17

How to Cite

Silva, R., Guan, Y. and Swartz, T. (2017) ’BAYESIAN DIAGNOSTICS FOR TEST DESIGN AND ANALYSIS’, Journal on Efficiency and Responsibility in Education and Science, vol. 10, no. 2, pp. 44–50. https://doi.org/10.7160/eriesj.2017.100202

Issue

Section

Research Paper