BAYESIAN DIAGNOSTICS FOR TEST DESIGN AND ANALYSIS

Authors

  • Rajitha Silva Department of Statistics, University of Sri Jayewardenepura, Sri Lanka
  • Yuping Guan Pelesys Learning Systems Inc, Canada
  • Tim Swartz Department of Statistics and Actuarial Science, Simon Fraser University, 8888 University Drive, Burnaby, V5A1S6, Canada

DOI:

https://doi.org/10.7160/eriesj.2017.100202

Keywords:

classical test theory, empirical bayes, item response theory, markov chain monte carlo, JAGS

Abstract

This paper attempts to bridge the gap between classical test theory and item response theory. It is demonstrated that the familiar and popular statistics used in classical test theory can be translated into a Bayesian framework where all of the advantages of the Bayesian paradigm can be realized. In particular, prior opinion can be introduced and inferences can be obtained using posterior distributions. In addition, the use of the JAGS programming language facilitates extensions to more complex scenarios involving the assessment of tests and questionnaires.

References

An, X., Yung, Y.-F. (2014) ‘Item response theory: what it is and how you can use the IRT procedure to apply it’. SAS Institute Inc., Cary NC, Paper SAS364-2014. [online], Available:  https://pdfs.semanticscholar.org/d85a/7025441f5685b287b53234ce6456dcd40192.pdf [5 Jun 2017].

Brozova, H., Rydval J. (2014) ‘Analysis of exam results of the subject ”Applied Mathematics for It”’, Journal on Efficiency and Responsibility in Education and Science, vol. 7, no. 3-4, pp. 59-65. https://doi.org/10.7160/eriesj.2014.070303

Cai, L., Choi, K., Hansen, M., Harrell, L. (2016) ‘Item response theory’. Annual Review of Statistics and Its Application, vol. 3, pp. 297-321. https://doi.org/10.1146/annurev-statistics- 041715-033702

Choi, J. (2017) ‘A review of PROC IRT in SAS’, Journal of Educational and Behavioral Statistics, vol. 42, no. 2, pp. 195-205. https://doi.org/10.3102/1076998616664568

DeVellis, R.F. (2012) Scale Development: Theory and Applications, Third Edition, Applied Social Methods Research Series, Editors L. Bickman and D.J. Rog, Sage, Los Angeles.

Fan, X. (1998) ‘Item response theory and classical test theory: an empirical comparison of their item/person statistics’, Educational and Psychological Measurement, vol. 58, no. 3, pp. 357-381. https://doi.org/10.1177/0013164498058003001

Fox, J-P. (2010) Bayesian Item Response Modeling: Theory and Applications, Statistics for Social and Behavioral Sciences Series, Editors S.E. Fienberg and W.J. van der Linden, Springer, New York.

Guler, N., Uyanik, G.K., Teker, G.T. (2014) ‘Comparison of classical test theory and item response theory in terms of item parameters’, European Journal of Research on Education, vol. 2, no. 1, pp. 1-6. [Online], Available: http://iassr2.org/rs/020101.pdf [5 Jun 2017].

Hambleton, R.K., Jones, R.W. (1993) ‘Comparison of classical test theory and item response theory and their application to test development’, Educational Measurement: Issues and Practice, vol. 12, no. 3, pp. 38-47.  https://doi.org/10.1111/j.1745-3992.1993.tb00543.x

Jabrayilov, R., Emons, W.H.M., Sijtsma, K. (2016) ‘Comparison of classical test theory and item response theory in individual change assessment”, Applied Psychological Measurement, vol. 40, no. 8, pp. 559-572. https://doi.org/10.1177/0146621616664046

Kohli, N., Koran, J., Henn, L. (2015) ‘Relationships among classical test theory and item response theory frameworks via factor analytic models’, Educational and Psychological Measurement, vol. 75, no. 3, pp. 389-405. https://doi.org/10.1177/0013164414559071

Levy, R., Mislevy, R.J. (2016) Bayesian Psychometric Modeling, Chapman & Hall/CRC Statistics in the Social and Behavioral Science Series, Boca Raton.

Lunn, D., Jackson, C., Best, N., Thomas, A., Spiegelhalter, D. (2013) The BUGS Book: A Practical Introduction to Bayesian Analysis, Chapman & Hall/CRC Texts in Statistical Science Series, Boca Raton.

Plummer, M. (2015). JAGS Version 4.0 User Manual, [Online], Available: http://www.uvm.edu/∼bbeckage/Teaching/DataAnalysis/Manuals/manual.jags.pdf [5 Jun 2017].

Raykov, T., Marcoulides, G.A. (2016) ‘On the relationship between classical test theory and item response theory: from one to the other and back’, Educational and Psychological Measurement, vol. 76, no. 2, pp. 325-338. https://doi.org/10.1177/0013164415576958

Sijtsma, K. (2009) ‘On the use, misuse, and the very limited usefulness of Cronbach’s alpha’, Psy- chometrika, vol. 74, no. 1, pp. 107-120.  https://doi.org/10.1007/s11336-008-9101-0

Skoda, J., Doulik, P., Hajerova-Mullerova, L. (2006) ‘Zasady spravne tvorby, pouziti a hodnoceni didaktickych testu v pripave budoucich ucitelu’. [Online], Available: http://cvicebnice.ujep.cz/cvicebnice/FRVS1973F5d [5 Jun 2017]

Swartz, T.B. (2011) ‘Bayesian clustering with priors on partitions’, Statistica Neerlandica, vol. 65, no. 4, pp. 371-386. https://doi.org/10.1111/j.1467-9574.2011.00490.x

Yuan, W., Deng, C., Zhu, H., Li, J. (2012) ‘The statistical analysis and evaluation of exam- ination results of materials research methods course’, Creative Education, vol. 3, pp. 162-164. https://doi.org/10.4236/ce.2012.37B042

Vol 10 No 2 (2017)

Additional Files

Published

2017-07-17

How to Cite

Silva, R., Guan, Y. and Swartz, T. (2017) ’BAYESIAN DIAGNOSTICS FOR TEST DESIGN AND ANALYSIS’, Journal on Efficiency and Responsibility in Education and Science, vol. 10, no. 2, pp. 44–50. https://doi.org/10.7160/eriesj.2017.100202

Issue

Vol. 10 No. 2 (2017)

Section

Research Paper

License

Authors declare with this manuscript intended for publication to ERIES Journal that:

  • all co-authors agree with the publication of the manuscript even after amendments arising from peer review;
  • all co-authors agree with the posting of the full text of this work on the web page of ERIES Journal and to the inclusion of references in databases accessible on the internet;
  • no results of other researchers were used in the submitted manuscript without their consent, proper citation, or acknowledgement of their cooperation or material provided;
  • the results (or any part of them) used in the manuscript have not been sent for publication to any other journal nor have they already been published (or if so, that the relevant works are cited in this manuscript);
  • submission of the manuscript for publication was completed in accordance with the publishing regulations pertaining to place of work;
  • experiments performed comply with current laws and written consent of the Scientific Ethics Committee / National Animal Care Authority (as is mentioned in the manuscript submitted);
  • grant holders confirm that they have been informed of the submitted manuscript and they agree to its publication.

Authors retain copyright and grant ERIES Journal right of first publication with the work simultaneously licensed under a Creative Commons Attribution License that allows others to share the published work with an acknowledgement of its initial publication in ERIES Journal. Moreover, authors are able to post the published work in an institutional repository with an acknowledgement of its initial publication in ERIES Journal. In addition, authors are permitted and encouraged to post the published work online (e.g. institutional repositories or on their website) as it can lead to productive exchanges, as well as earlier and greater citation of published work.