Development of Computer-Based Chemical Five-Tier Diagnostic Test Instruments: A Generalized Partial Credit Model
DOI:
https://doi.org/10.7160/eriesj.2024.170108Keywords:
Five-tier, chemical diagnostic, computer-based test, Generalized Partial Credit ModelAbstract
This study focuses on developing a five-tier chemical diagnostic test based on a computer-based test with 11 assessment categories with an assessment score from 0 to 10. A total of 20 items produced were validated by education experts, material experts, measurement experts, and media experts and obtained an average index of the Aiken test > 0.70. The validation results were tested on 580 respondents and analyzed using the Generalized Partial Credit Model (GPCM) Item Response Theory (IRT) type. The results of the analysis show that all of the items meet the requirements to be said to be valid for the model; the evidence of the value this: RMSEA < 0.08, CFI > 0.87, SRMR < 0.10, GFI > 0.90, NFI > 0.90, NNFI > 0.90, IFI > 0.90, TLI > 0.90, and RFI > 0.90, and all items were obtained has a p.S_X2 value greater than 0.05 which indicates that all items developed are fit and by the GPCM model. The construct reliability (CR) value is 0.99, which suggests the construct is reliable. The most challenging item is item 9, and the most accessible item is item 4
References
Aiken, L. R. (1980) 'Content validity and reliability of single items or questionnaires', Educational and Psychological Measurement, Vol. 40, No. 4, pp. 955-959. https://doi.org/10.1177/001316448004000419
Anam, R. S., Widodo, A., Sopandi, W. and Wu, H. K. (2019) 'Developing a five-tier diagnostic test to identify students’ misconceptions in science: an example of the heat transfer concepts', Elementary Education Online, Vol. 18, No. 3, pp. 1014-1029. https://doi.org/10.17051/ilkonline.2019.609690
Barke, H. D., Hazari, A. and Sileshi Yitbarek (2009) Misconception in Chemistry, Berlin: Springer.
Bayuni, T. C., Sopandi, W. and Sujana, A. (2018) 'Identification misconception of primary school teacher education students in changes of matters using a five-tier diagnostic test', Journal of Physics: Conference Series, Bandung, Vol. 1013, p. 012086 https://doi.org/10.1088/1742-6596/1013/1/012086
Chi, S., Wang, Z., Luo, M., Yang, Y. and Huang, M. (2018) 'Student progression on chemical symbol representation abilities at different grade levels (Grades 10-12) across gender', Chemistry Education Research and Practice, Vol. 19, No. 4, pp. 1055-1064. https://doi.org/10.1039/c8rp00010g
Cornick, J. E. (2015) 'Factor structure of the exercise self-efficacy scale', Measurement in Physical Education and Exercise Science, Vol. 19, No. 4, pp. 208-218. https://doi.org/10.1080/1091367X.2015.1074579
Crocker, L. and Algina, J. (2008) Introduction to Classical and Modern Test Theory, 2nd edition, Mason: Cengage Learning.
Dagnall, N., Denovan, A., Parker, A., Drinkwater, K. and Stephen Walsh, R. (2018) 'Confirmatory factor analysis of the inventory of personality organization-reality testing subscale', Frontiers in Psychology, Vol. 9, 1116, pp. 1-12. https://doi.org/10.3389/fpsyg.2018.01116
Debelak, R., Stobl, C. and Zeigenfuse, M. D. (2022) An Introduction to the Rasch Model with Examples in R, New York: CRC Press. https://doi.org/10.1201/9781315200620
Demars, C. (2010) Item Response Theory, New York: Oxford University Press.
Derkach, T. M. (2021) 'The origin of misconceptions in inorganic chemistry and their correction by computer modelling', Journal of Physics: Conference Series, Kryvyi Rih, Vol. 1840, pp. 1-13. https://doi.org/10.1088/1742-6596/1840/1/012012
Dewanti, S. S., Hadi, S. and Nu’man, M. (2021) 'The application of item response theory in analysis of characteristics of mathematical literacy test items', İlköğretim Online, Vol. 20, No. 1, pp. 1226-1237. https://doi.org/10.17051/ilkonline.2021.01.119
Dewi, F. C., Parlan, P. and Suryadharma, I. B. (2020) 'Development of four-tier diagnostic test for identifying misconception in chemical equilibrium', AIP Conference Proceedings, Vol. 2215, pp. 1-6. https://doi.org/10.1063/5.0000510
Drasgow, F. and Mattern, K. (2006) 'New tests and new items: Opportunities and issues', in Bartram, D. and Hambleton, R. K. (ed.) Computer-Based Testing and the Internet, Chichester: John Wiley & Sons Ltd.
Ghozali, I. and Fuad (2008) Structural equation modeling: teori, konsep, dan aplikasi dengan Program Lisrel 8.80, Semarang: Badan Penerbit Universitas Diponegoro.
Groen, M. M. van and Eggen, T. J. H. M. (2020) 'Educational Test Approaches: The Suitability of Computer-Based Test Types for Assessment and Evaluation in Formative and Summative Contexts', Journal of Applied Testing Technology, Vol. 21, No. 1, pp. 12-24.
Hambleton, R. K. (2006) 'Psychometric Models, Test Designs and Item Types for the Next Generation of Educational and Psychological Tests', in Bartram, D. and Hambleton, R. K. (ed.) Computer Based Testing and the Internet, Chichester: John Wiley & Sons Ltd.
Hambleton, R. K. and Swaminathan, H. (1985) Item Response Theory: Principles and Applications, New York: Springer Science+Business Media. https://doi.org/10.1007/978-94-017-1988-9
Hambleton, R. K., Swaminathan, H. and Rogers, H. J. (1991) Fundamentals of Item Response Theory Library, London: Sage Publications.
Harza, A. E. K. P., Wiji, W. and Mulyani, S. (2021) 'Potency to overcome misconceptions by using multiple representations on the concept of chemical equilibrium', Journal of Physics: Conference Series, Jawa Barat, Vol. 1806, pp. 1-6. https://doi.org/10.1088/1742-6596/1806/1/012197
Igbaria, M., Zinatelli, N., Cragg, P. and Cavaye, A. L. M. (1997) 'Personal computing acceptance factors in small firms: A structural equation model', MIS Quarterly, Vol. 21, No. 3, pp. 279-305. https://doi.org/10.2307/249498
Istiyono, E., Dwandaru, W. S. B., Erfianti, L. and Astuti, W. (2020) 'Applying CBT in physics learning to measure students’ higher order thinking skills', Journal of Physics: Conference Series, Yogyakarta, Vol. 1440, p. 012061. https://doi.org/10.1088/1742-6596/1440/1/012061
Istiyono, Edi, Dwandaru, W. S. B., Fenditasari, K., Ayub, M. R. S. S. N. and Saepuzaman, D. (2023) 'The development of a four-tier diagnostic test based on modern Test theory in physics education', European Journal of Educational Research, Vol. 12, No. 1, pp. 371-385. https://doi.org/10.12973/eu-jer.12.1.371
Jusniar, J., Effendy, E., Budiasih, E. and Sutrisno, S. (2021) 'Eliminating misconceptions on reaction rate to enhance conceptual understanding of chemical equilibrium using EMBE-R strategy', International Journal of Instruction, Vol. 14, No. 1, pp. 85-104. https://doi.org/10.29333/IJI.2021.1416A
Kang, T. and Chen, T. T. (2011) 'Performance of the generalized S-X2 item fit index for the graded response model', Asia Pacific Education Review, Vol. 12, No. 1, pp. 89-96. https://doi.org/10.1007/s12564-010-9082-4
Kay, C. C., Yiin, H. K., Chu, C. K. and Hong, K. Y. (2010) 'Misconceptions in the teaching of chemistry in secondary schools in Singapore & Malaysia', Proceedings of the Sunway Academic Conference, Petaling Jaya, No. 1996, pp. 1-10.
Kimmons, R., Clark, B. and Lim, M. (2017) 'Understanding web activity patterns among teachers, students and teacher candidates', Journal of Computer Assisted Learning, Vol. 33, No. 6, pp. 588-596. https://doi.org/10.1111/jcal.12202
Kiray, S. A. and Simsek, S. (2020) 'Determination and evaluation of the science teacher candidates’ misconceptions about density by using four-tier diagnostic test', International Journal of Science and Mathematics Education, Vol. 19, No. 5, pp. 935-955. https://doi.org/10.1007/s10763-020-10087-5
Kwahk, K. Y. and Lee, J. N. (2008) 'The role of readiness for change in ERP implementation: Theoretical bases and empirical validation', Information and Management, Vol. 45, No. 7, pp. 474-481. https://doi.org/10.1016/j.im.2008.07.002
Lamichhane, R., Reck, C. and Maltese, A. V. (2018) 'Undergraduate chemistry students’ misconceptions about reaction coordinate diagrams', Chemistry Education Research and Practice, Vol. 19, No. 3, pp. 834-845. https://doi.org/10.1039/c8rp00045j
Linden, W. J. van der (2018) Handbook of Item Response Theory Volume Three: Applications, Boca Raton: CRC Press.
Linden, W. J. and Hambleton, R. K. (1997) Handbook of Modern Item Response Theory, New York: Springer.
Llanos, J., Fernández-Marchante, C. M., García-Vargas, J. M., Lacasa, E., De La Osa, A. R., Sanchez-Silva, M. L., De Lucas-Concuegra, A., Garcia, M. T. and Borreguero, A. M. (2021) 'Game-based learning and just-in-time teaching to address misconceptions and improve safety and learning in laboratory activities', Journal of Chemical Education, Vol. 98, No. 10, pp. 3118-3130. https://doi.org/10.1021/acs.jchemed.0c00878
Lowyck, J. (2014) Handbook of Research on Educational Communications and Technology New York: Springer. https://doi.org/10.1007/978-1-4614-3185-5
Marsh, H. W., Balla, J. R. and Mcdonald, R. P. (1988) 'Goodness-of-fit indexes in confirmatory factor analysis: The effect of sample size', Psychological Bulletin, Vol. 103, No. 3, pp. 391-410. https://doi.org/10.1037/0033-2909.103.3.391
Martin, F. and Betrus, A. K. K. (2019) Digital media for learning: Theories, processes, and solutions digital media for learning: Theories, processes, and solutions, Cham: Springer Nature Switzerland. https://doi.org/10.1007/978-3-030-33120-7
Merriënboer, J. J. G. van and Bruin, A. B. H. de (2014) 'Research Paradigms and Perspectives on Learning', in Spector, J. M., Merrill, M. D., Elen, J. and Bishop M. J. (ed.) Handbook of Research on Educational Communications and Technology, New York: Springer.
Mills, C. n. and Breithaupt, K. J. (2016) Educational Measurement From Foundations to Future, New York: The Guilford Press.
Millsap, R. E. and Kwok, O. M. (2004) 'Evaluating the impact of partial factorial invariance on selection in two populations', Psychological Methods, Vol. 9, No. 1, pp. 93-115. https://doi.org/10.1037/1082-989X.9.1.93
Muraki, E. (1992) 'A generalized partial credit model: Application of an EM algorithm', Applied Psychological Measurement, Vol. 16, No. 2, pp. 159-176. https://doi.org/10.1177/014662169201600206
Murniningsih, Muna, K. and Irawati, R. K. (2020) 'Analysis of misconceptions by four tier tests in electrochemistry, case study on students of the chemistry education study program UIN Antasari Banjarmasin', Journal of Physics: Conference Series, Vol. 1440, pp. 1-6. https://doi.org/10.1088/1742-6596/1440/1/012008
Mutlu, A. and Sesen, B. A. (2015) 'Development of a two-tier diagnostic test to assess undergraduates’ understanding of some chemistry concepts', Procedia - Social and Behavioral Sciences, Vol. 174, pp. 629-635. https://doi.org/10.1016/j.sbspro.2015.01.593
Nicoll, G. (2001) 'A report of undergraduates’ bonding misconceptions', International Journal of Science Education, Vol. 23, No. 7, pp. 707-730. https://doi.org/10.1080/09500690010025012
Oriondo, L. L. and Antonio, E. M. D. (1984) Evaluating Educational Outcomes (Tests, Measurement and Evaluation), Manila: Rex Book Store.
Özmen, H. (2004) 'Some student misconceptions in chemistry: A literature review of chemical bonding', Journal of Science Education and Technology, Vol. 13, No. 2, pp. 147-159. https://doi.org/10.1023/b:jost.0000031255.92943.6d
Pokorný, M. (2023) 'Experience with online learning of mathematics in primary education', International Journal of Emerging Technologies in Learning (IJET), Vol. 18, No. 02, pp. 203-213. https://doi.org/10.3991/ijet.v18i02.35401
Potvin, P. (2023) 'Response of science learners to contradicting information: A review of research', Studies in Science Education, Vol. 59, No. 1, pp. 67-108. https://doi.org/10.1080/03057267.2021.2004006
Prodjosantoso, A. K., Hertina, A. M. and Irwanto (2019) 'The misconception diagnosis on ionic and covalent bonds concepts with three tier diagnostic test', International Journal of Instruction, Vol. 12, No. 1, pp. 1477-1488. https://doi.org/10.29333/iji.2019.12194a
Putra, A. S. U., Hamidah, I. and Nahadi (2020) 'The development of five-tier diagnostic test to identify misconceptions and causes of students’ misconceptions in waves and optics materials', Journal of Physics: Conference Series, Vol. 1521, No. 2. https://doi.org/10.1088/1742-6596/1521/2/022020
Rupp, A. A. and Zumbo, B. D. (2006) 'Understanding parameter invariance in unidimensional IRT models', Educational and Psychological Measurement, Vol. 66, No. 1, pp. 63-84. https://doi.org/10.1177/0013164404273942
Rusmini, Suyono, Jatmiko, B. and Yonata, B. (2021) 'The diagnosis of misconception on the concept of acid-base theory in prospective teacher students used a three-tier test', Journal of Physics: Conference Series, Vol. 1899, No. 1, pp. 1-7. https://doi.org/10.1088/1742-6596/1899/1/012061
Suparman, A. R., Rohaeti, E. and Wening, S. (2022) 'Development of attitude assessment instruments towards socioscientific issues in chemistry learning', European Journal of Educational Research, Vol. 11, No. 4, pp. 1947-1958. https://doi.org/10.12973/eujer.11.4.1947
Suparman, A. R., Rohaeti, E. and Wening, S. (2023) 'Effect of computer based test on motivation: A meta-analysis', European Journal of Educational Research, Vol. 12, No. 4, pp. 1583-1599. https://doi.org/10.12973/eu-jer.12.4.1583
Tie, B., Chen, G. and He, J. (2022) 'Validation of the inflexible eating questionnaire in a large sample of Chinese adolescents: psychometric properties and gender-related differential item functioning', Eating and Weight Disorders, Vol. 27, No. 3, pp. 1029-1037. https://doi.org/10.1007/s40519-021-01239-9
Tien, L. T. and Osman, K. (2017) Overcoming Students’ Misconceptions in Science, Singapore: Springer Nature. https://doi.org/10.1007/978-981-10-3437-4
Treagust, D., Chittleborough, G. and Mamiala, T. (2003) 'The role of submicroscopic and symbolic representations in chemical explanations', International Journal of Science Education, Vol. 25, No. 11, pp. 1353-1368. https://doi.org/10.1080/0950069032000070306
Üce, M. and Ceyhan, İ. (2019) 'Misconception in chemistry education and practices to eliminate them: Literature analysis', Journal of Education and Training Studies, Vol. 7, No. 3, pp. 202. https://doi.org/10.11114/jets.v7i3.3990
Vladusic, R., Bucat, R. B. and Ozic, M. (2022) 'Understanding covalent bonding - a scan across the Croatian education system', Chemistry Education Research and Practice, Vol. 23, No. 4. https://doi.org/10.1039/D2RP00039C
Wang, F., Franco-Penya, H.-H., Kelleher, J. D., Pugh, J. and Ross, R. (2017) 'An analysis of the application of simplified silhouette to the evaluation of k-means clustering validity', International Conference Machine Learning and Data Mining in Pattern Recognition, Vol. 13, pp. 291-305. https://doi.org/10.1007/978-3-319-62416-7_21
Wang, Z., Yan, W., Zeng, C., Tian, Y. and Dong, S. (2023) 'A unified interpretable intelligent learning diagnosis framework for learning performance prediction in intelligent tutoring systems', International Journal of Intelligent Systems, Vol. 2023, pp. 1-20. https://doi.org/10.1155/2023/4468025
Widarti, H. R., Permanasari, A., Mulyani, S., Rokhim, D. A. and Habiddin (2021) 'Multiple representation-based learning through cognitive dissonance strategy to reduce student’s misconceptions in volumetric analysis', TEM Journal, Vol. 10, No. 3, pp. 1263-1273. https://doi.org/10.18421/TEM103-33
Additional Files
Published
How to Cite
Issue
Section
License
Copyright (c) 2024 Achmad Rante Suparman, Eli Rohaeti, Sri Wening
This work is licensed under a Creative Commons Attribution 4.0 International License.
Authors declare with this manuscript intended for publication to ERIES Journal that:
- all co-authors agree with the publication of the manuscript even after amendments arising from peer review;
- all co-authors agree with the posting of the full text of this work on the web page of ERIES Journal and to the inclusion of references in databases accessible on the internet;
- no results of other researchers were used in the submitted manuscript without their consent, proper citation, or acknowledgement of their cooperation or material provided;
- the results (or any part of them) used in the manuscript have not been sent for publication to any other journal nor have they already been published (or if so, that the relevant works are cited in this manuscript);
- submission of the manuscript for publication was completed in accordance with the publishing regulations pertaining to place of work;
- experiments performed comply with current laws and written consent of the Scientific Ethics Committee / National Animal Care Authority (as is mentioned in the manuscript submitted);
- grant holders confirm that they have been informed of the submitted manuscript and they agree to its publication.
Authors retain copyright and grant ERIES Journal right of first publication with the work simultaneously licensed under a Creative Commons Attribution License that allows others to share the published work with an acknowledgement of its initial publication in ERIES Journal. Moreover, authors are able to post the published work in an institutional repository with an acknowledgement of its initial publication in ERIES Journal. In addition, authors are permitted and encouraged to post the published work online (e.g. institutional repositories or on their website) as it can lead to productive exchanges, as well as earlier and greater citation of published work.
