Assessing and Classification of Academic Efficiency in Engineering Teaching Programs
DOI:
https://doi.org/10.7160/eriesj.2021.140104Keywords:
efficiency, higher education, machine learning, predictive evaluationAbstract
This research uses a three-phase method to evaluate and forecast the academic efficiency of engineering programs. In the first phase, university profiles are created through cluster analysis. In the second phase, the academic efficiency of these profiles is evaluated through Data Envelopment Analysis. Finally, a machine learning model is trained and validated to forecast the categories of academic efficiency. The study population corresponds to 256 university engineering programs in Colombia and the data correspond to the national examination of the quality of education in Colombia in 2018. In the results, two university profiles were identified with efficiency levels of 92.3% and 97.3%, respectively. The Random Forest model presents an Area under ROC value of 95.8% in the prediction of the efficiency profiles. The proposed structure evaluates and predicts university programs’ academic efficiency, evaluating the efficiency between institutions with similar characteristics, avoiding a negative bias toward those institutions that host students with low educational levels.
References
Agasisti, T., Munda, G. and Hippe, R. (2019) ‘Measuring the efficiency of European education systems by combining Data Envelopment Analysis and Multiple-Criteria Evaluation’, Journal of Productivity Analysis, Vol. 51, No. 2, pp. 105–124. https://doi.org/10.1007/s11123-019-00549-6
Aldamak, A. and Zolfaghari, S. (2017) ‘Review of efficiency ranking methods in data envelopment analysis’, Measurement, Vol. 106, pp. 161–172. https://doi.org/10.1016/j.measurement.2017.04.028
Alsabawy, A. Y., Cater-Steel, A. and Soar, J. (2016) ‘Determinants of Perceived Usefulness of e-Learning Systems’, Computers in Human Behavior, Vol. 64, pp. 843–858. https://doi.org/10.1016/j.chb.2016.07.065
Amara, N., Rhaiem, M. and Halilem, N. (2020) ‘Assessing the research efficiency of Canadian scholars in the management field: Evidence from the DEA and fsQCA’, Journal of Business Research, Vol. 115, pp. 296–306. https://doi.org/10.1016/j.jbusres.2019.10.059
Aparicio, J., Cordero, J. M. and Ortiz, L. (2019) ‘Measuring efficiency in education: The influence of imprecision and variability in data on DEA estimates’, Socio-Economic Planning Sciences, Vol. 68, 100698. https://doi.org/10.1016/j.seps.2019.03.004
Aparicio, J., Perelman, S. and Santín, D. (2020) ‘Comparing the evolution of productivity and performance gaps in education systems through DEA: an application to Latin American countries’, Operational Research: An International Journal. https://doi.org/10.1007/s12351-020-00578-2
Benicio, J. and Mello, J. C. S. de (2015) ‘Productivity Analysis and Variable Returns of Scale: DEA Efficiency Frontier Interpretation’, Procedia Computer Science, Vol. 55, pp. 341–349. https://doi.org/10.1016/j.procs.2015.07.059
Berens, J., Schneider, K., Gortz, S., Oster, S. and Burghoff, J. (2019) ‘Early Detection of Students at Risk - Predicting Student Dropouts Using Administrative Student Data from German Universities and Machine Learning Methods’, Journal of Educational Data Mining, Vol. 11, No. 3, pp. 1–41. https://doi.org/10.5281/zenodo.3594771
Bianchi, N. and Giorcelli, M. (2020) ‘Scientific Education and Innovation: From Technical Diplomas to University Stem Degrees’, Journal of the European Economic Association, Vol. 18, No. 5, pp. 2608-2646. https://doi.org/10.1093/jeea/jvz049
Breiman, L. (2001) ‘Random Forests’, Machine Learning, Vol. 45, No. 1, pp. 5-32. https://doi.org/10.1023/A:1010933404324
Charnes, A., Cooper, W. W. and Rhodes, E. (1978) ‘Measuring the efficiency of decision making units’, European Journal of Operational Research, Vol. 2, No. 6, pp. 429–444. https://doi.org/10.1016/0377-2217(78)90138-8
Clayman, C. L., Srinivasan, S. M. and Sangwan, R. S. (2020) ‘K-means Clustering and Principal Components Analysis of Microarray Data of L1000 Landmark Genes’, Procedia Computer Science, Vol. 168, pp. 97-104. https://doi.org/10.1016/j.procs.2020.02.265
Corlu, M. A. and Aydin, E. (2016) ‘Evaluation of Learning Gains Through Integrated STEM Projects’, International Journal of Education in Mathematics, Science and Technology, Vol. 4, No. 1, pp. 20–29. https://doi.org/10.18404/ijemst.35021
de Morais, A. M., Araújo, J. M. F. R. and Costa, E. B. (2014) ‘Monitoring student performance using data clustering and predictive modelling’, In 2014 IEEE Frontiers in Education Conference (FIE) Proceedings, pp. 1–8. https://doi.org/10.1109/FIE.2014.7044401
Delahoz-Dominguez, E., Zuluaga, R. and Fontalvo-Herrera, T. (2020) ‘Dataset of academic performance evolution for engineering students’, Data in Brief, Vol. 30, 105537. https://doi.org/10.1016/j.dib.2020.105537
Duan, S. X. (2019) ‘Measuring university efficiency: An application of data envelopment analysis and strategic group analysis to Australian universities’, Benchmarking: An International Journal, Vol. 26, No. 4, pp. 1161–1173. https://doi.org/10.1108/BIJ-10-2017-0274
Durairaj, M. and Vijitha, C. (2014) ‘Educational data mining for prediction of student performance using clustering algorithms’, International Journal of Computer Science and Information Technologies, Vol. 5, No. 4, pp. 5987–5991.
Ferro, G. and D’Elia, V. (2020) ‘Higher Education Efficiency Frontier Analysis: A Review of Variables to Consider’, Vol. 13, No. 3, 140-153. https://doi.org/10.7160/eriesj.2020.130304
Fuentes, R., Fuster, B. and Lillo-Bañuls, A. (2016) ‘A three-stage DEA model to evaluate learning-teaching technical efficiency: Key performance indicators and contextual variables’, Expert Systems with Applications, Vol. 48, pp. 89–99. https://doi.org/10.1016/j.eswa.2015.11.022
Ghasemi, N., Najafi, E., Hoseinzadeh Lotfi, F. and Movahedi Sobhani, F. (2020) ‘Assessing the performance of organizations with the hierarchical structure using data envelopment analysis: An efficiency analysis of Farhangian University’, Measurement, Vol. 156, 107609. https://doi.org/10.1016/j.measurement.2020.107609
Gralka, S., Wohlrabe, K. and Bornmann, L. (2019) ‘How to measure research efficiency in higher education? Research grants vs. publication output’, Journal of Higher Education Policy and Management, Vol. 41, No. 3, pp. 322–341. https://doi.org/10.1080/1360080X.2019.1588492
Granadillo, E. D. L. H., Gomez, J. M. and Herrera, T. J. F. (2019) ‘Methodology with multivariate calculation to define and evaluate financial productivity profiles of the chemical sector in Colombia’, International Journal of Productivity and Quality Management, Vol. 27, No. 2, pp. 144–160. https://doi.org/10.1504/IJPQM.2019.100141
Heidrich, L., Victória Barbosa, J. L., Cambruzzi, W., Rigo, S. J., Martins, M. G. and dos Santos, R. B. S. (2018) ‘Diagnosis of learner dropout based on learning styles for online distance learning’, Telematics and Informatics, Vol. 35, No. 6, pp. 1593–1606. https://doi.org/10.1016/j.tele.2018.04.007
Hoeg, D. G. and Bencze, J. L. (2017) ‘Values Underpinning STEM Education in the USA: An Analysis of the Next Generation Science Standards’, Science Education, Vol. 101, No. 2, pp. 278–301. https://doi.org/10.1002/sce.21260
ICFES (2020) Estructura general del examen Icfes Saber Pro - Portal Icfes, [Online], Available at: https://www.icfes.gov.co/web/guest/acerca-examen-saber-11 [26 Oct 2020].
Johnes, J. (2006) ‘Data envelopment analysis and its application to the measurement of efficiency in higher education’, Economics of Education Review, Vol. 25, No. 3, pp. 273–288. https://doi.org/10.1016/j.econedurev.2005.02.005
Khan, S. U., Khan, Z. and Hameed, G. (2019) ‘Efficiency Assessment of Public Education & Health Sector in Selected Middle-Income Countries with Special Reference to Millennium Development Goals (MDGs)’, Journal of Applied Economics and Business Studies, Vol. 3, No. 1, pp. 41–60. https://doi.org/10.34260/jaebs.313
Klumpp, M. (2018) ‘The Index Number Problem with DEA: Insights from European University Efficiency Data’, Education Sciences, Vol. 8, No. 2, pp. 79. https://doi.org/10.3390/educsci8020079
Kolo, K. D., Adepoju Solomon A. and Alhassan, J. K. (2015) ‘A Decision Tree Approach for Predicting Students Academic Performance’, International Journal of Education and Management Engineering, Vol. 5, No. 5, pp. 12-19. https://doi.org/10.5815/ijeme.2015.05.02
Kuah, C. T. and Wong, K. Y. (2011) ‘Efficiency assessment of universities through data envelopment analysis’, Procedia Computer Science, Vol. 3, pp. 499–506. https://doi.org/10.1016/j.procs.2010.12.084
González, L., Minguillón, J., Martínez, T. and Meneses, J. (2018) ‘Institutional support to provide freshmen with flexible learning paths at course and semester level in open higher education’, Working Paper UOC.
Long, P. and Siemens, G. (2014) ‘Penetrating the fog: analytics in learning and education’, Italian Journal of Educational Technology, Vol. 22, No. 3, pp. 132–137.
Menardi, G. (2011) ‘Density-based Silhouette diagnostics for clustering methods’, Statistics and Computing, Vol. 21, No. 3, pp. 295–308. https://doi.org/10.1007/s11222-010-9169-0
Mikušová, P. (2017) ‘Measuring the efficiency of the Czech public higher education institutions: an application of DEA’, Journal on Efficiency and Responsibility in Education and Science, Vol. 10, No. 2, pp. 58–63. https://doi.org/10.7160/eriesj.2017.100204
Najadat, H., Althebyan, Q. and Al-Omary, Y. (2019) ‘Higher Education Units Assessment Based on Data Envelopment Analysis and Clustering Techniques’, In 2019 International Arab Conference on Information Technology (ACIT), pp. 171–177. https://doi.org/10.1109/ACIT47987.2019.8991103
Nazir, A. (2019) ‘A critique of imbalanced data learning approaches for big data analytics’, International Journal of Business Intelligence and Data Mining, Vol. 14, No. 4, pp. 419–457. https://doi.org/10.1504/IJBIDM.2019.099961
Oyelade, J., Isewon, I., Oladipupo, O., Emebo, O., Omogbadegun, Z., Aromolaran, O., Uwoghiren, E, Olaniyan, D., Olawole, O. (2019) ‘Data Clustering: Algorithms and Its Applications’, In 2019 19th International Conference on Computational Science and Its Applications (ICCSA), pp. 71–81. https://doi.org/10.1109/ICCSA.2019.000-1
Pawsey, N., Ananda, J. and Hoque, Z. (2018) ‘Rationality, accounting and benchmarking water businesses: An analysis of measurement challenges’, International Journal of Public Sector Management, Vol. 31, No. 3, pp. 290–315. https://doi.org/10.1108/IJPSM-04-2017-0124
Ramzi, S., Afonso, A. and Ayadi, M. (2016) ‘Assessment of efficiency in basic and secondary education in Tunisia: A regional analysis’, International Journal of Educational Development, Vol. 51, pp. 62–76. https://doi.org/10.1016/j.ijedudev.2016.08.003
Sharma, J. and Yarlagadda, P. K. (2018) ‘Perspectives of ‘STEM education and policies’ for the development of a skilled workforce in Australia and India’, International Journal of Science Education, Vol. 40, No. 16, pp. 1999–2022. https://doi.org/10.1080/09500693.2018.1517239
Singh, I., Sabitha, A. S. and Bansal, A. (2016) ‘Student performance analysis using clustering algorithm’, In 2016 6th International Conference - Cloud System and Big Data Engineering (Confluence), pp. 294–299. https://doi.org/10.1109/CONFLUENCE.2016.7508131
Sinuany‐Stern, Z., Mehrez, A. and Hadad, Y. (2000) ‘An AHP/DEA methodology for ranking decision making units’, International Transactions in Operational Research, Vol. 7, No. 2, pp. 109–124. https://doi.org/10.1111/j.1475-3995.2000.tb00189.x
Sun, Q., Liu, H., & Wang, W. (2020). ‘Evaluating the Quality of GANs Based on F1 Harmonic Mean’, Procedia Computer Science, Vol. 166, pp. 463–468. https://doi:10.1016/j.procs.2020.02.064
Suresh, K. M., Asokan, P. and Vinodh, S. (2016) ‘Application of design for Six Sigma methodology to an automotive component’, International Journal of Six Sigma and Competitive Advantage, Vol. 10, No. 1, pp. 1. https://doi.org/10.1504/IJSSCA.2016.080446
Suter, L. E. and Camilli, G. (2019) ‘International Student Achievement Comparisons and US STEM Workforce Development’, Journal of Science Education and Technology, Vol. 28, No. 1, pp. 52–61. https://doi.org/10.1007/s10956-018-9746-0
Thanassoulis, E., Dey, P. K., Petridis, K., Goniadis, I. and Georgiou, A. C. (2017) ‘Evaluating higher education teaching performance using combined analytic hierarchy process and data envelopment analysis’, Journal of the Operational Research Society, Vol. 68, No. 4, pp. 431–445. https://doi.org/10.1057/s41274-016-0165-4
Torabi, R., Moradi, P. and Khantaimoori, A. R. (2012) ‘Predict Student Scores Using Bayesian Networks’, Procedia - Social and Behavioral Sciences, Vol. 46, pp. 4476–4480. https://doi.org/10.1016/j.sbspro.2012.06.280
Torres-Samuel, M., Vásquez, C. L., Luna, M., Bucci, N., Viloria, A., Crissien, T. and Manosalva, J. (2020) ‘Performance of Education and Research in Latin American Countries through Data Envelopment Analysis (DEA)’, Procedia Computer Science, Vol. 170, pp. 1023–1028. https://doi.org/10.1016/j.procs.2020.03.079
Tran, C.-D. T. T. and Villano, R. A. (2019) ‘Environmentally-adjusted efficiencies of Vietnamese higher education institutions: a multi-stage bootstrap DEA method’, International Journal of Operational Research, Vol. 36, No. 3, pp. 413–439. https://doi.org/10.1504/IJOR.2019.103126
Visbal-Cadavid, D., Martínez-Gómez, M. and Escorcia-Caballero, R. (2020) ‘Exploring University Performance through Multiple Factor Analysis: A Case Study’, Sustainability, Vol. 12, No. 3, 924. https://doi.org/10.3390/su12030924
Visbal-Cadavid, D., Martínez-Gómez, M. and Guijarro, F. (2017) 'Assessing the efficiency of public universities through DEA. A case study', Sustainability, Vol. 9, No. 8, 1416. https://doi.org/10.3390/su9081416
Visbal-Cadavid, D., Mendoza, A. M. and Hoyos, I. Q. (2019) ‘Prediction of efficiency in Colombian Higher Education institutions with Data Envelopment Analysis and Neural Networks’, Pesquisa Operacional, Vol. 39, No. 2, pp. 261–275. https://doi.org/10.1590/0101-7438.2019.039.02.0261
Witte, K. D. and López-Torres, L. (2017) ‘Efficiency in education: a review of literature and a way forward’, Journal of the Operational Research Society, Vol. 68, No. 4, pp. 339–363. https://doi.org/10.1057/jors.2015.92
Wolszczak-Derlacz, J. (2017) ‘An evaluation and explanation of (in)efficiency in higher education institutions in Europe and the U.S. with the application of two-stage semi-parametric DEA’, Research Policy, Vol. 46, No. 9, pp. 1595–1605. https://doi.org/10.1016/j.respol.2017.07.010
Additional Files
Published
How to Cite
Issue
Section
License
Copyright (c) 2021 Enrique De La Hoz, Rohemi Zuluaga, Adel Mendoza
This work is licensed under a Creative Commons Attribution 4.0 International License.
Authors declare with this manuscript intended for publication to ERIES Journal that:
- all co-authors agree with the publication of the manuscript even after amendments arising from peer review;
- all co-authors agree with the posting of the full text of this work on the web page of ERIES Journal and to the inclusion of references in databases accessible on the internet;
- no results of other researchers were used in the submitted manuscript without their consent, proper citation, or acknowledgement of their cooperation or material provided;
- the results (or any part of them) used in the manuscript have not been sent for publication to any other journal nor have they already been published (or if so, that the relevant works are cited in this manuscript);
- submission of the manuscript for publication was completed in accordance with the publishing regulations pertaining to place of work;
- experiments performed comply with current laws and written consent of the Scientific Ethics Committee / National Animal Care Authority (as is mentioned in the manuscript submitted);
- grant holders confirm that they have been informed of the submitted manuscript and they agree to its publication.
Authors retain copyright and grant ERIES Journal right of first publication with the work simultaneously licensed under a Creative Commons Attribution License that allows others to share the published work with an acknowledgement of its initial publication in ERIES Journal. Moreover, authors are able to post the published work in an institutional repository with an acknowledgement of its initial publication in ERIES Journal. In addition, authors are permitted and encouraged to post the published work online (e.g. institutional repositories or on their website) as it can lead to productive exchanges, as well as earlier and greater citation of published work.
