Assessing and Classification of Academic Efficiency in Engineering Teaching Programs

Authors

DOI:

https://doi.org/10.7160/eriesj.2021.140104

Keywords:

efficiency, higher education, machine learning, predictive evaluation

Abstract

This research uses a three-phase method to evaluate and forecast the academic efficiency of engineering programs. In the first phase, university profiles are created through cluster analysis. In the second phase, the academic efficiency of these profiles is evaluated through Data Envelopment Analysis. Finally, a machine learning model is trained and validated to forecast the categories of academic efficiency. The study population corresponds to 256 university engineering programs in Colombia and the data correspond to the national examination of the quality of education in Colombia in 2018. In the results, two university profiles were identified with efficiency levels of 92.3% and 97.3%, respectively. The Random Forest model presents an Area under ROC value of 95.8% in the prediction of the efficiency profiles. The proposed structure evaluates and predicts university programs’ academic efficiency, evaluating the efficiency between institutions with similar characteristics, avoiding a negative bias toward those institutions that host students with low educational levels.

References

Agasisti, T., Munda, G. and Hippe, R. (2019) ‘Measuring the efficiency of European education systems by combining Data Envelopment Analysis and Multiple-Criteria Evaluation’, Journal of Productivity Analysis, Vol. 51, No. 2, pp. 105–124. https://doi.org/10.1007/s11123-019-00549-6

Aldamak, A. and Zolfaghari, S. (2017) ‘Review of efficiency ranking methods in data envelopment analysis’, Measurement, Vol. 106, pp. 161–172. https://doi.org/10.1016/j.measurement.2017.04.028

Alsabawy, A. Y., Cater-Steel, A. and Soar, J. (2016) ‘Determinants of Perceived Usefulness of e-Learning Systems’, Computers in Human Behavior, Vol. 64, pp. 843–858. https://doi.org/10.1016/j.chb.2016.07.065

Amara, N., Rhaiem, M. and Halilem, N. (2020) ‘Assessing the research efficiency of Canadian scholars in the management field: Evidence from the DEA and fsQCA’, Journal of Business Research, Vol. 115, pp. 296–306. https://doi.org/10.1016/j.jbusres.2019.10.059

Aparicio, J., Cordero, J. M. and Ortiz, L. (2019) ‘Measuring efficiency in education: The influence of imprecision and variability in data on DEA estimates’, Socio-Economic Planning Sciences, Vol. 68, 100698. https://doi.org/10.1016/j.seps.2019.03.004

Aparicio, J., Perelman, S. and Santín, D. (2020) ‘Comparing the evolution of productivity and performance gaps in education systems through DEA: an application to Latin American countries’, Operational Research: An International Journal. https://doi.org/10.1007/s12351-020-00578-2

Benicio, J. and Mello, J. C. S. de (2015) ‘Productivity Analysis and Variable Returns of Scale: DEA Efficiency Frontier Interpretation’, Procedia Computer Science, Vol. 55, pp. 341–349. https://doi.org/10.1016/j.procs.2015.07.059

Berens, J., Schneider, K., Gortz, S., Oster, S. and Burghoff, J. (2019) ‘Early Detection of Students at Risk - Predicting Student Dropouts Using Administrative Student Data from German Universities and Machine Learning Methods’, Journal of Educational Data Mining, Vol. 11, No. 3, pp. 1–41. https://doi.org/10.5281/zenodo.3594771

Bianchi, N. and Giorcelli, M. (2020) ‘Scientific Education and Innovation: From Technical Diplomas to University Stem Degrees’, Journal of the European Economic Association, Vol. 18, No. 5, pp. 2608-2646. https://doi.org/10.1093/jeea/jvz049

Breiman, L. (2001) ‘Random Forests’, Machine Learning, Vol. 45, No. 1, pp. 5-32. https://doi.org/10.1023/A:1010933404324

Charnes, A., Cooper, W. W. and Rhodes, E. (1978) ‘Measuring the efficiency of decision making units’, European Journal of Operational Research, Vol. 2, No. 6, pp. 429–444. https://doi.org/10.1016/0377-2217(78)90138-8

Clayman, C. L., Srinivasan, S. M. and Sangwan, R. S. (2020) ‘K-means Clustering and Principal Components Analysis of Microarray Data of L1000 Landmark Genes’, Procedia Computer Science, Vol. 168, pp. 97-104. https://doi.org/10.1016/j.procs.2020.02.265

Corlu, M. A. and Aydin, E. (2016) ‘Evaluation of Learning Gains Through Integrated STEM Projects’, International Journal of Education in Mathematics, Science and Technology, Vol. 4, No. 1, pp. 20–29. https://doi.org/10.18404/ijemst.35021

de Morais, A. M., Araújo, J. M. F. R. and Costa, E. B. (2014) ‘Monitoring student performance using data clustering and predictive modelling’, In 2014 IEEE Frontiers in Education Conference (FIE) Proceedings, pp. 1–8. https://doi.org/10.1109/FIE.2014.7044401

Delahoz-Dominguez, E., Zuluaga, R. and Fontalvo-Herrera, T. (2020) ‘Dataset of academic performance evolution for engineering students’, Data in Brief, Vol. 30, 105537. https://doi.org/10.1016/j.dib.2020.105537

Duan, S. X. (2019) ‘Measuring university efficiency: An application of data envelopment analysis and strategic group analysis to Australian universities’, Benchmarking: An International Journal, Vol. 26, No. 4, pp. 1161–1173. https://doi.org/10.1108/BIJ-10-2017-0274

Durairaj, M. and Vijitha, C. (2014) ‘Educational data mining for prediction of student performance using clustering algorithms’, International Journal of Computer Science and Information Technologies, Vol. 5, No. 4, pp. 5987–5991.

Ferro, G. and D’Elia, V. (2020) ‘Higher Education Efficiency Frontier Analysis: A Review of Variables to Consider’, Vol. 13, No. 3, 140-153. https://doi.org/10.7160/eriesj.2020.130304

Fuentes, R., Fuster, B. and Lillo-Bañuls, A. (2016) ‘A three-stage DEA model to evaluate learning-teaching technical efficiency: Key performance indicators and contextual variables’, Expert Systems with Applications, Vol. 48, pp. 89–99. https://doi.org/10.1016/j.eswa.2015.11.022

Ghasemi, N., Najafi, E., Hoseinzadeh Lotfi, F. and Movahedi Sobhani, F. (2020) ‘Assessing the performance of organizations with the hierarchical structure using data envelopment analysis: An efficiency analysis of Farhangian University’, Measurement, Vol. 156, 107609. https://doi.org/10.1016/j.measurement.2020.107609

Gralka, S., Wohlrabe, K. and Bornmann, L. (2019) ‘How to measure research efficiency in higher education? Research grants vs. publication output’, Journal of Higher Education Policy and Management, Vol. 41, No. 3, pp. 322–341. https://doi.org/10.1080/1360080X.2019.1588492

Granadillo, E. D. L. H., Gomez, J. M. and Herrera, T. J. F. (2019) ‘Methodology with multivariate calculation to define and evaluate financial productivity profiles of the chemical sector in Colombia’, International Journal of Productivity and Quality Management, Vol. 27, No. 2, pp. 144–160. https://doi.org/10.1504/IJPQM.2019.100141

Heidrich, L., Victória Barbosa, J. L., Cambruzzi, W., Rigo, S. J., Martins, M. G. and dos Santos, R. B. S. (2018) ‘Diagnosis of learner dropout based on learning styles for online distance learning’, Telematics and Informatics, Vol. 35, No. 6, pp. 1593–1606. https://doi.org/10.1016/j.tele.2018.04.007

Hoeg, D. G. and Bencze, J. L. (2017) ‘Values Underpinning STEM Education in the USA: An Analysis of the Next Generation Science Standards’, Science Education, Vol. 101, No. 2, pp. 278–301. https://doi.org/10.1002/sce.21260

ICFES (2020) Estructura general del examen Icfes Saber Pro - Portal Icfes, [Online], Available at: https://www.icfes.gov.co/web/guest/acerca-examen-saber-11 [26 Oct 2020].

Johnes, J. (2006) ‘Data envelopment analysis and its application to the measurement of efficiency in higher education’, Economics of Education Review, Vol. 25, No. 3, pp. 273–288. https://doi.org/10.1016/j.econedurev.2005.02.005

Khan, S. U., Khan, Z. and Hameed, G. (2019) ‘Efficiency Assessment of Public Education & Health Sector in Selected Middle-Income Countries with Special Reference to Millennium Development Goals (MDGs)’, Journal of Applied Economics and Business Studies, Vol. 3, No. 1, pp. 41–60. https://doi.org/10.34260/jaebs.313

Klumpp, M. (2018) ‘The Index Number Problem with DEA: Insights from European University Efficiency Data’, Education Sciences, Vol. 8, No. 2, pp. 79. https://doi.org/10.3390/educsci8020079

Kolo, K. D., Adepoju Solomon A. and Alhassan, J. K. (2015) ‘A Decision Tree Approach for Predicting Students Academic Performance’, International Journal of Education and Management Engineering, Vol. 5, No. 5, pp. 12-19. https://doi.org/10.5815/ijeme.2015.05.02

Kuah, C. T. and Wong, K. Y. (2011) ‘Efficiency assessment of universities through data envelopment analysis’, Procedia Computer Science, Vol. 3, pp. 499–506. https://doi.org/10.1016/j.procs.2010.12.084

González, L., Minguillón, J., Martínez, T. and Meneses, J. (2018) ‘Institutional support to provide freshmen with flexible learning paths at course and semester level in open higher education’, Working Paper UOC.

Long, P. and Siemens, G. (2014) ‘Penetrating the fog: analytics in learning and education’, Italian Journal of Educational Technology, Vol. 22, No. 3, pp. 132–137.

Menardi, G. (2011) ‘Density-based Silhouette diagnostics for clustering methods’, Statistics and Computing, Vol. 21, No. 3, pp. 295–308. https://doi.org/10.1007/s11222-010-9169-0

Mikušová, P. (2017) ‘Measuring the efficiency of the Czech public higher education institutions: an application of DEA’, Journal on Efficiency and Responsibility in Education and Science, Vol. 10, No. 2, pp. 58–63. https://doi.org/10.7160/eriesj.2017.100204

Najadat, H., Althebyan, Q. and Al-Omary, Y. (2019) ‘Higher Education Units Assessment Based on Data Envelopment Analysis and Clustering Techniques’, In 2019 International Arab Conference on Information Technology (ACIT), pp. 171–177. https://doi.org/10.1109/ACIT47987.2019.8991103

Nazir, A. (2019) ‘A critique of imbalanced data learning approaches for big data analytics’, International Journal of Business Intelligence and Data Mining, Vol. 14, No. 4, pp. 419–457. https://doi.org/10.1504/IJBIDM.2019.099961

Oyelade, J., Isewon, I., Oladipupo, O., Emebo, O., Omogbadegun, Z., Aromolaran, O., Uwoghiren, E, Olaniyan, D., Olawole, O. (2019) ‘Data Clustering: Algorithms and Its Applications’, In 2019 19th International Conference on Computational Science and Its Applications (ICCSA), pp. 71–81. https://doi.org/10.1109/ICCSA.2019.000-1

Pawsey, N., Ananda, J. and Hoque, Z. (2018) ‘Rationality, accounting and benchmarking water businesses: An analysis of measurement challenges’, International Journal of Public Sector Management, Vol. 31, No. 3, pp. 290–315. https://doi.org/10.1108/IJPSM-04-2017-0124

Ramzi, S., Afonso, A. and Ayadi, M. (2016) ‘Assessment of efficiency in basic and secondary education in Tunisia: A regional analysis’, International Journal of Educational Development, Vol. 51, pp. 62–76. https://doi.org/10.1016/j.ijedudev.2016.08.003

Sharma, J. and Yarlagadda, P. K. (2018) ‘Perspectives of ‘STEM education and policies’ for the development of a skilled workforce in Australia and India’, International Journal of Science Education, Vol. 40, No. 16, pp. 1999–2022. https://doi.org/10.1080/09500693.2018.1517239

Singh, I., Sabitha, A. S. and Bansal, A. (2016) ‘Student performance analysis using clustering algorithm’, In 2016 6th International Conference - Cloud System and Big Data Engineering (Confluence), pp. 294–299. https://doi.org/10.1109/CONFLUENCE.2016.7508131

Sinuany‐Stern, Z., Mehrez, A. and Hadad, Y. (2000) ‘An AHP/DEA methodology for ranking decision making units’, International Transactions in Operational Research, Vol. 7, No. 2, pp. 109–124. https://doi.org/10.1111/j.1475-3995.2000.tb00189.x

Sun, Q., Liu, H., & Wang, W. (2020). ‘Evaluating the Quality of GANs Based on F1 Harmonic Mean’, Procedia Computer Science, Vol. 166, pp. 463–468. https://doi:10.1016/j.procs.2020.02.064

Suresh, K. M., Asokan, P. and Vinodh, S. (2016) ‘Application of design for Six Sigma methodology to an automotive component’, International Journal of Six Sigma and Competitive Advantage, Vol. 10, No. 1, pp. 1. https://doi.org/10.1504/IJSSCA.2016.080446

Suter, L. E. and Camilli, G. (2019) ‘International Student Achievement Comparisons and US STEM Workforce Development’, Journal of Science Education and Technology, Vol. 28, No. 1, pp. 52–61. https://doi.org/10.1007/s10956-018-9746-0

Thanassoulis, E., Dey, P. K., Petridis, K., Goniadis, I. and Georgiou, A. C. (2017) ‘Evaluating higher education teaching performance using combined analytic hierarchy process and data envelopment analysis’, Journal of the Operational Research Society, Vol. 68, No. 4, pp. 431–445. https://doi.org/10.1057/s41274-016-0165-4

Torabi, R., Moradi, P. and Khantaimoori, A. R. (2012) ‘Predict Student Scores Using Bayesian Networks’, Procedia - Social and Behavioral Sciences, Vol. 46, pp. 4476–4480. https://doi.org/10.1016/j.sbspro.2012.06.280

Torres-Samuel, M., Vásquez, C. L., Luna, M., Bucci, N., Viloria, A., Crissien, T. and Manosalva, J. (2020) ‘Performance of Education and Research in Latin American Countries through Data Envelopment Analysis (DEA)’, Procedia Computer Science, Vol. 170, pp. 1023–1028. https://doi.org/10.1016/j.procs.2020.03.079

Tran, C.-D. T. T. and Villano, R. A. (2019) ‘Environmentally-adjusted efficiencies of Vietnamese higher education institutions: a multi-stage bootstrap DEA method’, International Journal of Operational Research, Vol. 36, No. 3, pp. 413–439. https://doi.org/10.1504/IJOR.2019.103126

Visbal-Cadavid, D., Martínez-Gómez, M. and Escorcia-Caballero, R. (2020) ‘Exploring University Performance through Multiple Factor Analysis: A Case Study’, Sustainability, Vol. 12, No. 3, 924. https://doi.org/10.3390/su12030924

Visbal-Cadavid, D., Martínez-Gómez, M. and Guijarro, F. (2017) 'Assessing the efficiency of public universities through DEA. A case study', Sustainability, Vol. 9, No. 8, 1416. https://doi.org/10.3390/su9081416

Visbal-Cadavid, D., Mendoza, A. M. and Hoyos, I. Q. (2019) ‘Prediction of efficiency in Colombian Higher Education institutions with Data Envelopment Analysis and Neural Networks’, Pesquisa Operacional, Vol. 39, No. 2, pp. 261–275. https://doi.org/10.1590/0101-7438.2019.039.02.0261

Witte, K. D. and López-Torres, L. (2017) ‘Efficiency in education: a review of literature and a way forward’, Journal of the Operational Research Society, Vol. 68, No. 4, pp. 339–363. https://doi.org/10.1057/jors.2015.92

Wolszczak-Derlacz, J. (2017) ‘An evaluation and explanation of (in)efficiency in higher education institutions in Europe and the U.S. with the application of two-stage semi-parametric DEA’, Research Policy, Vol. 46, No. 9, pp. 1595–1605. https://doi.org/10.1016/j.respol.2017.07.010

Additional Files

Published

2021-03-31

How to Cite

De La Hoz, E., Zuluaga, R. and Mendoza, A. (2021) ’Assessing and Classification of Academic Efficiency in Engineering Teaching Programs’, Journal on Efficiency and Responsibility in Education and Science, vol. 14, no. 1, pp. 41-52 https://doi.org/10.7160/eriesj.2021.140104.

Issue

Section

Research Paper