ALTERNATIVE APPROACHES TO EFFICIENCY EVALUATION OF HIGHER EDUCATION INSTITUTIONS

Evaluation of efficiency and ranking of higher education institutions is very popular and important topic of public policy. The assessment of the quality of higher education institutions can stimulate positive changes in higher education. In this study we focus on assessment and ranking of Slovak economic faculties. We try to apply two different quantitative approaches for evaluation Slovak economic faculties Stochastic Frontier Analysis (SFA) as an econometric approach and PROMETHEE II as multicriteria decision making method. Via SFA we examine faculties’ success from scientific point of view, i.e. their success in area of publications and citations. Next part of analysis deals with assessing of Slovak economic sciences faculties from overall point of view through the multicriteria decision making method. In the analysis we employ panel data covering 11 economic faculties observed over the period of 5 years. Our main aim is to point out other quantitative approaches to efficiency estimation of higher education institutions.


Introduction
Nowadays, many assessments and rankings of higher education institutions (HEI) and their faculties are published by wide range of agencies and organizations.Due to the higher education globalization, the focus has shifted to worldwide rankings and assessments.Evaluation of HEI seems to become a very popular and important supporting decision tool.University leaders believe that good rankings help to maintain and build institutional position and reputation, students and postgraduates exploit rankings to make a university choice, last but not least the rankings influence stakeholders' decisions about accreditation, sponsorship or employee recruitment.Ranking is also important from national and international partnerships and collaborations point of view.Leaders often verify a potential partner's rank prior to entering into discussion about research and academic partnerships.HEI assessments and rankings provided by wide range of agencies and organizations are based on different ranking systems; different indicators or metrics are used to measure higher education activities.In Slovak Republic, there are two agencies dealing with assessment and ranking of HEI: Accreditation Commission and ARRA (Academic Ranking and Rating Agency).ARRA is an independent Slovak civil association established in 2004 by former student leaders and personalities from the academic field with the objectives of assessing the quality of Slovak higher education institutions and to stimulate positive changes in Slovak higher education (Furková, 2013).In this study we also decided to focus on assessment and ranking of HEI, namely we chose economic sciences (group EKONOM -Slovak economic faculties) following ARRA classification of higher education institutions and their faculties.According to ARRA, the faculties are classified into eleven field-specific groups in order to compare only faculties that have the same orientation and similar working conditions (ARRA, 2013).The selection of criterions and classification of faculties in characteristic groups is based on the results of the development in the area of the assessment of higher education institutions in Slovakia as well as on international trends.The criterions (indicators) are divided into two main groups; education and research.The criterions are defined in ARRA (2013).Faculties gain a certain number of points for each group of indicators.The faculty with the highest value of the indicator in a group gets 100 points and other faculties in the group get points calculated as a linear proportion of the value of their indicator to the value of the highest indicator.Assigning summary point scores to faculties, the scores being expressed as an average of points for all indicators, i.e., the ranking of faculties in individual groups is based on average point score for all groups of indicators (ARRA, 2013).Annually, based on this methodology ARRA prepares and publishes report assessing Slovak higher education institutions from overall point of view (Furková, 2013).In our analysis we try to exploit two different quantitative approaches for evaluation Slovak economic faculties -Stochastic Frontier Analysis (SFA) as an econometric approach and PROMETHEE II as multicriteria decision making method.Via SFA we examine faculties' success from scientific point of view, i.e. their success in area of publications and citations (Publications and Citations (criterions VV1 -VV3a).Used SFA methodology is based on econometric theory where pre-specified functional form is estimated and inefficiency is modelled as an additional stochastic term (Kumbhakar et al., 2000).Applying SFA for measuring technical efficiency of economic faculties within the time period of 5 years we try to evaluate a quality of scientific research activities.The analysis is based on production function principle context; evaluated faculties are treated as producers of output (publications and citations) given some inputs.In our analysis various versions of stochastic frontier production function panel data models are applied.We estimated levels of technical efficiency for each economic faculty and the differences in estimated scores, parameters and ranking of faculties are compared across different panel data models.Next part of analysis deals with assessing of Slovak economic faculties from overall point of view through the multicriteria decision making method.From set of multicriteria decision making methods we chose PROMETHEE II method.This method requires defining a set of criterions and set of alternatives.As criterions were opted ARRA criterions and Slovak economic faculties -group EKONOM represented alternatives.PROMETHEE II method provides us final ranking of faculties and the comparison with ARRA ranking is also presented.Main aim of our analysis is to point out alternative approaches to efficiency estimation of HEI.

Materials and Methods
In this part of paper we briefly present proposed methodologies for assessments and rankings of higher education institutions.Firstly, we will discuss SFA methodology (according to Furková, 2013) and the second part will introduce PROMETHEE II method (according to Furková et al., 2012).

Stochastic Frontier Analysis
Stochastic frontier analysis has become a popular tool for production analysis.Stochastic frontier models date back to studies of Aigner, Lovell and Schmidt (1977) and Meeusen and van den Broeck (1977), who independently proposed a stochastic frontier production function model with a two-part composed error term.In the production context (this approach could be also used in cost context), where its use is most common, this error is composed of a standard random error term, representing measurement random factors, and a onesided random variable representing technical inefficiency, i.e. the distance of the observation from the production frontier.This technical efficiency reflects the ability of a unit (firm, country or school) to obtain maximal output from a given set of inputs.If the unit is 100 % efficient, it lays on the production frontier itself and this measure is bounded between zero and one.The general form of the stochastic frontier production function model for panel data can be formulated as follows: ( ) where Y it denotes the production at the t-th observation (t = 1,2,...T) and T is number of time periods) for the i-th unit (i = 1,2,...T) and N is number of units), X it is the corresponding matrix of inputs and other explanatory variables, β is a vector of unknown parameters to be estimated, vit is symmetric random variable reflecting effect of statistical noise, ui is time invariant technical inefficiency term.Compound error term is then formulated as ε it = v t -u i .This random effect model can be estimated using Maximum Likelihood Estimation (MLE) method, Generalized Least Square Method or Method of Moments.Using Maximum Likelihood Estimation method requires making distribution assumptions for stochastic terms.Usually we assume that vit are random variables to be normally distributed (vit ~ iid N(0, σ 2 v )) and ui are non-negative time-invariant random variables to be half normal distributed (u i ~ iid N+(0, σ 2 u )) or truncated normal distribution (u i ~ iid N+(m,σ 2 u )) can be also considered.The next step is to obtain estimates of the technical efficiency of each unit.
The problem is to extract the information that ε it contains on u i (we have estimates of ε it = v t -u i , which obviously contain information on u i ).A solution of this problem is obtained from the conditional distribution of u i given by ε it .This procedure is known as JLMS decomposition (for more details see Jondrow et al., 1982).An alternative minimum squared error predictor estimator can be also used for separation the inefficiency effect from the statistical noise (for more details see Kumbhakar et al., 2000or Furková, 2009).Once the point estimates of u i are obtained, estimates of the technical efficiency of each unit can be obtained by substituting them into equation ( 2).If the production frontier is specified as stochastic, the appropriate measure of individual technical efficiency becomes: which defines technical efficiency as the ratio of observed outputs quantities to the maximum outputs quantities attainable in an environment characterized by exp{v it }.
If we allow efficiency changes in time, inefficiency component will consist of two parts, namely cross-section component (u i ) and time component (β t ): and time invariant production efficiency model given by equation ( 1), we can reformulate as follows: ( ) where uit is now time variant technical inefficiency term of compounded error term ε it = v it -u it , remaining variables are defined in model (1) as well.
There are proposed various approaches to estimate time varying production frontier model given by equations ( 3) and ( 4) (for more details see Kumbhakar et al., 2000).Battese and Coelli (see Coelli et al., 2005) presented a model where they modelled the inefficiency component in (4) according to following exponential time function: where η is unknown parameter to be estimated.The function value is determined by value of parameter η and number of observations.This model can be estimated by using MLE method.The likelihood function of this model is a generalization of the likelihood function for the conventional model (for more details see e.g.Kumbhakar et al., 2000).Estimates of the technical efficiency of each unit at time t can be obtained by substituting estimates of u it into following equation:

Multi-attribute decision making methods
Multicriteria decision making problems can be divided into main groups according to definition of the feasible set of the alternatives.The first is the case when we have a finite number of criteria but the number of the feasible alternatives (the alternatives are determined by the system of requirements constraints) is infinite.These problems belong to the field of multiple criteria optimization.On the other hand the problems of the type when the number of the criterions and alternatives is finite, and the alternatives are given explicitly are called multi-attribute decision making problems (MDMP).The theory of MDMP is established very well and the possibilities of real applications (evaluation of investment alternatives, evaluation of credibility of bank clients, rating of companies, consumer goods evaluation and many others) are very large.
The multi-attribute decision making problem is usually defined by criterion matrix as follows: where X 1 , X 2 , ..., X n is the set of n alternatives, Y 1 , Y 2 , .., Y k is the set of k criterions, y ij is the criterion value of the alternative X i , i=1,2,...,n, j=1,2,...,k.In the matrix each column belongs to a criterion and each row describes the performance of an alternative, i.e. each element of the matrix y ij is a single numerical value representing the performance of alternative i on criterion j.The essential part of the multi-attribute decision making problem is setting the type of the criterions (minimization or maximization) and assigning the criteria weights.The weight w i reflects the relative importance of criteria and is assumed to be positive.The weights of the criteria are usually determined on subjective basis.They represent the opinion of a single decision maker or synthesize the opinions of a group of experts using a group decision technique, as well.The main goal of the multi-attribute decision making techniques can be complete or partial ranking of the alternatives.The multi-attribute decision making methods are based either on the Multi-attribute Utility Theory (MAUT) or Outranking Methods.The family of MAUT methods consists of aggregating the different criteria into a function, which has to be maximized.The MAUT methods are based on the utility functions.The utility functions can be applied to transform the raw performance values of the alternatives against diverse criteria to a common, dimensionless scale.In the practice, the intervals [0, 1] are used for this purpose and more preferred performance obtains a higher utility value of the utility function.From this group of methods can be mentioned e.g.WSA -Weighted Sum Approach or SMART -Simple Multi-Attribute Rating Technique.The concept of outranking methods is based on pairwise outranking assessments and having determined for each pair of alternatives whether one alternative outranks another, these pairwise outranking assessments can be combined into a partial or complete ranking.The most popular families of the outranking method are the ELECTRE, the PROMETHEE methods or TOPIS method.Selected MDMP method -PROMETHEE II used in our analysis will be briefly outlined in next part.The implementation of the PROMETHEE (Preference Ranking Organization Method for Enrichment Evaluation) method requires the knowledge of criterion matrix (7), weights of the criteria and preference functions of criteria with their parameters for measuring the strength of the preference of the pairs of alternatives with respect to the given criterion.PROMETHEE method can provide partial ranking of alternatives (PROMETHEE I) or complete alternatives ranking (PROMETHEE II).The procedure of the method can be summarized as follows.The first, the alternatives are compared in pairs for each criterion.The preference for the alternative is expressed by a number from the interval [0,1] (0 for no preference or indifference and 1 for strict preference).The preference function F i relating the difference in performance to preference is selected by the decision maker (for more details see e.g.Jablonský and Dlouhý, 2004).Next a multicriteria preference index is formed for each pair of alternative as a weighted average of the corresponding preferences for each criterion.The index π(X i , X j ) expresses the preference of alternative X i over alternative X j considering all criteria and can be defined as: In order to rank the alternatives, the following precedence flows are defined: Positive outranking flow: Negative outranking flow: The positive outranking flow expresses how much each alternative is outranking all the others.The higher positive outranking flow, the better the alternative and it represents the power of this alternative.The negative outranking flow expresses how much each alternative is outranked by all the others.The smaller negative flow, the better the alternative and it represents the weakness of this alternative.The PROMETHEE II method provides complete ranking of the alternatives according the net outranking flow which is defined as follows: All alternatives are now comparable, the alternative with the highest ( ) i X φ can be considered as the best one.The PROMETHEE I method offers partial ranking based on the comparison of the positive and negative outranking flows (for more details see e.g.Jablonský and Dlouhý, 2004).

Results
The first part of analysis was based on an econometric approach -SFA (according to Furková, 2013).Models of SFA defined by equations ( 1) and ( 4), (5) were applied in order to evaluate the scientific research activities of economic faculties through estimated levels of efficiency.Our balanced panel data set of  In order to estimate the model with time varying technical efficiency (Model3 and Model4) we formulated model given by equation ( 13): where u it is defined by equation ( 5).In all models v it is reflecting effect of statistical noise and u i or u it are random variables reflecting time invariant or time varying technical inefficiency respectively.Remaining variables have been defined before.The parameters of the models defined by equation ( 12) and ( 13) have been jointly estimated by the MLE method using FRONTIER 4.1 (Coelli, T.J., 1996).The MLE method requires making the distributional assumptions for stochastic terms.We made following distributional assumptions: In all models for separation the inefficiency effect from the statistical noise was used Battese and Coelli point estimator (see Coelli et al., 2005).The individual technical efficiency estimates were obtained by substituting the inefficiency effects into the equations ( 2) and ( 6).The final estimates of the parameters of all frontier models are listed in Table 1.The gamma parameter 2 value e.g. in Model1 is 0,7782 which suggests that 77,82 % of the disturbance term is due to inefficiency and its statistical signification (Model1 and Model2) implies that the technical inefficiency term is a significant addition to the model.Model1 and Model2 are both based on the assumption of time invariant technical efficiency but the models were estimated under different distributional assumptions for inefficiency term, in the first one was used the half normal distribution assumption (Model1) while in the second model was used truncated normal distributional assumption (Model2).Therefore additional estimated parameter µ is listed for Model2 in Table1.
Parameter µ is found to be not significant; therefore half normal specification is preferred for the distribution of u.In addition this conclusion was also confirmed by results of performed LR test.If the model is estimated by the method of ML method, hypothesis concerning individual coefficients can be tested using LR test.Our null hypothesis was set as H 0 : µ = η = 0, which implies time invariant half normal inefficiency effects and this hypothesis was confirmed.Maximized values of the log-likelihood function of all models are listed in last row of Table 1.The value of log-likelihood expresses the likelihood of observing the sample observations as a function of the unknown parameters (for more details see e.g.Coelli et al., 2005).Applied time variant efficiency models provide individual efficiency 2 Estimation of stochastic frontier production function model by MLE method requires parameterization of the log-likelihood function.Frontier 4.1 utilises Values of the parameter must lie between 0 and 1, if γ = 0 then all deviations from the frontier are due to noise, while γ = 1 means all deviation are due to technical inefficiency (for more details see Coelli, 1996or Coelli et al., 2005).measures for the faculties in each year but it is not possible to present these extensive results due to insufficient space.Due to this fact we present only results of Model1 (see Figure 1).The scores can move between 0 and 1, where the highest value implies a perfectly efficient faculty.Source: own calculations The second part of our analysis is aimed to exploitation of another quantitative approach for assessment of Slovak economic faculties via multi-attribute decision making methods namely PROMETHEE II method.Contrary to the first presented approach where the analysis was oriented to faculties' success from scientific point of view, i.e. their success in area of publications and citations, this part examines Slovak economic faculties from overall point of view.As criterions, ARRA criterions were opted again and in addition to SFA approach we added criterion Applications for Study (SV6-8).
As alternatives were chosen Slovak economic faculties -group EKONOM and observed period was again from 2008 to 2012.We supposed the same importance of all criterions and all calculations originated from SANNA (System for Analysis of Alternative) (Jablonský and Dlouhý, 2004).PROMETHEE II method is based on the principle of evaluating alternatives based on preference relation.Selected preference functions of criterions with their parameters for measuring the strength of preference of the pairs of alternatives with respect to the given criterion are given in Table 2. Six different types of preference function are proposed in the original PROMETHEE definition.

Conclusion
We presented SFA methodology and PROMETHEE II method as a contribution to the discussion about quantitative measurement of evaluation of HEI.In the first part of the paper we briefly discussed process of assessment and ranking of HEI in Slovak Republic.Following ARRA indicators of assessment we have exploited chosen quantitative approaches to evaluation of units.These methodologies were also presented in the second part.However, there are other multi-attribute decision making methods which could be used depending on the aim of the analysis, obtained data, etc.Moreover, the analysis might be enriched by using further quantitative parametric (e.g.DFA -

11 1
Slovak economic faculties observed over a period from 2008 to 2012 includes 55 observations in total.All data are based on information from statistics of ARRA (for more details see ARRA, 2013).The output and input selection is a crucial step in our analysis and as our goal in the first part of analysis is the evaluation of the scientific research activities, we chose as output variable aggregated ARRA indicator Publications and Citations (VV1 -VV3a) -Y1.As inputs which would significantly influence scientific research activities were chosen aggregated ARRA indicators Students and Teachers (SV1 -SV4) -X1, PhD Studies (VV4a -VV6) -X2 and Grants (VV7 -VV10) -X3 ((for more details see ARRA, 2013).Next important step of the analysis is to choose appropriate form of the production function.We decided to use more flexible translogaritmic function instead of more traditional Cobb -Douglas production function.The first part of SFA analysis was based on the assumption of time invariant technical efficiency.We applied SFA panel data models with time invariant technical efficiency assumption (Model1 and Model2) and the analysis was based on the estimation of the model given by equation (12): 1 Faculty of Management, Comenius University in Bratislava; Faculty of Business Economy, University of Economics in Bratislava; Faculty of Economic Informatics, University of Economics in Bratislava; Faculty of Economics, Technical University of Košice; Faculty of Management, University of Prešov; Faculty of Business, University of Economics in Bratislava; Faculty of National Economy, University of Economics in Bratislava; Faculty of Economics, Matej Bel University in Banská Bystrica; Faculty of Economics and Management, Slovak University of Agriculture in Nitra; Faculty of Business Management, University of Economics in Bratislava; Faculty of Operation and Economics of Transport and Communications, University of Žilina.

Figure 1 :
Figure 1: Efficiency scores and ranking of faculties according to SFA

Figure 1
Figure 1 provides efficiency estimates and ordering of economic faculties according to Model1.The highest score of efficiency in scientific activities (see Figure1) achieved Faculty