Journal of Management and Decision Making Volume 2016 (1), Article ID 12016, 10 pages DOI: 10.22218/JMDM/12016
1 Faculty of Agriculture, University of Novi Sad, Novi Sad, Serbia.
2 Jozef Stefan Institute, Ljubljana, Slovenia.
3 Faculty of Organization and Informatics, University of Zagreb, Croatia.
Full implementation of the Analytic Hierarchy Process (AHP) assumes solving the hierarchy of a given decision-making problem with multiple criteria and alternatives. At the local nodes the decision maker sets his/her judgments about the relative preferences of given decision elements and creates so-called comparison matrices. These local matrices are evaluated and the local priorities of the decision elements are calculated. The weighted additive summation produces the final priorities of the bottom decision elements (usually decision alternatives) versus the goal at the top of the hierarchy. In turn, various local consistency measures can be calculated and propagated during the synthesis process to obtain the global, hierarchy-wise consistency of the decision maker. In group contexts, decision makers commonly demonstrate different (in)consistencies, both locally and/or globally. A recently published AHP synthesis method (Srdjevic and Srdjevic, 2013) is used in this research to check sensitivity of the final AHP solution derived in the group context if only consistency measures are employed in measuring the quality of the decisions being made by individuals in the group. A real-life case-study example is described in this paper. The three e-learning management systems are assessed by the three experts in subject area and four levels hierarchy is used to perform sensitivity of the final solution if the two different aggregation schemes are applied. We show that if only the consistencies are used as criteria for identifying the best local priority vectors, one must be careful in selecting local vectors for the synthesis. Selection of matrices from decision makers is especially sensitive at higher levels of the hierarchy because differences in demonstrated consistency can lead to rank reversal of the evaluated alternatives.
Keywords: Group decision-making; Analytic Hierarchy Process; Consistency measures; e-Learning management system.
There are several commonly used consistency measures associated with the AHP in the individual and group contexts. In both contexts we can talk about local consistencies that are computed at all the nodes of a hierarchy, and about the global consistency that is computed for the hierarchy as a whole. The author of the AHP, (T. Saaty, 1980), defined a method to compute the hierarchy-wise (global) consistency ratio (HCR) by appropriately weighting the local consistency ratios (CRs) computed for all the pair-wise comparison matrices in the hierarchy. The HCR is used with a prioritization technique known as the eigenvector method, strongly advocated as the best one (Saaty, 2003). It is interesting to note that there is no significant discussion in the literature regarding global consistency such as, e.g., HCR, which is probably because it is a hot topic, with possible controversies among scientists and practitioners.
The most general and prioritization-method-independent consistency measure is the standard (quadratic) Euclidean distance (ED) that is applicable to all measurement frameworks. In the AHP, the ED can be computed locally, i.e., for a given local matrix in the hierarchy by summing for all the matrix positions’ squared differences between the numerically defined decision-maker's judgments and the computed weights of the related (compared) decision elements. By summing all the local EDs, it is easy to obtain information about the overall, hierarchy-wise consistency labeled hereafter as the HED, analogously to the HCR.
A globally meaningful consistency measure reported in AHP applications (e.g., Srdjevic 2005; Dong et al., 2008; Srdjevic and Srdjevic, 2011 and 2013) is known as the order reversal measure, commonly labeled as the minimum violation (MV) criterion (Golany and Cress, 1993). This criterion indicates the total number of rank reversals when numerical judgments in a local matrix are compared with the ratios of computed weights for the corresponding elements (used at a time when the judgment has been made). To create a unique framework for comparisons, the MV values can be normalized to make them independent of the sizes of the matrices within the hierarchy. By summing all the local (normalized) MV values, it is easy to compute the HMV for the whole hierarchy, similar to the HCR and the HED. In this way, we argue that a unique, hierarchy-wise framework can be created in measuring the decision maker’s consistency.
On the other hand, there are numerous AHP applications in group contexts, but not that much reported research on how group consistencies should be computed. Aggregations of individual judgments into the final decisions are well described in many papers (e.g., Aguaron et al., 2003; Moreno-Jimenez et al., 2008). In this work we present the results of several aggregations we used to derive the final weights of alternatives in the group context based on assessment of the global consistency measures HCR, HED and HMV. We asked three experts to assess the usability characteristics of the e-learning management systems (LMS): Blackboard 6, Clix 5.0 and Moodle 5.2.1. The same hierarchy as in (Srdjevic et al., 2012) is used and after individual AHP applications the final weights of the LMSs are compared based on the demonstrated consistencies of the experts. The final weights of the LMSs computed for all three experts are aggregated geometrically into the final group weights. The group result is then compared with the result obtained for the so-called virtual decision maker. Following an idea published in (Srdjevic and Srdjevic, 2013), an artificial (virtual) decision maker is constructed in such a way that at each node of the hierarchy the most consistent decision maker (expert) is selected and his/her comparison matrix is taken to be propagated through the standard AHP synthesis. Because the standard skeleton of the AHP pair-wise comparison matrices is created by inserting into the skeleton the most consistent matrices from the group of experts, the synthesis process consequently propagates the best local consistencies of the group, regardless who of the decision makers is involved at what local node of the hierarchy. As expected, the hierarchy-wise consistency is higher than the hierarchy-wise consistency of the most consistent decision maker. The final group decision (alternative weights) obtained for the virtual decision maker differed from the individually derived decisions. Although this decision is artificial, it can be compared with the decisions obtained with other aggregation techniques. In the 3D consistency space (HCR-HED-HMV) it offers an ideal-point benchmark for other methods with various options for further research agenda.
In the presented case-study example, we performed a sensitivity analysis aimed at recognizing if and how the changes in selecting the locally best priority vectors can influence the final result - ranking the alternatives. Multiple exchanges of local priority vectors from decision makers not differing in local consistencies by more than 10% did not produce the order reversal of alternatives if the exchanges were made at the bottom level of the hierarchy. Only in one case did it happen after the exchange of priority vectors was made at one node of a sub-criteria level. The local consistencies of two decision makers at that node were small which implies that measuring the consistency is important task, especially from the global hierarchy point of view, rather than just one local matrix (as most researchers concentrate on). Note that the AHP philosophy is the whole hierarchy, not any single matrix. Our experiments show that the sensitivity of the final solution is very dependent on the local consistencies of the decision maker, either in individual or group AHP applications. This intuitively logical conclusion is especially important in making and interpreting decisions related to the allocation of resources. The changes in weights are not of so much importance if the ordinal preferences of the alternatives are of interest, unless their final weights incur the reversal of their ranks. This effect has been the core part of the presented case-study example in this paper.
In Section 2 of the paper we briefly describe the AHP method. The AHP's application contexts, individual and group, are described in Section 3. The case-study example is presented in Section 4, while in Section 5 we derive the main conclusions. At the end of the paper is the cited literature and two appendices.
The AHP multi-criteria method enables an assessment of a hierarchy of decision elements in straightforward manner by performing pair-wise comparisons of the hierarchy elements at all levels, following the rule that at a given level the elements are compared with respect to the elements in the higher level by using the fundamental 9-point importance scale (Saaty, 1980). The created comparison matrices represent the input to the so-called prioritization model, which outputs the weights of the compared elements. The weights (summing up to 1) are usually called priorities and are mathematically represented as local priority vectors, one for each judgment matrix. There are several well-known prioritization models, generally distinguished as matrix and optimization models (Srdjevic, 2005; Kou and Lin, 2014). An interesting discussion on robust estimation of priorities in the AHP is given in (Lipovetsky and Conklin, 2002). Priority estimation in various group decision
making contexts is extensively elaborated in many papers (e.g. Lipovetsky, 2009; Yu and Lai, 2011), including few recently published (Scala et al., 2016; Srdjevic et al., 2016).
The core of the standard AHP is a set of local pair-wise comparison matrices created during the judgment process, a method for extracting the priorities of decision elements from these matrices, and a synthesis procedure where local priorities are multiplied by adjacent priorities from the upper level in the hierarchy and summed in a downward direction to obtain the final utilities of alternatives at the bottom level versus the goal at the top level of the hierarchy. The major feature of the AHP is that it involves a variety of tangible and intangible goals, attributes, and other decision elements. In addition, it reduces complex decisions to a series of pair-wise comparisons; implements a structured, repeatable, and justifiable decision-making approach; and builds a consensus.
The standard AHP (Saaty, 1980) uses the consistency coefficient CR to indicate the inconsistency of the decision maker. The other commonly used consistency measures are the total Euclidean distance, the minimum violations measure, and some more measures associated with prioritization methods. E.g., the geometric consistency index is used with the logarithmic lest squares prioritization method (Aguaron et al., 2003), and so-called natural consistency measure μ is used with the fuzzy preference programming prioritization method (Mikhailov, 2000). For group contexts, Dong et al. (2010) proposed to use the geometric cardinal consensus index while decision makers are searching for consensus. More on consistency in group contexts can be found in recent references (e.g. Benitez et al., 2016; Blagojevic et al., 2016).
The AHP was originally developed to support individual decision-making. In the last two decades the AHP is also aimed at supporting the decision-making processes in group contexts. In the latter cases, various aggregation schemes are applicable such as commonly used AIJ - aggregation of individual judgments and AIP - aggregation of individual priorities (Forman and Peniwati, 1998). An interesting iterative procedure for aggregating individual priorities is proposed in Dong et al. (2010). More recently Dong et al. (2015) proposed the methodology for the individual selection of numerical scale and prioritization method as an iterative consensus reaching model in investigating AHP group decision-making problems. Good overview of AHP applications in group contexts can be found in (Ishizaka and Labib, 2011), various applications in (Xu and Cai, 2011; Tavana et al., 1996) and most recently in Blagojevic et al. (2016). If there is more than one DM, the overall priorities of the alternatives can only be computed after the individual opinions of all the DMs have been elicited. There are different approaches to the aggregation of individually obtained priorities (Ossadnik et al., 2016). We shall elaborate two approaches:
(1) Geometric aggregation of the final individual priorities. This method is known as the geometric mean method (GMM); a version of this method applicable in AHP group use is known as the aggregation of (final) individual priorities - AIP;
(2) Aggregation of the locally best individual priorities before the final AHP synthesis is performed. This method is referred to as the MGPS algorithm (after the key terms ‘multi-criteria group prioritization synthesis’, Srdjevic and Srdjevic, 2013).
The first approach is based on the suggestion that there are three possibilities to directly aggregate the individual opinions of all the decision makers in the group. The first is to aggregate the individual judgments for each set of pair-wise comparisons into an 'aggregate hierarchy'. The second is to synthesize the individual hierarchies and aggregate the resulting priorities. The third option is to aggregate the derived individual priorities in each node of the hierarchy. The first is commonly referred to as aggregating the individual judgments (AIJ), and the second as aggregating the individual priorities (AIP). The third option is less used. It is generally agreed that AIJ and AIP are philosophically different circumstances, and whether AIJ or AIP should be used depends on whether the group intends to behave as a synergistic unit or as a collection of individuals. In situations where inconsistencies and/or significant disagreements may occur while eliciting individual judgments, the commonly used AIP method employs weighted geometric averaging. (Forman and Peniwati, 1998) provide an interesting discussion on the issue of aggregation and argue that, in most cases of group decision-making, the use of weighted geometric averaging does not violate the Pareto principle.
Aggregation of the local individual priorities by using the MGPS algorithm (Srdjevic and Srdjevic, 2013) realizes a concept analogous to the one proposed in Srdjevic (2005) for individual AHP applications where the best local priority vectors are selected based on the consistency performance of several of the most popular prioritization methods. In MGPS, judgment matrices in all nodes of a given hierarchy, as elicited from the decision makers, are used to compute local priority vectors. In turn, consistency measures ED and MV are computed for each matrix and for each decision maker. Assuming that
the consistency measures are evaluation criteria with associated weights, the individually obtained priority vectors are subjected to the multi-criteria evaluation aimed to identify a single decision maker at each single node of hierarchy with the best demonstrated consistency. Corresponding locally best priority vectors are synthesized to obtain the final AHP group decision.
As far as group aggregations are considered, an additional issue is how to obtain the individuals' weights if they are not to be equally weighted, and how to use them in the aggregation. How to differentiate decision makers is challenging and hardly justified (Blagojevic et al., 2016), and therefore our decision in this research was to associate equal weights with all the decision makers participating in the group for the evaluation of the e-learning management systems.
4.1 Problem statement and its hierarchy
The problem is stated so as to assess and rank by applicability the three e-learning management systems (LMS) based on three typical qualitative criteria and a number of qualitative sub-criteria (Pipan et al., 2010; Pipan et al., 2013; Srdjevic et al., 2012). Three experts are asked to perform the decision-making processes by applying the AHP model.
The four-level hierarchy, shown in Figure 1, is used as in Srdjevic et al. (2012). Description of decision elements is given in Appendix A.
Fig. 1 Hierarchy of the decision problem (Srdjevic et al., 2012)
The group of three experts in the e-learning field, and in particular with a significant knowledge of usability and user experience requirements (standards), took part in individual assessments of the aforementioned decision elements with the AHP. The experts are treated as independent evaluators and are identified hereafter as decision makers DM1, DM2 and DM3. Notice that the assessments of the evaluator DM1 are reported in Srdjevic et al. (2012) and replicated here in a group context with two more experts involved in evaluating the same problem.
4.2 Individual evaluations and group aggregations
Each decision maker filled 15 matrices with numbers from scale in Table 1 and the eigenvector prioritization method is used to compute the local weights of the decision elements for all the comparison matrices. The local consistency measures ED, MV and CR are computed and associated with the corresponding matrices. Along with the synthesis of local priority vectors for each decision maker to obtain the final weights of LMSs, corresponding hierarchy-wise values of the consistency measures, labeled accordingly as HED, HMV and HCR, are also computed for all the decision makers.
With equal weights being assigned to the decision makers, the AIP method is applied firstly to derive the final group decision. Secondly, based on the consistency measures summarized in Table 1, the MGPS algorithm is applied to derive a group decision, assuming the aggregation of the most consistent local matrices taken from the members of the group.
Based on the demonstrated consistencies in Table 1, a virtual decision maker representing the group is created, as given in Appendix B. To apply the MGPS algorithm, five local matrices are taken from the DM1, four matrices from the DM2, and the remaining six matrices are taken from the DM3 as indicated in Table 1. Finally, a standard AHP synthesis is performed for
the virtual decision maker to obtain the group weights of the e-learning management systems. The results of the performed AHP syntheses, aggregations and hierarchy-wise consistencies are presented in Tables 2 and 3, respectively.
Table 1 Local consistencies of the decision makers with highlighted best values by matrices
Table 2 LMSs (alternatives) weights computed for the individual experts and the group weights
obtained by the aggregated methods AIP and MGPS
Table 3 Hierarchy wise consistencies of individual experts and a virtual expert created within the MGPS framework
Srdjevic B., Srdjevic Z., Pipan M., Arh T., Balaban I.