Words: 7233
We study the process by which model-based decision support systems (DSSs) influence managerial decision making in the context of marketing budgeting and resource allocation. We focus on identifying whether and how DSSs influence the decision process (e. g. , cognitive effort deployed, discussion quality, and decision alternatives considered) and, as a result, how these DSSs influence decision outcomes (e. g. , profit and satisfaction both with the decision process and the outcome). We study two specific marketing resource allocation decisions in a laboratory context: sales effort allocation and customer targeting.
We find that decision makers who use high-quality, model-based DSSs make objectively better decisions than do decision makers who only have access to a generic decision tool (Microsoft Excel). However, their subjective evaluations (perceptions) of both their decisions and the processes that lead to those decisions do not necessarily improve as a result of DSS use. And expert judges, serving as surrogates for top management, have a difficult time assessing the objective quality of those decisions. Our results suggest that what managers get from a high-quality DSS may be substantially better than what they see.
To increase the inclination for managerial adoption and use of DSS, we must get users to “see” the benefits of using a DSS. Our results also suggest two ways to bridge the perception-reality gap: (1) improve the perceived value of the decision process by designing DSSs both to encourage discussion (e. g. , by providing explanation and support for alternative recommendations) as well as to reduce the perceived complexity of the problem so that managers invest more cognitive effort in exploring additional options and (2) provide feedback on the likely market/business outcomes of various decision options.
Key words: DSS; marketing models; decision quality; decision process; resource allocation ********** Introduction The determination and allocation of a budget (of time or resources, financial or otherwise) is a pervasive human activity. For example, we must all determine our budget for food, necessities, and leisure activities and allocate those budgets within those categories. We must also determine how much of our time we will work each week and how much of our remaining time we will spend with our children, surfing the Internet, watching television, and the like. Firms continually face such resource allocation challenges.
They must determine how much to spend on new product development and how to allocate those funds across projects and time. Charitable organizations must determine what their development budget should be and what past donors or prospects to target. Manufacturers must decide how much plant capacity to invest in and where that capacity should be placed. The determination of the budget and the allocation of that budget are tasks that are straightforward to define conceptually and mathematically, but not at all that simple for humans to perform “optimally” without some decision aid.
Indeed, such decisions helped form Simon’s (1955) view of satisficing behavior, where he states that “there is a complete lack of evidence that in actual human choice situations these computations can be or are in fact performed” (p. 105). It is perhaps not surprising, therefore, that a search on Google on June 14, 2004, for “resource allocation” and “software” turned up nearly 390,000 links. While it would seem then, that in an area of such importance, we would have substantial and definitive evidence about the benefits and costs of using decision support aids for resource allocation in various application domains, such is not the case.
For example, Agarwal et al. (1992) describe critical elements missing in systems to help support the choice of management information system (MIS) projects under resource constraints, a resource allocation task; Muckstadt et al. (2001) describe at least five key elements that they claim are missing in the design of systems to support resource allocation decisions in a supply chain. Indeed, there appears to be only modest evidence to support the belief that model-based DSSs can help improve business decisions of any sort (Sharda et al. 988, Benbasat and Nault 1990, Todd and Benbasat 1999). We focus here on one domain for specificity: organizational resource allocation decisions in marketing: how large the marketing budget should be (e. g. , for advertising, sales promotion, and sales force effort) and how that budget should be allocated over geographies, products, market segments, and time. And while there is some evidence of the effectiveness of model-based systems to support such decisions, the adoption rate of such systems by firms remains far below potential (Wierenga and Van Bruggen 2000).
A study by Accenture (2001) points out that more than two-thirds of the more than $1 trillion spent by the Global 1000 on marketing is allocated without any return on investment (ROI) justification, much less supported by a DSS. Is the apparent low level of adoption of decision support models in marketing because of their inherent lack of value, because their value (perceived or actual) is not sufficiently high for the adopting organization to incur the costs that the adopting individuals may be forced to bear, or because of some combination of these factors?
We study these issues by exploring how DSSs influence the decision process (e. g. , cognitive effort deployed, discussion quality, and decision alternatives considered) and, as a result, how these DSSs influence decision outcomes (e. g. , profit and satisfaction both with the process and the outcome). We study two specific marketing resource allocation decisions, i. e. , sales effort allocation, and customer targeting. We define a DSS as a packaged software application that uses analytical models to transform business data into numerical and graphical reports to help users make business decisions more easily and effectively.
In our conceptualization of DSSs the presence of built-in analytical decision models is essential, distinguishing a DSS from a more general-purpose tool like Excel. Also, DSSs for resource allocation differ on analytical model sophistication, ranging from relatively simple descriptive response models to sophisticated normative optimization models providing problem-specific recommendations. In this study, we investigate the effects of two quite sophisticated DSSs for resource allocation. There have been several studies on the effects and effectiveness of marketing DSSs, including DSSs designed for resource allocation. 1) Most have focused primarily on exploring whether the use of a DSS improves the performance of decision makers as measured by decision quality (typically based on outcome variables such as sales, profit, or market share computed endogenously from the model) or by decision makers’ satisfaction and confidence in the results of using the DSS. Only a few studies have examined how a DSS affects the decision process, and the few that have, have not investigated how the DSS influences both the process and the outcomes. The studies report mixed results regarding DSS effects on outcomes.
Most studies in the marketing literature report that DSSs improve marketing resource allocation decisions, with the notable exception of the study by Chakravarti et al. (1979), which concluded that the use of a DSS had a detrimental effect on decision quality. However, the broader DSS research reports mixed findings in laboratory studies on the effects of DSSs on decision outcomes (see Sharda et al. 1988, Benbasat and Nault 1990). Of the 11 studies that Sharda et al. (1988) reviewed, 6 showed improved performance because of DSS use, 4 showed no difference, and in 1 study performance actually decreased for DSS users.
We note three issues with respect to the past studies. (1) Most studies have not tracked the decision processes associated with DSS use. (2) It is possible that DSS can improve objective decision outcomes without having a positive effect on the subjective evaluations of these decisions and vice versa, and it would be useful to understand the separate nature of these two effects. (3) Field studies have used DSSs to address real managerial problems, but have lacked effective experimental control (e. g. Fudge and Lodish 1977), making it difficult to demarcate the drivers of DSS performance, while lab studies have imposed sound experimental controls, but have addressed relatively simple and contrived problems. We designed our research to balance the benefits of experimental control (internal validity) with those of real-world applicability (external validity) by conducting a laboratory study where we used field-tested DSSs and real-world cases, for which actual outcomes are both known and have been reported in the academic literature.
In view of the limitations of past studies, we designed our research to incorporate the following three features. (1) Broad assessment of DSS impact. We incorporate objective measures, exogenous expert judgments, and subjective perceptions of DSS impact, including both multiple dependent (outcome) and mediating (process) variables. We evaluate whether a DSS influences outcomes, and also study how that influence is moderated by changes in the decision process. (2) Two different DSSs.
We study the effects of two DSSs that have different foci, although both address resource allocation problems: customer targeting for ABB Electric and sales force allocation for Syntex Labs (described in subsequent sections). Having two different DSSs should allow us to more readily identify results/relationships that are both common and unique across these two types of resource allocation models. (3) Realistic study context. Unlike most previous studies, we do not compare a DSS versus a non-DSS treatment, an unrealistic comparison.
Instead, subjects in our non-DSS condition (which we will refer to as the Excel-only condition throughout for clarity) have access to and can manipulate (through Excel) the same data as the DSS subjects, as is the case in natural settings. Excel is a DSS generator that enables our subjects to do estimation and optimization tasks, both of which are useful for addressing the resource allocation problems faced by our subjects. However, Excel does not automatically enable people to develop problem representations to fully exploit the available information.
Our DSSs, unlike a more general-purpose tool like Excel, build in the DSS developers’ expertise in the design, potentially helping users to both better understand how to represent the problem as well as how to develop specific, defensible recommendations. The theoretical underpinnings of our study rest on the cost-benefit framework of cognition (Payne et al. 1993), applied to the field of decision support by Todd and Benbasat (1999), and on the fit-appropriation model (FAM) developed to understand the effects of group support systems (Dennis et al. 2001).
From these theories, we postulate that DSSs that have task-technology fit (i. e. , are of high quality) can improve decision quality / accuracy and / or improve decision outcomes. However, these theories also suggest that decision makers prefer less effort to more effort. Blending these theoretical perspectives, we develop hypotheses about the kinds of effects the DSSs will have on marketing resource allocation decisions. Our results show that decision makers with access to high-quality, model-based DSSs make objectively better decisions than do those with access only to Excel.
However, subjective evaluations of both the decisions made and the decision processes that lead to those decisions, do not necessarily improve as a result of DSS use. In particular, even expert judges have difficulty assessing the objective quality of those decisions. We also find that DSSs can lead to better objective decision outcomes without seeming to affect some aspects of the decision process, such as the number of decision alternatives considered.
However, to improve subjective evaluations of decision quality, a necessary precondition for DSS adoption, the decision process must be influenced in such a way that it is viewed favorably. The Impact of Model-Based DSS: Hypothesis Development To assess whether and how a DSS leads to different decision outcomes, we must distinguish the effects of a DSS on various impact measures and separate the effects of a DSS on the decision process from its effects on decision outcomes.
DeLone and McLean (1992), seeking a measure of general information systems (IS) impact or success, report “rather than finding none, there are nearly as many measures as there are studies” (p. 61). They identify several categories of IS impact and make an important distinction between the effect of the IS on the individual and the effect on the organization. The individual-level impact of an IS is closely related to the user’s perceived performance given IS use, but the impact can also involve providing a user with a better understanding of the decision problem or changing the user’s perception of the usefulness of the IS.
The organizational impact of an IS relates to its effect on organizational performance, which can be measured in terms of firm profit, sales, or related performance metrics, an area in need of research (DeLone and McLean 1992). This taxonomy of IS success measures applies to the DSSs we study: We address both the individual and the organizational impact of DSSs and distinguish two categories of impact variables–DSS impact on decision process and their impact on decision outcomes.
Within both impact categories, we distinguish between objective outcomes and subjective evaluations. Combining these two dimensions, we study the impact of DSSs on four groups of variables: (1) the characteristics of the team decision process, (2) the (individual) subjective evaluation of the team decision process, (3) objective decision outcomes for the team, and (4) the (individual) subjective evaluation of the decision outcomes (see Figure 1).
We investigate the impact of DSS on specific measures within each of these four classes of variables. Decisions emerge from an underlying decision process, which can be characterized by the amount of cognitive effort that people devote to problem solving, the quality of the discussions they have during the decision process, the decision alternatives they consider, and so on. Both the decision outcomes and the decision process, in turn, will be influenced by the context in which decision makers operate.
This context can be described by the characteristics of the decision environment, the characteristics of the decision makers who must resolve problems, and the characteristics of the available DSSs, especially how appropriate they are for the tasks faced by the decision makers. Click for Full Size A priori, we can expect decision models to have a positive effect on decision outcomes for several reasons. Decision makers have cognitive limitations in acquiring and processing information (Tversky and Kahneman 1974, Hogarth and Makridakis 1981, Bazerman 1998).
When confronted with large amounts of information in short timeframes, they use heuristic approaches to solve problems, which trigger various cognitive biases that could diminish decision quality. An example heuristic is anchoring and adjustment. Decision makers who apply this heuristic start from an initial “anchor” point and adjust it to arrive at a decision. When there are strong anchors, adjustments from an anchor point tend to be sub-optimal (Slovic and Lichtenstein 1971, Mowen and Gaeth 1992).
Many techniques have been proposed in the literature to “debias” the decision-making process, including task simplification, providing training and feedback to decision makers, and providing simulators to generate multiple alternative explanations (Croskerry 2003). A DSS can potentially be a debiasing tool to reduce several types of bias (Arnott 2002). In resource allocation tasks, DSSs can help managers cope with large amounts of information and integrate that information in a consistent way (Dawes 1979).
In particular, a DSS can help managers choose good resource allocation strategies by consistently weighting the available options according to specified criteria, whereas humans tend to alter the weights they assign to different variables by using heuristics. At the same time, a DSS can underweight important idiosyncratic elements (e. g. , the strategic desirability of an option) relevant to a particular resource allocation problem.
Given these advantages and limitations of DSSs, it is perhaps not surprising that several researchers have demonstrated that a combination of a DSS and human decision making outperforms unaided decision makers (Blattberg and Hoch 1990, Hoch 1994, Hoch and Schkade 1996). The main explanation for this finding is that DSSs cause changes in the (imperfect) processes by which decisions are made (Silver 1990). Thus, good decision support technologies should be designed to provide decision makers with capabilities needed to extend their bounds of rationality (Todd and Benbasat 1999).
However, the mere availability or use of DSSs will not automatically lead to better decisions because decision makers make effort-accuracy tradeoffs (Payne et al. 1993) in their decision processes, and these trade-offs affect the quality of the decision outcomes. The literature suggests that effort is the most important factor influencing strategy selection (Todd and Benbasat 1999). If a DSS enables a higher quality decision process with no more effort than the current process, that higher quality DSS process is more likely to be adopted.
However, if a DSS allows the decision maker’s existing decision process to be executed with less effort, then the decision quality may not improve. It is generally easier for decision makers to assess efficiency gains (e. g. , savings in cognitive effort) from DSS use than it is for them to assess decision-quality improvements, especially if they are inexperienced and unfamiliar with the DSS. Only if a DSS is intrinsically of high quality and makes it easy to deploy higher quality decision processes will DSS use improve both objective decision outcomes and decision efficiency.
However, if users do not recognize the intrinsic quality of the DSS or the value of the outcomes it helps generate, they may not be satisfied. The fit-appropriation model (FAM) (Dennis et al. 2001) proposes that the effects of (group) DSSs are influenced by two factors. The first is the fit between the task and the DSS, i. e. , the task-technology fit. The second is the appropriation support the group members receive in the form of training, facilitating, routinizing, or software restrictions to help them incorporate the system effectively into their decision-making process.
FAM proposes that task-technology fit is a necessary, but not sufficient, condition to improve decision performance. Without proper appropriation support, performance is less likely to improve significantly even when task-technology fit is high. That is, the effect of task-technology fit on performance will be moderated by appropriation support. Appropriation itself, in turn, is affected by the fit (a good fit is more likely to lead to faithful appropriation). Empirical results show that even without appropriation support performance may still be influenced positively, whereas the subjective evaluations (e. . , satisfaction with the decision) may not be (positively) influenced (Dennis et al. 2001). Hence, the FAM model suggests that DSSs can be expected to improve objective decision outcomes if they show a sufficiently high level of task-technology fit. Given decision makers’ natural tendency to prefer effort reduction, and the fact that merely following the recommendation of a high-quality DSS offers both low effort and high decision quality, we can expect high-quality DSSs to improve objective outcomes (incremental return/profit).
For our study, we selected DSSs that have high task-technology fit, a necessary condition for improving objective decision performance. We also chose the level of appropriation support to reflect the conditions under which resource allocation software is typically used by marketing managers, as few companies today maintain a large analytic staff to provide extensive support for managers. Rather they operate in environments characterized by moderate appropriation support for software (e. g. , telephone/ web support, remote diagnostics, and so on).
We investigate the effects of the availability of model-based DSSs relative to the use of the general purpose decision tool, Microsoft Excel. Excel is a DSS generator that enables estimation and optimization, both of which are useful for the tasks faced by our subjects. However, Excel does not help users with problem representation (i. e. , what to estimate and how, or what to optimize and how). Without the appropriate problem representation, the available data may not be exploited to the fullest.
In contrast, our DSSs provide a sort of blueprint, embedding an analytical process for problem formulation and representation that allows the users to more fully exploit the available data. In a sense, Excel is just a toolkit whereas our DSSs are toolkits that come with a blueprint, permitting users to exploit the toolkit most effectively. Thus, although Excel is less restrictive than our DSSs, it offers no guidance to fully exploit its capabilities in a specific problem context. Therefore, the Excel-only aid has a lower task-technology fit than the two DSSs used in the study.
The two DSSs in our study are similar to each other in that they both support resource allocation decisions, and both show a high level of task-technology fit and moderate appropriation support. However, they differ in important ways: the DSS for ABB is nondirective (i. e. , it gives no feedback, nor does it generate specific recommendations) whereas the Syntex DSS provides both a specific recommendation and a projected profit impact of that recommendation, relative to the current allocation. Goodman (1998) and Wigton et al. 1986) show that such feedback can play both an informational role (promoting knowledge acquisition) as well as a motivational role (providing a reward cue for increasing cognitive effort investment). In the framework of Balzer et al. (1989), user interactions with the Syntex DSS (but not with the ABB DSS) provide “cognitive feedback,” linking the task with the environmental performance measures. Note that the Syntex DSS allows users to conduct “what-if” analyses, experimenting with different constraints and observing their impact on expected profits.
The ABB DSS only offers users additional information in terms of computed probabilities, but does not include options to explicitly encourage users to experiment with or analyze multiple scenarios, as was the case with the Syntex DSS. Even though both DSS have moderate levels of appropriation support (e. g. , no direct training), Syntex provides more appropriation support than ABB, and could lead to higher effort deployment, which in turn could lead to better outcomes. However, we do not postulate these differences as formal hypotheses.
Based on the above reasoning, we propose the following hypotheses about the impact that the use of high-quality DSS will have on various measures of performance. (2) HYPOTHESIS 1. Model-based DSSs will improve objective decision outcomes relative to the Excel-only tool. HYPOTHESIS 1A. DSSs will generate more incremental returns/profits relative to the Excel-only tool. HYPOTHESIS 1B. DSSs will result in more favorable experts’ evaluation of the decisions relative to the Excelonly tool. Because we have a moderate level of appropriation support, we posit the following.
HYPOTHESIS 2. DSSs will have no effect on the subjective evaluation of the decision outcomes relative to the Excel-only tool. HYPOTHESIS 2A. DSSs will have no differential effects on decision satisfaction relative to the Excel-only tool. With only moderate appropriation support, it would be difficult for decision makers to judge the value of the available DSS, and therefore they would not be able to fully assess how their decision-making efforts improve decision quality. Therefore, decision makers are more likely to reduce effort at the expense of decision accuracy.
It is also possible that the availability of a model to facilitate discussions could lead decision makers to (incorrectly) pursue quick consensus when, in fact, it may be in their best interest to generate more alternatives, and evaluate them more autonomously (Miranda and Saunders 1995). In view of the above arguments, we expect less effort, but more focused and higher quality discussions (because of the high-quality DSS), which also means that fewer decision alternatives will be generated or evaluated carefully. We, therefore, hypothesize the following. HYPOTHESIS 3. DSSs will have mixed effects on the decision process.
HYPOTHESIS 3A. DSSs will lead to less effort devoted to problem solving than the Excel-only tool. HYPOTHESIS 3B. DSSs will enhance the quality of the discussions as compared to the Excel-only tool. HYPOTHESIS 3C. DSSs will lead to fewer decision alternative generated than the Excel-only tool. Appropriation support involves training, facilitating, routinizing, or software restrictions to help users incorporate a DSS within their decision-making process. We can expect moderate levels of appropriation support to lead to less intensive decision processes with mixed effects on the subjective evaluation of these processes.
On the one hand, if decision makers spend less effort on the task because of the facilitation offered by the DSS, they may not discover the full complexity of the task. On the other hand, they may view the decision problem as complex, and because they do not spend much effort developing a full understanding of the problem, the complexity remains even after DSS use. Given these two countervailing effects, we hypothesize no significant net effect of the DSS on perceived problem complexity. And because decision makers may use the DSS for effort reduction, it may reduce the amount of learning that occurs because of DSS use.
Finally, the DSS should help decision makers improve along at least one dimension: lower effort or higher decision quality. Therefore, we expect DSSs will be perceived as being useful. HYPOTHESIS 4. DSSs will have mixed effects on the subjective evaluation of the decision process. HYPOTHESIS 4A. DSSs will have no effect on perceived problem complexity relative to the Excel-only tool. HYPOTHESIS 4B. DSSs will lead to less perceived learning relative to the Excel-only tool. HYPOTHESIS 4C. DSSs will be perceived as useful relative to the Excel-only tool. Methodology To test the hypotheses, we applied six criteria to select our methodology.
First, we conducted our research in a laboratory setting because we wanted our decision context to be replicable to permit statistical model building and hypothesis testing. Second, our study required a realistic decision context to enhance external validity. Third, we wanted our hypotheses to be testable across DSS designs. Therefore, we included two realistic resource allocation scenarios, which had DSSs associated with them: the ABB Electric case and the Syntex Labs (A) case, both of which received the Edelman Prize from INFORMS as outstanding examples of the practice of management science.
Papers describing these models (Gensch et al. 1990, Lodish et al. 1988) include the actual market response to the resource allocation decisions implemented by the respective firms. Therefore, it is possible, ex post, to estimate likely decision effectiveness. Fourth, our subjects should be real decision makers or have had sufficient training in the domain to understand the issues associated with resource allocation decisions. Fifth, subjects should have the background and capability to understand and use spreadsheets and market response models.
Sixth, our subjects must not be experts (e. g. , analysts) in the use of DSSs, because our hypotheses concern decision making by typical managers. These criteria lead us to consider business school undergraduates, MBAs, and company executives. Pilot tests with undergraduates showed that they did not have sufficient background to understand the problem context. We were not able to locate a large enough group of executives who were sufficiently homogeneous in background and skill level to meet our needs.
Pilot studies with MBA students who had taken core marketing and management science courses showed that such students not only were able to understand both the context (marketing resource allocation) and the approach (spreadsheet tools and response model-based decision support), but were also sufficiently homogenous along other dimensions to make them appropriate subjects. We adapted software implementations of the ABB and Syntex model from Lilien and Rangaswamy (1998). To mimic the organizational reality and group decision process associated with such decisions in practice, we used two-person teams as the experimental unit.
We randomly assigned each team to one of eight experimental conditions to analyze and develop recommendations for both cases. All groups received identical data (described in the cases) in the form of spreadsheets and had the full functionality of Microsoft Excel available to them. The groups differed in (1) whether their spreadsheet included an embedded DSS model that allowed them to analyze the data (if they chose to) using a resource allocation model and (2) the order in which they analyzed the cases–ABB followed by Syntex or Syntex followed by ABB.
We briefly describe these two cases and the associated models. ABB Electric Case. The decision problem was to allocate a supplementary marketing budget to the “top 20” customers (out of 88) to be recommended by the subjects. The data reported how each of these customers rated the four suppliers (including ABB) on criteria such as invoice price, technical specs of the products, availability of spare parts, and so on. Subjects who had access to the DSS were also able to run a multinomial logit analysis to determine the probability of each customer buying from each of the four suppliers.
The subjects could then use the results of the model analysis in any way they thought was appropriate (e. g. , sort customers according to their probability of purchasing from ABB and construct an index of vulnerability or attractiveness) in identifying the target customers. To provide a common decision anchor, all subjects were told the company had historically targeted its marketing programs at its largest customers, but that a company consultant had introduced the concept of targeting customers by “switchability. The new idea was to target those customers whose likelihood of purchase indicated that they were “sitting on the fence” with respect to purchasing from ABB (i. e. , where ABB was either a narrow first choice or was the second choice by a narrow margin), and pay less attention to those customers who were already either loyal to competitors or were loyal to ABB. Switchability segmentation conflicted with the prior company resource allocation strategy, which was to target purely based on sales potential of the customers, putting more effort on customers with higher sales potential.
Figure 2 summarizes the key data as well as the results from running the multinomial logit model (available to groups that had access to the DSS). Syntex Labs (A) Case. The Syntex case describes the situation that Syntex Labs faced in 1982, when it had 430 sales representatives in the United States and were adding 40 reps per year. The company had 7 different products and the stated management plan was to continue adding 40 reps per year, and to allocate those reps to those 7 products proportionally to the current allocation of representatives.
The company was concerned both about the total size of its sales force and the allocation of the sales effort across products, because a relatively new product, Naprosyn, was popular and appeared to be underpromoted relative to the resources allocated to other products. The case describes the concept of a response model and the hiring of a consultant who led a team of Syntex executives through the calibration of that response model.
All subjects received data on the current level of effort, the allocation of that sales effort to products, the current sales of these products, the profitability of the products, the current overall profitability of the firm, and the results of the response model calibration session. This latter information–the answer to questions such as “What would the percentage sales of Naprosyn be with a 50% increase in its sales force allocation from (current) 96. 8 sales reps to 145. 2? ” Answer: “26% increase” (Cell I9 in Figure 3)–was provided to the respondents in Excel template format as presented in Cells G9-J15 in Figure 3.
The DSS-supported group also had access to an optimization model that generates the recommendations in Columns D and E in Figure 3. That model allowed subjects to determine the “optimal” (short-term profit maximizing) sales force size and effort allocation, either on an unconstrained basis or under user-specified constraints. Those constraints could be placed either on the total size of the sales force or on individual products. Subjects were given a common decision anchor to allocate any increase in sales effort proportionally to the most recent level of effort allocation, a widely used process in practice.
The ABB DSS and the Syntex DSS differ both in their designs and in respect to the problem context in which each is used. The ABB DSS does not make specific recommendations about which customers to target under various user-selected criteria (the user has to develop those criteria), nor does it provide any expected outcomes associated with a resource allocation decision. In contrast, the Syntex DSS, through a profit optimization tool, makes specific recommendations for the sizes of the sales force and effort allocation, and also provides the resulting level of expected profit (computed from sales response functions).
Thus, Syntex is a directive DSS, which provides its users with specific feedback on the expected sales and profit outcomes of alternative resource allocations. Experimental Procedure. Subjects analyzed both the ABB and the Syntex cases in two-person teams. We systematically manipulated the availability of a DSS as shown below. To address order effects, half the groups did the ABB case first followed by the Syntex case, while the other groups did the reverse. Table 1 summarizes the resulting eight experimental conditions.
Our experimental procedure consisted of the following five steps. Step 1. Background and Qualifications. After entering the lab, each subject completed a pre-experimental questionnaire with questions about demographics (age, gender, and so on), work background, and computer and Excel experience. We used this information to check (post hoc) whether the teams in the various experimental conditions had similar backgrounds and qualifications. Click for Full SizeStep 2. Case 1. Subjects as a group received their first case and a tutorial illustrating how the related software worked.
The tutorials that were given to subjects with the DSS contained additional information about running the DSS. After they had analyzed the case, all groups completed forms summarizing their recommendations and their justifications for those recommendations. Step 3. Postanalysis Questionnaire 1. After completing their recommendation form for the first case, all subjects (individually) completed a postanalysis questionnaire that asked for their subjective evaluations of their case analysis, the associated discussions, their recommendations, and their assessment of the available software.
Step 4. Case 2. The same as Step 2, but now for the second case. Step 5. Postanalysis Questionnaire 2. The same as Step 3, but now for the second case. At the end of the exercise, the subjects were debriefed and asked not to discuss the case with anyone. Subjects. There were 112 first year MBA students who participated in our study, making 56 groups, with 7 groups per experimental condition. We paid each subject $25 for their (approximately) 3 hours of participation. We told all groups they were eligible to win one of three group prizes, depending on their performance. Click for Full Size
Measures. We classify the variables used in the study as (1) experimental factors (independent variables) and the following dependent DSS impact variables: (2) group decision process variables, (3) subjective evaluation of the group decision process variables, (4) objective decision outcome variables, and (5) subjective evaluations of decision outcomes variables. (We also collected information on problem-solving style and computer and Excel efficiency and found no significant differences between experimental groups. ) Below, we describe these variables and their measurement.
Experimental Factors. We systematically manipulated the following two experimental factors. (1) DSS Availability. Yes–1 or No–0, for the two DSSs being used; namely, ABB and Syntex. Unobtrusively collected tracking data showed that the decision groups that had the DSSs available always used them. (2) Order. ABB first/Syntex second = 0; Syntex first/ABB second = 1. To control for order effects, we had half the teams start with the ABB case and other half start with the Syntex case in a manner that made order independent of the two experimental factors overall. DSS Impact.
In Tables 2a-d, we summarize the process and outcome variables that we used to measure DSS impact. Wherever feasible, as noted in the tables, we used or adapted scales from previous research, although for several constructs, as indicated in Tables 2b-d, we had to develop new measures because well-tested scales either did not exist or were inappropriate for our context. We used LISREL 8. 3 (Joreskog and Sorbom 1993) to assess the quality of the seven subjectively measured multiple-item measurement instruments (e. g. , subjective evaluations of effort, discussion quality, decision alternatives considered, complexity, learning, perceived sefulness of the DSS, and satisfaction with the decision). We specified one confirmatory factor analysis model. The chi square for this model is 171. 09 (p = 0. 81). The comparative fit index is 1, above the generally accepted level of 0. 90. The value of the standardized root mean square residual is 0. 058. All factor loadings are highly significant (minimum t-value is 4. 40, p < 0. 001, and most t-values are above 10) and larger than 0. 50 (except for one loading, which was 0. 35). These findings support the convergent validity of the items.
The correlations between the seven constructs are significantly different from unity, which supports their discriminant validity. In view of the small sample size of our data set, we also developed confirmatory factor models separately for each group of impact measures (see Figure 1), i. e. , for decision process, subjective evaluation of decision process, and subjective evaluation of decision outcome (Churchill 1979). These analyses yielded results similar to that of the overall confirmatory factor model. The Cronbach alpha reliabilities of the factors are presented in Tables 2b-d and range from 0. 6 to 0. 96. As objective outcomes of using the DSSs, we measured the incremental revenue obtained by the teams and the quality of the recommendations and their justifications as judged by outside experts. The items listed in Table 2a require additional description. Incremental Return Computation. For both cases, there is information in the research papers cited earlier about the resource allocation plans actually adopted by the firms and the incremental return (profits in the Syntex case and incremental sales revenue for ABB) that can be attributed to these plans.
That information allows us to calibrate a scoring rule to determine what the incremental return would be for any recommendation made by a team. Click for Full SizeWhile the Syntex model is an “optimization model,” it also permits the user to specify different constraints on the total amount to spend and the upper and lower bounds on per-product spending, to change projected product profit margins, to run sensitivity analyses on different response functions, and the like.
Hence, no single “optimization” is right, and the subjects are urged to run scenarios and consider organizational and resource constraints in making their recommendations. The ABB model is not an optimization model; rather it provides information about the likely response of individually targeted customers and prospects. The test condition, in which the DSS provides purchase probabilities, allows knowledgeable users to develop better targeting plans rule than in the control condition, where those probabilities are not provided.
It is important to note that in this research, as in management practice, there is no single optimal solution; hence, we frame our assessment procedure both in terms of incremental return calculation (for objective results) and expert judgments (for subjective assessments). ABB Incremental Return Calculation. For ABB, we used the market results reported in Gensch et al. (1990, Table 3, p. 16). * There is no impact on incremental revenue from additional effort deployed on customers who are loyal to ABB or loyal to competitors.
Specifically, if either ABB or a competitor had a purchase likelihood statistically significantly higher than that of its closest competitor, ABB saw no gain in targeting these customers. * There is a 30% gain from customers who had a slightly lower probability of purchasing from ABB (but not significantly so) than from their most preferred supplier. ABB would then see a 30% gain, on average, from targeting these customers (called switchables). * There is a 31% gain from customers who had a slightly higher probability of purchasing from ABB (but not significantly so) than from their next most preferred supplier.
ABB would then see a 31% gain, on average, from targeting these customers (called competitives). We used the choice probabilities computed by the model to identify the largest 20 of the vulnerable customers (switchables and competitives). We then computed the expected incremental sales from each targeted customer as: zero if not a switchable or competitive customer, and otherwise equal to adjustment factor * ((1 – P(buying from ABB)) * max sales potential).
This formula determines the incremental return that ABB gets either by retaining a customer who would otherwise likely have switched to a competitor, or by gaining a new customer who currently marginally prefers one of the competitors. We computed the adjustment factor (=0. 40) to obtain the overall sales increase from switchables and competitives of 30. 5% to be consistent with the actual results that ABB realized. DSS users had the information in Figure 2 to work with; they had to develop their own financial targeting rule, albeit with better information than the Excel-only users.
Syntex Incremental Return Calculation. Syntex’s actual market performance (three years forward) closely matched what the managerially generated judgmental response functions had predicted. Hence, we used the following estimate of profit per product: Profit for Product i = [base sales[. sub. i] X Response[. sub. i]([X. sub. i]/[base [X. sub. i]]) X margin i] – [[X. sub. i] X salesman unit cost], where Click for Full Size[X. sub. i] is the sales force effort level deployed on product i, and Response[. sub. i]([X. sub. i]/base [X. sub. ]) is the judgmentally calibrated response function assessed at [X. sub. i]. We summed these profit figures over all products to yield an overall company profit for a team’s recommendation. As an example for Naprosyn, if the recommendation is for 145 reps (approximately 1. 5 X 96. 8 reps), then, from Row 9 of Figure 3, we get Naprosyn profit = $214,400,000 X 1. 26 X 0. 70 – $63,000 X 145 =$179,965,000. Note that the DSS automates the estimation of the response function and invokes Excel’s Solver optimization module to help with such calculations (i. e. , the estimation of the 1. 6 response factor above resulting from the 50% increase in the sales force allocation to Naprosyn). As noted earlier, the DSS also permits the user to impose upper or lower limits on overall sales force spending or on spending on individual products. Excel-only users had all the input data needed to build the response function (i. e. , data in cells G9-J15 in Figure 3), but not the response functions themselves. Click for Full SizeExpert Rater’s Evaluations. All groups completed a recommendation form for each case, along with their justifications for these recommendations.
We transcribed and typed these recommendation forms to make them appear uniform, and gave them to three expert raters for evaluation. We removed references to the form of the DSS that the respondents had available so that the raters would not know whether the groups had had access to a DSS to aid their decisions. The raters were senior faculty members in marketing and management science at two leading universities and were knowledgeable about the specific problem context and resource allocation issues in general. We provided the raters with the cases and the accompanying software, but provided no indication of “right” answers.
We then asked the raters to score the overall quality of the recommendations on a scale of 1-100. Results To test the significance of the effects of our experimental manipulations, we conducted a series of analysis of variances (ANOVAs) with repeated measurements. Within each team, we treated measures for the two subjects as the repeated measures for the same dependent variables. As independent variables, we included DSS availability and order. Further, we included the interaction between DSS availability and order to analyze whether the effect of a model would be different if a case were analyzed first or second.
The structure of the ANOVA model is as follows: DSS impact variable = [[beta]. sub. 0] + [[beta]. sub. 1] * DSS availability + [[beta]. sub. 2] * order + [[beta]. sub. 3] * DSS availability * order. The significance levels we report in this section are based on an analysis of the complete model that includes the two main effects and their interactions. DSS Impact on Objective Decision Outcomes We start by testing Hypothesis 1; namely, that the use of a DSS improves objective decision outcomes.