Words: 9798
| | ? Research Articles Conceptualizing and Measuring Democracy: A New Approach Michael Coppedge and John Gerring, with David Altman, Michael Bernhard, Steven Fish, Allen Hicken, Matthew Kroenig, Staffan I. Lindberg, Kelly McMann, Pamela Paxton, Holli A. Semetko, Svend-Erik Skaaning, Jeffrey Staton, and Jan Teorell. 1 In the wake of the Cold War, democracy has gained the status of a mantra. Yet there is no consensus about how to conceptualize and measure regimes such that meaningful comparisons can be made through time and across countries. In this prescriptive article, we argue for a new approach to conceptualization and measurement.
We ? rst review some of the weaknesses among traditional approaches. We then lay out our approach, which may be characterized as historical, multidimensional, disaggregated, and transparent. We end by reviewing some of the payoffs such an approach might bring to the study of democracy. n the wake of the Cold War, democracy has gained the status of a mantra. Perhaps no other concept is as central to policymakers and scholars. Yet there is no consensus about how to conceptualize and measure regimes such that meaningful comparisons can be made through time and across countries. Skeptics may wonder if such comparisons are possible at all.
While this conclusion may seem persuasive, one must also consider the costs of not comparing in a systematic fashion. Without some way of analyzing regime types through time and across countries we have no way to mark progress or regress on this vital matter, to explain it, to reveal its consequences, or to affect its future course. This may account for the ubiquity of crossnational indices such as Freedom House, Polity, and Democracy/ I Michael Coppedge is Professor of Political Science and a Faculty Fellow of the Helen Kellogg Institute for International Studies at the University of Notre Dame (coppedge. 1@nd. du). He has served as Chair of the APSA Task Force on Indicators of Democracy and Governance. (Although this collaboration has many synergies with the APSA task force, this article is not a product of the task force and does not necessarily represent the views of its members. ) John Gerring is Professor of Political Science at Boston University (jgerring@bu. edu). He served as a member of The National Academy of Sciences’ Committee on the Evaluation of USAID Programs to Support the Development of Democracy in 2006???07 and as President of the APSA’s Organized Section on Qualitative and MultiMethod Research in 2007???09.
The authors’ names are listed alphabetically to re? ect equal contributions. The authors thank the Kellogg Institute at Notre Dame and the Pardee Center for the Study of the Longer-Range Future at Boston University for their generous support. The other collaborators???David Altman (Ponti? cia Universidad Catolica de Chile), Michael Bernhard (University of Florida), Steven Fish (University of California, Berkeley), Allen Hicken (University of Michigan), Matthew Kroenig (Georgetown University), Kelly McMann (Case Western Reserve University), Pamela Paxton (University of
Texas at Austin), Holli Semetko (Emory University), Svend-Erik Skaaning (Aarhus University), Jeffrey Staton (Emory University), and especially Staffan I. Lindberg (Gothenburg University and the University of Florida) and Jan Teorell (Lund University)???are participants in an ongoing effort to implement the approach described here and have made many substantive contributions to this article. The authors also thank others who generously commented on various drafts of the manuscript: Andre Bachtiger, Tabitha Benney, David Black, Christian Davenport, John Dryzek, Amitava Dutt, Tasha Fair? eld, Tiago
Fernandes, Robert Fishman, Archon Fung, Larry Garber, Carlos Gervasoni, Clark Gibson, Jack Goldstone, John Grif? n, Rita Guenther, Jonathan Hartlyn, Macartan Humphreys, Jo Husbands, Sebastian Karcher, Phil Keefer, Fabrice Lehoucq, Jim Mahoney, Mick Moore, Gerardo Munck, Peter Nardulli, Guillermo O’Donnell, Anibal Perez-Linan, Sabine MacCormack, Kunle Owolabi, David Samuels, Margaret Sarles, Fred Schaffer, Andreas Schedler, Mitchell Seligson, Rich Snyder, Otton Solis, Paul Stern, Strom Thacker, J. Samuel Valenzuela, Nicholas van de Walle, and Jeremy Weinstein. June 2011 | Vol. 9/No. 2 247 doi:10. 017/S1537592711000880 ? ? | | ? Research Articles ? | Conceptualizing and Measuring Democracy Dictatorship in the work of social science and in the policy world. Clearly, there is a desire to compare. Indeed, a great deal depends upon our ability to analyze regime types through time and across space. Billions of dollars in foreign aid intended to promote democracy and governance in the developing world is contingent upon judgments about how democratic a polity is at the present time, its recent history, future prospects, and the likely causal effects of giving or withholding assistance, which e discuss in our conclusion. Likewise, a large portion of work in political science deals with these same issues, i. e. , the causes, consequences, and trajectories of regimes around the world. For both policymakers and academics???not to mention those living in the developing world, who are affected by rich-world policies???the conceptualization and measurement of democracy matters. How, then, can this task be handled most effectively? In this prescriptive article, we argue for a new approach to conceptualization and measurement. We ? rst review some of the weaknesses among traditional approaches.
Then we lay out our approach, which may be characterized as historical, multidimensional, disaggregated, and transparent. An online appendix includes a detailed questionnaire with all of the indicators envisioned for this project. 2 We next review some of the payoffs our approach might bring to the study of democracy. One is that it could foster greater agreement on the disaggregated components of democracy, even while recognizing the inherent impossibility of agreement on one way of assembling these components into an overall conception of “democracy. ” Another payoff is improved speci? ity in measurement, which would make indicators of democracy more reliable, more useful for the evaluation of policies, and appropriate for drawing conclusions about the sequencing of democratization. Similarly, our suggested approach may be useful for testing claims about the consequences of regime type for various outcomes of concern, e. g. , economic development, social policy, and international relations. Arguments for a New Approach Commentary and criticism directed at democracy indices are legion. 3 Here, we touch brie? y on six key issues of conceptualization and measurement: de? ition, precision, coverage and sources, coding, aggregation, and validity and reliability tests. Our discussion focuses on the most prominent efforts including the Political Rights, Civil Liberty, Nations in Transit, and Countries at the Crossroads indices, all sponsored by Freedom House; the Polity2 variable from the Polity IV database; a binary measure of democracy and dictatorship (“DD”) constructed by Adam Przeworski and colleagues; a binary measure constructed by Michael Bernhard, Timothy Nordstrom, and Christopher Reenock (“BNR”); a multidimensionsal index produced by the
Economist Intelligence Unit (“EIU”); and the Bertelsmann Transformation Index (“BTI”) funded by the Bertelsmann Foundation. 4 Of these, Polity2 (Polity IV) and Political Rights (Freedom House) are by far the most commonly used, having been cited hundreds of times according to the Web of Science and thousands of times according to Google Scholar. Glancing reference will be made to other indices in an increasingly crowded ? eld, and many of the points made in the following discussion probably apply more broadly. 5 However, it is important to bear in mind that each index has its own particular strengths and weaknesses.
The following exercise does not purport to provide a comprehensive review but merely to call attention to the sort of problems that are endemic to this sector. 6 In our discussion, index/indices will be understood as a highly aggregated, composite measure of democracy, while indicator(s) will be understood as a more speci? c, disaggregated element of democracy. For example, the Polity2 variable is an index while turnout is an indicator. De? nition Democracy, understood in a very general way, means rule by the people. This seems to be a common element to all usages of the word and claims a long heritage stretching ack to the Classical age. 7 All usages of the term also presume sovereignty. A polity, however large or small, must enjoy some degree of self-government in order for democracy to be realized. Beyond these core de? nitional elements there is great debate. The debate has both descriptive and normative overtones; it is about what actually occurring polities are (or reasonably could be) and about what they should be. 8 Since de? nitional consensus is necessary for obtaining consensus over measurement, the goal of arriving at a single universally accepted measure of democracy is, in some very basic sense, impossible.
If one cannot agree on what X is, one cannot measure X in an authoritative fashion. As an example, one might consider the Polity2 index, which rates the United States as fully democratic throughout the twentieth century and much of the nineteenth century. This is a fair conclusion if one disregards the composition of the electorate???from which women and blacks were generally excluded???in one’s de? nition of democracy. 9 Similar challenges could be levied against other indices that omit consideration of attributes that some regard as de? nitional. These omissions are particularly glaring where democracy is de? ned in a minimal fashion.
By the same token, conceptions of democracy that are more encompassing may be criticized for including elements that fall far from the core meaning of the term. For example, the Political Rights index includes questions pertaining to corruption, civilian control of the police, the absence of widespread violent crime, willingness to grant 248 Perspectives on Politics ? | | ? political asylum, the right to buy and sell land, and the distribution of state enterprise pro? ts. 10 The authors of the index would argue that it measures freedom, not democracy; nevertheless, it is frequently used as an index of democracy.
In other instances, it is not the extensiveness of the de? nition that is problematic so much as the judgments embraced within an index. For example, the EIU index regards mandatory voting as re? ecting negatively on the quality of democracy in a country. While this provision infringes upon individual rights and in this respect may be considered undemocratic, it also enhances turnout and, arguably, the quality of representation, and thus might be considered a component of democracy. Its status in enhancing rule by the people is therefore controversial, depending on one’s conception of democracy (a subject addressed later).
What is clear is that the methodological problems affecting contemporary indices begin at the level of de? nition. Since de? nitional consensus is necessary for obtaining consensus over measurement, the goal of arriving at a universally accepted summary measure of democracy may be illusory. Other problems of measurement ? ow, in no small part, from this de? nitional conundrum. ever, can be deceiving. Polity scores, for example, bunch up at a few places (notably ? 7 and ? 10), suggesting that the scale is not as sensitive as it purports to be.
Likewise, because Polity2 combines six underlying factors, a country’s score is the product of its subscore on these six factors (plus the weighting system that is applied to these factors). This means that two countries with the same score may have varying underlying components, which is to say that the quality of democracy may be quite different. The EIU index is by far the most sensitive of the extant indices, and does not appear to be arbitrarily bunched??? though it is of course subject to the problem of comparability (since, like Polity, aggregate scores are the product of underlying components). 4 In sum, the precision or reliability of all indices is too low to justify con? dence that a country with a score a few points higher is actually more democratic. 15 Note that most extant indices are bounded to some degree, and therefore constrained: there is no way to distinguish the quality of democracy among countries that have perfect negative or positive scores. This is acceptable so long as there really is no difference in the quality of democracy among these countries???an assumption that might be questioned. In 2004, Freedom House assigned the highest score on its
Political Rights index to countries as dissimilar as Andorra, Bulgaria, Denmark, Israel, Mauritius, Nauru, Panama, South Africa, Uruguay, and the United States. It seems probable that there are substantial differences in the quality of democracy among these diverse polities. Even if they are all highly democratic, they are democratic in different ways, e. g. , consensual versus majoritarian. 16 Precision Many of the leading democracy indices are insensitive to important gradations in the degree or quality of democracy across countries or through time. If one purpose of any measurement instrument is discrimination,11 extant emocracy indices fall short of the ideal. At the extreme, binary measures such as DD and BNR reduce democracy to a dummy variable. Of course, this is useful for certain purposes,12 such as analyzing the duration of democratic regimes. However, this dichotomous coding lumps together polities that exhibit quite different regime qualities. 13 For example, the DD index recognizes no distinctions within the large category of countries that have competitive elections and occasional leadership turnover. Papua New Guinea and Sweden thus receive the same score, despite evident differences in the quality of lections, civil liberties, and barriers to competition afforded in these two settings. (There is, however, a more differentiated coding for dictatorships, which are divided in the most recent version of DD into monarchic, military, and civilian. ) Continuous measures appear to be more sensitive to gradations of democracy/autocracy because they have more ranks. Freedom House scores democracy on a sevenpoint index (14 points if the Political Rights and Civil Liberties indices are combined???a questionable aggregation technique). Polity provides a total of 21 points if the Democracy and Autocracy scales are merged (also a ? awed ggregation procedure, but one suggested by the data providers), creating the Polity2 variable. Appearances, how- Coverage and Sources Many democracy indices are limited in temporal or country coverage. Nations in Transit, produced by Freedom House, covers only the post-communist states. Countries at the Crossroads, also produced by Freedom House, covers seventy countries (beginning in 2004) that are deemed to be strategically important and at a critical juncture in their trajectory. The Political Rights and Civil Liberty indices are the most comprehensive Freedom House endeavor, stretching back to 1972 and including most sovereign and emisovereign states. DD begins in 1946, BNR begins in 1919, BTI begins in 2003, and EIU begins in 2006. Most indices are updated annually???or, in the case of EIU, biannually. Only a few democracy indices stretch back further in historical time???notably, Polity (1800-), and the index of democratization created by Tatu Vanhanen (1810???present). 17 We suspect that the enduring value of Polity stems partly from its comprehensive historical coverage???though it excludes states with fewer than 500,000 inhabitants and makes no attempt to include colonies prior to independence, even if they enjoy substantial self-government. June 2011 | Vol. /No. 2 249 ? ? | | ? Research Articles ? | Conceptualizing and Measuring Democracy The limited reach or bias of most indices is partly a product of the data sources they rely on. The Political Rights and Civil Liberty indices rely heavily on secondary accounts such as the New York Times and Keesing’s Contemporary Archives for coding in the 1970s and 1980s. These historical sources, while informative, do not provide equally comprehensive coverage of every country in the world. In later eras, these indices have relied much more on country expert coding. However, the change from one source of evidence to another???coupled with some ossible changes in coding procedures???may have compromised the continuity of the time-series. No effort has been made to revise previous scores so that they are consistent with current coding criteria and expanded knowledge of past regimes. 18 Some indices such as the EIU rely heavily on polling data, which is available on a non-comparable and highly irregular basis for 100 or so nation-states. This means that data for these questions must be estimated by country experts for all other cases, about half of the population. (Procedures employed for this estimation are not made publicly available. 19 In an attempt to improve coverage and sophistication, EIU imputes a large quantity of missing data???a dubious procedure wherever data coverage is thin, as it seems to be for many of the EIU variables. While surveys of the general public are important for ascertaining the attitudes of citizens, the problem is that systematic surveys of relevant topics are not available for every country in the world, and in no country are they available on an annual basis. Moreover, use of such surveys severely limits the historical reach of any democracy index, since the origin of systematic surveying stretches ack only a half-century (in the US and parts of Europe) and is much more recent in most countries. Coding Many indices (including Freedom House, EIU, and BTI) rely heavily on expert coding. Such judgments can be made fairly reliably only if there are clear and concrete coding criteria. 20 Unfortunately, this is not always the case. To make this discussion more explicit, let us take a look at a few questions from the Nations in Transit survey, conducted by Freedom House. Five sub-questions are posed in answering the question, “Is the country’s governmental system democratic? “: 1.
Does the Constitution or other national legislation enshrine the principles of democratic government? 2. Is the government open to meaningful citizen participation in political processes and decision-making in practice? 3. Is there an effective system of checks and balances among legislative, executive, and judicial authority? 4. Does a freedom of information act or similar legislation ensure access to government information by citizens and the media? 5. Is the economy free of government domination? 21 These are not easy questions to answer, and their dif? culty stems from the ambiguous terms in which they are posed.
Wherever human judgments are required for coding, one must be concerned about the basis of the respondent’s decisions. In particular, one wonders whether coding decisions about particular topics???e. g. , press freedom???may re? ect an overall sense of how democratic a country is rather than an independent evaluation of the level of press freedom. In this respect, “disaggregated” indices may actually be considerably less disaggregated than they appear. It is the ambiguity of the questionnaires underlying these surveys, and their reliance on the subjective judgment of experts, that foster this sort of premature aggregation.
Aggregation Since democracy is a multi-faceted concept all composite indices must wrestle with the aggregation problem??? which indicators to combine into a single index, whether to add or multiply them, and how much to weight them. It goes without saying that different solutions to the aggregation problem lead to quite different results. 22 This is a very consequential decision. Typically, aggregation rules are additive, with an (implicit or explicit) weighting scheme. Others recommend that one should consider the various sub-components of democracy as necessary (non-substitutable) or mutually constitutive (interactive). 3 Evidently, in order for any aggregation scheme to be successful, rules must be clear, they must be operational, and they must re? ect an accepted de? nition of democracy. Otherwise, the resulting concept is not valid. Although most indices have fairly explicit aggregation rules, they are sometimes dif? cult to comprehend and consequently to apply (e. g. , Polity). They may also include “wild card” elements, allowing the coder free rein to assign a ?nal score, in accordance with his or her overall impression of a country (e. g. , Freedom House). A more inductive approach may also be taken to the aggregation problem.
Michael Coppedge, Angel Alvarez, & Claudia Maldonado apply an exploratory factor analysis of a large set of democracy indicators, identifying two dimensions which they label Contestation and Inclusiveness. 24 Other writers analyze extant indices as re? ections of a (unidimensional) latent variable. 25 These inductive approaches allow for the incorporation of diverse data sources and may provide uncertainty estimates for each point score. However, problems of de? nition are implicit in any factor-analytic or latent-variable index, for an author must decide which indicators to include in the sample??? equiring a judgment about which extant indices are measuring “democracy” and which are not???and how to 250 Perspectives on Politics ? | | ? interpret commonalities among the chosen indicators. This is not solvable simply by referring to the labels assigned to the indicators in question, as many of the most wellknown and widely regarded democracy indices are packaged as “rights,” “liberties,” or “freedom” rather than democracy. Moreover, while factor-analytic and latentvariable approaches allow for the incorporation of multiple sources of data, thereby reducing some sources of error, they remain biased by any systematic error common to he chosen data sources. Another approach to the aggregation problem is to collect primary data at a disaggregated level, letting end-users decide whether and how to aggregate it. Democracy assessments (also termed audits) provide detailed indicators for a single country. 26 This sort of detailed inquisition into the quality of democracy is very much in the spirit of our proposed approach, except that the collected data are limited to a single country (and often to recent years). Several speci? c topics integral to democracy have been uccessfully measured on a global scale (refer to the online appendix). And several broader ventures to measure democracy in a comprehensive and disaggregated fashion have been proposed, but not fully implemented. 27 These efforts at disaggregation push in the right direction. However, they are problematic on one or several accounts: (a) the number of indicators may be very small; (b) the resulting indicators may still be highly abstract and hence dif? cult to operationalize; (c) the underlying components, while conceptually distinct, may be gathered in such a way as to ompromise their empirical independence; (d) the information necessary to code the indicator may not be available across nations or prior to the contemporary era; or (e) the indicators may not be released to the general public (so no use can be made of them). Consider the Polity index, which is ostensibly disaggregated into ? ve components: competitiveness of participation, regulation of participation, competitiveness of executive recruitment, openness of executive recruitment, and constraints on executive. Although each of these components is described at length in the Polity codebook,28 it is dif? ult to say precisely how they would be coded in particular instances, or how the stated aggregation principles lead to an overall score for a given country in a given year. 29 Even in disaggregated form,30 the Polity index is highly abstract, and therefore open to diverse interpretations. The two principal Freedom House measures???Civil Liberties and Political Rights???are similarly dif? cult to get one’s arms around. Indeed, the notion of “political rights” is scarcely less abstract than the core concept of democracy and commonly interpreted as synonymous with it. Since 2006, Freedom House has released coding scores for he components of Civil Liberties and Political Rights. The Political Rights index is shown to be the product of (a) Electoral Process, (b) Pluralism and Participation, and (c) Functioning of Government. The Civil Liberties index comprises (a) Freedom of Expression, (b) Association and Organizational Rights, (c) Rule of Law, and (d) Personal Autonomy and Individual Rights. This represents a step towards greater clarity and disaggregation. However, intercorrelations among the seven components are extremely high???Pearson’s r ? 0. 86 or higher. This by itself is not necessarily problematic; it is possible that all democratic or nondemocratic) things go together. However, the high inter-correlations of the Freedom House indicators coupled with their ambiguous coding procedures suggest that these components may not be entirely independent of one another. It is hard to exclude the possibility that country coders have a general idea of how democratic each country is, and that this idea is re? ected in consistent scores across the multiple indicators. Components that are scored separately on a questionnaire may not be independently coded. The new EIU index does a better job of disaggregating component variables, which are reported for ? e dimensions: electoral process and pluralism, civil liberties, the functioning of government, political participation, and political culture. 31 Correlations are still quite high, ranging from . 74 to . 93 (except for the cultural variable, which is more distinct). Moreover, the speci? city of the questions makes the claim of independence among these ? ve variables plausible. Unfortunately, EIU is unwilling to divulge data for the sixty speci? c questions that compose the ? ve dimensions, so it is dif? cult to judge the accuracy and independence of the index. It may be useful, or it may e not, but we cannot know for sure. Moreover, we shall never be able to judge the content of the ? ve dimensions because answers to the component questions are unavailable. Also, the chosen data sources (in large part, survey data), are not extendable into the past or to countries where surveys were not taken in a particular year. Many of the values have to have been imputed. Validity and Reliability Tests Worries about validity in extant democracy indices are nourished by periodic appraisals focused on speci? c countries. A recent study by scholars of Central America alleges major ? ws in coding for Costa Rica, El Salvador, Honduras, Guatemala, and Nicaragua in three crossnational indices???Polity, Vanhanen, and Mark Gasiorowski??? errors that, the authors suspect, also characterize other indices and other countries. 32 Surprisingly, inter-coder reliability tests are not common practice among democracy indices. Freedom House does not conduct such tests, or at least does not make them public. Polity used to do so, but it required a good deal of hands-on training before coders reached an acceptable level of coding accuracy. This suggests that other coders might not reach the same decisions simply by reading June 2011 | Vol. /No. 2 251 ? ? | | ? Research Articles ? | Conceptualizing and Measuring Democracy Polity’s coding manual. And this, in turn, points to a potential problem of conceptual validity: key concepts may not be well matched to the empirical data. These critiques notwithstanding, defenders of contemporary indices often point out that the extant indices are highly intercorrelated. Indeed, the correlation between Polity2 and Political Rights is a respectable 0. 88 (Pearson’s r). Yet on closer examination consensus across the two dominant indices is largely the product of countries lying at the democratic extreme???Canada, Sweden, the United
States, et al. When countries with perfect democracy scores are excluded from the sample the correlation between these two indices drops to 0. 78. And when countries with the top two scores on the Freedom House Political Rights scale are eliminated, Pearson’s r drops again???to 0. 63. This is not an impressive level of agreement, especially when one considers that scholars and policymakers are usually interested in precisely those countries lying in the middle and bottom of the distribution???countries that are undemocratic or imperfectly democratic. 33 It follows that differences across indices may produce ivergent ? ndings in empirical work where democracy is a key variable. Indeed, Gretchen Casper and Claudiu Tu? s show that few explanatory variables (beyond per capita income) have a consistently signi? cant correlation with democracy when different democracy indices are employed. 34 Thus, we have good reasons to suspect that extant indices suffer problems of validity and reliability and that these problems are consequential. They impact what we think is going on in the world. A New Approach The task of constructing a global index of democracy that is valid and precise, and universally acknowledged as such, s well nigh impossible, for all the reasons we have discussed. Existing measures of democracy are especially inadequate for measuring small changes and differences in the quality of autocracy/democracy; empirically analyzing relationships among various elements of democracy; and evaluating the effectiveness of targeted democracy promotion efforts. Polity, Freedom House, and their counterparts are overstretched insofar as they are applied for these sorts of tasks. At the same time, extant indices perform some important functions well. Sometimes, one needs to identify major regime changes, or gross differences in levels of democracy.
Sometimes, one needs to measure trends in the average level of democracy at a global level. For these purposes, extant indices provide a rough empirical estimate of a complex and multivalent concept. Our proposal, therefore, is not to do away with extant indices but rather to create a new set of indicators that cuts the material at a different angle and thus serves some- what different purposes. Four features, considered together, distinguish our approach to conceptualizing and measuring democracy. First, we propose to extend indicators of democracy back through modern history, wherever possible.
Second, we propose a multidimensional approach to the problem of conceptualizing democracy. Third, we propose to collect information relevant to democracy at a highly disaggregated level. Fourth, we propose a strategy for data collection and presentation that should enhance the precision, validity, transparency, and legitimacy of the resulting indicators. Before introducing these features we must further clarify the scope conditions of the proposal. Our principal concern is with the operation of democracy within large and fairly well-de? ned political units (e. g. , nation-states) which we shall refer to as polities.
The sizeable population of these units presumes that representative institutions will be signi? cant in the political process (insofar as they are democratic), though it certainly does not preclude more direct forms of citizen governance existing side-by-side with representative institutions. We are less concerned with democracy within very small communities (e. g. , neighborhoods, school boards, corporations), in contexts where the political community is vaguely de? ned (e. g. , transnational movements), or on a global level (e. g. , the United Nations). This is not to say that the concept of democracy hould be restricted to formal and well-de? ned polities. It is simply to clarify our approach to de? nition and measurement, and to acknowledge that different strategies might be required for different subject areas. History The industry of democracy and governance indices has been predominantly prospective rather than retrospective in its general orientation. New indicator projects are launched almost monthly, most of them focused on tracking some aspect of democracy or governance going forward in time. While policymakers are rightly concerned with the course of future events, researchers (and, one might add, ell-informed policymakers) realize that this requires a sound understanding of the past. One cannot understand the future of democracy in the world and how to shape it unless one understands the forces that produced the regime types that populate the world today. The longer the time series, the more likely it is that one will be able to pin down democratization trends, causes, and effects. 35 This may seem like a hopeless task. However, the advantage of our topic (vis-a-vis other historical measurement tasks such as national income accounts) is that much of the evidence needed to code features of democracy is well reserved in books, articles, newspapers archives, and memories. Democracy is a high-pro? le phenomenon. While a secretive regime may hide the true value of goods and 252 Perspectives on Politics ? | | ? These are seen as de? ning features of democracy, not simply as aids to political competition. The liberal model takes a “negative” view of political power insofar as it judges the quality of democracy by the limits placed on government. Principles and procedures must be established so as to ensure that rule by the majority does not result in the oppression of minorities or the loss of individual liberties. 8 The majoritarian conception of democracy (or responsible party government) re? ects the principle that the will of the majority should be sovereign. The many should prevail over the few. To facilitate this, political institutions must centralize and concentrate, rather than disperse, power (within the context of competitive elections), which means that majoritarian democracy is in tension with liberal democracy in many respects, e. g. , strong and centralized parties, a unitary rather than federal constitution, plurality rather than proportional electoral laws (or PR with high statutory thresholds), and so forth.
Even so, many aspects of democracy are compatible with both conceptions such as civil liberties, due process, human rights, and transparency. 39 The participatory conception of democracy is usually viewed as a lineal descendant of the “direct” (i. e. , nonrepresentative) model of democracy, as derived from the experience of Athens???though elements of this model may also be discerned in “republican” thought and in the experience of many small communities throughout the world and throughout human history. 40 The motivation for participatory democracy is uneasiness about delegating complete authority to representatives.
Direct rule by citizens is preferred, wherever practicable. And within the context of representative government, the participatory component is regarded as the most democratic element of the polity. This model of democracy thus highlights the importance of voting, but also of citizen assemblies, party primaries, referenda, social movements, public hearings, town hall meetings, and other forums of citizen engagement. 41 The deliberative conception of democracy focuses on the process by which decisions are reached in a polity. A deliberative process is one in which public reasoning ocused on the common good motivates political decisions???as contrasted with emotional appeals, solidary attachments, parochial interests, or coercion. In this conception, democracy requires more than a mindless aggregation of existing preferences; there should be respectful dialogue at all levels???from preference formation to ? nal decision???among informed and competent participants who are open to persuasion. 42 “The key objective,” writes David Held, “is the transformation of private preferences via a process of deliberation into positions that can withstand public scrutiny and test. 43 Some political institutions have a speci? cally deliberative function, such as consultative bodies (hearings, panels, assemblies, courts); polities with these sorts of institutions might be judged more deliberative than those without them. However, the services, it cannot disguise the existence of an election. And various features of an election that might prejudice the outcome towards the incumbent, while sometimes dif? cult to discern, are also dif? cult to obscure completely. This, again, is because virtually everyone living in that country, studying that country, or covering that country or some foreign news organization or aid organization has an interest in tracking this result. Standardized polls are helpful in this fact-gathering exercise, but not necessary (as they are for many other measurement features associated with the topic of governance). Thus, we regard the goal of historical data-gathering as essential and also realistic, even if it cannot be implemented for every indicator that is gathered for the present era. Some historical indicators are better than none at all. Likewise, if it can be shown that the kind of indicators that are available for the past are highly correlated with better” indicators that be produced only for recent years, cost-saving proxy indicators can be constructed. Conceptions There is no consensus on what democracy at large means, beyond the prosaic notion of rule by the people (refer to our opening discussion). Political theorists have been emphasizing this point for some time, and empiricists would do well to take this lesson to heart. 36 Even so, there appears to be some consensus over the various plausible conceptions of this protean term. In surveying an immense literature, six key models seem paramount.
These may be summarized as electoral, liberal, majoritarian, participatory, deliberative, and egalitarian. Each represents a different way of understanding what “rule by the people” means. Thus, while no single conception can reasonably purport to embody all the meanings of democracy, these six conceptions taken together offer a fairly comprehensive accounting of the concept of democracy as it is employed today. The electoral conception of democracy???also known as contestation, competition, elite minimal, realist, or Schumpeterian???is the idea that democracy is achieved through competition among leadership groups, which vie for the lectorate’s approval during periodic elections before a broad electorate. Parties and elections are the crucial instruments in this largely procedural account of the democratic process. Of course, many additional factors might be regarded as important for ensuring and enhancing electoral contestation, e. g. , civil liberties, an active media, a written constitution, an independent judiciary (to enforce the rules of the game), and so forth. However, these factors are viewed as secondary to electoral institutions. 37 The liberal (sometimes called consensus or pluralist) conception of democracy stresses the intrinsic importance f transparency, civil liberty, rule of law, horizontal accountability (effective checks on rulers), and minority rights. June 2011 | Vol. 9/No. 2 253 ? ? | | ? Research Articles | Conceptualizing and Measuring Democracy Table 1 Conceptions of democracy Principles Question Institutions Contestation, competition Are government offices filled by free and fair multiparty elections? Elections, political parties, competitiveness and turnover Limited government, multiple veto points, horizontal accountability, individual rights, civil liberties, transparency Is political power decentralized & constrained?
Multiple, independent, and decentralized, with special focus on the role of the media, interest groups, the judiciary, and a written constitution with explicit guarantees Majority rule, centralization, vertical accountability Does the majority (or plurality) rule? Consolidated and centralized, with special focus on the role of political parties IV. Participatory Government by the people Do ordinary citizens participate in politics? Election law, civil society, local government, direct democracy V. Deliberative Government by reason Are political decisions the product of public deliberation? Media, hearings, panels, other eliberative bodies VI. Egalitarian Political equality Are all citizens equally empowered? Designed to ensure equal participation, representation, protection, and politically relevant resources I. Electoral (aka elite, minimal, realist, Schumpeterian) II. Liberal (aka consensus, pluralist) III. Majoritarian (aka responsible party government) ? more important issue is the degree of deliberativeness that can be discerned across all powerful institutions in a polity (not just those explicitly designed to serve a deliberative function) and among the citizenry. 44 The egalitarian conception of democracy addresses the oal of political equality. An egalitarian polity is one that achieves equal participation, equal representation, equal protection (civil liberties extended to all and due process for all), and equal resources (such as income, education, and health). Resources are presumed to be a key feature of political empowerment; where resources are not equally shared it is dif? cult to imagine a polity in which citizens enjoy equal political power. Political equality thus presumes social equality???though perhaps not perfect social equality (small differences in resources may not be consequential). 45
A summary of these six conceptions of democracy is contained in Table 1, along with key institutions implied by each conception. Naturally, this schema does not capture all the important debates or theoretical distinctions at play within the concept of democracy. It does not capture the distinction between direct and representative democracy, or among different principles of representation. 46 It does not address normative theories of democracy. 47 It does not address whether, or to what extent, a theory of democracy should be realistic, as opposed to ideal. 48 It does not capture the intellectual history or etymology of he concept, though traces of that history will be glimpsed in each conception. 49 All typologies are limited in some respects, and this one is limited in many respects. Still, the simple six-part taxonomy summarized in Table 1 captures a good deal of the action surrounding current debates on democracy. In some respects, the various conceptions of democracy are complementary. Thus, a polity scoring highly on several conceptions might be deemed more democratic than a polity scoring highly on only one. Yet there are also potential con? icts across conceptions. The most obvious s between the liberal and majoritarian conceptions, which adopt contrary perspectives on most institutional components: fragmented power satis? es the liberal ideal but inhibits the possibility of majority rule. One can easily perceive con? icts across other conceptions as well. For example, fragmented institutions (the liberal ideal) may diminish the capacity of government to redress political inequalities and may inhibit the political power of less advantaged citizens, contravening the egalitarian model of democracy. If one were to reach for a coherent de? nition of democracy (at large), these potential con? cts would have to be reckoned with. One might argue, for example, that electoral democracy encompasses all there is to the concept of democracy???a minimal de? nition. Another might argue 254 Perspectives on Politics ? | | ? that electoral democracy serves as a necessary attribute of democracy, in which case other conceptions are meaningless without it. Arguably, neither participation nor deliberation is democratic apart from a framework of multiparty elections. 50 Similarly, it is often argued that participation does not serve democracy when education and resources are unequally distributed.
Of course, many different combinations of conceptions can be envisioned. One might even take the position that all six conceptions of democracy are required but that each must be approached with moderation: democracy as an admixture of diverse conceptions. There are thick and thin approaches to democracy, as to other concepts. 51 We do not propose any particular de? nition of democracy (at large). We leave this to others. Our intention here is to capture various possible conceptions of democracy without making judgments about how they might be combined or how they might contribute to a summary index. 2 Our claim is that these six conceptions describe our subject in a fairly encompassing fashion and that each conception is logically distinct and???at least for some theorists???independently valuable. That is, some writers believe that enhanced avenues for participation are good for democracy even in the absence of electoral or liberal dimensions of democracy. Some writers believe that more equal access to resources has a bene? cial impact on democracy (through political equality) even in the absence of electoral or liberal dimensions of democracy.
Moreover, we suspect that there is a good deal of divergence across these six conceptions among the world’s polities. Some will be particularly strong on the electoral conception; others will be strong on the liberal conception, and so forth. Thus the typology of ideal-types provided in table 1 may prove a useful empirical device, allowing one to chart variation in political institutions through time and space. treaty agreements and regular international-system constraints). 2. Authority: the degree to which central governmental authority is preeminent throughout the territory laimed as part of the polity. (If the government has no authority outside the capital then the rest of the territory is not governed by whatever rules that polity has established. ) 3. Elective government: the extent to which executive functions are handled by of? cials chosen by election. 4. Male suffrage: the extent to which adult males have the right to vote in elections. 5. Female suffrage: the extent to which adult females have the right to vote in elections. 6. Turnout: the level of participation in elections and other of? cially sponsored consultations. 7.
Regular elections: the extent to which elections are held regularly and on schedule, according to the constitution. 8. Free elections: the extent to which parties and candidates can gain access to the ballot, compete for votes in an environment free of government interference, and have their votes counted and allocated fairly. Also, the extent to which citizens are able to register to vote. 9. Access to media and campaign ? nance: the extent to which all parties/candidates are granted equal access to the media and to campaign ? nance, proportional to their support in the electorate. 10.
Executive rule of law: the extent to which the executive (and persons and agencies under his/her control) follows the law, as de? ned by the constitution, treaties, statutes, and as interpreted by the judiciary. 11. Executive constraints: the extent of effective constraints on the executive (whether by elective or non-elective bodies). 12. Legislative power: the extent to which the legislature controls the executive???with parliamentary systems understood as de? ning one end of the continuum and systems dominated by a separately selected executive unaccountable to the legislature, or systems in which the legislature is entirely absent, e? ning the other end. 13. Judicial independence: the extent to which the highest judicial bodies are independent of the executive and other outside in? uences. 14. Judicial review: the extent to which the highest judicial bodies are able to review acts of legislation and other governmental actions in the light of constitutional provisions, and the extent to which such decisions are respected by other bodies. 15. Party strength: the extent to which parties are institutionalized (rather than simply the vehicle for speci? c candidates) and centralized (in organizational structure, electoral behavior, and legislative ehavior). Components and Indicators Previously, we made the case for greater disaggregation in measuring the concept of democracy. The six conceptions discussed in the previous section offer one step in that direction. A second step is the identi? cation of mid-level components. 53 This second level of disaggregation speci? es more fully the referents of the six conceptions of democracy and moves us closer to the very speci? c concepts that can be measured most validly and reliably. Just as people prefer different conceptions of democracy, they have different preferences about which components to include and exclude.
Our list, which probably errs on the side of conceptual inclusiveness, is as follows: 1. Sovereignty: the degree to which a polity is able to govern itself in its domestic and foreign policies, free from interference from other polities (aside from June 2011 | Vol. 9/No. 2 255 ? ? | | ? Research Articles ? | Conceptualizing and Measuring Democracy 16. Party ideology: the extent to which parties have wellde? ned, consistent, and coherent ideologies. 17. Party system size: the number of parties gaining seats in the national legislature and party system fractionalization (weighting parties by their relative size). 8. Electoral system proportionality: the absence of barriers to representation for small parties, both as a product of district magnitude, thresholds, and other statutory restrictions. 19. Competitiveness: the closeness of the vote between the two highest vote-getters in a national election. 20. Turnover: the change in (a) party control and (b) individual control over (c) the executive and??? if different???(d) the most powerful of? ce in the land. 21. Media development: the extent to which major media outlets are independent, free to air diverse political views, and able to reach the citizenry. 22.
Civil society independence: the extent to which civil society (excluding parties and media) is independent of the state and able to voice opinions critical of political leaders. 23. Civil society political engagement: the extent to which civil society (excluding parties and media) is engaged in politics???both electoral and consultative. 24. Subnational government elections: the extent to which there are subnational governments elected through free and fair elections. 25. Unevenness in democratic development: whether some subnational governments are signi? cantly more or less respectful of civil liberties and free and fair lections than others in the polity. 26. Direct democracy: the extent to which opportunities exist for citizens to engage directly in policymaking (e. g. , through referenda). 27. Civil liberty: the extent to which citizens enjoy freedom of speech and freedom from politicallymotivated persecution by government. 28. Property rights: the extent to which property rights are protected. 29. Religious freedom: the extent to which freedom of religion is guaranteed. 30. Equal resources: the extent to which resources such as income, education, and health???which may impact the possibility of participating in politics??? are widely and equally available. 1. Gender equality: the extent to which women achieve equal representation in the legislature and other high positions within government. 32. Ethnic equality: the extent to which underprivileged ethnic groups (de? ned by race, religion, caste, or other ascriptive characteristics) are granted formal rights to suffrage and to positions of power within the government, as well as the extent to which such groups actually vote and gain represen- tation in the legislature and other high positions within government. 33. Inclusive citizenship: the degree to which all citizens and permanent residents enjoy the protections of the law.
The ? nal step in disaggregating the concept of democracy is the identi? cation of low-level indicators. These are measurable aspects of dimensions and components that lie very close to the empirical bone. In identifying indicators we look for features that perform some function that brings the political process into closer alignment with the core meaning of democracy (rule by the people); pertain to one or more of the six conceptions of democracy (as summarized in Table 1); and are measurable???directly or indirectly (via coder judgments) across polities and through time.
Our online appendix provides a comprehensive list of these indicators. To give the reader a feel for what we are proposing an exemplary set of indicators pertaining to formal and descriptive representation is included as Table 2. At this point, several clari? cations are in order. First, although components and indicators must be closely associated with at least one of the six conceptions of democracy (as stipulated earlier), they may be associated with multiple conceptions of democracy. These components have a many-to-many rather than one-to-one correspondence with the six conceptions of democracy.
This is because a given institution may perform a variety of functions. For example, a free and independent press may enhance the competitiveness of elections (electoral democracy), the rule of the majority (majoritarian democracy), the meaningful participation of citizens (participatory democracy), and the empowerment of out-groups (egalitarian democracy). That said, a free and independent press is directly and de? nitionally entailed by the liberal model of democracy, which is why it is listed in the second row of Table 1. In any case, the six conceptions of democracy overlap at the level of components and indicators. We do ot list all the possible correspondences between components or indicators and conceptions here because they are complex and because different scholars would make different selections for different purposes. If one were interested in a thin concept of electoral democracy, for example, then elective government, free elections, and regular elections would probably suf? ce. A thicker concept of electoral democracy might add male and female suffrage, party strength, party ideology, party system size, electoral system proportionality, turnover, competitiveness, subnational government elections, and even civil liberty and media development.
Second, some components and indicators are undoubtedly more important in guaranteeing a polity’s overall level of democracy than others. This of course depends upon one’s model of democracy. Inclusion on our list does not presuppose a judgment of relative importance. All it means 256 Perspectives on Politics ? | | ? is that a particular component or indicator is relevant to the operationalization of at least one conception of democracy. Third, components and indicators of democracy sometimes con? ict with one another. For example, protection of individual liberties can impose limits on the will of he majority; and the existence of strong civil society organizations may have the effect of pressuring government to restrict the civil liberties enjoyed by marginal groups. 54 Likewise, the same institution may be differently viewed within different models of democracy. For example, the common practice of mandatory voting is clearly offensive to the liberal model (where individual rights are sacrosanct and include the right not to vote), but is vindicated by the participatory model (since it has a demonstrated effect in boosting turnout where sanctions are more than nominal).
Such contradictions are implicit in democracy’s multidimensional character, as discussed in the previous section. No wide-ranging empirical investigation can avoid con? icts among democracy’s diverse attributes. However, with separate indicators for these different components it will be possible to examine potential tradeoffs empirically???an important substantive issue for policymakers and academics to grapple with. Fourth, we do not assume that components or indicators of democracy are always associated with desirable policy outcomes. Democracy is undoubtedly an important element of good governance, but it is certainly not the um-total of good governance. Thus, to say that an indicator or component provides a measure of democracy (according to one or more conceptions of democracy) is not to say that it advances the cause of justice or the good. 55 The causal inter-relationship of democracy to other desired outcomes is an empirical matter. Finally, our proposed set of components and indicators, while fairly comprehensive, is by no means entirely comprehensive. The protean nature of “democracy” resists closure; there are always potentially new components/ indicators that, from one perspective or another, may be ssociated with this essentially contested term. Moreover, some conceptions of the concept are dif? cult to capture empirically, and virtually impossible to track over time and across countries on a global scale. This limits the scope of any empirical endeavor. Table 2 An exemplary category of indicators: Formal and descriptive representation Studies, datasets General questions Indicators* IDEA Global Survey of Voter Turnout Paxton et al. 2003 IPU Chronicle of Parliamentary Elections IDEA Quota Database Paxton, Hughes, and Green 2008 US Department of State Human Rights Reports
Cederman et al. 2010 Elkins and Ginsberg 2011 Krook 2009 US Office of Personnel Management Investigations Service 2001 Who is allowed to vote and who votes? Who is eligible for public office and who actually attains it? Does the legislature reflect population characteristics? Formal Representation???Suffrage What percent of all adult citizens are allowed to vote per election law? Are there a significant percent of citizens who cannot vote in practice? Is there restricted male suffrage? Is there universal male suffrage? Is there restricted female suffrage? Is there universal female suffrage?
Formal Representation???Turnout What percent of all adult citizens voted? Descriptive Representation??? Eligibility for Public Office What restrictions does the constitution place on eligibility to serve in the legislature? Do candidate eligibility rules in practice exclude women? Do candidate eligibility rules in practice exclude members of certain ethnic or religious groups? Descriptive Representation??? Attainment of Public Office Percent female national legislators. Presence of a gender quota. Presence of a gender quota with placement mandates. Presence of a gender quota with anctions for noncompliance. Percent minority national legislators. Presence of a minority quota. Presence of a female president or prime minister Presence of a minority president or prime minister Data Collection and Dissemination The viability of a dataset hinges critically on its method of data collection and distribution. It is therefore important to detail these issues of strategy, which are as vital to the product as basic decisions about conceptualization and measurement. Two types of coders are envisioned in any data-collection project of this magnitude. The ? rst is the country expert, a All items below pertain to major national elections??? parliamentary and (if applicable) presidential elections. (Presidential elections may not be relevant if the presidency is a weak office with little policymaking power. ) June 2011 | Vol. 9/No. 2 257 ? ? | | ? Research Articles ? | Conceptualizing and Measuring Democracy person who has resided in and studied a particular country for many years and has an intimate feel and detailed knowledge of that polity. The second is the institutional expert, a person who has expertise in some category of indicators listed in the online appendix. These two nowledge-sets must be merged, and this involves a good deal of cross-communication between the two sorts of coders such that each learns from, and is informed by, the other. The coding of a particular indicator must be consistent with general guidelines laid down for that indicator (a judgment the institutional expert is best placed to resolve) as well as with the context of a particular time and place (a judgment that the country expert is best placed to decide). In establishing the de? nition of each indicator, a wide range of institutional experts on a particular subject ought to be consulted.
Once de? ned and operationalized, a single institutional expert can serve as the coordinator for this indicator across all polities and time-periods, ensuring spatial and temporal consistency. In establishing the precision of each coding of an indicator, more than one country expert must be enlisted. This also provides the basis for inter-coder reliability tests, which are essential for the development of any scale where some degree of measurement error (perhaps deriving from personal biases or diverse understandings of a topic) is anticipated. In addition, the coding system should include “Remarks” ? eld where the coder lists primary sources that proved useful in arriving at a particular code, and any commentary on the coding that might help with its interpretation. Note that some indicators are fairly objective, such as turnout rates in an election (although sometimes coders will need to make judgments among rival turnout statistics). Others require more subjective interpretations, e. g. , minorities holding seats in the national legislature, which requires judgments about what constitutes a minority and who quali? es for membership in that category.
Other indicators, listed in the online appendix, require even more coder judgment. Unfortunately, there is no way to dispense with subjective judgments entirely if one is to address the multifaceted nature of the key concept, democracy. This is especially the case when it concerns expectations rather than facts, such as whether the next election is likely to be held on schedule. After coding is complete, it is important to establish an open and transparent system of display and dissemination. This might include a blog- or Wiki-style format in which interested individuals could comment on the scores rovisionally assigned to the country or countries or indicator(s) that they know well. This commentary might take the form of additional information???perhaps unknown to the country expert???that speaks to the viability of the coding. Or it might take the form of extended discussions about how a particular question applies to the circum- stances of that country. Naturally, a few cranky participants may be anticipated. However, the Wikipedia experience suggests that there are many civic-minded individuals, some of them quite sophisticated, who may be enlisted and may have a lot to add to the discussion. At he very least, it may provide further information upon which to base estimates of uncertainty. (Final decisions about coding must be left to whatever larger committee of experts is associated with a project. ) It is important that the process of revision be continual. Even after the completed dataset is posted, users should be encouraged to contribute suggestions for revision and these suggestions should be systematically reviewed. Finally, wherever indicators are employed to provide scoring for higher-level concepts???i. e. , components or conceptions of democracy???the aggregation rules should be fully transparent, and hence eplicable. Likewise, endusers could easily construct their own aggregate indices based upon the low-level indicators provided by the project, using their own chosen aggregation principles. Payoffs Having described what a dataset of new democracy indicators might look like, the payoffs anticipated from such an index may now be brie? y enumerated. We begin with the aggregation problem. Many problems of conceptualization and measurement stem from the decision to represent democracy as a single point score (based on a binary, ordinal, or interval index). Of course, summary measures have their uses. Sometimes ne wants to know whether a country is democratic or non-democratic or how democratic it is overall. However, the goal of summarizing a country’s regime type is elusive. As we have seen, extant democracy indices suffer from serious problems of conceptualization and measurement. While many new indices have been proposed over the past several decades???all purporting to provide a single point score that accurately re? ects countries’ regime status???none has been successful in arriving at an authoritative and precise measurement of this challenging concept. In our view, the traditional approach falls short because ts self-assigned task is impossible. The highly abstract and contested nature of democracy impedes effective operationalization. This is not a problem that can be solved???at least not in a conclusive fashion. Naturally, one can always impose a particular de? nition upon the concept, insist