Using Qualitative Information to Improve CausalInference

Using the Rosenbaum (2002, 2009) approach to observational studies, we show how qualitative information can be incorporated into quantitative analyses to improve causal inference in three ways. First, by including qualitative information on outcomes within matched sets, we can ameliorate the consequences of the difficulty of measuring those outcomes, sometimes reducing p-values. Second, additional information across matched sets enables the construction of qualitative confidence intervals on effect size. Third, qualitative information on unmeasured confounders within matched sets reduces the conservativeness of Rosenbaum-style sensitivity analysis. This approach accommodates small to medium sample sizes in a nonparametric framework, and therefore it may be particularly useful for analyses of the effects of policies or institutions in a small number of units. We illustrate these methods by examining the effect of using plurality rules in transitional presidential elections on opposition harassment in 1990s sub-Saharan Africa.

Observational studies in political science are often beset by problems that can lead to fragile and biased estimates of causal effects. Most fundamentally, important confounding variables that affect both the treatment variable and the outcome variable may be unmeasured, and even measured confounding and outcome variables may only be poorly measured. Many of these observational studies are also “medium-n,” having fewer observations than is needed for large-sample techniques to provide accurate approximations.

 Moreover, this sample size problem afflicts more large-n studies than is generally recognized. Large-n data sets often contain units that are incomparable on measured confounding variables, and this lack of overlap between treatment and control units results in analyses that rely upon extrapolation for causal inference. We may guard against this by restricting a study to a smaller set of similar observations (Brady and Collier 2004) or by removing these incomparable observations by preprocessing the data through matching (Ho et al. 2007). But what often remains after limiting the scope of the analysis in this way is a medium-n study.

 We present a set of methods to mitigate these problems and improve causal inferences in medium-n studies through a formal synthesis of qualitative information1 and quantitative analysis. This synthesis is conducted within the Rosenbaum (2002, 2009) randomization inference-based approach to observational studies, which enables nonparametric inference with small sample sizes.
 We first demonstrate the basic technique using pairs of units that have been matched on measured confounders, as it simplifies the presentation and allows for an analogy to a repeated use of the comparative method (Lijphart 1975). We then show that these techniques can be extended to some of the more complicated matching strategies in Rosenbaum (2002, 2009).

This approach can integrate qualitative information with a quantitative analysis to improve causal inference in three ways. First, we can ameliorate the effects of difficultto-measure outcomes by converting qualitative information into ordinal measurement of outcomes within the matched sets, which can reduce p-values. Second, additional information on the ranks of the sizes of the absolute within-set differences, allows us to present qualitative confidence intervals – that is, qualitative descriptions of effect sizes that have the same properties as conventional confidence intervals.2 Third, qualitative information on unmeasured confounders within matched sets facilitates a sensitivity analysis that is less conservative than the typical Rosenbaum-style sensitivity analysis. This approach is feasible because of the medium-n sample size and because results from nonparametric statistics help identify what information will provide the most leverage.

 While this approach has many benefits, identifying the information that maximizes statistical power also identifies information that would maximize bias if mismeasured. Because our procedure partially couples the measurement and analysis processes, it introduces opportunities to corrupt the analysis. We propose in the conclusion to minimize this threat by explicitly separating and outsourcing the measurement stage.

 We focus on treatment effects for a binary treatment. It is straightforward, however, to adapt our methods for other causal questions, such as treatment effects of continuous treatments or multiple treatments and interactions or multiple outcomes, in this framework. We refer readers to Rosenbaum (2009) for a discussion of these topics, or Caughey, Dafoe, and Seawright (2013) for a recent approach to multiple outcomes.
 

We demonstrate these points through a medium-n study of whether using plurality rules in transitional presidential elections in sub-Saharan Africa in the 1990s increased the severity of opposition harassment in the period leading up to the election. The appendix presents the qualitative information from the comparative case studies that is incorporated into the analysis. We find evidence strongly suggestive of a positive effect of plurality rules on opposition harassment, even after accounting for threats to causal inference. With the full matching implemented in the penultimate section, our approach obtains a one-sided p-value of 4.2% with only 9 units, and a sensitivity analysis accounting for unmeasured confounding demonstrates that this p-value is unlikely to rise above 10%.

 This method differs from existing approaches to “mixed methods” for bolstering quantitative analyses with qualitative case studies. In many of these approaches, case studies are used to illustrate an argument and provide a “plausibility check” (Dunning 2012; Fearon and Laitin 2008; George and Bennett 2005). Lieberman (2005) suggests a nested approach in which an unsatisfactory large-n analysis is followed by a model-building small-n analysis. Qualitative Comparative Analysis (Ragin 2000) provides a method that accommodates many comparisons and causal factors with a small sample size.
 Our approach differs from these approaches by formally incorporating qualitative information into a standard statistical framework. Our approach is also more flexible than other formalized procedures for integrating qualitative information, such as Herron and Quinn (2014), which assume binary outcomes or parametric models and often require the elicitation of Bayesian priors.

 The article proceeds as follows. The next section introduces our running example of transitional presidential elections in 1990s sub-Saharan Africa, the formal notation, and randomization inference for pair-matched binary outcome data. Then, in each of the following sections, we introduce qualitative information to the analysis to elaborate on our formal mixed-method procedure for improving causal inference in medium-n studies. We first incorporate within-pair and between-pair information on the outcome through the signed-rank statistic to generate p-values and qualitative confidence intervals.
 We then show how full matching and the Quade statistic can further reduce p-values and how qualitative information on unmeasured confounders reduces the conservativeness of Rosenbaum-style sensitivity analysis. The supporting information (SI) presents R code for our analyses. The conclusion discusses implications for practice and guidelines for researchers using these methods.
 

An Illustrative Example and Notation 

To demonstrate these methods, we explore the effect of plurality electoral rules on opposition harassment in multiparty presidential elections in sub-Saharan Africa in the 1990s that marked transitions away from authoritarian rule. To demonstrate these methods, we explore the effect of plurality electoral rules on opposition harassment in multiparty presidential elections in sub-Saharan Africa in the 1990s that marked transitions away from authoritarian rule. These transitional elections were watershed events at which citizens of these countries, often for the first time in their lives, had the opportunity to replace an authoritarian incumbent at the ballot box. But they were also precarious moments in which incumbents might employ violence against the opposition in order to stay in power.
 Twenty-four sub-Saharan countries held these transitional elections in the 1990s, and four of these 24 used plurality rules under which a candidate must obtain more votes than any other candidate in order to be declared the winner.3 The other countries used some form of runoff rules, which stipulate that should no candidate meet a given vote share threshold (usually 50%) in the first round, weaker candidates are eliminated and the top two finishers compete in a second-round election.4 This rule and other elements of the election framework were determined by the authoritarian incumbent, with varying degrees of input from opposition representatives and civil society groups through national conferences and constitutional review committees. Foreign constitutional scholars, social scientists, and other experts on democratic institutions were often sponsored by foreign donors’ democracy promotion programs to offer advice (Nwajiaku 1994; van Cranenburgh 2011). As we elaborate below, we believe ex ante that plurality rules might increase opposition harassment. Our question is therefore whether using plurality rules raised the likelihood and intensity of opposition harassment in these countries’ transitional elections.5 We begin with an incumbent authoritarian regime that has agreed to hold multiparty presidential elections in the face of pressures for political liberalization. The regime wants to hold on to power by having its favored candidate win the election, and to this end, it allocates its finite resources to a combination of opposition cooptation and harassment. We assume that harassment cannot reliably convert opposition supporters into voters for the regime’s favored candidate, and that harassment can suppress voting by some but not all opposition supporters.
 While all are aware of widespread dissatisfaction with the regime, not enough information is available about support for specific challengers to the authoritarian incumbent to ensure Duvergerian coordination in the transitional elections. This means that under plurality rules, a potential challenger who does not have the resources to win a majority but might be able to win a plurality may compete in the election and divide opposition support, reducing the vote margin needed to win the election. For the incumbent authoritarian regime, this makes opposition harassment more likely to be decisivefor the outcome of the election and an attractive strategy, particularly if the harassment can be targeted at the supporters of the opposition candidate who is likely to have the most support.
 With a runoff provision, the incumbent authoritarian regime could try to place in the top two rather than win a majority of votes cast in the first round. But this strategy is dangerous because the opposition would gain the opportunity to coordinate behind a single candidate for the second round; the regime’s favored candidate may also place third and be ineligible for the runoff election.
 Therefore, the incumbent regime’s strategy will be to try to win an outright majority in the first round by drawing potential challengers and their supporters into its coalition, which in turn encourages weak challengers to contest the election in order to be co-opted by the regime, even if they do not have the resources to muster a majority.7 Opposition harassment could help the incumbent by reducing turnout and therefore the number of votes needed to comprise a majority, but resources would need to be diverted from co-optation. Moreover, unlike plurality rule, under which harassment can change the threshold for an incumbent win, harassment does not change the requirement of a majority under runoff rules. This means that opposition harassment is relatively less effective than cooptation under runoff rules and is less likely to be decisive.
 Consequently, we expect plurality rules to lead to greater opposition harassment than would runoff rules.
 An empirical study of this proposed plurality effect has several difficulties shared by many observational studies. In addition to the small sample size, we are likely to have significant unmeasured confounding because we do not know what information was available to the key actors who set the electoral rule or know how they weighed different considerations. For example, strong opposition to the incumbent might have increased the amount of opposition harassment under either set of electoral rules and might have also increased the likelihood of using plurality rules.Moreover, and mostfundamentally, the outcome variable of opposition harassment is difficult to measure.
 The remainder of the article tackles these concerns.

Conclusion

 For many questions in political science, researchers face the challenges of difficult-to-measure outcomes, imbalance on measured and unmeasured confounders, and small sample size after removing incomparable unitsfrom the study. Analyses of the effects of country-level institutions on large-scale social or political outcomes are particularly vulnerable to these problems, since these institutions are generally chosen endogenously through complex political processes and the population of units is limited. But, as this article has demonstrated, the small sample sizes of these observational studies makes feasible the use of qualitative information to improve causal inferences.
 In our analysis of the effect of presidential electoral rules on opposition harassment in African countries undergoing regime transition in the 1990s, comparative case studies allowed us to rank within- and between-set differences in the severity of opposition harassment and rank the direction of within-set differences in unmeasured strength of opposition. The techniques described in this article provide a principled way in which to use the qualitative information we learned from these brief case studies to improve our analysis. We showed that by incorporating case knowledge within the Rosenbaum (2002, 2009) approach, we could improve power and potentially reduce p-values, provide qualitative confidence intervals, and reduce sensitivity to unmeasured confounders.
 By showing how and how much additional information can improve causal inference, we offer statistically grounded guidelines for how mixed-methods researchers should direct their efforts in data collection for smalland medium-n studies. The first step is to understand the treatment assignment process and to focus on measurement of the more important matching variables rather than improving measurement of the outcome variable.
 After matching with these variables, researchers should then focus on signing and ranking differences in outcomes within concordant pairs or sets in order to, but do no more than, establish within-set and between-set rankings. Existing data sets can be very helpful starting points for both of these steps, and researchers need only to focus on the outcome variable for the cases in the matched sets and not the entire sample. Our methods also point to which set of cases is likely to define the bounds of a qualitative confidence interval for some specified level, so that the researcher can focus on characterizing more precisely the difference in outcomes among those likely cases. Finally, deep knowledge beyond the information encoded in quantitative data sets should be used to assess the relative probabilities of treatment assignment within these sets in order to strengthen and clarify the credibility of a study.
 One concern is that the link between the analysis and the partial coding may allow researchers to tailor their analyses to obtain particular results. This hazard can be reduced by outsourcing the partial coding decisions. Experts who do not know the treatment variable of interest could be tasked with measurement of the outcome variable, and other experts who do not know the outcome variable of interest could be assigned to code the unmeasured confounders. This procedure would effectively decouple the analysis from the coding required for the analysis and enable the formal registration of the study as discussed in Humphreys, de la Sierra, and van der Windt (2013). Transparency about the matching procedure and careful documentation of the sources used in the comparative case studies will also enable replication and scrutiny of the researcher’s coding by others (Lieberman 2010).
 More generally, we have shown that even with a small sample size, randomization inference allows qualitative information to be incorporated in a nonparametric statistical framework. Unlike other mixed-methods approaches, our method formally integrates qualitative information with quantitative analysis. Moreover, the formal synthesis does not require parametric assumptions or the elicitation of Bayesian priors. This allowed us to provide evidence suggesting that plurality rules may increase the severity of opposition harassment and to characterize the lower bound for the size of that effect. That we obtained this result with onlyfour countries with plurality rules points to potential gains from expanding the study.