Cost-Effect-Analysis: Quant. uncertainty, transparent, customize

improve GiveWell’s (and others') tools/analysis” and widen the scope of causes/interventions; permit Montecarlo, transparency, and allow user input of moral and epistemic parameters

This workspace

This is an organizing space started by David Reinstein of Rethink Priorities (on his own time). See Who is involved? for other contributors and discussants. The project as a whole is a joint effort with Sam Nolan and others (see Who is involved?).

Introduction

We give the motivation for working on tools to improve CEAs. We try to the current state of research and impact in this area... ... if you would be interested in contributing, you shouldn't have too much difficulty finding suitable work to do.

Why tooling for CEAs?

Quantifying impact is a cornerstone of Effective Altruism. Currently, GiveWell is generally considered a gold standard in the EA community. However, GiveWell's analyses have some limitations which the EA community may want to reconcile.

1. Perhaps the most critical issue with GiveWell's analysis is that the analysis does not formally consider uncertainty. Representing uncertainty is important among EAs, particularly when it comes to determining the value of research, epistemics and forecasting (For an EA org in this space, see QURI). More practically, representing uncertainty may will help donors and policymakers consider the 'risk versus return' of each intervention, and consider how confident they should be in the evaluations. (See also Why make uncertainty explicit?)
2. A second issue is that the model may have bugs in it, may input wrongly coded data, or may have internal inconsistencies. Making the computations more explicit and transparent could facilitate checking and the use of tools to improve reliability and reduce errors. See Pedant for an EA project in this space focusing on 'type checking'.

3. The third issue is that currently the Cost Effectiveness Analysis (CEAs) provided by organizations such as GiveWell are very daunting and confusing to understand. The underlying model can be hard to tease out from a collection of cell references and formulas. Improving the way that users understand and interact with these models could improve accessibility to EAs and to the research community.

That being said, GiveWell is already a positive outlier in terms of the quality of CEAs. Our project also aims to create tools and examples that help create CEAs for non-GiveWell organizations, particularly longtermist ones, which have often escaped more rigorous analysis of Cost Effectiveness up to now.

Specifically, this project investigates the efficacy of the following innovations:

• Representing CEAs using code (See Code representations of GW models and Pedant)

• Using alternatives to spreadsheets (MS Excel etc) for representing CEAs, particularly those that can handle uncertainty (See @Guesstimate and @Causal.app)

• Presenting these in 'visual dashboard/BI' ways that 'make the uncertainty clear' and enable intuitive comparisons

We further discuss this case, and responses, under:

Key writings and resources

Limitations of GiveWell

... and in other sections below.

Semi-aside: I think this would be really good both for directly considering the most valuable global health interventions, and for building the sophistication and epistemics of EA and policymaking communities. It is a really good example to illustrate Fermi Monte Carlo estimation, as well as making concrete, and opening up some very interesting and meaningful considerations of what to measure and value.