Limitations of GiveWell
Explaining and showing the state of GW model and presentation, and noting how it might be improved
(Discussion in progress)
Givewell presents a model that is transparent and explained in some detail. However, it is not as easy to follow as it might be, and in some ways difficult to adjust, check, or run simulations on.
They produce and share their model in Google spreadsheets such as the one HERE, which I've 'forked' HERE to be able to comment on and play with. Note that you need to copy ('fork') it to be able to view the underlying formulas. The calculations are done in the various cells of the spreadsheet. The formulae look like:
GW provides a route to specifying your own moral weights and discount rates, involving a blank row and changing an indicator in a dropdown. This then permits you to see the "results" on another sheet. You can go back and forth to see how these change, or how it compares to GW.
However, it's not easy to see and compare this 'all in one place'.
Limitations (which can be addressed by our proposed approach)
The modeling and presentation is not terribly succinct. It stretches over multiple pages, sheets, and rows, and to get all the details requires some amount of 'clicking and chasing and following footnotes.' It's hard to find explicit statements of the relevant equations and to compare these across models. Further muddling the reader, there are elements in the spreadsheet that actually do not affect the final outputs ... such as 'amount donated'.
They don't explicitly model and present and present uncertainty. (see 'why make uncertainty explicit' below) They don't give a 'statistical distribution of likely impact (for each charity) under explicitly stated and justified parameter distributions'. This makes it hard for donors and policymakers to consider the 'risk versus return' of each intervention, and consider how confident they should be in the evaluations.
The model is not easily checkable for bugs, nor is there explicit code one could use to do 'type-checking' (see Pedant) and other systematic checks. Code could also make various robustness checks and simulations easier to do, and make the work more replicable and extendable
While users can plug in their own parameters (see 'types of uncertainty' below), this requires a few steps ... copying the sheet and consulting certain columns, etc. And even after doing so it's not so easy to go back and forth between assumptions and outcomes (or outcome distributions). Dashboards like the ones built into Causal.app make this far easier.
The current setup might lead to
Unnoticed errors (see (Possible errors and misunderstandings: examples from GW and beyond)... to be expanded
Limited ability for GiveWell to assess a larger set of charities without sacrificing analytical rigor. But if they can make their uncertainty explicit and transparent, they may feel more comfortable considering charities with harder-to-measure and more long-range benefits.
Relatedly, GW may be ignoring "high risk high return 'hitspace' interventions"
Lack of donor, foundation, and researcher engagement with the model,
Lack of confidence that people 'understand the model', people moving outside the GW model because they can't figure out how to get it to embody their assumptions
Last updated
