@RyanKidd Thanks! I agree, I might not be the best person to do this with my limited experience. I searched EA forum and LW and only found those CE estimates:
- CAIS
- arb's AISC estimate
- Nuno sempre work on long termist orgs
- MATS self assessment

Which is indeed limited. Given this I think it's a good idea to do this, even to just put out something and let other people work on this to improve this. Worst case scenario, this will end up being something along lines of "How good are AIS orgs at communicating their findings (measured by citations)"

To improve quality I will try to get feedback from some organizations and EAs.

My rough plan is to break down the theories of change that are pursued in AIS down into:

• Evals

• Field-building

• Governance

• Research

And measure impact in each category seprately.

Evals can be measured by quality and number of evals, relevance to ex-risks. It seems pretty straightforward to differentiate a bad eval org from a good eval org—engaging with major labs, having a lot of evals, and a relation to existential risks.

Field-building—having a lot of participants who do awesome things after the project.

Research—I argue that the number of citations is also a good proxy for the impact of a paper. It's definitely easy to measure and is related to how much engagement a paper received. In the absence of any work done to bring the paper to the attention of key decision makers, it's very related to the engagement.

I'm not sure how to think about governance.