Tuesday’s post made a case for what might be called “ensemble” approaches to policy formulation. The idea is borrowed from ensemble weather forecasts. Comparing independent numerical weather predictions (NWP) both provides opportunity to improve accuracy (the “consensus”of the various model runs) and characterize the uncertainty (as revealed by the spread or divergence of the different runs). The extension to the policy arena suggests that society is better served by capturing the full diversity of policy opinion from independent sources, again with the aim of charting an optimal policy course and at the same time gauging the degree of polarization on any given policy issue. In 2004, James Surowiecki provided an engaging account bearing on all this in his wonderful book The Wisdom of Crowds: Why the Many Are Smarter Than the Few and How Collective Wisdom Shapes Business, Economies, Societies and Nations.
When it comes to the ensemble forecasts, those using the technique are free to choose which numerical weather prediction (NWP) models will be included and which will be excluded. The choice might be made on the basis of the rigor of the science used in the model, its individual performance based on quantitative metrics, its cost, its transparency, accessibility, and other benchmarks. Those constructing the individual NWP models making up such ensembles are presumably motivated by similar criteria.
In the same way, when contributing our perspectives to any policy discussion, you and I might do well to discipline ourselves bit… to reflect on the legitimacy of our personal thought process, as well as the legitimacy of the overall procedure, involving collection and aggregation of many individual views. We shouldn’t engage irresponsibly. We shouldn’t feel free to suggest whatever comes to mind, or what serves our short-term interests at the expense of the others. We shouldn’t feel free to suppress dissent. And so on.
My colleague Paul Higgins, Director of the American Meteorological Society’s Policy Program and a Senior Policy Fellow, has given this some thought. He’s articulated a set of criteria for establishing legitimacy in the expression of views, and has passed it around for informal reaction.
Paul emphasizes that this material is “draft,” but to me and several others it looks pretty good! At my request, he’s graciously allowed me to reproduce his list here, in hopes of getting more feedback. In particular please look at it through the lens of its bearing on ensemble formulation of policy (regulation, or legislation, for example). Here’s what Paul has to say:
There is great need for the scientific community to effectively and credibly express views on a range of topics. Such views may include objective assessments of scientific knowledge and understanding or subjective expressions of opinion (e.g., community goals and priorities and what “should” be).
In broad general terms, the legitimacy (and power) of views expressed by any group is enhanced through a process that ensures:
1) The full range of credible and defensible views contained within the community are fully and fairly considered in the assessment process
2) Relevant external views are sought and considered fairly
3) Minority held views are sought and, when credible, included
4) All participants have an equal opportunity to contribute (NB: not the same as saying all participants will contribute equally because …)
5) All suggestions are assessed and included (or excluded) based on their merit (i.e., their substantive contribution to the assessment’s validity)
6) Views expressed are free from self-interest to the maximum extent possible
7) Any notable remaining self-interest is identified explicitly and clearly within the assessment itself
8) The potential for participation bias in the assessment process is recognized and accounted for (e.g., the AMS Policy Statement on Geoengineering recognized the possibility that only proponents of geoengineering would be willing to invest the time required of participants)
9) The group possesses (or has access to) sufficient subject matter expertise to fairly and accurately assess all relevant information
10) Opportunities for broader community input and review are included to the extent possible (e.g., member/public comment periods)
11) Independent validation and oversight occurs when possible (e.g., the AMS Council provides this role in the AMS statement process)
12) Criticism and dissent are welcome and encouraged
13) Scientists wishing to provide criticism or dissenting views are able to do so in ways that guard against potential retribution or the potential perception of it
14) Hard choices and sacrifices do not involve non-participants in the process (i.e., such calls are not credible)
15) Assessment efforts are led by a credible, trusted, and capable source (i.e., individuals and institutions committed to the principles articulated here and capable of ensuring they are met to the maximum extent possible)
Your thoughts? Paul (and I) would welcome your comments and suggestions. You can reach him at phiggins@ametsoc.org or comment on this post. Or both!
Bill:-
Looks like a pretty comprehensive and well-crafted list. The only obvious [at least to me] omission is any mention of uncertainty. To me, this is the Charybdis and Scylla that much of the current “discussion” about AGW can’t get past. How can reasonable scientists continue to make so many bold pronouncements in the face of such great uncertainty?
In the course I teach on “Systems Thinking,” I teach my students that the optimal way to solve wicked problems is to take small steps and monitor the outcomes. I have always believed that humility is one of the most important virtues a scientist must possess – and uncertainty is one of those factors that should lead us all to be more humble in our estimations of reality. How much more so when we try to extrapolate from our narrow [and noisy!] world of fact to make definitive policy pronouncements.
One more comment (I promise). There is something that is a sort of a corollary to 8) and 9) that should be added:
9.5) The [policy] statement should not go beyond the expertise available to the group.
Why? Let’s look at a very pertinent example. Let’s assume that the AMS looks at global temperature data and the suite of models and their predictions and concludes that temperature is moving upward and, IF NOTHING CHANGES, is going to lead to a very different world than the one we live in. Then 9.5 would allow a statement to be made about the science – something like, “based on data and model predictions we believe that IF NOTHING CHANGES in energy usage per capita, that the temperature rise will lead to a very different world with unknown consequences” [For this example, I’m ignoring the poor performance of the models, esp. at prediction]. Of course, this wouldn’t be very satisfying to the activist members of AMS, but to go further would require that they enlist their brethren and sisters in the ag community, the economic community, the social sciences … to determine what those consequences might be.
If they wanted to include an economic consequences statement, they would need to (based on 8) and 9)) get a consensus of some sort from among the economics community, and not just get some self-serving statement from self-aggrandizing publicity hound.
Seems like a lot of work, doesn’t it? Personally, I prefer it that way. As citizens, we have a right and a duty to speak out about what we believe. But I see no reason why my voice should be louder than a poor cotton picker in Issaquena County, MS; or a banker in Charlotte; or a logger in the state of Washington, when we’re talking about something none of us are experts in. When our professional societies speak, their voices are louder – both because of a presumed expertise as well a sort of social “constructive interference.” Thus, the bar for any sort of definitive statement by a professional society should be set very high, to make sure that they don’t overstep their professional bounds.