“Everybody talks about impact, but nobody does anything about it.[1]”
Have admired David Guston and his political science/policy research from afar. You should too – but probably from way closer-up.
By way of helping you decide whether you want to do just that, consider one of his recent publications – a short article in the Summer 2024 edition of NASEM’s Issues in Science and Technology, entitled What We Talk About When We Talk About Impact. This is by no means his most significant work (his c.v. lists half a dozen books, myriad book chapters, over thirty peer-reviewed publications, and an extensive body of editorial pieces, spanning a broad range of topics). But like his other perspectives, this is robustly structured, meticulously crafted, a fascinating read – and addresses an important subject – impact. We all want our work to matter – and matter more. David Guston shows us how.
Impact has a long history, and even a prehistory (before there was the word, there was impact – think Chicxulub). With great oversimplification, “impact” became a thing for scientists when science became expensive and started depending on financial support from the public – mostly non-scientists. Scientists had to start explaining the value of their work to people who by and large were making less money than scientists and struggling to make ends meet. Immediately after World War II, the value of science and technology was obvious. The atomic bomb, radar, and penicillin had helped the Allies win the war. There was a honeymoon period. Scientists said, “give us lots of money and don’t ask too many questions or interfere, and someday you’ll be glad you did.” Remarkably, both sides to this social contract kept their part of the bargain (and benefited commensurately) for decades.
But as costs have mounted and competing claims for public funds emerged, the bloom has come off this rose. One milestone: in the late 1990’s the National Science Foundation required proposals to address not only the intellectual merit of the proposed work but also the broader impact. Scientists had long accepted the need to justify the former but at first resisted and have never really embraced the latter.
You could see that problem coming. An on-the-ground-experience from the 1970’s: I was a branch chief in NOAA’s Wave Propagation Laboratory. We developed remote-sensing techniques for observation and study of the atmosphere. We were good at what we did and knew it. We had swagger. We had a great laboratory director – C. Gordon Little, a man of scientific acumen, integrity, and vision. Morale was generally high, but there were a few rumblings about rank and promotion and fairness. In response, Gordon decided that each of us (130 strong) should anonymously rank every lab employee, including ourselves, on “their net positive impact (otherwise undefined!) on the laboratory.” He decided further that the employee survey results and the branch chief results (there were seven of us) should be separately tabulated.
We all thought (especially the handful of us in management) his idea insane. The lab groups were diverse (acoustic-, optical-, radar-, radiometry- atmospheric studies, etc.). The work ranged from remote-sensing theory to technique development to application in research to tech-transfer-to- the-service-providing-elements of NOAA or other federal agencies. Some WPL groups and employees were largely NOAA-funded; others funded primarily by other agencies. Absent any specificity to the “impact” criterion, the rankings would be entirely subjective. The results would surely prove divisive and tear the laboratory apart! But Gordon persisted.
The results were amazing – almost miraculously so. The 130 employees, including the managers, were ranked. The variances for the rankings were miniscule. No one had a variance greater than one position up-or-down in the rankings. The managers were high or at the top of the heap; and there was no detectable difference between the managerial rankings of the staff and the overall peer ranking of the staff. The mood at the debriefing was remarkably celebratory[2].
But – germane to the broader impacts discussion here – in the Q&A near the end of the meeting, Gordon was asked how he would evaluate any work on the tech transfer element of the laboratory’s mission. He casually remarked – “well, by definition, we would wait until there had been actual uptake of a technology by some NOAA services line office or another federal agency.”
Perhaps a dozen members of the lab were immediately dismayed. They knew full well that this process took years, not months, if it happened at all – and was subject to the vicissitudes of politics and funding and wholly out of WPL’s control. Gordon quickly walked back his statement, but some damage was done. At least one of our rising stars in such work left the lab to take a job in another institution not long after.
To use a football analogy (after all, it’s that season), this was a fumble on the five-yard line.
Back to David Guston.
David Guston provides much-needed help. He lays out a thumbnail history of impact. He identifies ways research has impact: directly shaping policy goals and language; changing general thinking; education and training of professionals; interaction with lay knowledge (offering explanatory detail on each of these).
But it’s the evaluation, the measurement of impact that he gives the most attention. He points out the many difficulties of evaluation, but instead of throwing up his hands in dismay he provides a framework that allows approach to the challenge with the same discipline scientists apply to their core research areas. He gives attention to the pathway between the research in questions and societal outcomes (a so-called knowledge value connective or KVC). And he provides an impact catechism (and links and references to a universe of proto-catechisms, themselves of interest) – a set of questions to be applied to the research:
- What kind(s) of impacts (category/type) are you aiming at?
- What scope (extensivity) and depth (intensivity) of impact are you planning for?
- What specific audience(s) are you addressing or constructing?
- What (causal) model do you have in mind for creating impact?
- How are you creating opportunities for impact?
- Who or what (KVC) connects your outputs to impacts and outcomes?
- How are you participating in, researching, or keeping track of (intermediate) impacts along the way?
- How will you tell the story of the impact that you have with humility and accuracy?
And speaking of humility (as in the last bullet), Guston doesn’t present these questions as prescriptive, let-alone holy writ – he’s merely saying that if as a starting point researchers ponder these questions as applied to their work, if faculties and other research groups discuss them, out of those conversations will flow a more robust understanding of “impact,” its worth as a concept, and the worth of research in light of that concept[3].
A kind of DIY approach to building and articulating your impact in a way that will allow you to stand tall versus hem-and-haw.
The idea bowls you over, doesn’t it?
[1] With apologies to Charles Dudley Warner, who made the same observation about the weather (and Mark Twain, who usually gets the credit for it).
[2] In hindsight, I think this positive result has a lot to do with the attention lab management gave to promotions and career development – unmatched by any other group in my professional experience. We dealt with promotions at least quarterly. Branch chiefs would prepare promotion packages, promoting, say, someone from GS-12 to GS-13. There would be a group discussion of the individual case. Then there would follow two discussions: is this the GS-12 from the lab we would choose to be promoting at this time? Where does this person rank relative to the current GS-13’s? The process put a spotlight on branch chiefs who were either too slow or overeager to recognize their talent. Branch chiefs out of line with consensus would be encouraged (sometimes told) to either hold back an action for a quarter or put forward an overdue action at the next quarter.
[3] And perhaps an improved catechism
