Science Impact
To what end are you working?
To what end are you working?
To what end are you working? Presumably for the principle that science’s sole aim must be to lighten the burden of human existence.
If the scientists, brought to heel by self-interested rulers, limit themselves to piling up knowledge for knowledge’s sake, then science can be crippled and our new machines will lead to nothing but new impositions.
Berthold Brecht (1898–1956) Gallileo, Scene 14
The fundamental purpose of the scientific endeavour is change. Even the most basic or blue-sky research seeks to lay the foundations of knowledge that can one day be translated to the advantage of society. Yet, though fundamental, how science quantifies and understands impact is complex.
In general, most modern researchers will be familiar with impact assessments that tend to consider only the direct impacts, that is, in isolation of broader effects that might lie beyond their field of research, or that seem too theoretical.
Meanwhile, many research organisations have moved to a more integrated model of impact management. Unlike impact assessment, whereby impacts are considered in isolation of their broader context, impact management is a strategic approach to identifying benefits throughout the life of a research project or investment. While this move is welcomed, we argue that the current practices of research assessment and evaluation remain insufficiently focused on providing value to all stakeholders throughout the research value chain.
Most, if not all, science, technology,
engineering, and mathematics (STEM) research nodes invest in a merit-based assessment which is grounded in technocratic, rather than a whole-of-research value chain, focus. 1 In many instances – and unlike the social sciences – STEM scientists object to research funding being coupled to education or outreach efforts. 2
A study by Tretkoff, cited in Nagy et al., posited that “many scientists are unhappy with the broader impacts requirements, and feel they should be funded based on the quality of their research, not for outreach” … and… ”many physicists feel they don't have the expertise to do outreach activities.” One respondent reported “she thinks education and outreach should be encouraged but shouldn't be a requirement for research funding.”
It was reported, “[I]ndeed, some scientists, especially those applying for their first grants, find the broader impacts requirement confusing, burdensome and punitive.” 3 In most instances, while STEM scientists undertake impact assessments, generally these assessments are undertaken by members of the research team and do not extend beyond academia. 4 The overall findings of the Nagy et al.'s research indicated that the “broader impacts criterion is interpreted by many scientists as an introduction of extraneous political, cultural, or economic concerns into basic research”. 5
Putting it simply, we contend (a) that
In many instances – and unlike the social sciences – STEM scientists object to research funding being coupled to education or outreach efforts.
there does not appear to be any real effort by STEM scientists to adopt a holistic approach to managing research impacts across the entire research value chain; and (b) a conflict of interest exists if the assessment relies on this core network node, if for no other reason than that core nodes reinforce the status quo.
Arguably, the current instruction for undertaking impact assessments is idiosyncratic and biased. Apart from a few examples6, the research sector has only been “dabbling” in an impact approach to research investment and delivery. Rather than considering their broader impact activities7,8 primary research investigators tend to adopt established metrics and data collection methods
There is a constructive and lively discussion to be had about how we shift cultures, values, attitudes and behaviours to improve research investment and the delivery of impact.
that are nuanced from a personal and/ or an institutional perspective.
It will likely be considered a controversial call to recommend the research sector move away from long entrenched traditions, complete with values, rewards, systems, and processes, to a system that establishes a strategic management of impact. We do, however, feel that a discussion for such a shift is necessary. Such a shift is required to make researchers more accountable for their impact (or lack thereof ), especially given the significant level of public investment in research, most notably, within STEM fields.
There is a constructive and lively discussion to be had about how we shift cultures, values, attitudes and behaviours to improve research investment and the delivery of impact for the benefit of the nation and the world.
Research assessment and evaluation methodologies
Part of the problem remains that, except for a few emerging methodologies, there is an absence of effective systems which encourage the adoption of a holistic approach to research impact assessment. Ideally, this can be achieved by overhauling the faculty-reward system with a broad shift in the attitudes of research teams and, in particular, of professors, especially those on promotion and tenure committees. 9
We recognise that scientists are not solely responsible for delivering impact; the ‘research' team charged with the delivery of impact is much wider than individual researchers alone. Responsible Research and Innovation (RRI) approaches, and the role of knowledge brokers, boundary organisations and boundary individuals who are regularly identified as ‘boundary spanners' or ‘boundary agents' , also play an important role in providing a nexus between science and end users, thus assisting in securing a “social licence to operate”. 11,12 There is also a need for strategic management approaches to assess research impacts, including holistic engagement strategies that go beyond immediate academia.
By identifying and linking activities throughout the research value chain, and by gathering data from a range of stakeholders, it is possible to assess a spectrum of positions and perspectives in relation to the management of impact assessment. By adopting a mixed method of assessment and applying it through a broad impact-driven lens, research impact assessments will not only reinforce knowledge production, they will recognise and value public engagement in science as essential for best practice across research institutions, funding bodies and communities alike.
Leaving academia to transform the innovation system has arguably resulted in the assessment process being turned into an academic exercise, one where the focus lies on finding metrics/ indicators13, over other methods such as Responsible Research & Innovation (RRI). 14
Schomberg cited in Owen et al., 15 describes Responsible Research and Innovation as: a transparent, interactive process by which societal actors and innovators become mutually responsive to each other with a view on the (ethical) acceptability, sustainability and societal desirability of the innovation process and its marketable products (in order to allow a proper embedding of scientific and technological advances in our society).
There is a growing international shift toward the measurement of the broader impact of science, particularly in the context of publicly funded research organisations.
[An] innovative impact management system will require stakeholders to recast how they talk about, measure and manage risks, and positive and negative impacts.
To date, academic research initiative has been measured on inputs (such as external funding, dedicated research space, infrastructure) and, more recently, on outputs such as international databases ranking publication performance, and citation indices.
Transforming from an idiosyncratic and linear approach to one that endorses a holistic approach to assessing research impacts, is not only sensible but necessary. It requires a dramatic expansion of who are considered to be the stakeholders in research, including for example, local community members with local knowledge, civil society, policymakers, formal interest groups, and funding bodies.
An assessment methodology that can engage these periphery stakeholders will expand an academic's perception of what constitutes a research team, and will make science more accountable, while garnering more public support for research funding.
Shifting focus, impacts management
Notwithstanding a general apathy by the STEM sciences to strategically manage broad impacts, there is a growing international shift toward the measurement of the broader impact of science16, particularly in the context of publicly funded research organisations.
The UK Research Excellence Framework, the Swedish Research Council, the National Science Foundation (United States of America), and the International School of Research Impact Assessment have all flagged a deliberate shift away from traditional models of impact.
This movement goes beyond solely assessing contributions to academic knowledge (i.e. the delivery of ‘excellent science') and is increasingly focused on how science has delivered tangible benefits (social, environmental, economic) to the societies in which research organisations operate.
As with any large structural change, to reach this goal of adopting a wideview innovative impact management system will require stakeholders to recast (with consensus) how they talk about, measure and manage risks, and positive and negative impacts.
Tretkoff, in Nagy et al., makes the argument: Such a system includes good governance, meriting inclusiveness, openness, fairness, transparency, and accountability across the research value chain. Further it has been claimed priority should be given for identifying valid and feasible ways to assess research impact more objectively. 17… and … [A]n important contribution to the field would be the development of multiple well-defined measures, consisting of both hard and soft data, which can be systematically analysed”. 18
Nagy, like Dowd et al. and Owen et al., argues the need for the whole research value chain to change. 19,20,21 This includes not only the “business/ corporate” side of the chain, but also the funders and investment decisionmakers who need to approach their processes from an impact perspective.
Similarly, the European Commission recognised the benefit of managing broad impacts and adopted an extended impact assessment (EXIA) framework when considering policy development. 22 The purpose of EXIA is to carry out a more in-depth analysis of the potential impacts of the policy proposed on the economy, society and the environment; and to consult with interested parties and relevant experts according to the minimum standards for consultation. 23
It is critical to remember that changing our understanding of impact will not only benefit the quality of research undertaken, and galvanise the social licence for science, it will improve the outcomes for the whole of the value chain that relies on, or interacts with, the knowledge generated, including
Changing our understanding of impact will not only benefit the quality of research… it will improve the outcomes for the whole of the value chain that relies on the knowledge generated.
Broader impacts refer to specific, desired societal outcomes, such as the participation of underrepresented groups in STEM; public scientific literacy; and partnerships between academia, industry, and others.
governments who are keen to get returns on their expenditure.
According to the International School on Research Impact Assessment (ISRIA): As governments, funding agencies and research organisations worldwide seek to maximise both the financial and non-financial returns on investment in research, the way the research process is organised and funded is becoming increasingly under scrutiny. There are growing demands and aspirations to measure research impact (beyond academic publications), to understand how science works, and to optimise its societal and economic impact. In response, a multidisciplinary practice called research impact assessment is rapidly developing. Given that the practice is still in its formative stage, systematised recommendations or accepted standards for practitioners (such as funders and those responsible for managing research projects) across countries or disciplines to guide research impact assessment are not yet available. 24
As such, ISRIA posits that, “most research and funding institutions are simply lacking in their capacity to meet growing demands and aspirations to measure research impact (beyond academic publications), to understand how science works, and to optimise its societal and economic impact”.
Similarly, the US National Science Foundation established a merits-based policy whereby all funding proposals submitted to the agency would be evaluated on two criteria: intellectual merit and broader impacts. 25 Broader impacts refer to specific, desired societal outcomes, such as the participation of underrepresented groups in science, technology, engineering, and
Transforming from a linear research-to-impact approach to an oscillating and strategically motivated approach to managing impacts, requires human capital and social capital.
mathematics (STEM); enhancing STEM education; public scientific literacy and engagement; and partnerships between academia, industry, and others. 26
Transformation
As Australia's national science agency, CSIRO takes the assessment of the impact it delivers very seriously. Since 2015, CSIRO evaluations have focused primarily on benefit-cost analyses (BCAS), with resulting return on investment (ROI) calculations. We accept that BCA is not always the most relevant form of analysis for specific types of impact; an issue we are addressing through the update of our evaluation guide.
The updated guide will reflect the growing maturity of the impact approach within CSRIO; and will encourage the consideration of evaluation methodologies that are more closely aligned with the types of impacts sort through specific research activities, rather than defaulting to the BCA approach. This updated guide will be available publicly (through the CSIRO website) in early 2020.
For example, we recognise there is a benefit of considering the social return on investment (SROI). Social return on investment is a principles-based method for measuring extra-financial value (i.e., environmental and social value not currently reflected in conventional financial accounts) relative to resources invested. Developed from traditional cost-benefit analysis and social accounting, SROI is a participative approach that captures in monetised form the value of a wide range of outcomes, whether these already have a financial value or not. 27
At CSIRO we argue that articulating the impact pathway (inputs, activities, outputs, outcomes, impacts) associated with the evaluation target is essential to recognising and managing impacts throughout the research value chain.
Transforming from a linear research-to-impact approach to an oscillating and strategically motivated approach to managing impacts, requires human capital and social capital that include boundary organisations and/or individuals. Building relationships through engagement provides opportunities to link academia with industry, interest groups and individuals who have an interest in the research project.
Engaging boundary organisations enables them to more quickly integrate scientific findings and practitioner experiences to create usable knowledge about science and research. In this way they can: (a) assist in research communications, thereby enhancing literacy and salience, and informing policy decisions and societal actions; and (b) amplify public value creation and promote even broader impacts, much like a ripple effect. 28
Bridging and bonding networks are fundamental elements of social capital, and they are essential for managing impacts. Some networks link people who are similar in crucial respects and tend to be inward looking, hence the term bonding social capital; while others encompass different types of people and tend to be outward looking, hence the term bridging capital. 29
Networks are a critical component for building social capital, since “dense networks of social interaction appear to foster sturdy norms of generalised reciprocity”. 30
Social capital, inclusive of social organisations' reciprocity, norms and trust, facilitates action and cooperation between stakeholders for mutual
‘Stocks’ of social capital reflect the level of social interaction, networks and relations, trust and reciprocity that exist within a research community.
benefit 31 . Thus, in a research context,
‘stocks' of social capital reflect the level of social interaction, networks and relations, trust and reciprocity that exist within a research community 32 .
With increased stocks of social capital, typically research value chains will have reciprocal interactions and increased trust that are directed towards mutual benefit 33,34 and assist in managing impacts across the research value chain.
The work of Dowd et al. illustrates the importance of social integration throughout the research value chain, highlighting the role of network analysis as a means of identifying those social ties that exist within a dynamic network. It is these dynamic networks and relationships that aid the adoption and transfer of science impacts across the whole research value chain.
Therefore, maximum value is derived when all stakeholders in this system engage, and remain engaged, throughout the life of the project. Herein lies the role of the boundary organisation or knowledge broker.
The theory of boundary organisations draws upon the social constructivist conviction that the boundaries between science and non-science are contingent, and socially constructed.
Typically, boundary organisations, teams or individuals link academics with others in the research value chain. These organisations and/or individuals play a very important role ensuring information and knowledge transfer by managing interactions between science and politics, economics and society 35. Boundary spanners are well placed to identify and assist with research evaluation and the management of impact.
The Minnipa Agricultural Centre and Rural Solutions South Australia is one example where staff (including scientists) live locally and engage with community interest groups in civic science projects. These organisations operate as brokers between science, policy, administration and end users – farmers – and employ individuals, often agronomists who can engage meaningfully with end users.
Conclusion
In this paper we have raised issues that are designed to stimulate a discussion about STEM research and managing research impacts. We argue that the current practices of research assessment and evaluation are not broad enough to provide value to all stakeholders throughout the research value chain.
Unlike current impact assessment in which research ‘impacts' are considered in isolation of their broader potential or intended benefits, we argue for a shift to impact management, which is a strategic approach to identifying impacts throughout the life of a research project. Moreover, if we are to seriously address the challenges inherent in impact assessment of STEM research, we need first to be creative in the way we engage with the research value chain.
Second, we need to conceptualise and develop practical ways to measure broad impacts across the research value chain. Finally, by recognising the valuable role boundary organisations, teams and individuals play in the management of impacts, we impress upon the reader to consider the benefits for working collaboratively and strategically when managing broad research impacts.