AFIRE (Accelerator For Innovation & Research Funding Experimentation) is a new collaboration between RoRI, the Innovation Growth Lab (IGL) and a consortium of our partners to boost leadership and institutional capacity for the design, implementation and synthesis of experiments with research and innovation funding.

Applicants as reviewers: a guide to distributed peer review

A practical guide for funders exploring or refining Distributed Peer Review, with checklists, setup tips, and lessons from real-world trials

The project in detail

Our aims are to enhance:

• Awareness of experimental possibilities and findings in the funding ecosystem.
• Motivation to experiment, and to evaluate the effectiveness of interventions
• Capacity to experiment, encompassing both technical expertise around issues such as data collection and analysis, as well as an understanding of the frontiers of research on research and innovation funding so interventions can be targeted to the most pressing needs.
• Scale and reach of experiments. The Accelerator will support getting experiments up and running, with greater ambition in running experiments at larger scale, at greater levels of robustness and in ways that support sharing and dissemination of post-experiment evaluation.
• The culture of research and innovation funding, across the whole sector, so it is more experimental, creative and evidence-informed.

Interest in experiments is growing, but different research funders are at different stages of readiness as reported by the RORI–SNSF—EMBO—IGL workshop, December 2021 (internal report ‘RoRI Funder Lab: improving research funding processes through experimentation’).  AFIRE will provide a broad platform which is accessible to funders at all levels of engagement, combined with structured support to work with funders to increase the speed and ambition of their engagement with experiments. The project will build on RoRI’s successful collaboration with IGL and, through them, bring in expertise in experimental methods, workshop facilitation and their experience working with innovation agencies.

AFIRE will consolidate and extend RoRI’s role as a leading broker for research funding experimentation. The project will be led by Tom Stafford, senior research fellow at RoRI and professor of cognitive science at the University of Sheffield, in partnership with Albert Bravo-Biosca and team at the Innovation Growth Lab (IGL). Speaking about plans for AFIRE, Tom Stafford said:

“Good funder experiments take rigour and imagination. It’s exciting to be part of a project which is able to draw on both, from across the RoRI network of expertise. Together we’ll be able to support more experiments and better use of experimental evidence to improve research funding.”

Project team

• Tom Stafford (RoRI/Sheffield) – Project lead
• Theodore Hodapp (Gordon and Betty Moore Foundation) – Co-Chair
• Stian Westlake (UKRI) – Co-Chair
• Cindy Lopez-Bento (KU Leuven)
• Ines Rombach (RoRI, Sheffield)
• Anna Butters (RoRI, Sheffield)
• James Wilsdon (RoRI/UCL)
• Amanda Kvarven (RoRI/UCL)
• Stephen Curry (RoRI/Imperial)
• Albert Bravo-Biosca (IGL, Nesta)
• James Phipps (IGL, Nesta)

Partners and steering group

Project partners include: 

• The Gordon and Betty Moore Foundation
• Luxembourg National Research Fund (FNR)
• Innovation Growth Lab (IGL)
• Wellcome Trust
• UK Research and Innovation (UKRI)

During RoRI’s pilot phase, we identified three priority topics for funder experimentation:

• Navigating the Grey Zone – gathering data on reviewer confidence / uncertainty
• Designing Panel Rules – experiments on structuring the review process to mitigate biases
• Partial Randomisation – building on an area of substantial existing funder interest and prior RoRI work

Scoping work on these and related topics is ongoing with a vanguard of our experimentally-minded partners, including: Swiss National Science Foundation (SNSF), Novo Nordisk Foundation (NNF), Luxembourg National Research Fund (FNR), Austrian Science Fund (FWF) and Health Research BC. 

AFIRE is designed to bring in other partners, and foster new, more ambitious experiments, creating a structured pathway for research and innovation funders to begin, expand and/or report the findings of their experimental interventions in priority-setting, grantmaking, review and evaluation.

Timeline and outputs

The AFIRE platform will consist of:

• Forum – An arena for peer exchange and learning about experiments at ideation, design stage or in progress. The first event in this series was held in May 2024. Invite only, and conducted under the Chatham House Rule, these informal discussions provide a safe space in which the real work of experimental approaches can be discussed in detail.

• Sprints – On-demand small-group mini-workshop series tackling a particular issue, question, or experimental stage. Sprints would involve more intensive and targeted activity with the intention to achieve substantive progress over a short period of time. They would bring together a group of partners who are developing experiments within the same topic area or who are at a similar stage in the experimental process. Each sprint would consist of a series of sessions facilitated by our team, with participants expected to work independently and make progress on their projects in-between sessions. The starting point would vary for each sprint, and the approach, format and frequency of meetings would be adjusted depending on the identified objective, participants’ needs and their availability. Some sprints may bring together multiple partners developing experiments around a similar challenge (e.g., biases) while others may focus instead on helping partners make progress on a particular experimentation stage (e.g., developing a logic model and seeking user feedback). While the intention would be for sprints to bring together multiple partners in order to benefit from peer learning opportunities, there would be the option of having sprints that only include one active partner if it is difficult to align timetables and priorities across multiple partners. 

• Structured support
  • For experimental design, implementation, analysis – Ongoing assistance for partners who are developing and implementing experiments. This could take different forms, such as one-off advice provision, light-touch continued support for a particular project, or intensive deep dive interventions with specific partners, triggered at different stages of the experimental process when additional support is required to ensure progress (for instance workshops on ideation, trial design, analysis etc). In addition, brokerage for partners to identify external researchers and other delivery partners to collaborate on experimental projects.
  • For in-house capacity-building/experiment promotion – The project would run events for partners that would support capacity building and internal advocacy for experimental approaches.

• Evidence accumulation & synthesis – continuing RoRI’s role as a repository for sector knowledge and site of evidence synthesis, e.g. hosting the live audit of trials on partial randomisation, coordinating common reporting standards, tracking outputs and publications from conducted trials.

The project would develop a roadmap to structure partner engagement within the forum, sprint and structured support; based on a prioritisation of needs and commensurate expectations on work required from the partners.