| The scientific evaluation of organic food and farming (OFF) projects has been challenging, under two aspects.
Firstly, it has been observed in the past that traditional evaluation criteria may not consider appropriately the system-oriented, innovative and interdisciplinary approach of OFF-proposals.
Secondly, the success of trans-national research programmes has been highly dependent on a transparent evaluation procedure, which clearly distinguishes between scientific evaluation and political selection.
Our objective here in a new CORE Organic report was:
- to assess in what extent the evaluation criteria and procedures used for the CORE Organic pilot call challenge these perspectives
- to provide further recommendations to improve the future call procedures.
|
 |
To achieve these aims, a feedback evaluation (questionnaire) involving the different target groups involved in the Core Organic pilot call (expert panel members, national call contact persons and governing board members) has been implemented and extended by a literature review of the latest assessment practices advocated in the research evaluation field.
The survey showed that the proposed set of nineteen evaluation criteria, clustered within six main categories, fulfilled the expectations of most target groups involved with the CORE Organic pilot call. However, some of the respondents wished to have a stronger focus on aspects of interdisciplinarity.
Some of the comments of the applicants showed that handling the gap between the scientific evaluation and the final selection of OFF research projects is the main challenge for a successful evaluation of trans-national research in the future. To overcome this conflict, different measurements were suggested.
The analysis of the scientific evaluation of the proposals submitted in the pilot call showed that rejected proposals had received insufficient scores regarding methodological criteria. Proposals with high prioritisation showed higher scores for the criterion relevance. In order to improve chances to succeed in future calls, proposals should therefore focus on the criteria methodology and relevance.
In contrast to the survey with the actors involved in the pilot call, the review on the literature revealed further potential for optimising the evaluation criteria and call procedure. Even though almost all the literature on the topic of peer reviewing or research evaluation addresses new challenges from a conceptual perspective, the analysis shows that criteria and procedures used in the CORE Organic pilot call only partially address the new challenges raised by a strong change of perspective in research practices, as much as the specificity of OFF: Interdisciplinary and innovative aspects should be addressed in a more appropriate way. The decision process should be open to a wider community of experts in order to assess cooperation and trans-national aspects and to non-governmental representatives to provide wider scrutiny. The way national priorities are integrated in the decision process should be rethought and the transparency of the procedure should be increased.
We suggest to invent mechanisms to allow the funding of few “risky” research projects, to facilitate new incomers to enter the arena and to promote curiosity-issue research projects. Other assessment steps could be implemented for the projects identified as more innovative and less robust. This may include the tutorial of the ongoing research projects and results evaluation.
Furthermore the set of evaluation criteria should be refined in order give them clearer definitions and boundaries which would allow both, to develop new and more suitable sub-criteria to deal with “interdisciplinary and innovation”, and to reshape in an appropriate way the actual main criteria categories. Particularly, the creation of a main category called interdisciplinarity is of crucial interest. This process involves splitting of others main categories, as methodology, Scientific innovation, and Consortium.
Report on Scientific evaluation of trans-national projects – Between credibility and national preferences (PDF)
Prepared by Thomas Alföldi and Urs Niggli, FiBL, Stéphane Bellon and Julien Blanc, INRA
