Bollettino telematico di filosofia politica

PDF

Index

Abstract
Introduction
1. Background: The context of the reform
2. The Agreement on Reforming Research Assessment and the centrality of quality
2.1 Diversity
2.2 Qualitative evaluation
2.3 Responsible metrics
3. Open Science: Just science done right?
4. Science and quality
5. Science and truth
6. Science and collaboration
7. Conclusions. Towards a definition of research quality
Acknowledgements
References

Keywords

Open Science, Responsible Research Assessment, Qualitative Assessment, Research Quality

Cite as

Di Donato F., “What we talk about when we talk about research quality. A discussion on responsible research assessment and Open Science”, Bollettino telematico di filosofia politica, March 2024. https://commentbfp.sp.unipi.it/quality-fdd/; DOI: 10.5281/zenodo.10650139.

Abstract

The current reform of the research assessment system focuses on quality and qualitative assessment, and establishes a strong nexus between the two concepts. This article explores the definition of quality proposed in the Agreement on Reforming Research Assessment (ARRA) through an analysis of its principles and core commitments, and suggests that this definition is in line with the definition of Open Science provided in the UNESCO Recommendation on Open Science. Furthermore, a re-reading of The Conflict of the Faculties of Immanuel Kant is proposed to help understand the terms of the definition of Open Science and the link between the latter and research quality from a philosophical-political perspective. Finally, the article introduces a definition of quality in line with the ARRA, emphasising several aspects to be explored in its possible future developments.

Introduction

The crisis of the current research assessment system has been at the centre of academic debate for many years but it is only recently that the issue has been brought to the forefront of the agendas of public institutions financing research. In fact, the existing evaluation criteria placed excessive emphasis on quantity and on productivity, leaving aside research quality, collaborative open research methods, and the wider impact of research on society.

The article starts from this consideration (section 1), which led to the ongoing reform coordinated by the Coalition for Advancing Research Assessment (CoARA). The Agreement on Reforming Research Assessment (ARRA), which is analysed in section 2, firstly recognises the need to value the variety and diversity of contributions to research and science. Quality is not associated (solely) with publication (of articles) but rather with  the diverse contributions that researchers make to science and society; the practices contributing to robustness, openness, transparency and inclusiveness of research, and the research activities which include teaching and leadership among others. Secondly, it proposes a vision that relies on qualitative judgement based on peer reviewing, only supported by quantitative indicators. In the ARRA a link is thus significantly established between quality as a principle, and qualitative assessment, based on peer review. In particular, quality is defined as the originality of ideas, professional research conduct, and results beyond the state-of-the-art, and is strongly connected to both commitments 1 and 2, as it entails rewarding a variety of research missions, and implies that research is carried out through transparent research processes and methodologies and through research management allowing systematic re-use of previous results, and in a collaborative manner. Quality is moreover linked to other principles, such as diversity, collaboration and inclusion.

Notably, the definition of quality which emerges in the ARRA is closely linked to that of Open Science, and in particular to that given by the UNESCO Recommendation on Open Science of 2021. Section 3 explores this link, showing how this definition focuses as much on the diversity of activities related to science as on the values and principles associated with Open Science, in line with the ARRA’s principles and commitments.

Sections 4, 5 and 6 aim to provide the philosophical foundations of the definitions of quality and of Open Science which emerge in the previous sections. In particular, a philosophical-political reflection on the elements that constitute the quality of research is proposed, in order to better understand its definition. In particular, section 4 focuses on the definition of quality starting from the reflection of Robert Maynard Pirsig who focused his entire research career on defining quality, and who, like in the Agreement, identifies quality with a procedural definition, namely the integrity of the scientific method. How is this method defined? Is it a mere procedural question, or does it concern the very essence of scientific research? Sections 5 and 6 propose an answer to these questions, reconstructing Kant’s contribution in this regard. In the late The Conflict of the Faculties of 1798, the philosopher, who in 1784 had praised Frederick II, the sovereign who did not fear the light and who allowed freedom of ‘public use of reason’, tackles the problem of research quality discussing the conditions for public research to be defined as such. In fact, according to Kant, research is such if it is autonomous and continuously subject to criticism by experts, who who are considered to be so on the basis of scientific merit, and therefore cannot be other than open, collaborative and inclusive. In this perspective, the term Streit (conflict), i.e. a scientific debate between peers, takes on a central meaning.

In the conclusions, a dual definition of quality is proposed, as a ‘contextual event’ and as a ‘continuous process, which must never end’. In both cases, the way to assess it is through qualitative judgement, based on peer reviewing, which can also be supported by quantitative criteria if contextual and relevant for the purpose of the assessment. In this sense, a more in-depth discussion on what is meant and recognised as peer review in scientific communities would be crucial, and the contribution of the CoARA Working Group (WG) on this topic will certainly be important. However, in the context of the ARRA an aspect which still deserves to be emphasised is that a necessary condition for science to be defined as such is for evaluation to be in the hands of scientific communities, for science is no longer science if it is subject to heteronomous reasons.

1. Background: The context of the reform

While the reform of research assessment has been on the international political agenda for several years now, it is only since the COVID-19 pandemic that the need to rethink evaluation mechanisms has become a central and unavoidable problem. And it is only in 2020-21 that policy makers and research funders initiated concrete actions in this direction in a coordinated approach.

As a matter of fact, the World Health Organization (WHO) Director-General explicitly called on “all countries, companies and research institutions to support open data, open science, and open collaboration so that all people can enjoy the benefits of science and research” (WHO Director-General, 2020) already at an early stage of the COVID-19 pandemic, in April 2020.  Robert Therry went even further a year later, in September 2021, exposing how the current scientific system was inadequate to producing high-quality scientific knowledge, and showing that less than 25% of the material scientific information included in the WHO COVID-19 Living Guidelines came from mainstream publications. “They [publishers] proved useless, just when we needed them most”, he asserted (Therry R., 2021).

What Therry was pointing to is the profound crisis in scholarly publishing that we have witnessed over the last decades. While in 2017 the annual margin profit of commercial publishers stood at 36%, outperforming large players such as Facebook and Amazon (Buranyi S., 2017), recent research has shown that the annual cost of journal subscriptions amounts to approximately 10 billions dollars. In addition, a study commissioned by the Directorate-General for Research and Innovation of the European Commission calculated the amount of money we, as citizens, lose each year because research data are not properly managed, which exceeds 26 billions euro (Schimmer R., Geschuhn K.K., Vogler A., 2015. See also Crace J., 2023). It is clear that if there is an equilibrium in these numbers, it is not a fair one. The pandemic turned out to be a social experiment that proved yet another point: the publication and evaluation system is based on the prestige of publishing in closed journals. As Covid has shown, as well as being extremely costly, this system keeps results sealed, has extremely long publication delays, does not provide data and is not transparent. Specifically, as underlined by Pontika and others the “current assessment criteria are believed to focus too heavily on inappropriate criteria related to productivity and quantity as opposed to quality, collaborative open research practices, and the socio-economic impact of research” (Pontika N., Klebel T., Correia A., Metzler H., Knoth P., and Ross- Hellauer T., 2022, p. 888).

Jean-Claude Guédon had already shown in 2001 that the so-called ‘serials crisis‘ depends on evaluation, and in particular on the progressive imposition of bibliometric indicators as suitable criteria to measure research quality (Guédon J.C., 2001)¹. As a matter of fact, research assessment in STEM is mostly based on bibliometric indexes and most of the Social Sciences and Humanities disciplines rely on a selected “top class” list of journals. The direct and side effects of such a system have been extensively studied and are now widely known (Baccini A., De Nicolao G., Petrovich E., 2019; Fyfe A., Coate K., Curry S., Lawson S., Moxham N., Røstvik C.M., 2017; Morales E., McKiernan E., Niles M.T., Schimanski L., Alperin J.P., 2021; Kulczycki E. et al, 2018; Kulczycki E., 2023). Altogether, this system leads to an increasing number of publications due to the fact that researchers are forced to publish, so quantity prevails over quality; and because they pursue high citation rates and aspire to publish in so-called “prestigious” venues.

Several studies have demonstrated the unfairness, inefficiency and unsustainability of the above described system. What needs to be emphasised for the purpose of this argument, is that the current evaluation system does not guarantee scientific quality, and this has various consequences: “The emphasis on publication quantity is training researchers to skimp on detail and rigour in favour of hype and speed. It is also discouraging them from diversifying their methods: for instance, by complementing quantitative research with qualitative findings or vice versa, which is a rewarding but time-consuming effort; and by collaborating with those outside their immediate specialized networks, whose diverse viewpoints and expertise may not fit discipline-focused forms of assessment. The future generation is therefore being pushed away from transdisciplinary research and robust investigative practices. The chances to rebel are small, given that publication-obsessed cultures privilege those who have long held academic jobs and lack incentives to address prejudice, ageism, bullying, misogyny and racism.” (Leonelli S., 2023, p 11). Moreover, phenomena such as retraction and fraud are extremely widespread. As Bishop, who spent a lot of time studying this phenomenon, pointed out, “fraud is a far more serious problem than most scientists recognise” (Bishop C., 2023. See also Judson H., 2004)²; Similarly, Oransky, who dealt with the phenomenon of retractions, observed how “researchers will do anything to publish papers in some journals, including even creating fake authors” since “[…] publishing papers in certain journals is the only way to earn grants, tenure, and promotions” (Oransky J., 2020, p. 142). Overall, it has become clear that, independently from the measures, indicators or metrics we decide to adopt, there is a method problem. As the well known Goodhart’s law states, “when a measure becomes a target, it ceases to be a good measure” (Goodhart C., 1975) – and therefore the current system can be considered neither a valid nor an appropriate way to assess research quality and “gaming the metrics” is at the heart of the game in itself.

In very recent years, however, we have seen the emergence of international initiatives led by academics which have focused the internal debate on the need to rethink evaluation, its principles and processes, and which have acquired significant political value. Initiatives such as the San Francisco Declaration on Research Assessment (DORA) of 2012, the Leiden Manifesto for research metrics of 2015 (Hicks D., Wouters, P., Waltman, L., et al, 2015), the Hong Kong Principles of 2020 (Moher D., Bouter L., Kleinert S., et al, 2020) and, more recently, the SCOPE Framework for Research Evaluation (INORMS, 2022) have recognised the need to improve the ways in which scientific research outputs are evaluated by funding agencies, academic institutions, and other parties. Moreover, they have provided recommendations to improve research evaluation by assessing the intrinsic merit of research and abandoning the inappropriate use of bibliometric indicators such as the Journal Impact Factor and the h-index. These initiatives have also identified principles on the basis of which the adoption of practices supporting research integrity should be encouraged, recalling the values and principles once shared by the scientific community, such as those described by Merton in his 1942 essay “The Normative Structure of Science”.

The above mentioned issues have taken on a global impact, and are at the centre of international debates and policies. As Section 3 of this article shows, The UNESCO Recommendation on Open Science (UNESCO, 2021) provides relevant inputs, and the G7 Ministries of Science have established a working group dedicated to Research assessment and incentives for Open Science (G7 Science Ministers, 2023; G7 Open Science Working Group (OSWG), 2023). A discussion on reforming research assessment is alive in China, too (Zhang L., Sivertsen G., 2020).

The European Commission and the Council have been producing and supporting the production of reports, communications and conclusions – all of which are oriented towards a better understanding and a re-conceptualisation of the research assessment system³. A crucial step in this process has been the publication of the report “Towards a reform of the research assessment system” in November 2021, at the end of a 9 month consultation process led by the European Commission (European Commission, Directorate-General for Research and Innovation, 2021). The report is based on two main assumptions: that the research process is changing, becoming less linear, more open, interdisciplinary and collaborative (to tackle global challenges), with a multiplicity of outputs, and is more and more based on “team science”; and that the traditional evaluation system is not suited to reflect this change. What is more, the report goes as far as asserting that the current research assessment system is the main barrier towards the adoption of Open Science practices, because research assessment is still based on the concept of rewarding publications only, preferably in “prestigious” venues, and focuses only on quantitative aspects, while collaborative processes and different types of results remain outside of its scope, and are therefore, in practice, hidden. A reform process, which increases the efficiency, impact, and social responsibility of research, is therefore needed.

What comes out is clearly shown in this slide from the Open Science activist Jon Tennant. There is a divergence between the path to a successful career and the adoption of Open Science practices, and the current assessment system is the general frame defining the social and technical barriers.

Fig. 1 – John Tennants’ slide in “Open Science is just science done right” (Tennant, J.,2018, p. 21)

The Council conclusions of June 10th, 2022 “Research assessment and implementation of Open Science” precisely address this contradiction and indicate some actions to reconcile these diverging trends (Competitiveness Council, 10 June 2022). The content of the conclusions reflects the main elements of the Agreement on Reforming Research Assessment. The next section examines its contents in detail.

2. The Agreement on Reforming Research Assessment and the centrality of quality

Published in July 2022, the Agreement on Reforming Research Assessment(ARRA) is  the result of a co-creation process which started in December 2021, led by the European Commission⁴.

The text of the ARRA that the signatories undertake to respect is a very short document of twelve pages containing the principles, commitments and timeframe for reforms, plus eight pages of  Annexes. The ARRA sets out the guidelines for a coalition of organisations willing to work together to implement such a reform.

In fact, signing the ARRA is the first step to enter the Coalition for Advancing Research Assessment (CoARA), which was launched at the end of 2022.

Fig. 2 – Signatories of the ARRA vs. CoARA membership

Specifically, CoARA aims to enable systemic reform based on common principles and to facilitate information exchange and mutual learning, especially through several Working Groups focused on specific topics – including peer review, open infrastructures and multilingualism – and through National Chapters whose activities are focused on the communication and uptake of the ARRA principles and commitments at the national level. Signatory organisations of the ARRA are invited to share with the community how their organisation has started the process of implementing the Core Commitments according to an Action Plan with defined milestones within one year of signing the Agreement (extended to 1,5 years for early signatories).

In January 2024, CoARA had approximately 600 members from 50 countries, representing Universities, research performing organisations and research funders, as well as Academies, learned societies, and associations of researchers. The actions that the signatories of the agreement commit to⁵  are laid out in the four core commitments which form the heart of the paradigm shift defined here.

2.1 Diversity

The first commitment is to “recognise the diversity of contributions to, and careers in, research in accordance with the needs and nature of the research” (ARRA, 2022. p. 4). It entails shifting the focus from (high impact) publications, so as to cover the entire spectrum of scientific research publications (Piwowar H., 2013). What is acknowledged here is that considering solely the count (or value) of papers, which nowadays have become a sort of more or less defined but certainly immutable snapshot of any kind of work, can yield an extremely partial judgement in terms of a person’s skills and possible professional developments – but also of a scientific idea. In this context, a scientific contribution is increasingly conceived as a living system that integrates many types of content (protocols, ethical discussions, data collection, data publication, results discussion, code, etc.) in an extremely changing and dynamic way over time, almost in a context of “rolling release”. The scope of this commitment is clear: “Changes in assessment practices should enable recognition of the broad diversity of:

• valuable contributions that researchers make to science and for the benefit of society, including diverse outputs beyond journal publications and irrespective of the language in which they are communicated;
• practices that contribute to robustness, openness, transparency, and the inclusiveness of research and the research process including: peer review, teamwork and collaboration;
• activities including teaching, leadership, supervision, training and mentoring” (ARRA, 2022. pp. 4-5).

Research objects which should be assessed include contributions, practices and activities, and the scope of the commitment is not to provide a detailed list of them. On the contrary: “ […] the aim is to allow organisations to broaden the spectrum of what they value in research, while acknowledging that this may vary across disciplines and that each individual researcher should not be expected to contribute to all activities at once.” (ARRA, 2022. p. 5) The first point is therefore about avoiding to identify and assess research quality – the content – depending on the label under which can be categorised, and to assess objects, processes and activities whose format and type may vary over time.

2.2 Qualitative evaluation

The second commitment requires to “base research assessment primarily on qualitative evaluation for which peer review is central, supported by responsible use of quantitative indicators.

Purpose: This commitment will enable the move towards research assessment criteria that focus primarily on quality, while recognising that responsible use of quantitative indicators can support assessment where meaningful and relevant, which is context dependent.” (ARRA, 2022. p. 5)

In this commitment, “qualitative evaluation” is introduced to define the type of evaluation to which the ARRA aspires, as opposed to quantitative evaluation. The adjective “qualitative” is tautologically defined in the Oxford English Dictionary as “relating to quality” (Oxford Learner’s Advanced Dictionary of Current English, 2000, p. 1035) and occurs seven times (plus seven more in the Annexes) in the ARRA, where it is defined as “based on peer-reviewing”, and contrasting with quantitative indicators. More importantly, in the definition, qualitative evaluation also means focusing primarily on “quality”. It is worth noticing that in the Agreement, the term “quality” occurs twenty times (plus another thirteen in the Annexes), and is defined as the first of the Principles for assessment criteria and processes.

“Quality” and “qualitative judgement” are at the core of the ARRA vision which states that “the assessment of research, researchers and research organisations recognises the diverse outputs, practices and activities that maximise the quality and impact of research. This requires basing assessment primarily on qualitative judgement, for which peer review is central, supported by responsible use of quantitative indicators.”(ARRA, 2022. p. 2. Emphasis is mine)

The centrality of peer review as an essential means of assessing the quality of research is enshrined in the second commitment, which defines peer review as the rigorous application of the scientific method.

Peer review is a form of evaluation that consists in the procedure by which an academic submits a text to the judgement of other academics (the so-called “peers”) who establish its validity; as a technical term, it corresponds to the specific evaluation process that precedes the publication of the work in a scientific journal, the acceptance of a presentation at a conference, or the allocation of funds by funding agencies. In essence, peer review is a means of “measuring” the quality of knowledge and it is there to guarantee that knowledge is true and trustworthy. In practice, it is one of the driving forces behind research funding: as a criterion for publishing a scientific result or not, funding a project or accepting a contribution to a conference, peer review influences recruitment and career progression in universities and research institutes, as well as public and private research funding.

Peer review is a distinctive feature of the modern academic system. In spite of it having no legal value, it is not only accepted as part of the profession by generations of scholars, but is also considered to be the element that establishes and characterises scientific knowledge. When we buy a medicine, we choose it among those that have passed evaluation by specialists; and when we do research, we put our trust, to varying degrees, in sources that have already been published, and therefore validated, because they have passed the filter of the scientific community. And these sources are only deemed to be results, and hence publishable, if they meet current professional standards. Therefore, peer review contributes both to the construction of scientific authority, and to the distribution of scientific knowledge (Di Donato F., 2007).

Although the birth of peer reviewing practices is traced back to the Royal Society of Edinburgh in 1731, strikingly enough, the formalisation of peer review only took place in recent times, in the transition from ‘refereeing’ to actual ‘peer review’, which took place in the 70s of the last century (Baldwin M., 2018). “Over time, peer review adopted different forms to respond to priorities and needs of the scientific community. For instance, peer review was initially single-blind, meaning that the authors do not know the identity of the reviewers, while reviewers know the identity of the authors. In the 70s, sociology journals started to adopt double-blind peer review, whereby authors and reviewers do not know each other’s identity” (Seeber M., Klemenčič M., Meoli M., Sin C., 2023, p. 121). The refereeing process became closed in very recent times. A transition that was far from free of criticism, as Baldwin shows (Baldwin M., 2018; see also Csiszar A., 2016. Seeber, M. 2022).

Despite the fact that peer review dynamics and practices obey different and specific logics depending on the discipline, peer review is based on a single general principle: it is through publications that one becomes comparable to all the others (Di Donato F., 2007).

This does not mean that peer review has not been and is not exempt from criticism (Di Donato F., 2007; Ziman J., 1996; Rowland F., 2002.; McKiernan G., 2003; Williamson A., 2003; Poeschl U., 2004; Grivell L., 2006). A summary list of the problems encountered may include: the high cost of the process; biassed judgements (often subjective); certain forms of abuse and misconduct (by evaluators), such as prejudicing competitors or plagiarism; the poor scientific value of the process (for example, it has often proved to be an inadequate tool to find errors); the slowdown of scientific communication; and opacity (in the case of “blind” or “double blind” peer review). But eventually, many an author has extended Churchill’s famous quote about democracy to peer review, describing it as “the worst of all systems […] to the exclusion of any other” (Dall’Aglio P., 2006, p. 8).

In fact, peer review in the ARRA is recognised as “the most robust method known for assessing quality and has the advantage that it is in the hands of the research community. It is important that peer review processes are designed to meet the fundamental principles of rigor and transparency⁶: expert assessment, transparency, impartiality, appropriateness, confidentiality, integrity and ethical considerations, gender, equality and diversity” (ARRA, 2022, p. 5).

The above mentioned characteristics, i.e. expert assessment, transparency, impartiality, appropriateness, confidentiality, integrity and ethical considerations, gender, equality and diversity are defined in the Global Research Council document cited in the text and are fundamental features of peer review. So not all forms of peer review are fine. We will return to this point in the following sections, where we will discuss which forms of peer review actually guarantee the quality of research.

To address the biases and imperfections to which any method is prone, the research community keeps a permanent discussion of peer reviewing practices alive, which also includes the possibility of exploring and developing revised and new criteria, as well as tools and processes for assessing research quality. Investing in research on research (INORMS, 2022) is thus essential to avoid that “a measure becomes a target”, again. In order to encourage researchers to perform peer reviewing, a very time-consuming activity often scarcely recognised in assessment processes, the ARRA states that researchers should be recognised for this activity, and that their contributions as reviewers of peers’ work should be valued as a fundamental part of their research activity, and an essential part in the construction of scientific knowledge.

Research and a permanent open debate on peer review practices are necessary to keep the mechanism efficient and alive.

2.3. Responsible metrics

According to the third commitment of the ARRA, the inappropriate use of indicators such as JIF and h-index must be abandoned, as their misuse has proven to be harmful and toxic for the communication of scientific research. What is meant by inappropriate use is clarified here, i.e. “moving away from using metrics like the Journal Impact Factor (JIF), Article Influence Score (AIS) and h-index as proxies for quality and impact” (ARRA, p. 6). Publication- and author-based metrics, which rely on publication venue, format or language, cannot be trusted as a means to assess research quality and/or impact. Furthermore, they are neither inclusive nor global, focusing mainly on English as a lingua franca, and on European and United States research.

Last but not least, the fourth commitment advocates for not using university rankings as a means to assess research or researchers’ quality. This “will help the research community and research organisations regain the autonomy to shape assessment practices, rather than having to abide by criteria and methodologies set by external commercial companies. This could include retaining control over ranking methodologies and data” (ARRA, p. 6).

Both the third and the fourth commitments include a negative definition of quality: publication- and author-based metrics, and university rankings do not guarantee research quality. But what is research quality then?

In the Oxford Advanced Learner’s English Dictionary “quality” is defined as: “1. The standard of sth when it is compared to other things like it; how good or bad sth is. 2. A high standard” (The Oxford Learner’s Advanced Dictionary of Current English,  2000. p. 1036), while in the Collins Dictionary of the English Language it is defined as “the degree or standard of excellence, esp a high standard”⁷. In the latter sense, quality is defined through another concept, excellence. However, the term “excellence” does not appear in the ARRA, whereas the concept of “quality” is at its core. The term “research quality” appears in the Glossary, where it is referred to the definition included in the Principles for assessment criteria and processes (ARRA, 2022, p. 15 and p. 3).

Quality is the first principle on which assessment must be based: “Focus research assessment criteria on quality. Reward the originality of ideas, the professional research conduct, and results beyond the state-of-the-art. Reward a variety of research missions, ranging from basic and frontier research to applied research. Quality implies that research is carried out through transparent research processes and methodologies and through research management allowing systematic re-use of previous results. Openness of research, and results that are verifiable and reproducible where applicable, strongly contribute to quality. Openness corresponds to early knowledge and data sharing, as well as open collaboration including societal engagement where appropriate. Assessment should rely on qualitative judgement for which peer review is central, supported by responsibly used quantitative indicators where appropriate.” (ARRA, 2022. p. 3)

The proposed definition of quality is a procedural one, which affects the methods more than the content. While “originality” and “results beyond the state of the art” are only mentioned but not defined in the text, considerable space is given to the definition of professional research conduct, which is based on the transparency of both processes and methods and on the FAIRness of research management, which means managing research data and processes so as to have findable, accessible, interoperable and reusable data⁸.

In the definition of the ARRA, openness is a necessary condition for quality and a means to enable collaboration at various levels, within the scientific community and society as a whole. In the same section, where the Principles for assessment criteria and processes are outlined, further principles are complementary to research quality. The first is impact, which “implies effects of a scientific, technological, economic and/or societal nature that may develop in the short, medium or long-term, and that vary according to disciplines and research types (e.g. basic and frontier research vs. applied research)” (ARRA, pp. 3-4)⁹. Others include diversity, i.e. recognizing “the diversity of research activities and practices, with a diversity of outputs, and reward early sharing and open collaboration”, inclusiveness, i.e. “use assessment criteria and processes that respect the variety of scientific disciplines, research types (e.g. basic and frontier research vs. applied research), as well as research career stages (e.g. early career researchers vs. senior researchers), and that acknowledge multi-, inter-, and trans-disciplinary as well as inter-sectoral approaches, when applicable. Research assessment should be conducted commensurately to the specific nature of scientific disciplines, research missions or other scientific endeavours”, and collaboration (ARRA, pp. 3-4).

Once again, a link is established between quality as a principle, and “qualitative assessment” based on peer review, which is the first step through which research and knowledge are validated. This is a very important point. It is relevant to note that the reform entails a major change of perspective, which implies a sort of counter-revolution. For years, the adoption of quantitative indicators and citation indexes has been sold to the scientific community as a necessary step towards objective indicators which would be free from the arbitrary subjective assessment of peers, who, as humans, have demonstrated being incapable of being “objective”. The rationale is that peer review is random and subjective, and subject to various biases in both its closed – double-blind – and open forms, whereas quantitative metrics are a response to this arbitrary and aleatory “subjectivity”. And in public debates on the ongoing reform of research assessment, very often this argument is raised.

But it is nowadays clear that quantitative indicators are far from objectively assessing research quality. Numbers were synonymous with (greater) objectivity. However, this has proven not to be the case, and it is clear that numbers need to be put into context to have a significant meaning. Moreover, bibliometric parameters merely aggregate subjective judgements which are, in the case of the JIF, calculated on the journal or, in the case of the h-index, on the author’s production, and are totally independent from the content. Nevertheless, the fact that quantitative metrics have proven incapable of assessing research quality does not in itself imply that qualitative judgement could or should be able to do so.

To unravel this problem, it is important to return to the definition of research quality, and its connection to the concept of qualitative evaluation. This will be done through a twofold strategy.

First, we will analyse the relationship between the definition of quality research proposed in the ARRA and the definitions of Open Science. It is the definition of the Agreement itself which indicates this equivalence, starting with the attention paid to the multiplicity and diversity of outputs produced in the course of scientific research. Next, to ground and support the first strategy, an analysis of the concept of quality from a philosophical point of view is proposed in sections 4, 5 and 6.

3. The quest for quality

We have seen that  there is a close relationship between quality research and Open Science both in the Agreement and in its founding documents. This connection is made explicit through the definition of quality. But what exactly is meant by Open Science? There are various definitions, and it is important to clarify which is the vision indicated by the Agreement.
Open Science encompasses diverse assumptions and activities and is often defined as an umbrella – or rather a mushroom – term, “in the sense that it consists of many visible practical activities (often digital) and that it also concerns, below its surface, many underlying institutional issues such as integrity, infrastructure and evaluation” (Rafols I., Meijer I., Molas-Gallart J., 2024, p. 4).

Fig. 3 – Eva Mèndez’s Open science mushroom (Mèndez E., 2021)

Fecher and Friesike have identified five schools of thought, based on their focus and different interpretations of the term. These include “the infrastructure school (which is concerned with the technological architecture), the public school (which is concerned with the accessibility of knowledge creation), the measurement school (which is concerned with alternative impact measurement), the democratic school (which is concerned with access to knowledge) and the pragmatic school (which is concerned with collaborative research)” (Fecher, Friesike, 2013, p. 3).

The first mention of the term dates back to Chubin’s 1985 essay “Open Science and Closed Science: Tradeoffs in a Democracy” in which he takes up the principles set out by Merton in his 1942 essay “The Normative Structure of Science”, examining their function and recognising a tension between these values, in particular between communalism, and the various forces involved in the development of science, for whom the absence of sharing has its own benefits.

This distinction and tension was taken up in the early 2000s by Paul David (David P., 2000 and David P., 2014). David identified two opposing models according to which scientific and technological research activities are organised at the macro-level in modern economies, and he is in fact the first to give a definition of Open Science. The first model – the “Open Science” one – is based on the principle of collaboration. According to David, the “Republic of Open Science” is not a recent novelty, but a fundamental trait of the modern scientific revolution that has its roots in the Middle Ages. The emergence of the institutions and organisational features of Open Science is a legacy of feudalism to capitalism – but it is in the 1980s – 1990s that international organisations establish principles and develop guidelines to protect researchers’ easy and broad access to scientific data and high-quality information generated by publicly funded bodies, and that recommendations begin to circulate through implementations. When defining the first model, David reconstructs the main events in this regard (and the history of the Open Access movement). This model has remained and stays alive thanks to scientists who, individually and as a collective, act with “good scientific conduct conjoined with good technique” (David P., 2014, p. 22) and who aim to transmit the scientific ethos to future scientists who see them as mentors. But he also recognises that Open Science culture and practices are exposed and vulnerable – a vulnerability that has become evident over the past three decades. David observes how the idea of science as a common good seems natural but is at the same time controversial, and how in reality this model is fragile. The institutional infrastructures of Open Science seem to have a measure of plasticity, which however has its limitations.

The second model is that of commercially-oriented Research & Development (R&D), based on proprietary information. The cornerstone of this second model is competition and “new economy” and “intellectual capitalism” are its key words. According to this model, the free and open circulation of data and information is perceived as a threat. The competition model is winning, because competition is seen as synonymous with innovation. In particular, intellectual property rights have been strengthened both domestically and internationally and restrictions on access to scientific data (such as patents and tighter copyright-based restrictions) have increased. A new institutionalised system of fruitful interactions with proprietary activities and market-oriented R&D leading away from Open Science has developed. It is a historical process that has put Open Science under attack.

David believes that these conditions have exacerbated the decline in the effectiveness of the scientific research system as a whole, and that the State must support the weak model: “The main lessons and implications for the future vitality of Open Science institutions that can be drawn from the foregoing selective sketch of the experiences of the past 15 years is that research communities of this kind possess not only the technical and organizational ingenuity, but also the organizational capabilities to apply them to sustain their culture and protect their characteristically efficient collaborative modes of conducting socially valuable exploratory, fundamental research. To go on doing this, however, they must be adequately supported by external, public and charitable sources of funding, and nurtured by “top down” public policy actions that reinforce and help them reproduce the ethos of open science in successive generations of university-trained researchers.” (David P., 2014, p.3)

In recent years – and in particular after the COVID-19 pandemic -, research programs conducted in accordance with the principles of Open Science are increasingly widely recognised as fundamental contributions to respond to societal needs, support economic growth and improve collective well-being. A growing number of funders and decision-makers have now realised that they need to follow David’s recommendation and financially support the Open Science model, that is to say the weakest model, which is also the most focused on the quality of science. This shift is based on the common recognition that this model is the one that, through its transparency paradigm, ensures research integrity and reusability are achieved and maintained. It is indeed obvious that if all is transparent, it is harder to fraud and the solidity of the scientific process can be verified.

That Open Science is an old idea may be a shared assumption, but only in recent times did we reach unified definitions. The 2015 OECD report “Making Open Science a reality” identified Open Science and its practices as an important driver of innovation and economic development. The same view is supported in the European Commission’s book “Open Innovation, Open Science, Open to the world” of the same year, where “Open Science represents a new approach to the scientific process based on cooperative work and new ways of diffusing knowledge by using digital technologies and new collaborative tools. The idea captures a systemic change to the way science and research have been carried out for the last fifty years: shifting from the standard practices of publishing research results in scientific publications towards sharing and using all available knowledge at an earlier stage in the research process.

Open Science is to science what Web 2.0 was to social and economic transactions: allowing end users to be producers of ideas, relations and services and in doing so enabling new working models, new social relationships and leading to a new modus operandi for science. Open Science is as important and disruptive a shift as e-commerce has been for retail. Just like e-commerce, it affects the whole ‘business cycle’ of doing science and research – from the selection of research subjects, to the carrying out of research and to its use and re-use – as well as all the actors and actions involved up front (e.g. universities) or down the line (e.g. publishers)”. (European Commission, 2015, p. 37)

As Sabina Leonelli notes (Leonelli S., 2023, pp.17-18), the definition of OS provided by the European Commission in 2015, illustrates how the idea of ‘openness as sharing’ informs the construction of an OS ecosystem. The Commission’s emphasis is first and foremost on designing, developing and promoting tools, and infrastructures to make data findable, accessible, interoperable and reusable. The most relevant example is the European Open Science Cloud (EOSC), an ecosystem of research infrastructures based on open procedures and standards, which will enable the pooling of data, resources and knowledge. This project is currently being implemented in specific Horizon Europe programmes.

More generally, Leonelli identifies in these definitions an object-oriented approach, which places transparency as the starting point and considers it as the enabling factor for sharing. In this vision “there is a specific direction of travel […], a choreography and prioritization of specific values as the best path towards openness, which arguably underpins many mainstream efforts to implement OS …. First, one needs to achieve transparency. This is often presented as the most immediate and urgent preoccupation for OS: the push to put everything online, thereby making it accessible to a wide variety of potential users. Second, one worries about the quality of what is being shared. Enter criteria and mechanisms for assessing the reliability of outputs and methods circulated on the web, ideally accompanied by sanctions for those who do not abide by such rules. […] Third, there is inclusion, intended as the opportunity for anybody with relevant interests and expertise to engage with and participate in research, and thus to utilize – and help scrutinize – the resources being shared. In most OS policy documents, the end goal and ultimate outcome of improving the transparency and quality (often cashed out as reproducibility) of research is an inclusive and fair research process, which fosters scientific engagement while also helping to discriminate between good and bad contributions to knowledge”. (Leonelli S., 2023, p. 20). Instead, the aim of OS is to create the conditions for quality research.

Vice versa, Leonelli proposes a model of ‘openness as judicious connection’, which starts from inclusion and arrives at transparency through quality: “To date, some parts of the OS movement – particularly its institutionalized, top-down incarnations – have paid too much attention to designing procedures and technologies for sharing, and this has come at the expense of strategies, training and procedures to assess who is included and excluded from such apparatus, understand why and with which implications, and mitigate eventual instances of epistemic injustice. To correct this trend, I propose to invert this conceptualization of the direction of travel for OS implementation […]. Instead, the implementation of OS needs to start from consideration of what it may take to make research more inclusive, diverse and just – rather than expecting such an outcome to naturally follow from the ‘right’ choice of software, infrastructures, standards, publishing platforms, or whatever other technological or institutional fix is being devised to facilitate access to resources (…). It is only through explicit consideration of the demarcation strategies presupposed and supported by OS systems that research quality can be reliably evaluated, and transparency pursued in ways that are informative, discerning and suited to the research context in question”. (Leonelli S., 2023, pp. 42-43) This inversion is represented in the following figure.

Fig. 4 – Core values in OS implementation: the proposed direction of travel (Leonelli S., 2023, p. 43)

The UNESCO Recommendation on Open Science, adopted on 23 November 2021 as part of the 41st session of the UNESCO General Conference, provides a different and more comprehensive definition which seems in line with Leonelli’s model.

The UNESCO Recommendation is the first normative document on Open Science, which was reached after a consultation process requested and initiated by the Member States two years earlier¹⁰. The 2021 Recommendation is therefore an important response to the needs of a community that aspires to greater and better cooperation, sharing and inclusion in scientific research. Moreover, scientific research must be conducted according to precise standards and responsible and ethical principles, which the Recommendation undertakes to specify and emphasise in the text¹¹.

The document opens with a preamble that recognises the most significant challenges, changes and problems shaping the environment, the economy and contemporary society in general, as well as the essential role of science, technology and innovation in addressing and finding solutions to these challenges. The urgent tone of the document is therefore immediately established, starting from the introductory text. The macro level definition of its context of origin and of the overall vision provides a deep understanding of the aspiration to use scientific practices that are as open as possible at international level.

In fact, from the outset, the fundamental principles and intrinsic values of Open Science are identified, including the availability and accessibility of knowledge, collaboration, as well as transparency and inclusion in all its forms, and their relevance for the advancement of science, technology, the economy and civil society and for the benefit of individuals and the planet is highlighted. This premise allows us to investigate the complex concept of Open Science, considering an overall picture which already identifies the underlying preconditions and pillars that guide it.

“Considering that more open, transparent, collaborative and inclusive scientific practices, coupled with more accessible and verifiable scientific knowledge subject to scrutiny and criticism, is a more efficient enterprise that improves the quality, reproducibility and impact of science, and thereby the reliability of the evidence needed for robust decision-making and policy and increased trust in science” (UNESCO, 2021, p. 2).

Several concepts are anticipated here: openness, transparency, collaboration, inclusivity. As well as the possibility of scrutiny and criticism. These elements are essential to ensure quality, reproducibility and impact, all of which make science trustworthy. In the UNESCO Recommendation all these elements are present and are placed on an equal footing.

Open Science therefore reveals itself as made up of principles and values and underpins all phases of the scientific process, not only the production of results, for it has a particular interest in adopting practices which are unrestricted as possible, with the aim to significantly increase the quality of research.

The Recommendation reiterates the inclusive definition of “science” already contained in the 2017 UNESCO Recommendation on Science and Scientific Researchers (UNESCO, 2017). This broad definition is particularly relevant since it identifies, with regard to the scientific ecosystem and the evaluation of research, the tension that exists between collaboration and competition. This tension, in fact, represents and reflects two complementary and concurrent fundamental aspects of the scientific practices adopted by the community, which should be balanced within the scientific landscape.

“As for the 2017 UNESCO Recommendation on Science and Scientific Researchers, the term ‘science’ signifies the enterprise whereby humankind, acting individually or in small or large groups, makes an organized attempt, in cooperation and in competition, by means of the objective study of observed phenomena and its validation through sharing of findings and data and through peer review, to discover and master the chain of causalities, relations or interactions; brings together in a coordinated form subsystems of knowledge by means of systematic reflection and conceptualization; and thereby furnishes itself with the opportunity of using, to its own advantage, understanding of the processes and phenomena occurring in nature and society” (UNESCO, 2021, p. 6).

However, the definition of Open Science has its focus on collaboration, which is an essential characteristic of it:

“For the purpose of this Recommendation, Open Science is defined as an inclusive construct that combines various movements and practices aiming to make multilingual scientific knowledge openly available, accessible and reusable for everyone, to increase scientific collaborations and sharing of information for the benefits of science and society, and to open the processes of scientific knowledge creation, evaluation and communication to societal actors beyond the traditional scientific community. It comprises all scientific disciplines and aspects of scholarly practices, including basic and applied sciences, natural and social sciences and the humanities, and it builds on the following key pillars: open scientific knowledge, open science infrastructures, science communication, open engagement of societal actors and open dialogue with other knowledge systems” (UNESCO, 2021, p. 7, emphasis mine).

The movements referred to in the UNESCO definition are varied and diverse, but they share the principles and values mentioned above, and their practices are based on these. They include the movement which gave birth to the Internet in the 1960-70s, and then to the design and implementation of the World Wide Web (Baran P., 1962; Cerf V.G., Kahn R.E., 1972; Kahn R.E., 1974; Berners-Lee T., 2002), that is to say transparent, sustainable and flexible open systems, based on an open network architecture with distributed layers. The design and implementation of the Internet occurred in a community of computer scientists within which the project was developed according to the proper scientific method and collectively (Mattelart A., 2001); an essential communication tool for scientists who are part of the “republic” was the Request For Comments (RFC), a series of notes inaugurated in 1969 as a tool through which researchers quickly and informally exchanged notes, observations, ideas, which are discussed, expanded, defined and then put into practice. These scientists made the principles of free thought, critical method, and sharing and cooperation with their peers an organised work practice – establishing open refereeing as a standard methodology. The practice of RFCs generated a beneficial effect, both by providing the rules of communication to the community, and by expanding the number of those who collaborate on the innovations introduced. In practice, open (copyright-free and free) access to RFCs fostered the growth of the Internet because it allowed for specifications to be discussed, modified and adopted both in universities and research centres, and in the private sector (Di Donato F., 2009. pp. 91-101).

The Free Libre Software movement represents a model for the development of information technologies, and is based on the idea of free access to both the source code and executable code of a program – and on the adoption of licences under which the author gives away some of his economic rights over a work (copyleft) to allow for its circulation, reworking and reuse. This model is very effective in practice, as demonstrated by the case of Linux, the operating system created in the early 1990s by the Finnish student Linus Torvalds which became an example of collaborative open research (Gonzalez-Barahona J., 2012). The Open content movement translated the logic of copyleft to general content (Lessig L., 1999; Lessig L., 2001; Lessig L., 2004) thanks to the design and development of the Creative Commons licenses, which are now widely applied to scientific content.

The Open Access movement brought to the attention the need for open scholarly content and tools (Budapest Open Access Initiative, 2002; Bethesda Statement on Open Access publishing, 2003; Berlin Declaration on Open Access to Knowledge in the Sciences, 2004). “The future Web” is stated in the Berlin Declaration “has to be sustainable, interactive, and transparent. Content and software tools must be openly accessible and compatible”. One of the first steps, which proved decisive in shaping the Open Access model, was the birth of ArXiv, the open archive founded by Paul Ginsparg at Cornell in 1991 and dedicated to hosting preprints of high-energy physicists. Ginsparg interpreted the community’s need to communicate quickly and efficiently which conflicted with the publishing mechanism. In fact, traditional paper journals were slow, expensive and hard to access. A few years later, in 1999, the Santa Fe Conventions initiated the definition of OAI-PMH (2001), the Open Archive Initiative metadata harvesting protocol, based on the Dublin Core open standard. The protocol allows to link archives and journals in an interoperable network. This was followed by the Budapest OA Initiative, also in 2001, and the Berlin Declaration in 2003, which marked further significant steps in terms of awareness, resulting in a formal commitment, on behalf of funding bodies to support this model.

Doing Open Science requires a change of approach in scientific culture, assuming participation and engagement as pillars of the methodology itself. Here, the collaborative nature of science is postulated, and openness is understood as connection, where epistemic and ethical considerations are intertwined.

“The core values of open science stem from the rights-based, ethical, epistemological, economic, legal, political, social, multi-stakeholder and technological implications of opening science to society and broadening the principles of openness to the whole cycle of scientific research” (UNESCO 2021, p. 17) and are listed with the principles of Open Science in the following image.

Fig. 5 – Open Science Values and Principles (UNESCO, 2021, p. 19).

The first values are quality and integrity: “open science should respect academic freedom and human rights and support high-quality research by bringing together multiple sources of knowledge and making research methods and outputs widely available for rigorous review and scrutiny, and transparent evaluation processes” (UNESCO, 2021, p. 17). Here again research quality is defined as openly scrutinised methods and outputs. Quality and integrity are then complemented by other values, including Collective benefit, Equity and fairness and Diversity and Inclusiveness, and which can be partially mapped into the Principles for general assessment criteria outlined in the ARRA (ARRA, 2022, p. 3). Values are also complemented by principles, providing a framework for enabling conditions and practices within which the above values are upheld. They include: Transparency, scrutiny, critique and reproducibility; Equality of opportunities; Responsibility, respect and accountability; Collaboration, participation and inclusion; Flexibility and f. Sustainability (UNESCO, 2021, pp. 18-19).

Actions to be promoted regarding research evaluation are included in the Recommendation Action n. V: Fostering a culture of open science and aligning incentives for open science, according to which it is necessary to “change the current research culture and to recognize researchers for sharing, collaborating and engaging with other researchers and society, and to support, in particular, early-career researchers in particular to drive this cultural change”. Just like in the ARRA, research is conceived as a process and evaluation systems should take into account the wide breadth of scientific missions, activities and scientific outputs “including high-quality FAIR data and metadata, well-documented and reusable software, protocols and workflows, machine-readable summaries of findings, and teaching, outreach and engagement of societal actors within the knowledge creation environment” (UNESCO, 2021. p. 28).

Moreover, evaluation and assessment systems should: i) be built on previous initiatives such as DORA, value all relevant research activities; ii) make participation in the research process more inclusive of society, respecting the diversity of disciplines, which requires different approaches to Open Science; and iii) promote “high-quality and responsible research […] to reduce scientific misconduct, including the fabrication and falsification of results, violation of scientific ethical norms, and plagiarism” (UNESCO, 2021. p. 29). The definition of the UNESCO Recommendation, which places the emphasis on the movements and their practices, seems at least partly in line with the paradigm shift proposed by Leonelli, as opposed to the transparency-centred one.

As Pierre Mounier noted, Open Science is “an ecosystem producing a ‘milieu’ of knowledge and supported by a community which shares value”, and the challenge today is to connect the building blocks of the ecosystem. This involves facing technological challenges, socio-cultural challenges, and organisational challenges (Mounier P., 2022. The specific quote is page 1).

Notably, the UNESCO recommendation underlines two points. First, “it highlighted the diversity of activities related to OS, particularly by giving more prominence than previous definitions to other issues such as open engagement and dialogue with non-academics and marginalized actors (process-oriented practices)[…]. Second, it made explicit the values and principles associated with OS.” (Rafols, I., Molas-Gallart, J., & Meijer, I., 2024, emphasis mine).

These values and principles are reflected in the Agreement, which was published one year after the UNESCO Recommendation. In fact, Open Science is a necessary precondition of quality research, but a definition of quality is still lacking. Before returning to the Agreement, we must return to the question: what is quality? The answer proposed in the conclusions is the result of a philosophical discussion of this concept. In fact, it has been noticed that, although research quality is at the core of the current debate and reform, there is no general agreement on the meaning of quality. While all seem and pretend to know what it is, at the same time they all mean something else¹².

4. Science and quality

“Quality … you know what it is, yet you don’t know what it is. But that’s self-contradictory. But some things are better than others, that is, they have more quality. But when you try to say what the quality is, apart from the things that have it, it all goes poof! There’s nothing more to talk about. But if you can’t say what Quality is, how do you know what it is, or how do you know that it even exists? If no one knows what it is, then for all practical purposes it doesn’t exist at all. But for all practical purposes it really does exist. What else are the grades based on? Why else would people pay fortunes for some things and throw others in the trash pile? Obviously some things are better than others … but what’s the ‘betterness’? … So round and round you go spinning mental wheels and nowhere finding any place to get traction.” (Pirsig R.M., 1974, p. 188)

That quality is easy to recognise but difficult to define seems to be a common problem. The quote is from Robert M. Pirsig’s first novel. He worked on searching for a definition of quality for over thirty-five years, starting from The Zen and the Art of Motorcycle Maintenance. In his 1974 novel Pirsig begins a philosophical research on quality which originates from the paradox described in the above quotation, i.e. the fact that, although we are all able to recognise quality, it is not easy to define it.

The novel uses a rhetorical device. The narrator, Robert, sets off on a motorcycle trip with his son. He has just undergone compulsory medical treatment with electroshock, and during the journey, he has a new and different contact with reality and absorbs the sensations of the ride (the surrounding nature, the motorbike, and the relationship with his son and friends). And then there is Phaedrus, his other and former self before the electroshock, whose story the protagonist tells and discovers during his philosophical and personal motorbike journey through the United States. The story told in the novel, however, is not an artifice, it is true; and the journey helps him to remember his life before electroshock. Memories resurface, and with them the motive that drove his former self, Phaedrus, to madness, which originated with the question of what quality is.

In addressing the problem of defining quality, Pirsig applies a twofold strategy. On the one hand, he puts into practice a practical experiment with a group of students in which he decides to abolish grades and to let the students assess the quality of their essays by themselves. This is the “pars destruens” of the reasoning. On the other hand, Pirsig applies the correct approach (the scientific method) to a practical problem: the maintenance of his motorcycle. This is the “pars construens” of his discourse, in which the phases and elements of the scientific Galilean method are analysed and a revisitation of Western scientific thought – that Pirsig aspires to reconnect to the Eastern one – is proposed. It is worth reconstructing both the arguments he develops in order to arrive at a preliminary definition of quality.

1. The pars destruens is described both in The Zen and the Art of Motorcycle Maintenance and in the essay Subjects, Objects, Data, and Values written twenty-five years later (Pirsig R.M., 1999).

As a Professor of Rhetorics, in Bozeman, Pirsig has a contract with the Montana State government to teach communication, and this is what leads him to the problem of defining quality. At the beginning of the academic year, he asks his students to define quality, and no one succeeds in doing so. He therefore decides to do an experiment: he gives a group of essays to several students and asks each student to evaluate them based on their own criterion of quality. He then asks students to judge a few papers day after day – until they feel they know what quality is. In parallel, he decides to abolish grades for one term in order “to provide [them] an environment in which that mule can turn into a free man.” (Pirsig, R.M., 1974, p. 202)
In the beginning, the students, not knowing how their work would be evaluated, give their best, and demonstrate that even if they do not know what quality is (they are not capable of defining it), they are able to recognise it. They have not been asked to say in any conceptual way what the quality of the object is, but they understand that when you see it, you know it.

At the end of the experiment, Pirsig asks the students to provide their judgement in the form of a report; what he finds out is that the final judgement reflects their starting level (he did not change the system). As expected, the relative rankings of the students are correlated with each other and with those of the teacher, meaning that the experiment reflects the existing assessment system. The transformation of a mule into a free man hoped for by the Professor of Rhetoric does not happen, because the students have shown that they are not capable of expressing an autonomous judgement, therefore replicating the existing one.

“As a result of his experiments he concluded that imitation was a real evil that had to be broken before real rhetoric teaching could begin. This imitation seemed to be an external compulsion. Little children didn’t have it. It seemed to come later on, possibly as a result of school itself. That sounded right, and the more he thought about it the more right it sounded. Schools teach you to imitate. If you don’t imitate what the teacher wants you get a bad grade. Here, in college, it was more sophisticated, of course; you were supposed to imitate the teacher in such a way as to convince the teacher you were not imitating, but taking the essence of the instruction and going ahead with it on your own. That got you A’s. Originality on the other hand could get you anything…from A to F. The whole grading system cautioned against it.” (Pirsig, R.M., 1974, p. 195)

Pirsig thus observes that students imitate their teachers – and that this mechanism stems from education and schooling. The same phenomenon is observed by the physicist Carlo Bernardini, professor of Physics at Padua University, when he curates and organises a series of Laboratories of Scientific Knowledge in a nursery school near Florence in Italy, between 1982 and 1987. The aim is to train children in approaching science and to introduce them to scientific methods from an early age (Sgobino D., Barbetti S. (eds), 2013).

Bernardini observed that children are particularly willing to use inductive forms of reasoning, which fail to develop adequately due to resistance from the school and family community (Bernardini C., 2009, p. 7). In the Laboratories, Bernardini implements a “trial and error” methodology in which the rules have to be discovered, and are the point of arrival. The importance of inductive rule discovery is fundamental. Behaviour refers to the forms of interaction between individuals which strengthen the trustworthiness of results through the verification process known as “intersubjectivity”. Therefore, a model for training new teachers should trigger individual and collective processes of research and clarification through observation, experimentation and collective discussion, to foster autonomy of thought, and not to penalise mistakes. Bernardini also observed the importance of the methodological exercise consisting in the practice of building efficient mental representations. It is a skill which has more to do with decision-making than mnemonics. Such a model is not pedagogical, but a scientific, rational methodology.

Furthermore, the professor observed that children between the ages of three and five had a much higher level of non-specific scientific knowledge than adults, for three reasons: 1. because they were capable of changing their minds when faced with the evidence of the facts; 2. because they were willing to question themselves; 3. and because they had no interest in altering the experiments. Conversely, adults present a kind of resistance to these forms of rationality, lost due to the current educational system.

Pirsig reaches the same conclusion with his experiment. In addition, Pirsig underlines another aspect, and that is the tension between originality on the one hand, and the need to build knowledge by building on previous results on the other. While originality is a characteristic of research quality, the entire system is built assuming that novel theories – especially paradigm breaking ones – need sufficient space and time to be discussed and validated by experts. And these only become scientific theories when the scientific community reaches a general consensus on them. Michael Polanyi identifies this tension in the essay The Republic of Science, in which he defines the “current motivational standards” of scientists. These are professional standards, “standards of scientific merit accepted by the scientific community” which are defined on the basis of three criteria: the first is plausibility, which means consistency with the entire knowledge system; the second is scientific value, i.e. accuracy, systematic importance, intrinsic interest of its subject. The third is originality, i.e. the “degree of surprise” that the communication of a result raises in the scientific community. Polanyi observes how plausibility and scientific value tend to reinforce conformity – while originality encourages dissent, and that it is this internal tension which drives and motivates scientific work (Polanyi M., 1962).

However, as noted, this is the “pars destruens” of Pirsig’s strategy, and the experiment ends there.

2. In the pars costruens of his argumentation, Pirsig revisits a history of philosophical thought and outlines a treatise on the scientific method, which has at its centre the problem of what quality is. This problem is reformulated as a question about the rational foundations of communication, and it is applied to a concrete case: the art of maintenance of a motorcycle.

“A motorcycle functions entirely in accordance with the laws of reason, and a study of the art of motorcycle maintenance is really a miniature study of the art of rationality itself.” (Pirsig, R.M., 1974, p. 97). The motorcycle therefore enables the reader to understand the application of the scientific method.

As a starting point, it is essential, when approaching a problem, to understand the rational idea and to work on the concepts. Pirsig assumes that putting concepts in order means establishing a hierarchical structure that produces a system. The motorbike is an example of an actual system which is used by the author as an illustration of the application of the method defined in other – actually far more philosophical – parts of the text. Tracing the history of Western thought from Socrates to Plato, through Galileo, Kant, Poincaré and Einstein, Pirsig discusses the Galilean scientific method, which he applies to ideas. This is a central point. In fact, the way forward in defining quality starts with a correct definition of the ways to arrive at this definition.

The correct approach is described in the text. The way to break through hierarchies or structures of thought – of the system – is through logic. Through inductive logic, one starts from observation to arrive at general conclusions; deductive logic follows the reverse path and starts from general knowledge to predict an observation. “Solution of problems too complicated for common sense to solve is achieved by long strings of mixed inductive and deductive inferences that weave back and forth between the observed machine and the mental hierarchy of the machine found in the manuals. The correct program for this interweaving is formalised as the scientific method” (Pirsig, R.M., 1974, p. 106).

The method Pirsig describes is the same as that taught to students and then applied at all grade levels, from primary school to university. When facing motorcycle maintenance, as with any scientific problem, one follows a work plan and keeps track of the steps.

“For this you keep a lab notebook. Everything gets written down, formally, so that you know at all times where you are, where you’ve been, where you’re going and where you want to get. In scientific work and electronics technology this is necessary because otherwise the problems get so complex you get lost in them and confused and forget what you know and what you don’t know and have to give up. In cycle maintenance things are not that involved, but when confusion starts it’s a good idea to hold it down by making everything formal and exact. Sometimes just the act of writing down the problems straightens out your head as to what they really are” (Pirsig, R.M., 1974, p. 107).

The first principle Pirsig mentions is one of the underlying principles for a good Research Data Management. Then, “the logical statements entered into the notebook are broken down into six categories: (1) statement of the question, (2) hypotheses as to the cause of the problem, (3) experiments designed to test each hypothesis, (4) predicted results of the experiments, (5) observed results of the experiments and (6) conclusions from the results of the experiments” (Pirsig, R.M., 1974, p. 107).

In the application of the method, Pirsig underlines the importance of adopting a cautious approach to the initial problem. The most difficult part is the formulation of the hypothesis. “The number of rational hypotheses that can explain a given phenomenon is infinite” (Pirsig, R.M., 1974, p. 119). To achieve this, one must observe reality carefully, and in order to do so, it is essential to be open-minded, autonomously making use of one’s own criteria of judgement.

The application of the scientific method to concepts aims to guide thought in a precise way. The true aim of the scientific method is to ensure that nature has not tricked you into believing that you know what you do not know, that is, the aim is to ascertain the truth. This point is very important. “The purpose of scientific method is to select a single truth from among many hypothetical truths. That, more than anything else, is what science is all about.” (Pirsig, R.M., 1974, p. 119. Emphasis is mine)

The purpose of the scientific method is to ascertain the truth, therefore a new fundamental element is introduced, a link between science and truth. This link is crucial to define both the quality of science (as a concept and a result) and of research (as a practice). But what is truth? And what is the best way to ascertain it?

5. Science and truth

The nexus between science, research and truth is at the heart of philosophical and scientific thought and is particularly addressed by Immanuel Kant in the late Conflict of the Faculties. Kant’s interest in the status of science – and more specifically of fundamental research – runs through his entire work. Indeed, it is possible to trace a common thread that runs through and links the works of Kant’s theoretical, practical and political philosophy, and that is the reflection on the status of philosophical knowledge (“scientific” knowledge par excellence)¹³. However, it is in The Conflict of the Faculties, the only work to have the university at its centre, that Kant makes explicit the conditions for science not to be subject to heteronomous laws, and also addresses the question of what is the interest of science, and of public and fundamental research.

The Introduction to The Conflict of the Faculties explains the reason why Kant decides to publish the three essays collected there, which is to define the space of what the Enlightenment essay defines as the ‘public use of reason’, in a more in-depth way¹⁴. If in 1784, under the power of Frederich II, research is free and there is no need for the conditions of the public use of reason to be imposed, in 1798 these conditions are discussed in detail and the university is a necessary condition for the exercise of the public use of reason¹⁵.

Although Kant’s intention is not to deduce a general theory of the university, The Conflict of the Faculties is far from being a simple historical narrative of the university of his time either; the philosophical work around the concept of university is aimed at saying that university is necessary; and at outlining a normative vision, that is, how the university, a necessary public institution like the State, should be¹⁶..

In examining the importance of the autonomy of science for its own progress, the essay defines the role of fundamental research and the reasons for its autonomy in the interest of science at large.

The university is designed starting from a unitary rational idea. Kant uses a metaphor to represent the university, comparing it to a factory. Its workers are public teachers, professors, who form, with the division of labour, a kind of common of scientists who work in autonomy (because only scientists can judge scientists). Just like in the introduction to the Methodenlehre of the first Critique, the construction of science takes place through cooperative and cumulative work. In this context, the purpose of science, following the metaphor built by Kant in the preface, is to build “an edifice in relation to the supplies given to us that is at the same time suited to our needs”, and to make it high enough to allow Man to see from a broader perspective.
What Kant is saying here is that the builders aspire to build a very tall tower, but cultural and linguistic differences prevent this from happening. As an alternative, Man will settle for building a house, or rather a home (in which he will live), which will nonetheless be high enough to provide him with an overarching view (Kant I., 1781-87, A 707 | B 735. p. 627).

Still, in this case, the process of research is compared to that of a factory, which needs a mechanism to make the educational system function. The university is divided into disciplinary fields through its faculties, which perform the dual function of taking on students and appointing “doctors”. When he defines the Faculties, Kant does not deviate from the traditional subdivision of the universitas magistrorum et scholarium into four faculties, one “lower” and three “higher”; a subdivision that descended from the statute of the first European university founded in Paris in 1215, and that the organisational model of the universities of the Germanic states of his time followed rather faithfully¹⁷.

The philosophical faculty was defined as “lower” as it was preparatory to the three higher and specialised faculties, which allowed access to professions. But the philosopher somehow distances himself from this denomination from the beginning, when he states that the origin of the subdivision and its name do not depend on the scientific community, but on the government.

What is original in this context is in fact the motivation which, in Kantian definition, underlies the use of adjectives qualifying the two classes of university faculties. The faculty of philosophy is called lower because it has to deal with the interest of science, and it is such, he anticipates, “because it can deal with its propositions as it sees fit.” (Kant I., 1798, A 10-11, p. 27) And he continues: “It is absolutely essential that the learned community at the university also contain a faculty that is independent of the government’s command with regard to its teachings; one that, having no commands to give, is free to evaluate everything, and concerns itself with the interests of the sciences, that is, with truth: one in which reason is authorised to speak out publicly. For without a faculty of this kind, the truth would not come to light (and this would be to the government’s own detriment); but reason is by its nature free and admits of no command to hold something as true (no imperative “Believe!” but only a free “I believe”).” (Kant I., 1798, A 9-10, p. 27-29)

The cited passage introduces three important points.

The first is the need for a philosophical faculty in the university system, and the necessary condition for its existence, namely freedom of criticism in any discipline. Kant considers the Faculty of Philosophy essential to the functioning of the entire university. The teachings of this faculty consisted of the so-called “artes liberales”, whose disciplines go beyond philosophy in the strict sense¹⁸. When he refers to the disciplines taught in the philosophical faculty, he therefore means both the mathematical, physical and natural sciences, and the human sciences (letters, history and philosophy), whose study at university level consisted, at the time, of what we would today call fundamental research. The Faculty of Philosophy: “does not consider all these areas (i.e. the doctrines and precepts of the higher faculties) as content, but as the object of its examination and criticism, having to aim for the benefit of the sciences.” (Kant I., 1798, A 9-10)

The text affirms the “absolute” necessity for a scientific community that deals with fundamental research, and the need for a training system based on education in the scientific and philosophical method, which is neither oriented towards professions nor driven by utility. It is a type of study that, unlike what is taught in the higher faculties, focuses on the interest of truth.

The second important point is that the faculty of philosophy does not give orders, but is free to express any opinion on all matters relating to the interests of science, that is, to truth. This is where the link between science and truth appears. Here, an equivalence between the interest of science and truth is established, along with the need for (unlimited) authorisation of the publicness of scientific discourse.
Truth is an evasive term in Kantian lexicon. In this context, it is repeatedly recalled and emphasised (Bahti T., 1987, pp. 442-443), without ever being associated with any specific content. In an objective sense, truth coincides with the idea to which research tends and approaches¹⁹; in a subjective sense, it has a double meaning: truth is the correct exercise of thought, in accordance with the principles of reason, and consists in sincerity in presenting one’s theses, without concealing doubts and difficulties for the sake of convenience.

The correct exercise of thought recalls the principles mentioned by Carlo Bernardini, which are a common point for his students, and which are expressed as the three maxims of sensus communis (an expression by which Kant means a “common sense” we all share) in the Critique of the power of judgement: “1. To think for oneself; 2. To think in the position of everyone else; 3. Always to think in accord with oneself. The first is the maxim of the unprejudiced way of thinking, the second of the broad-minded way, the third that of the consistent way.” (Kant I., 1788, A 156 | B 158. p. 174)

In the Critique of pure reason, he anticipates the theme of the conflict he will face in The Conflict of the Faculties. The philosopher observes how, in human nature, there is a certain “duplicity”, that is, a tendency to keep one’s thoughts hidden to others, a propensity for concealment and deception that also finds its place in scientific communication; a field in which, on the contrary, there should be no conflicting interests with the sincere expression of one’s thoughts. “For what can be more disadvantageous to insight than falsely communicating even mere thoughts, than concealing doubts which we feel about our own assertions, or giving a semblance of self-evidence to grounds of proof which do not satisfy ourselves?” (Kant I., 1781-87, B 777|A 749. p. 648) The least that can be asked in the evaluation of scientific arguments, that is to say the necessary condition of science, is that everything be done fairly – without pandering to any power.

From a subjective point of view, truthfulness is important, that is to say, that the exchange of knowledge is sincere, so as to create a space for reason to exercise. A space which, Kant adds, also requires “public and unlimited authorization” – the only way for criticism to reach maturity. Without the possibility of criticism, reason is in a state of nature, that is, at war; and the possibility of peace lies in “the freedom to exhibit the thoughts and doubts which one cannot resolve oneself for public judgement without thereupon being decried as a malcontent and a dangerous citizen. This lies already in the original right of human reason, which recognizes no other judge than universal human reason itself, in which everyone has a voice; and since all improvement of which our condition is capable must come from this, such a right is holy, and must not be curtailed.” (Kant I., 1781-87, B 780 | A 752, p. 650)

Another relevant point addressed by Kant has to do with who the peers are. In Answer to the question: What is Enlightenment? the distinction and contrast are already drawn between the figure of “Gelehrter” (scholar), who “address the public of readers through writings”, and that of professionals such as: “the ecclesiastic”, “the intendant of finance”, “a doctor”, whose examples refer precisely to the triad of higher faculties. In the introduction of The Conflict of the Faculties, Kant makes no distinction between higher and lower faculty professors within the faculty community. As public teachers, the first task of academics is to train those who will later become “professionals” and make up the state bureaucracy and ecclesiastical hierarchies.

Kant also observes that scientists belonging to the university constitute only a subset of the “scholars” or “scientists”; beside whom there are others, free from the corporation, who are either gathered in free associations (academies, scientific societies) and laboratories, or do science as amateur “Gelehrter”, without any prescription or public rule, by vocation (Kant I., 1798, A 5).

Therefore, the scientists working in the university are only a part of the scientific community, and the recognition of the status of “scientist” does not necessarily come from the university, nor does it require any special qualification. The only requirement to be considered a scientist and to be treated as such is to address the public through writings, that is, to publish one’s own works. For both university scientists and others, “publicity is given solely and exclusively by the public nature of the literary publication that forces the concrete institutional boundaries of a given university and which, from the beginning, is oriented in a cosmopolitical way” (Brandt R., 2003, pp. 24-25. English translation is mine).

Kant wonders if criticism can be unlimited, and its exercise open to students as well: “But should not the young, at least, who are entrusted to academic instruction, be warned about writings of that sort, and be protected from premature acquaintance with such dangerous propositions, until their power of judgement has matured or rather the doctrine that one would ground in them has become firmly rooted, in order vigorously to resist all persuasion to the contrary, from wherever it might come?” (Kant I., 1781-87, B 782 | A 754. p. 651). The professor of philosophy’s position in this regard is clear: the purpose of academic teaching is not merely to acquire knowledge of historical philosophy, but also to learn the philosophical method. Research is scientific if and only if it is autonomous and, at the same time, cumulative.

Truth changes over time, and consequently, so does research quality. Thus, truth and quality can be understood and conceived as a method, rather than content. And since science coincides with truth as a method, and the purpose of the lower faculty is to deal with scientific questions of all kinds, then the mission of the latter will consist in freedom to publicly express a judgement on any scientific question. This entails that continuous peer reviewing is needed.

For truth is the result of public scrutiny carried out following the correct method, starting from the factory of workers who cooperate in a common activity (higher education, whose purpose is to feed the state bureaucracy and ecclesiastical hierarchies) and in a collective work (namely, scientific research). Truth has no content, or at least it is not important to define it. The important thing to define is how to get there. Or rather: truth does have content, but the general formula of truth is generic, and the task of establishing it case by case and object by object is an ongoing one²⁰. In Kant’s argument, the interest of science, understood as truth, is the foundation of the entire research system, a system which is created collectively²¹.

6. Science and collaboration

This collaborative process through which scientists ascertain truth has precise rules. To refer to them, Kant uses the term Streit. The term Streit can literally be translated as “conflict”, or “dispute”. Kant defines the concept that gives the title to the entire Conflict of the Faculties in the two sections that close the first part of the essay: one is dedicated to the “illegitimate conflict” between the two classes of faculties; the other, to the “legitimate conflict” between them. The latter, in particular, consists of the dispute over the truth of the doctrines that the government “publishes” as statutes, and which sees scientists of the lower faculty confronting scientists of the higher faculties.
Kant specifies that the dispute cannot and should not be resolved by an amicable agreement; on the contrary, it needs a court and a judgement. Conflict must be conducted as a trial, according to laws (of reason) before a judge (of reason), in order to find the truth.
And he also specifies that the conflict can never stop, and the philosophical faculty is the one that must be armed for this at all times (Kant I., 1798, A 37). In stating this, the philosopher adds a particularly important point: scientists are not only allowed to express themselves publicly on certain issues; they are required to do so. And the process of validating and falsifying scientific ideas does not end but begins with their publication.

In conclusion, Kant uses a political metaphor to clarify the role of fundamental research in the system of scientific knowledge:

“The rank of the higher faculties (as the right side of the parliament of learning) supports the government’s statutes; but in as free a system of government as must exist when it is a question of truth, there must also be an opposition party (the left side), and this is the philosophy faculty’s bench. For without its rigorous examinations and objections, the government would not be adequately informed about what could be to its own advantage or detriment. But if the businessmen of the faculties should want, on their own initiative, to make changes in the decrees given for them to expound publicly, then the government in its vigilance could lay claim to [jurisdiction over] them as innovators who could be dangerous to it. It could not, however, pass judgment on them directly, but only in accordance with the most loyal verdict drawn from the higher faculties, since it is only through the faculty that the government can direct these businessmen to expound certain teachings.” (Kant I., 1798, A 41, pp. 57-59).

Abandoning the traditional higher-lower hierarchy in this section, Kant suggests the adoption of a French republican-parliamentary and revolutionary model. Unlike the English parliament where the government and the opposition benches are against and facing each other, the French parliament, which took on its current form during the French Revolution, is characterised by the opposition of right and left. “This conflict is quite compatible with an agreement of the learned and civil community in maxims which, if observed, must bring about a constant progress of both ranks of the faculties toward greater perfection, and finally prepare the way for the government to remove all restrictions that its choice has put on freedom of public judgment. In this way, it could well happen that the last would some day be first (the lower faculty would be the higher)-not, indeed, in authority, but in counseling the authority (the government). For the government may find the freedom of the philosophy faculty, and the increased insight gained from this freedom, a better means for achieving its ends than its own absolute authority.” (Kant I., 1798, A 42, p. 59)

Kant’s referral to the French parliament is no accident here. In fact, the distinction between right and left goes back to a vote of the French National Constituent Assembly on 11 September 1789, on granting the king a right of veto over laws. On this occasion, monarchists in favour of a prohibitive veto took the place of honour to the right of the President, while supporters of a suspensive veto only, those in favour of the revolution, lined up to the left. The right-wing solution, which allowed parliament to legislate exclusively according to the king’s will, would have failed to break the status quo of absolute monarchy. Thus, in accordance with a practice born in the national constituent assembly of revolutionary France, advocates of the government and the status quo would sit to the right, and those who stood against it and desired change sat to the left.
In the system outlined by Kant, the higher faculties are on the right side because they serve the government, and the lower faculty is on the left as a result of its mission to criticise. But the constitution which ideally regulates the parliament of science – whose activity, as a parliament, is public – is not that of the Prussian monarchy: it is “a free constitution as it must be where truth is at stake”. The faculty of philosophy therefore offers the government an unsolicited form of instruction which is public, antagonistic, and inspired by an ideal constitutional model that differs from the current regime. The opposition of this faculty indicates that the status quo, which is serviced by professors who are ministers of power, can and must be overcome. As Pievatolo points out, the faculty of philosophy’s “criticisms, in the long run, are not as inoffensive as one would like them to appear” (Pievatolo M.C., 2018).

By conflict, therefore, Kant means nothing more than free scientific debate – which starts from discussion within the scientific community, i.e. peer review, in the broadest sense. But why does he use the word conflict?

To answer this question, one should reconstruct the meaning of the term at the time; in the entry “Streit” of the Zedlers Universal-Lexicon of 1740, there is a broad sub-definition dedicated to the “conflict of men against their spiritual enemies”; in the definition, conflict is not necessarily military, but can also relate to the field of the mind and the sciences²².

The term Streit encompasses another technical expression, “Streitschriften”, which can be translated into polemical writings, meaning “a type of writing that seeks the truth or falsity of a matter that has been exposed by another” and demonstrates it publicly following rules that can only be learned by reason²³.

Contrary to their apparent meaning – the term Streit is related to the ‘elenchus’ (from the Greek ‘elengkhos’, confutation), a strategy employed by sophists and based on the eristic method²⁴-, the Streitschriften were based on the dialectical method²⁵, and on the idea that knowledge and science, whose goal is to discover the truth by applying the correct methodology, are collective and collaborative enterprises. In Kant’s reflection, the term Streit comes from the Christian “Kontroversientheologie”, a method and a practice which took on particular significance and strength in the Lutheran church, where the free interpretation of the texts was accompanied by the refutation of opponents’ theses, both in private and in public²⁶. Kant’s choice of the term can be thus explained on the basis of this custom, the enormous amount of polemical writings of the time and the development of a scientific practice oriented towards the search for truth and the correct interpretation of texts: the review of a book, for example, was not about making it known to the public, neither was it written (as is often the case today) to advertise the author or publisher and entice readers to buy it; instead, it was aimed at critically discussing the theses of colleagues within a community that had the interest of science in common and that was subject to shared rules. An ex-ante and ex-post peer reviewing whose common rules are:

“1) Present your arguments ‘without regard to the person’; 2) Fact-oriented argumentation; 3) Construct one’s own judgement impartially, without regard to one’s belonging to a party; 4) Oblige yourself only to the truth and your own conscience; 5) Be sincere in choosing topics; 6) Let meekness dominate towards the adversary; 7) Beware of malicious misunderstandings from your opponent; 8) Take into consideration the opponent’s arguments; 9) Prohibition of insulting the opponent’s person” (Goldenbaum U., 2004, p. 111).

In this context, the publication of writings was a means of validating one’s theses, by making them accessible to the entire community and exposing them to scientific refutation, to which scholars (philosophers, philologists, theologians, jurists, etc.) participated through publications. This implied not only making connections but also taking controversial decisions – and being prepared to revisit those decisions in light of failure or unexpected implications. By using the term Streit, Kant therefore indicates a precise mode of scientific debate – one that occurs within scholarly communities and is public, structured, and ruled by the scientific method.

Finally, here comes the third argument. This debate has another fundamental characteristic: it takes place among the faculties, and the government is merely present as an observer. “On the one hand”, Brandt writes, “there is the arbitrary act of the government – inspired by the “not bad idea”²⁷  mentioned above – which exercises, through the universities, a policy of interests; on the other, there is reason that escapes any particular arbitrariness, which does not work for the interests of a reason of state, but which places itself exclusively at the service of freedom and seeks truth for itself. So, … heteronomy is opposed to autonomy, respect for the authority of others to the motto ‘have the courage to use your own intelligence’” (Brandt R., 2003, p. 18).

The philosopher clarifies his position. The government is committed, it is true, to the doctrines stemming from the higher faculties. However, this commitment is not in the interest of these faculties understood as scientific societies (als gelehrten Gesellschaften), but only in its own interest, and it is for this very reason that the government does not enter into the details of the contents of their doctrines, as if he was acting as a scientist himself. The responsibilities of the higher faculties towards the government consist in training professionals who “circulate among the people as a civil community and, because they could impair the government’s influence over it, are subject to its sanction” (Kant I. 1798, A 40, p. 57).

Research, which is aimed at ascertaining the truth, is based on a continuous debate that must never end – a debate which sees scientists collaborate. Here, the autonomy of the community from any external intervention (political, religious) is postulated. To the question: who evaluates? Kant responds, the scientific community, because only scientists can judge scientists. If it were to be altered by reasons external to its own reason, namely the search for truth, science would no longer be such.

The process of evaluation and judgement is a critical and collaborative one. Collaboration between scientists is therefore a characteristic of the scientific method²⁸.

7. Conclusions. Towards a definition of research quality

The current research assessment reform has quality at its core and is linked to the adoption of Open Science practices, which entails a paradigm shift in the way science is done and shared. The UNESCO Recommendation prescribes something which was clear to scientists in the past, but no longer represents institutional scientific research, and it is for this very reason that the Recommendation was necessary. Hence, in the Recommendation, the problem is not posed in a merely descriptive manner, but relates to the very methods of science and, one might say, the very concepts of intelligence, research and knowledge.

This vision is closely linked to the discussion of what research quality is, a concept that ARRA places at the centre, together with that of qualitative assessment. In this vision, the development of infrastructures and multilingual transdisciplinary dialogue are at the heart of research practices, and competition plays a secondary role.

Focusing on reforming research evaluation on the basis of quality enables us to bring the purpose of research back into focus. From the very beginning of the history of philosophical thought, the aim of the research process has been to recognise the truth among the many hypotheses available. This objective is achieved by strictly applying the scientific method within communities of knowledge and making the research phases, processes, activities and objects transparent and verifiable, and by sharing processes and results. In this process, new knowledge is built on previous knowledge, which is first verified and validated by groups constituting partially overlapping networks of experts, the peers, and who follow specific practices. Then, over time, this knowledge undergoes further scrutiny by vast concentric circles of experts and finally by society as a whole, that is, by the entire readership, both present and future.

Opening science means correctly applying the scientific method to the entire scholarly communication workflow. It is based on collaboration, early sharing and opening up of processes and outputs. It is a prerequisite for research integrity. And this is what science done right is all about – high-quality research. Thus, this equivalence tells us that quality research is open, collaborative, transparent, and inclusive.

The ARRA clarifies in Commitment 2 that it is not simply a matter of extending the list of assessable products, or (commitment 6), of adding new criteria, processes and indicators. The reason for this is clarified through this powerful image provided by Pirsig: “if a factory is torn down but the rationality which produced it is left standing, then that rationality will simply produce another factory. If a revolution destroys a government, but the systematic patterns of thought that produced that government are left intact, then those patterns will repeat themselves.” (Pirsig R.M., 1974)

Since Science is the product of a collective and cumulative intelligence²⁹, scientificity is not about merit or content, or product, but about the method – or the process. This is why the current situation is one of cultural change, which requires a radical change of perspective. In fact, Open Science is not a theme or a chapter to be evaluated³⁰, rather it implies a revolution which entails evaluating research from an Open Science perspective³⁰.

This paradigm shift is clarified in the following passage from Weinberger: “When science was a type of publishing, it aimed at producing knowledge that was – like a publication -broken off from its source because it was embodied in a physical thing with a life of its own. The new issue of Nature arrives on the desk of the scientist, and she sighs in relief. Her research is out there at last. If, heaven forbid, a truck were to hit her this morning, the knowledge wouldn’t die with her. It now has a life of its own that can be tracked and weighed. But now that science is becoming a network, knowledge is not something that gets pumped out of the system as its product. The hyperlinking of science not only links knowledge back to its sources. It also links knowledge into the human contexts and processes that produced it and that use it, debate it, and make sense of it. The final product of networked science is not knowledge embodied in self-standing publications. Indeed, the final product of science is now neither final nor a product. It is the network itself—the seamless connection of scientists, data, methodologies, hypotheses, theories, facts, speculations, instruments, readings, ambitions, controversies, schools of thought, textbooks, faculties, collaborations, and disagreements that used to struggle to print a relative handful of articles in a relative handful of journals.” (Weinberger D., 2001. p. 106. Emphasis mine) The 2021 Scoping Report entitled “Towards a reform of the research assessment system” is precisely based on this assumption (European Commission, Directorate-General for Research and Innovation, 2021, p. 1).

Again, it is Pierre Mounier who underlines the challenges arising from this change of perspective, “which means, when considering or even evaluating open science initiatives, projects, services and tools, to flip the order or priorities and to pay attention first to the way they move in their ecosystem: how do they nurture from it, how do they fertilise it, how do they cooperate with others, rather than other criteria that are usually considered as more important; such as innovation, efficiency, excellence. And then, when we have a comprehensive representation of the full web of interactions and interdependencies, maybe we could start asking the right questions: is it sustainable? Is it inclusive? Is it creative? Is it alive?” (Mounier P., 2022. p. 6).

The procedural definition proposed by the ARRA therefore has a clear meaning and the paradigm shift it implies and, at the same time, promotes is much more evident. This definition is in line with a definition of Open Science based on the assumption “that OS practices can better support the quality of scientific outputs when they focus on the specific ways in which accessibility is provided, and particularly the strategies used within specific research situations to decide who counts as a contributor, how objects should be handled and interpreted, and what goals should be pursued. This framework takes research outputs such as data, models and articles as temporary signposts of the ongoing process of inquiry, whose function is to adequately support communication and learning within and beyond the research community. This is a process-oriented philosophy of science, which calls attention to the conditions under which outputs are produced, disseminated, stored and deployed, and conceptualizes scientific research as primarily aimed to advance active knowledge. Far from being solely a question of sharing resources, openness is thereby conceptualized as the opportunity to make and maintain connections among relevant stakeholders in the research process – whether these be professional researchers, other publics, non-human organisms or machines – in ways that help to develop ever more relevant forms of interaction with the world.” (Leonelli S., 2023, pp. 65-66. Emphasis mine) This means, in practical terms, valuing the diversity of contributions to science and research (including ‘outputs’, processes and activities) and basing research assessment on contextual qualitative judgement.

In order to propose a definition of quality, it is worthwhile resorting to Pirsig once again, according to whom quality “is the event at which the subject becomes aware of the object. […] Here is where the distinction between qualitative and quantitative loses meaning. […] Subjectivity and objectivity are not separate universes that have no connection to each other. Instead, they are separate phases of a single evolutionary process“. (Pirsig R.M., 1999, p. 7)

Four points are made in this definition:

The first is that Quality – like truth, despite being perceived as a constant, turns out to be very variable and people have differing opinions about it. In fact, just like truth, quality also varies over time. It is on this basis that Pirsig states that quality is an event.

The second aspect is that if quality occurs at a given time and can change over time, any evaluation is in itself finite. What is important, is that an endless process of discussing a scientific truth remains open alongside the finite assessment exercises.

The definition then sheds light on a third point, namely, it brings out two different notions of quality – or better, two distinct moments. “Dynamic Quality is a stream of quality events going on and on forever, always at the cutting edge of the present. But in the wake of this cutting edge are static patterns of value. These are memories, customs and patterns of nature. The reason there is a difference between individual evaluations of quality is that although Dynamic Quality is a constant, these static patterns are different for everyone because each person has a different static pattern of life history. Both the Dynamic Quality and the static patterns influence his final judgment. That is why there is some uniformity among individual value judgments but not complete uniformity” (Pirsig R.M., 1999. p. 9).

On the basis of Pirsig definition, we can distinguish two types of evaluation, the first of which is finite, and takes place at specific moments in time and involves people, projects, specific results or processes. In such a system, contextual evaluations (such as competitions, evaluation of projects and objects and processes) must take place according to open, verifiable and transparent criteria and processes, and must be based on open and transparent data and infrastructures. The ARRA is explicit on this³². These procedures can be forms of ex-post evaluation, made possible through filtering tools which combine opinions and information from many sources, which are under the control of the reader. But these assessment events are snapshots of particular moments in research practice, and should not be regarded as timeless scientific truths per se, but rather act as contextual scaffolds for scientific activities aimed at increasing knowledge. It is thus necessary to regularly reassess their validity, relevance and significance within research, i.e. their quality.

However, since truth and quality are events which can evolve over time, we must also bear in mind that public scientific debate can never end, and that a valid theory can subsequently be proven wrong, and vice versa, as “the greatest strength of the scientific method is that it always allows for new experiences, new ideas and a new evaluation of what is learned.” (Pirsig R.M., 1999, p. 9)³³. The idea of openness is thus linked to that of learning as going beyond one’s knowledge and boundaries. This continuous activity and the possibility of verifying processes and results is also fundamental, for scientific truth is a function of time – and its longevity is inversely proportional to the intensity of the scientific effort. In this dynamic we have seen that there are motivational elements pushing in different directions: while plausibility and scientific value tend to reinforce conformity, originality encourages dissent – an internal tension between orthodoxy and innovation (Polanyi M., 1962), which depends on the communities of practice of reference, and which must also be the subject of analysis. In practice, this tension can be observed in the dynamics behind the establishment of a new scientific theory, which needs time and a community that agrees with it to be established.

The fourth and last aspect worth emphasising is that, from this perspective, the distinction between subjective (qualitative) and objective (quantitative) loses its meaning³⁴.

Considering evaluation in both senses i.e. as contextual events and as a continuous process, the way in which quality is evaluated is through qualitative judgement, i.e. peer review, a rigorous and responsible verification of data, processes and outputs created by other scientists. Scientists, while freely choosing what to study and judging according to their own individual criteria, in fact, collaborate as members of a community who share a common method, and their duties include verifying that science is as such – that is, true – before validating and certifying it. These activities are coordinated by subsets of networks that certify scientific knowledge itself. And in the process, the general public is actively involved.

In the Open Science perspective, however, peer review should be reconceptualised: it cannot coincide with its technical sense as it was formalised in the 1970s, and must be redefined in light of certain elements. The characteristics recalled in the ARRA, include expert assessment, transparency, impartiality, appropriateness, confidentiality, integrity and ethical considerations, gender equality and diversity.

Forms of blind or double-blind peer review should therefore be questioned, as they are incompatible with the definition of quality proposed by the ARRA. On the contrary, peer review should be open, namely making reviewer and author identities known, publishing review reports (Seeber M., Klemenčič M., Meoli M., Sin C., 2023) and enabling greater participation and transparency – hence, trust – in the peer review process (Ross-Hellauer T., 2017)³³. Open peer review is currently used by the publishing platform Open Research Europe (ORE) of the European Commission³⁴, which publishes results deriving from projects it funded, and adopts an open and ex-post peer review model. A preliminary editorial control, which takes place in any case on outputs produced by projects already selected for funding, is followed by the actual peer review, with the publication of the reviewers’ reports and total transparency of the process.

Moreover, as it was the case with Kant, both in the ARRA and in the Open Science paradigm, ‘peer’ has a different meaning than professional rank or membership of some scientific institution, and qualifies anyone who is willing and able to make relevant contributions to scientific research. The only requirement to be considered a scientist and to be treated as such is to address the public through writings, i.e. to publish scientific works, i.e. to participate in the public scientific debate. More specifically, collaborative activities, as well as the sharing of scientific resources among scientists, institutions and within the community in general, are in total compliance with Sustainable Development Goal 17 – Partnership for the objectives. (UNESCO, 2021, p. 17)

Such a transformation has a considerable impact on the system of evaluation of science, not only from a theoretical point of view, but also and above all from a practical one: the revolution in research evaluation is that it is no longer necessary for peer review to precede publication and for it to be done by “a few experts”. Citizen science can be seen as a form of collective open peer review – which is nothing more than the public use of free and unlimited reason. The new forms of evaluation made possible by the media revolution we are experiencing can provide an impulse for the democratisation of scientific communication – and in particular for a broadening of the process on which the evaluation of results and processes is based. The more discussion there is about a text, the more data there is about its value³⁵. This has an important consequence on a philosophical-political level which consists in choosing to rely a little more on posteriors, providing them with as much information as possible, and with the tools to be able to select and choose what will be the best in their opinion.

The vision of quality and qualitative evaluation embodied by the ARRA and CoARA is therefore essential, but is also based on an important premise, namely that the evaluation of research should be in the hands of the scientific community itself. As we have seen, the link between the autonomy of research and its advancement has already been extensively emphasised by Kant. The ARRA clarifies several times that criteria, data, infrastructures and processes must be in the hands of the scholars and asserts the autonomy of research. As it was clear to Kant, science can be institutionalised as a “factory”, but science is such if and only if it remains autonomous, and free from any authority who might attempt to interfere in its debate (Pievatolo M.C., 2024).

In Kant’s definition of university, the hierarchy between the university classes is overturned: the philosophical faculty is lower because it is the place where a public discussion between scientists can and must take place; as such, it has primacy both over the government and the higher faculties. Kant is ironic about the ancillary role of the philosophical faculty, stating that it could indeed carry the burden of the theology faculty, but also the torch ahead of it. Philosophical activity is fundamental research, the exercise of a method which consists in subjecting any doctrine to criticism, and as such it is the fundamental precondition of all knowledge. It consists of free communities of peers who learn from their mistakes and constantly self-correct. The corresponding faculty is therefore lower as a foundation and necessary condition for science in general; it also has a fundamental role in the mechanism to guarantee freedom of research and teaching, and has the task of subjecting any doctrine to examination by passing it to the scrutiny of reason, that is, of public, open and free debate. It is therefore clear that responsible evaluation means nothing if the institutional model follows the right-wing model of the prohibitive veto of 1789, i.e. if evaluation agencies are not in the hands of researchers but in those of the government. This applies to both research assessment and Open Science. As Leonelli recognises “beyond an internal re-orientation of academic priorities and institutions, the biggest challenge – the elephant in the room – is the extent to which OS efforts are prone to instrumentalization by the political forces and economic structures within which science is unavoidably positioned.” (Leonelli, 2023, p. 68)

The sociologists of science Whitley and Gläser collected studies of “strong” national research evaluation systems, which regularly conduct institutionalised and public evaluations according to formalised rules and procedures to produce rankings that influence funding decisions, and “weak” ones, which do not (Whitley R., Gläser J., (eds.) 2007; Whitley R., Gläser J., Engwall L., (eds). 2010). The analysis of the sociologists showed that “when evaluation is in the hands of a single centralized authority that imposes its criteria, normalization and hierarchy are hard to avoid, unless other peripheral variables come into play.” (Pievatolo M.C., 2024, pp. 27-28). So changing the way we evaluate is not enough if we do not also discuss the evaluators themselves. The last point is at the core of a responsible research assessment reform. In fact, the ARRA requires the direct involvement of individual academics and of scientific communities in the definition of new criteria and processes (ARRA, 2022, pp. 3, 5, 6, 9)³⁹, but academic communities should assume collective ownership and control over the infrastructures necessary for successful reform. This last point is not as prominent in the ARRA as it should have been – and should be a central governing principle in the future CoARA.

Acknowledgements

This article is the result of a reflection on the reform of research assessment, a topic on which I have focused my research in recent years. In particular, between 2022 and January 2024, I participated in twenty public discussions at events organised around the Agreement on Reforming Research Assessment, and CoARA. The very stimulating discussions at these meetings have been crucial in bringing into focus the issues discussed here. I would therefore like to thank first of all the participants, the organisers and the speakers of these events, and in particular Silvia Bello, Stefano Bianco, Alessandro Carosi, Brunella Casalini, Roberto Caso, Riccardo Coratella, Alberto Credi,  Annagrazia Chiodetti, Giovanni De Simone, Pasqualina Dursi, Angelo Facchiano, Sergio Garattini, Silvia Gomez Recio, Alberto Granzotto, Claudio La Rocca, Marialuisa Lavitrano, Emma Lazzeri, Daniela Luzi, Francesca Masini, Tiziana Metitieri, Lucio Morettini, Fabrizio Pecoraro, Ginevra Peruginelli, Janne Pölönen, Emanuela Reale, Alex Rushforth, Stefano Sensi, Maddalena Pennisi, Menico Rizzi, Lisa Sella.

I am very grateful to Maria Chiara Pievatolo, who read and discussed the first versions of this text and allowed me to clarify points that were unclear. I am also grateful to Elena Giglia, who pointed out some logical jumps and made explicit the connections that needed to be clarified in the first version, and to Daniela Tafani, for her comments related to sections 5 and 6. The second version benefited from the reading of some colleagues, whom I would like to thank: Anestis Amanatidis for recommending a relevant reading; Elisa Guadagnini and Ludo Waltman, for their comments; Erzsébet Tóth-Czifra, for her timely and thought-provoking remarks, which allowed me to clarify some points that needed further investigation; and Vito Pirrelli, for his careful and valuable reading and suggestions to make the text clearer and more readable – which I hope I was able to take on board. Lottie Provost helped me to improve the English form, with careful and punctual work that gave the text its shape.

I would also like to thank Massimo Spadoni, with whom I have repeatedly discussed The Zen and the Art of Motorcycle Maintenance, a passion we share. Finally, this work is dedicated to my old friend Nicola Viegi, who in 1991 suggested that I read Pirsig’s novel, a book that has stayed with me ever since and from which my love for philosophy was born.

References

Agreement on Reforming Research Assessment, 2022. https://coara.eu/agreement/the-agreement-full-text.

Amin M., Mabe M., 2000. “Impact Factors: Use and Abuse”, Perspectives in Publishing, 1, 2000, pp. 1-6: http://www.elsevier.com/homepage/about/ita/editors/perspectives1.pdf.

ANVUR, 2023. “Valutazione della Qualità della Ricerca 2020-2024 (VQR 2020-2024)”: https://www.anvur.it/wp-content/uploads/2023/10/Bando-VQR-2020-2024_31ottobre.pdf

Bahti T., 1987. “Histories of the University: Kant and Humboldt”,Modern Language Notes, Vol. 102, No. 3, German Issue., Apr., 1987, pp. 437-460.

Baccini A., De Nicolao G., Petrovich E., 2019. “Citation gaming induced by bibliometric evaluation: A country-level comparative analysis”. PLoS ONE 14(9): e0221212: https://doi.org/10.1371/journal.pone.0221212.

Baldwin M., 2018. “Scientific Autonomy, Public Accountability, and the Rise of “Peer Review” in the Cold War United States”, Isis, volume 109, number 3:https://gwern.net/doc/statistics/peer-review/2018-baldwin.pdf.

Berlin Declaration on Open Access to Knowledge in the Sciences, 2004: https://openaccess.mpg.de/Berlin-Declaration.

Bernardini C., 2009. Adulti e bambini, in Bernardini C. (editor), Il tempo, le cose, la natura, II Ed., Edizioni Conoscenza: http://archivio.edizioniconoscenza.it/wp-content/uploads/2013/11/indice-Bernardini-03-sito.pdf.

Berners-Lee T., 2002. The World Wide Web – Past, present, future. Exploring Universality,http://www.w3.org/2002/04/Japan/Lecture.html.

Bethesda Statement on Open Access publishing, 2003: https://web.archive.org/web/20120311105112/http://www.earlham.edu/%7Epeters/fos/bethesda.htm.

Bishop C., 2023. Defence against the dark arts: a proposal for a new MSc course:https://deevybee.blogspot.com/2023/11/defence-against-dark-arts-proposal-for.html.

Bosman, J., Bruno, I., Chapman, C., Greshake Tzovaras, B., Jacobs, N., Kramer, B., … Veksler, L., 2017. The Scholarly Commons – principles and practices to guide research communication: https://doi.org/10.31219/osf.io/6c2xt.

Brandt R., 2003. Il conflitto delle facoltà. Determinazione razionale ed eterodeterminazione nell’università kantiana, in Bertani C., Pranteda M.A. (eds), Kant e il conflitto delle facoltà. Ermeneutica, progresso storico, medicina, Il Mulino, Bologna 2003, pp. 13-52.

Budapest Open Access Initiative, 2002: https://www.budapestopenaccessinitiative.org/.

Buranyi S., 2017. “Is the staggeringly profitable business of scientific publishing bad for science?”, The Guardian, 27 June 2017. https://www.theguardian.com/science/2017/jun/27/profitable-business-scientific-publishing-bad-for-science.

Cerf V.G., Kahn R.E., 1974. “A protocol for packet network interconnection”, IEEE Trans. Comm. Tech., COM-22, 5, pp. 627-641.

Chubin D.E., 1985. “Open Science and closed science: Tradeoff in a Democracy”, Science technology and Human Values, 1985, vol. 10, n.2, pp. 73-81.

Coalition for Advancing Research Assessment: https://coara.eu/.

COMMISSION RECOMMENDATION (EU) 2018/790 of 25 April 2018 on access to and preservation of scientific information: https://eur-lex.europa.eu/legal-content/EN/TXT/HTML/?uri=CELEX:32018H0790&from=EN.

Commission Communication of 30 September 2020 on a new European Research Area for R&I COM/2020/628 final: https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=COM%3A2020%3A628%3AFIN.

Competitiveness Council conclusions, 2022. “Research assessment and implementation of Open Science” adopted by the Council at its 3877th meeting held on 10 June 2022: https://www.consilium.europa.eu/en/press/press-releases/2022/06/10/council-provides-political-orientations-on-international-cooperation-open-science-and-european-missions/.

Conclusions for the Competitiveness Council of 27-28 May 2021 on attractive and sustainable researchers’ careers and working conditions: https://data.consilium.europa.eu/doc/document/ST-8621-2021-INIT/en/pdf.

Council Conclusions on the new ERA of December 1st 2020:https://data.consilium.europa.eu/doc/document/ST-13567-2020-INIT/en/pdf.

Competitiveness Council conclusions, 2023. “Council conclusions on high-quality, transparent, open, trustworthy and equitable scholarly publishing”, May 2023: https://data.consilium.europa.eu/doc/document/ST-8827-2023-INIT/en/pdf.

Crace J., 2023. “‘Too greedy’: mass walkout at global science journal over ‘unethical’ fees”, The Guardian, 7 May 2023: https://www.theguardian.com/science/2023/may/07/too-greedy-mass-walkout-at-global-science-journal-over-unethical-fees

Creative Commons licenses: https://creativecommons.org/licenses/list.en.

Curry S., 2012. “Sick of Impact Factors”, Reciprocal Space, http://occamstypewriter.org/scurry/2012/08/13/sick-of-impact-factors/.

Csiszar A., 2016. “Peer review: Troubled from the start”. Nature 532, 306–308:https://doi.org/10.1038/532306a.

David P., 2000. The Historical Origins Of ‘Open Science’,  An Essay on Patronage, Reputation and Common Agency Contracting in the Scientific Revolution, MERIT Working Papers 2014-082, United Nations University – Maastricht Economic and Social Research Institute on Innovation and Technology (MERIT): https://ideas.repec.org/p/sip/dpaper/06-038.html.

David P., 2014. “The Republic of Open Science. The institution’s historical origins and prospects for continued vitality”. Contributi del Centro Linceo Interdisciplinare ‘Beniamino Segre’, 2014.

Dall’Aglio P., 2006. “Peer review and journal models”, ArXiv, 31 August 2006: http://arxiv.org/abs/physics/0608307.

Di Donato F., 2006. “Università, scienza e politica nel Conflitto delle Facoltà di Kant”, Bollettino telematico di filosofia politica: https://zenodo.org/records/7543877.

Di Donato F., 2007. “Come si valuta la qualità nella repubblica della Scienza? Una riflessione sul concetto di peer review”, Bollettino telematico di filosofia politica, July 2007. http://eprints.rclis.org/12602/1/peerreviewfdd_it.pdf.

Di Donato F, 2009. La scienza e la rete. L’uso pubblico della ragione nell’età del Web, Firenze University Press: https://media.fupress.com/files/pdf/24/1998/3867.

Di Donato F., 2022. “Una questione di qualità o una formalità? L’Agreement on Reforming Research Assessment e il processo di riforma della valutazione della ricerca in Europa”, Bollettino telematico di filosofia politica: 10.5281/zenodo.7433047.

Directorate-General for Research and Innovation (European Commission), PwC EU Services, 2019. Cost of not having FAIR research data, https://publications.europa.eu/resource/cellar/d375368c-1a0a-11e9-8d04-01aa75ed71a1.0001.01/DOC_1.

European Commission, Directorate-General for Research and Innovation, 2015. Open innovation, Open Science, Open to the World – A vision for Europe, Publications Office, https://data.europa.eu/doi/10.2777/061652.

European Commission, Directorate-General for Research and Innovation,Cabello Valdes, C.(editor), Rentier, B.(editor), Kaunismaa, E.(editor), Metcalfe, J.(editor), Esposito, F.(editor), McAllister, D.(editor), Maas, K.(editor), Vandevelde, K.(editor), O’Carroll, C.(editor), 2017. Evaluation of research careers fully acknowledging Open Science practices : rewards, incentives and/or recognition for researchers practicing Open Science, Publications Office: https://data.europa.eu/doi/10.2777/75255.

European Commission, Directorate-General for Research and Innovation, Peters, I., Frodeman, R., Wilsdon, J. et al., 2017. Next-generation metrics – Responsible metrics and evaluation for open science, Publications Office: https://data.europa.eu/doi/10.2777/337729.

European Commission, Directorate-General for Research and Innovation, Schomberg, R., Britt Holbrook, J., Oancea, A., et al., 2019. Indicator frameworks for fostering open knowledge practices in science and scholarship, Publications Office: https://data.europa.eu/doi/10.2777/445286.

European Commission, Directorate-General for Research and Innovation, 2019. Future of scholarly publishing and scholarly communication – Report of the Expert Group to the European Commission, Publications Office https://data.europa.eu/doi/10.2777/83653.

European Commission, Directorate-General for Research and Innovation, Mendez, E., Lawrence, R., 2020. Progress on open science – Towards a shared research knowledge system – Final report of the open science policy platform, Lawrence, R. (editor), Publications Office: https://data.europa.eu/doi/10.2777/00139.

European Commission, Directorate-General for Research and Innovation, 2021. Towards a reform of the research assessment system – Scoping report, Publications Office: https://data.europa.eu/doi/10.2777/707440.

Fecher B. and Friesike S., 2013. “Open Science: One Term, Five Schools”, RatSWD_WP_218, SSRN: https://ssrn.com/abstract=2272036; http://dx.doi.org/10.2139/ssrn.2272036.

Figà-Talamanca A., 2000. L’Impact Factor nella valutazione della ricerca e nello sviluppo dell’editoria scientifica, in: IV Seminario Sistema informativo nazionale per la matematica SINM 2000: un modello di sistema informativo nazionale per aree disciplinari<7em>: Lecce, 2 ottobre 2000:http://siba2.unile.it/sinm/4sinm/interventi/fig-talam.htm.

Fyfe A., Coate K., Curry S., Lawson S., Moxham N., Røstvik C.M., 2017. Untangling Academic Publishing: A History of the Relationship between Commercial Interests, Academic Prestige and the Circulation of Research: https://doi.org/10.5281/zenodo.546100.

Fitzpatrick, K. (2019). Generous Thinking: A Radical Approach to Saving the University. Baltimore: Johns Hopkins University Press., https://doi.org/10.1353/book.99579.

Grivell L., 2006. “Through a glass darkly. the present and the future of editorial peer review”, EMBO reports 7, no. 6.

G7 Science and Technology Ministers’ Communique, Sendai, May 12-14, 2023: https://www8.cao.go.jp/cstp/kokusaiteki/g7_2023/230513_g7_communique.pdf.

G7 Open Science Working Group (OSWG), 2023. Annex 1: https://www8.cao.go.jp/cstp/kokusaiteki/g7_2023/annex1_os.pdf.

Global Research Council, 2018. Statement of Principles on Peer/Merit Review: https://globalresearchcouncil.org/fileadmin/documents/GRC_Publications/Statement_of_Principles_on_Peer-Merit_Review_2018.pdf.

Goldenbaum U., 2004. Appell an das Publikum. Die öffentliche Debatte in der deutschen Aufklärung 1687-1796, Akademie Verlag, Berlin.

Gonzalez-Barahona J., 2012. “A Brief History of Free, Open Source Software and Its Communities”, Computer, vol. 54, no. 02, pp. 75-79: DOI: 10.1109/MC.2020.3041887.

Goodhart C., 1975. “Problems of Monetary Management: The U.K. Experience”. Papers in Monetary Economics. 1975 ; 1 ; 1. – [Sydney]. – 1975, p. 1-20. Vol. 1. Sydney: Reserve Bank of Australia.

“Goodhart’s Law”, Wikipedia entry: https://en.wikipedia.org/wiki/Goodhart%27s_law.

Greco P., Carra L., 2015. “I fondamenti della Responsible Research and Innovation”, : https://www.scienzainrete.it/node/17155/revisions/68049/view.

Gregory F., 1989. “Kant, Schelling, and the administration of science”, Osiris, 5, 1989.

Guédon J.C., 2001. In Oldenburg’s long shadow. Librarians, Research Scientists, Publishers, and the control of scientific publishing, ARL: https://www.arl.org/wp-content/uploads/2001/12/in-oldenburgs-long-shadow.pdf.

Hanks P., McLeod W.T. (eds), 1986. Collins Dictionary of the English Language, second edition.  William Collins Sons & Co; Second Edition.

Hicks, D., Wouters, P., Waltman, L. et al., 2015. “Bibliometrics: The Leiden Manifesto for research metrics”, Nature 520, 429–431: https://doi.org/10.1038/520429a.

“h-index”, Wikipedia entry: https://en.wikipedia.org/wiki/H-index.

Horizon Europe Standard Application Form for Research and Innovation Actions and for Innovation Actions, version 7.0, September 2023: https://ec.europa.eu/info/funding-tenders/opportunities/docs/2021-2027/horizon/temp-form/af/af_he-ria-ia_en.pdf.

INORMS, 2022. SCOPE Framework for Research Evaluation: https://inorms.net/scope-framework-for-research-evaluation/.

“Journal Impact Factor”, Wikipedia entry: https://en.wikipedia.org/wiki/Impact_factor.

Judson H.F., 2004. The Great Betrayal, Orlando.

Kahn R.E., 1972. “Communications Principles for Operating Systems”, Internal BBN memorandum.

Kant I., 1784. Beantwortung der Frage: Was ist Aufklärung?, in Kant’s Gesammelte Schriften, Band VII, 1902, De Gryuter, Berlin. English translation: Answer to the Question: What is Enlightenment? In Kant. I., Gregor M. (Editor), 1996. Practical Philosophy, pp. 11 – 22, Cambridge University Press: https://doi.org/10.1017/CBO9780511813306.005.

Kant I., 1781-87. Kritik der reinen Vernunft, in Kant’s Gesammelte Schriften, Band III, 1902, De Gryuter, Berlin. English translation: Critique of Pure Reason, Guyer P., Wood A.W., (editors),1998, Cambridge University Press.

Kant I., 1788. Kritik der Urteilskraft, in Kant’s Gesammelte Schriften, Band V, 1902, De Gryuter, Berlin. English translation: Critique of the power of Judgement, Guyer P. (editor), 2000, Cambridge University Press.

Kant I., 1798. Streit der Fakultäten, in Kant’s Gesammelte Schriften, Band I, 1902, De Gryuter, Berlin. English translation: The Conflict of the Faculties, Gregor Mary (editor), 1979, Abaris Books.

Kropotkin P., 1902. Mutual Aid: A Factor of Evolution: https://theanarchistlibrary.org/library/petr-kropotkin-mutual-aid-a-factor-of-evolution.

Kulczycki E., et. al., 2018. “Publication Patterns in the Social Sciences and Humanities: Evidence from Eight European Countries”, Scientometrics 116 (1): 463–86. https://doi.org/10.1007/s11192-018-2711-0.

Kulczycki E., 2023. “Research Evaluation: Unraveling the Metrics-Driven Pressures”. International Higher Education, (115), 28–29: https://ejournals.bc.edu/index.php/ihe/article/view/16793.

Kulczycki, E., 2023a. The Evaluation Game. How Publication Metrics Shape Scholarly Communication, Cambridge University Press.

Lambe P.J., 1988. “Critics and Skeptics in the Seventeenth-Century Republic of Letters”, The Harvard Theological Review, Vol. 81, No. 3. (Jul., 1988).

Larivière, V., Sugimoto, C.R., 2019. The Journal Impact Factor: A Brief History, Critique, and Discussion of Adverse Effects. In: Glänzel, W., Moed, H.F., Schmoch, U., Thelwall, M. (eds),Springer Handbook of Science and Technology Indicators. Springer Handbooks. Springer, Cham. https://doi.org/10.1007/978-3-030-02511-3_1.

Lawrynowicz K., 1999. Albertina. Zur Geschichte der Albertus-Universität zu Königsberg in Preussen, Duncker & Humblot, Berlin.

Leonelli S., 2023. Philosophy of Open Science, Cambridge University Press. Elements of Philosophy.

Lessig L., 1999. Code and Other Laws of Cyberspace, Basic Books: ISBN 978-0-465-03913-5.

Lessig L., 2001. The Future of Ideas, Vintage Books: ISBN 978-0-375-50578-2.

Lessig L., 2004. Free Culture, Penguin: ISBN 978-1-59420-006-9.

Longman Dictionary of the English Language, 1984.

Mancuso, S., 2015. Brilliant Green: The Surprising History and Science of Plant Intelligence, Island Press.

Mattelart A., 2001. Histoire de la société de l’information, Edition la Découverte, Paris.

McClelland C., 1980. State, Society and University in Germany 1700-1914, Cambridge University Press, Cambridge.

McKiernan G., 2003. “Alternative peer review: Quality management for 21st century scholarship”, in Peer Review in the Age of Open Archives: http://www.public.iastate.edu/~gerrymck/APR-1.ppt.

Méndez, Eva (2021). Open Science por defecto. La nueva normalidad para la investi-gación. Arbor, 197(799): a587 https://doi.org/10.3989/arbor.2021.799002.

Merton R.K., 1973 [1942]. “The Normative Structure of Science”, in Merton R.K. (editor), The Sociology of Science: Theoretical and Empirical Investigations, Chicago: University of Chicago Press, pp. 267–278, ISBN 978-0-226-52091-9, OCLC 755754.

Moher D., Bouter L., Kleinert S., Glasziou P., Sham M.H., Barbour V., Coriat A.M., Foeger N., Dirnagl U., 2020. “The Hong Kong Principles for assessing researchers: Fostering research integrity”, PLoS Biol 18(7): e3000737. https://doi.org/10.1371/journal.pbio.3000737.

Morales E., McKiernan E., Niles M.T., Schimanski L., Alperin J.P., 2021. “How faculty define quality, prestige, and impact in research”, bioRxiv 2021.04.14.439880: https://doi.org/10.1101/2021.04.14.439880.

Mounier P., 2022. “Connecting the building blocks of Open Science: an ecological approach”, Septentrio Conference Series, (1): https://doi.org/10.7557/5.6772.

Ochsner M., 2020. Evaluation Criteria and Methodology in Ochsner M., Kancewicz-Hoffman N., Hołowiecki M., Mickiewicz A., Holm J. (Eds), Overview of Peer Review Practices in the SSH. ENRESSH Report.

Oransky I., 2020. Retraction Watch: What We’ve Learned and How Metrics Play a Role in Biagioli M., Lippman A., (editors), 2020. Gaming the Metrics. Misconduct and Manipulation in Academic Research, MIT press: https://mitpress.mit.edu/9780262537933/gaming-the-metrics/.

The Oxford Learner’s Advanced Dictionary of Current English, 2000.

Peruginelli G.,  Pölönen J, 2023. “The legal foundation of responsible research assessment: An overview on European Union and Italy”, Research Evaluation: https://doi.org/10.1093/reseval/rvad035.

Pievatolo M.C., 2005. “Il Menone di Platone”, Bollettino telematico di filosofia politica: https://btfp.sp.unipi.it/dida/menone/index.xhtml.

Pievatolo M.C., 2006-7. “La Repubblica di Platone”, Bollettino telematico di filosofia politica: https://btfp.sp.unipi.it/dida/resp/index.xhtml.

Pievatolo M.C., 2012. “La moneta della scienza: Trasimaco e gli indici bibliometrici”, Bollettino telematico di filosofia politica: https://btfp.sp.unipi.it/it/2012/05/la-moneta-della-scienza-trasimaco-e-gli-indici-bibliometrici/.

Pievatolo M.C., 2016-2021, “Immanuel Kant, La metafisica dei costumi. Un esperimento di lettura ipertestuale”, Bollettino telematico di filosofia politica: https://btfp.sp.unipi.it/dida/kant_mds/index.xhtml.

Pievatolo M.C., 2017, “Wilhelm von Humboldt: un frammento di università”, Bollettino telematico di filosofia politica. DOI: https://doi.org/10.5281/zenodo.1302781.

Pievatolo M.C., 2018. “Immanuel Kant: una critica dell’università”,  Bollettino telematico di filosofia politica: https://btfp.sp.unipi.it/dida/kant_s/index.xhtml.

Pievatolo M.C., 2024. “The scale and the sword”, Bollettino telematico di filosofia politica: https://commentbfp.sp.unipi.it/maria-chiara-pievatolo-the-scale-and-the-sword-statal-science-and-research-evaluation/.

Pirsig R.M., 1974. The Zen and the Art of Motorcycle Maintenance, Bantam Books: https://www.bartneck.de/projects/research/pirsig/zen.pdf.

Pirsig R.M., 1991. Lila: An Inquiry into Morals, Bantam Books, 1991 (revised edition published 2006),  ISBN 0-553-07873-9.

Pirsig R.M., 1999. Subjects, Objects, Data, Values, in Diederik Aerts, Jan Broekaert & Ernest Mathijs (eds.), Einstein Meets Magritte: An Interdisciplinary Reflection. Springer. pp. 79-98: https://pryazhnikov.files.wordpress.com/2008/03/sodv.pdf.

Piwowar H., 2013. “Value all research products”. Nature 493, 159 (2013). https://doi.org/10.1038/493159a.

Plato, Meno: http://data.perseus.org/citations/urn:cts:greekLit:tlg0059.tlg024.perseus-grc1:70a.

Poeschl U., 2004. “Interactive journal concept for improved scientific publishing and quality assurance”, Learned Publishing 1.

Polanyi M., 1962. “The Republic of Science: Its Political and Economic Theory”, Minerva 1:54-74: http://www.polanyisociety.org/mp-repsc.htm.

Pontika N., and Knoth P., 2015. Open Science Taxonomy. FOSTER. https://doi.org/10.6084/m9.figshare.1508606.v3.

Pontika N., Klebel T., Correia A., Metzler H., Knoth P., and Ross- Hellauer T., 2022. “Indicators of Research Quality, Quantity, Openness, and Responsibility in Institutional Review, Promotion, and Tenure Policies across Seven Countries”, Quantitative Science Studies, 3: 888–911: https://direct.mit.edu/qss/article/3/4/888/113769/Indicators-of-research-quality-quantity-openness.

Pozzo R., Oberhausen M., 2002. “The Place of Science in Kant’s University”, History of Science, XI 2002, p. 2.

Rafols I., Molas-Gallart J., 2022. Reforming research assessment in Spain requires greater university autonomy, November 2nd, 2022: https://blogs.lse.ac.uk/impactofsocialsciences/2022/11/02/reforming-research-assessment-in-spain-requires-greater-university-autonomy/.

Rafols, I., Molas-Gallart, J., & Meijer, I., 2024, February 2. “Monitoring Open Science as Transformative Change: Towards a Systemic Framework”, SSRN: https://doi.org/10.31235/osf.io/knhzt.

Rowland F., 2002. “The peer-review process”, Learned Publishing 15, no. 4.

Ross-Hellauer T., 2017. “What is open peer review? A systematic review” [version 2; peer review: 4 approved]. F1000Research, 6:588: https://doi.org/10.12688/f1000research.11369.2.

San Francisco Declaration on Research Assessment: https://sfdora.org/.

Schimmer R., Geschuhn K.K., Vogler A., 2015. “Disrupting the subscription journals’ business model for the necessary large-scale transformation to open access”: DOI:10.17617/1.3.

Seeber, M. 2022. Efficacy, Efficiency, and Models of Journal Peer Review: The Known and Unknown in the Social Sciences. In Engels T.C.E., Kulczycki E. (editors), Handbook on Research Assessment in the Social Sciences, Cheltenham: Edward Elgar Publishing, pp. 67–82.

Seeber M., Klemenčič M., Meoli M., Sin C., 2023. “Publishing review reports to reveal and preserve the quality and fairness of the peer review process”, European Journal of Higher Education, 13:2, 121-125, DOI: 10.1080/21568235.2023.2192549.

Sgobino D., Barbetti S. (eds), 2013. La traccia di una innovazione. Esperienze di educazione scientifica nel Centro 0-6 “Turri” di Scandicci, https://www.comune.scandicci.fi.it/cred/pdf/traccia.pdf.

“Standing on the shoulders of giants”, Wikipedia entry: https://en.wikipedia.org/wiki/Standing_on_the_shoulders_of_giants.

Stark W., 1992. “Die Formen von Kants Akademischer Lehre”, in Deutsche Zeitschrift für Philosophie, 40, 1992, pp. 543-562.

Tennant J.P., 2018. “Open Science is just science done right”. PUBMET Annual conference 20 – 21 September, Zadar, Croatia, Presentation: http://pubmet.unizd.hr/pubmet2018/sessions/open-science/.

Tennant J.P., Dugan J.M., Graziotin D. et al.,  2017. “A multi-disciplinary perspective on emergent and future innovations in peer review” [version 3; peer review: 2 approved]. F1000Research, 6:1151: https://doi.org/10.12688/f1000research.12037.3.

The Merriam-Webster Dictionary, 1974.

Therry R., 2021. Open science: Time to unlock the potential of the Digital Age: https://www.youtube.com/watch?v=TrZrRcCoQSo.

Turner R.S., 1971. “The growth of professorial research in Prussia, 1818-1848 Causes and Context”, Historical studies in the physical sciences, III 1971, pp. 137– 82.

UNESCO, 2017. Recommendation on Science and Scientific Researchers (RS|SR): https://www.unesco.org/en/recommendation-science.

UNESCO, 2021. Recommendation on Open Science: https://doi.org/10.54677/MNMH8546.

Weinberger D., 2012. Too Big to Know. New York: Basic Books. https://archive.org/details/toobigtoknowreth0000wein.

WHO Director-General’s Opening Remarks at the Media Briefing on COVID-19, 2020: https://www.who.int/director-general/speeches/detail/who-director-general-s-opening-remarks-at-the-media-briefing-on-covid-19—6-april-2020.

Whitley R., 2007 “The Consequences of Establishing Research Evaluation Systems for Knowledge Production in Different Countries and Scientific Fields.” In Whitley R. Gläser J., (eds.) 2007. The Changing Governance of the Sciences: The Advent of Research Evaluation Systems. Dordrecht: Springer Netherlands.

Whitley R. Gläser J., Engwall L., (eds). 2010. Reconfiguring Production Knowledge. Changing Authority Relationships in the Sciences and Their Consequences for Intellectual Innovation. Oxford: Oxford U.P.

Why UU is missing in THE ranking, 2023: https://www.uu.nl/en/news/why-uu-is-missing-in-the-the-ranking.

Wilkinson M.D. et al., 2016. “The FAIR Guiding Principles for scientific data management and stewardship”, Sci. Data 3:160018 DOI: 10.1038/sdata.2016.18.

Williamson A., 2003. “What will happen to peer review?”, Learned Publishing 16.

Zedlers J., 1731-1754. Grosses vollständiges Universal-Lexicon aller Wissenschafften und Künste: http://www.zedler-lexikon.de/.

Zhang L., Sivertsen G., 2020. “The New Research Assessment Reform in China and Its Implementation”, Scholarly Assessment Reports: https://www.scholarlyassessmentreports.org/articles/10.29024/sar.15/.

Ziman J., 1996. “Is science losing its Objectivity?”, Nature, 382, 1996 pp. 751-54.