Bollettino telematico di filosofia politica

PDF

Presentation for English-speaking readers and abstract

1. Introduction: an impossible dialogue?

Although it is open to public review, this article is not an open peer review of Andrea Bonaccorsi’s book La valutazione possibile.¹ His book is behind a paywall, which is detrimental to the transparency of the debate and, for some non-monetary facets,² to the interests of the author himself. Moreover, a debate can only be between peers if the parties are and see themselves as such. But this was not the case during the period when Bonaccorsi was a government-appointed member³ of the Governing Board of the ANVUR; and the very conclusions of his book show that he himself does not treat his fellow researchers as peers.

Why, Bonaccorsi wonders, has the Italian research evaluation system attracted so much criticism? According to him, the motives of its critics can be reduced to three:⁴

• the emotionality caused by the personal involvement of those who see research as something more than a job like any other;
• their misunderstanding of the technical and impersonal nature of the assessment of their work;
• for some, the fear that their illusion of being better than others will be exposed by the objective gaze of research evaluation and that their ‘heavy academic interests’ will be jeopardized.

Bonaccorsi, who reports that he had to be taught by Alessandro Pizzorno⁵ that “people can misinterpret the way they are judged”, observes that

all three explanations contain elements of truth. Among the opponents of evaluation, we inextricably find those who object out of a sincere love of science, those who rightly fear the undesirable effects of exclusion and discouragement, and those who defend themselves, their group and, more generally, their academic interests.⁶

When is the thinking of others likely to be reduced to a phenomenon to be explained psychologically, economically, socially or biologically, rather than a question to be answered? According to Robert K. Merton, an author often cited by Bonaccorsi,⁷ this happens when society is so divided that its groups develop impermeable, competing and mutually alien universes of discourse. Even the critics of evaluation seem so alien to Bonaccorsi that he degrades them from subjects of discussion⁸ to objects of explanation. If they resist evaluation, they must be either emotional, incapable of understanding, or dishonest with themselves and others. But if he sees us as such, it would be futile to try to discuss with him as equals: our arguments – reduced to a manifestation of emotionalism, inability to understand, or dishonesty – would not be taken seriously, not even to be refuted. Nevertheless, his book deserves to be examined as an attempt to provide a theoretical justification for political interference in research that is almost unprecedented, at least in states that claim to be governed by the rule of law.

2. What is meant by ‘research evaluation’?

Even we, as researchers discussing Bonaccorsi’s book, are ‘evaluating’ a ‘research product’ – to use ANVUR jargon;⁹ and likewise, when we cite or refer to a colleague’s text for reasons unrelated to academic strategies, we are performing an activity of ‘evaluation’ that is intrinsic to research itself. For Bonaccorsi, however, ‘evaluation’ means something more: “an exercise in the explication, formalization and aggregation of judgments already present in the communities of experts.”¹⁰

Who is doing this exercise? It is difficult to understand because Bonaccorsi usually uses passive or impersonal sentences¹¹ without ever giving an explicit operational definition of evaluation and its actors. There is, however, one passage in which he makes a passing reference to its actors, typically – as we shall see – confusing the political and legal discussion with the scientific one. It is worth quoting in full, albeit temporarily out of context:

We have seen that Supiot makes an important polemical argument: equivalence conventions are an operation performed by both the legal and the statistical system, but the former is a transparent and contestable operation, while the latter is opaque and not subject to discussion. This is a false argument. In the field of evaluation, the equivalence convention has been a subject of discussion for at least half a century, ever since de Solla Price began to study the long-run dynamics of science and Garfield introduced citation analysis. It is so much a subject of discussion that there are several scientific journals devoted exclusively to it. The existence of a lively debate obliges decision-makers (governments, ministries, agencies) to justify their decisions in a documented way.¹²

In the above passage, Bonaccorsi invokes, albeit selectively,¹³ the authority of Alain Desrosières. According to Desrosières, the purpose of statistics is to group individual objects by reducing the abundance of reality to taxonomies, the construction and adoption of which open the way to further actions, both cognitive and political.¹⁴ These taxonomies produce conventional classes of equivalence – the conventions of equivalence mentioned in the above quotation – which are and should always be discussed. Discussion, in turn, is destabilizing because it disrupts ordinary administration and its taxonomies by attempting to construct an alternative. But it is also unavoidable if the future is not to be confined to categories elaborated in the past and for the past. Yet, Desrosiéres, like the jurist Supiot¹⁵ and unlike Bonaccorsi, sees a tension here.¹⁶

One could compare this tension with the more general tension resulting from the fact that many debates bear simultaneously on substantial objects and on the very rules and modalities of the debate: the constitution, the functioning of the assemblies, the means for designating the representatives. Any constitution lays down the rules of its own modification. But that is the point: statistical information does not present itself in the same way. The “indisputable facts” that it is called upon to provide (but that it
has also helped to certify) do not contain the modalities of their own debate.¹⁷

Bonaccorsi’s argument that the statistical “facts” on which politics and administration rely are still subject to scientific debate overlooks a not insignificant detail: while scientific debate can continue indefinitely in search of a conclusion that can no longer be refuted,¹⁸ political and administrative debate must end with decisions that are imposed even on those who disagree. It is precisely for this reason that the rule of law tradition has developed forms, procedures, checks and balances that are not needed in scientific debate.¹⁹. Bonaccorsi himself admits it: however much they claim to document their decisions, “governments, ministries, agencies”²⁰ are “decision-makers”.

The research evaluation to which Bonaccorsi refers is therefore not the one carried out by scholars, among peers, but the one imposed by the state, by its ministries and politically appointed agencies, such as the Italian ANVUR. As can be seen from the exegesis of his text, Bonaccorsi takes this circumstance for granted, although he never explains it in detail, preferring instead to dwell on an attempt to demonstrate the theoretical correctness of his reconnaissance operation rather than its scientific and political justification.

3. Science as a social institution

When, as in Italy, the evaluation of research is a state evaluation, carried out by a government-appointed agency with strong and intrusive administrative powers, it is necessary to address the preliminary question of its legitimacy. In order to separate the scholarly discussion about the way in which research is evaluated from the question of the legitimacy of the evaluating agency, we will provisionally concede something controversial, namely that the evaluation based on Bonaccorsi’s canons provides an accurate picture of the quality of research in the country in which the evaluation takes place. Would a state evaluation of research that was, in hypothesi, scientifically flawless be legitimate in a republic that wanted to continue to uphold the tradition of the rule of law?

Bonaccorsi presents the evaluation of research, which he oversaw as a member of the ANVUR board, as a reconnaissance operation that merely makes explicit and formalizes canons already shared by the scientific communities subject to it. This exercise, he argues, can be based on R.K. Merton’s sociology of science.

For my part, I have no difficulty in starting with the main normative model of modern science, that of Robert K. Merton. In its best-known formulation, scientists are universalist, communitarian, disinterested and skeptical.²¹

Merton refused to present his sociology of science as a sociological theory of knowledge aimed at identifying the alleged social foundations of valid knowledge. He intended his sociology of science to be descriptive and empirical, despite some later charges of idealization²² against it.²³

Science is a deceptively inclusive word which refers to a variety of distinct though interrelated items. It is commonly used to denote (1) a set of characteristic methods by means of which knowledge is certified; (2) a stock of accumulated knowledge stemming from the application of these methods; (3) a set of cultural values and mores governing the activities termed scientific; or (4) any combination of the foregoing. We are here concerned in a preliminary fashion with the cultural structure of science, that is, with one limited aspect of science as an institution. Thus, we shall consider, not the methods of science, but the mores with which they are hedged about. To be sure, methodological canons are often both technical expedients and moral compulsives, but it is solely the latter which is our concern here. This is an essay in the sociology of science, not an excursion in methodology. Similarly, we shall not deal with the substantive findings of sciences (hypotheses, uniformities, laws), except as these are pertinent to standardized social sentiments toward science. This is not an adventure in polymathy.²⁴

In dealing with the institutionalized ethos of the modern scientific community, Merton understands the institution in its general sociological sense as a system of formal or informal and historically situated social norms. His sociological study of the scientific ethos is normative only in the sense that it examines the ways in which society influences thought and discovers and describes socially accepted norms. It is not normative in the sense that it claims to impose them, once discovered and formalized, as criteria for scientific verification. If Merton had understood his work in the latter sense, he would have had to brand as invalid, for example, the foundations of Western mathematics because they were developed in groups, such as the Pythagorean community, whose esoteric practices were very different from those of modern scientists.

However, let us concede, for the sake of argument, that the snapshot of the ethos of the scientific community that Bonaccorsi thinks he can take using Merton’s theory is perfectly accurate. Going far beyond Merton’s intention,²⁵ let us even concede that it can also form the basis of an evaluation of research carried out by a government agency such as ANVUR. And let us also imagine that such an agency evaluates research using a combination of the following criteria: bibliometric indices calculated on the basis of proprietary databases, anonymous peer review of texts that may be behind paywalls and, in the field of the humanities and social sciences, the opinions of expert committees and the inclusion in lists of scientific journals and journals of excellence (“di classe A”) based on their reports.²⁶ Finally, suppose that the government-appointed authority in charge of the research evaluation justifies itself to the researchers as follows: “I am not imposing anything on you other than what you are already imposing on yourselves: my criteria are indeed just an up-to-date elaboration of what you have already informally developed in the course of modernity.”²⁷ Would it be right?

An Answer to the Question: What is Enlightenment?²⁸ written by Immanuel Kant in 1783 and carefully avoided by Bonaccorsi²⁹ can help to develop at least two arguments that justify a negative answer.

4. Caesar est supra grammaticos

Kant’s essay on the Enlightenment (Ak VIII 39-40) considers the case of a society of clergymen, governed as an aristocracy or even as a democracy, which, after a process of free deliberation, commits itself to a certain unalterable creed. Is such a deliberation morally acceptable? Kant resolutely denies it: such a commitment would ‘violate the sacred rights of humanity’. Even if it were democratic, an ecclesial community that opted for such a perpetual commitment would inconsistently deprive future generations of the same right to debate and think for themselves that it has enjoyed in deliberating its own doctrine. An ethos that is codified as binding ceases to be a freely accepted practice and becomes a coercive legal norm. But this very obligation cuts off the root of its legitimacy, which is the free choice of the community of reference, not only yesterday or the day before, but also today and tomorrow.

Kant, who lived under an absolute monarchy, remarks that, in general, “the touchstone of whatever can be decided upon as law for a people lies in the question: whether a people could impose such a law upon itself (AK VIII 39); “what a people may never decide upon for itself, a monarch may still less decide upon for a people; for his legislative authority rests precisely on this, that he unites in his will the collective will of the people” (AK VIII 39-40). Similarly, what the scientific community cannot decide for itself, except by depriving future generations of the freedom of research that it has enjoyed, can still less be imposed on it by a government authority, even in the limited case where the evaluation methods are actually error-free.

The extension of Kant’s argument from religion to science is not based on an arbitrary interpretation. The essay on the Enlightenment continues as follows (AK VIII 40):

It even infringes upon his majesty if [the monarch] meddles in these affairs by honoring with governmental inspection the writings in which his subjects attempt to clarify their insight, as well as if he does this from his own supreme insight, in which case he exposes himself to the reproach Caesar non est supra grammaticos, [Caesar is not above the grammarians] but much more so if he demeans his supreme authority so far as to support the spiritual despotism of a few tyrants within his state against the rest of his subjects.

This passage suggests another question: what if the government’s assessment is not perfect, as we have conceded for the sake of argument, but is based on questionable criteria and metrics?

Kant was dealing with a monarch, albeit an enlightened one, but his arguments against monarchical interference can be extended to the sovereign people, and a fortiori to those who claim to be acting in their service by virtue of opaque government appointments.³⁰

According to the Enlightenment essay, the state tries to interfere in research either by having its agents participate as peers in the scientific conversation or by directly using its power to establish the supremacy a school of thought.

The answer to the first type of interference can be the same as that given by Bishop Piacentius to Sigismund of Luxembourg, who, not wanting to be corrected for his Latin errors, claimed to be above grammar as Emperor of the Holy Roman Empire: Caesar non est supra grammaticos. If a state evaluator of research presented himself as a peer in the scientific conversation, he could not impose his evaluation criteria by coercion and, above all, he would jeopardize the very authority that invested him. Wanting to discuss Latin grammar, Sigismund found himself in an embarrassing dilemma: either accept the correction of a subject against his own majesty, or impose his own error on grammar by the sword.

If the state chooses the second horn of the dilemma, it will have research evaluated by offering the use of its sword to those scientists most inclined to comply. The result will be a state-controlled science that, as in the darkest pages of 20th century history,³¹ will not only host and impose state errors, but will also systematically stifle any true and free scientific debate.

In the secret article of the Perpetual Peace (AK VIII 369), Kant draws a comparison between philosophy, an inferior faculty in the university system of his time, and two superior and vocational faculties, jurisprudence and theology, the former at the service of the reason of the state and the latter at the service of revealed truth.

The lawyer who has made his symbol the scales of right along with the sword of justice does not usually make use of the latter merely to keep all extraneous influences away from the former, but when one side of the scales refuses to sink he puts the sword into it (vae victis) and a lawyer who is not also a philosopher (at least in morality) is greatly tempted to do so, since his office is only to apply existing laws but not to investigate whether such laws themselves need to be improved, and he counts this rank of his faculty, which is in fact lower, as higher because it is accompanied by power (as is also the case with the other two faculties).³²

What Kant says about jurists who are not also philosophers can be extended to all scholars who are organic to political power. The scale stands, in particular, for justice and, in general, for the disinterested judgement that should be appropriete to research. But those who also wield the sword of service to the state will be tempted to throw it on the scale, like Brennus, to put an end to the debate with an iron argument. The majority will bend the knee; those who do not will suffer the consequences. Peer review will become impossible, both for the many who submit and for the few who dissent, who will face a debate that is no longer disinterested but inevitably agonistic.

The evaluator who is empowered by the sword of the state would like to have a scientific legitimacy by claiming that he is using as a metric the same scale that is recognized by the majority of researchers. If this is the case, however, it is one of two things:

1. If the state evaluation replicates the weighing already done by the scientific community and gets the same result, then it is superfluous because it reproduces the same value structures – or, less optimistically, academic power structures – that already exist.
2. on the other hand, if the state evaluator claims to use the same scale, but gets – and enforces – a different result, then he must have added another weight to the scale: the unspoken and unjustified.³³ weight of the sword.³⁴

There are not only empirical studies showing a correlation between state evaluation and academic conformism, but also a widespread institutional awareness of the lack of diversity it creates.³⁵ However, in our argument, which focuses on the future and freedom rather than the past and its burdens, this circumstance is only accidental. Even if researchers were free and strong enough to refuse to comply with the evaluation agency, our conclusion would remain: state evaluation of research is a despotic exercise.

5. “Stop, you are so beautiful!”: research evaluation as a theoretical issue

Although he served as a government official in a country whose constitution recognizes, in addition to general freedom of expression, a special freedom for the arts, sciences and their teaching,³⁶ Bonaccorsi takes the legitimacy of government evaluation for granted. The only criticism he acknowledges is theoretical: “The basic critical argument is that it is not possible to make qualitative judgements on research that have the qualities of independence and impartiality necessary for the evaluation to have the character of neutrality.”³⁷ Thus, from his point of view, the issue can be reduced to a struggle between two opposing positions.

1. Evaluation of research is possible because scientific communities have developed their own intersubjective criteria, which are substantiated by citations, as R.K. Merton’s sociology of science supposedly shows; the evaluator only needs to formalize an already socially shared ethos.
2. The evaluation of research “does not reflect reality”,³⁸ because the scientific community evaluates itself and builds its internal hierarchy on the basis of social power relations, which also influence the purposes and ways in which researchers cite each other.

This dichotomy pits Merton against authors as diverse as Kuhn, Bourdieu and Foucault. In Bonaccorsi’s book, such an aut aut, certainly more articulated³⁹ than summarized here, is central to his argument. Taken seriously, it leads the reader to believe that the critique of state research evaluation and its bibliometric indicators depends on relativistic positions that resist the evidence of numbers and reduce science to a system of hierarchies and power. Bonaccorsi’s aut aut, however, can be dialectically resolved into an et et, by asking whether relativism is really necessary to discover, as sociologists of science, hierarchies and power constellations in the academic communities we study.

Pierre Bourdieu, for example, would have given a negative answer. In his preface to the English edition of Homo academicus, he rejects postmodern irrationalism: a sociology that reflects on itself, its own academic practices, their social determination and their influence on its own scientific discourse can make itself epistemologically more self-aware and thus freer from this very determination.⁴⁰ According to Merton himself,⁴¹ the sociology of science describes the ethos of the scientific communities it studies, but it does not treat any particular historically situated ethos as a necessary and sufficient condition for the validity of science, because it is not a sociological theory of knowledge. If historians or sociologists of science describe the Pythagoreans as an esoteric sect, they are not obliged to treat the Pythagorean theorem as a power device; nor are mathematicians obliged, if they find its demonstration within the formal system of Euclidean geometry convincing, to recommend the closure of public universities and their transformation into sects. Sociological description – which insists on historically situated objects – is one thing, the validity of science is another.⁴² Bonaccorsi has difficulty distinguishing between the two, only because he inadvertently assimilates the ethos described by sociologists to a norm of validation prescribed by evaluators.

Let us suppose, for the sake of argument, that Bonaccorsi’s text was written some forty years ago, and that it photographs in hypothesi a happy situation⁴³ in which the ethos of the historically existing scientific community is perfectly congruent with the conditions of validity of science. That he is therefore right to write that

The criteria used for the ex-post evaluation should be the same as those used for the ex-ante selection of contributions when they are accepted for publication in scientific journals or in the series of scientific publishers.⁴⁴

Let us assume that his representation of the scientific community, based on his reduction of the Mertonian sociology of science, is unchallengeable and that there is a general consensus on the use of bibliometrics and anonymous reviews for evaluation purposes. Let us suppose, then, that scientists compete with good sportsmanship for recognition of their own originality through publication in prestigious journals, and that they cite the work of others only to pay their debts to them. In such a situation, would we have sufficient theoretical reasons to say “Verweile doch! Du bist so schön!” and to distill from the snapshot of that moment a set of formal criteria for evaluating research?

The answer is no, for at least two reasons:

1. The fatal gesture of freezing the moment in order to extract a metric from it crystallizes as universal and necessary something that is, as empirical, particular and contingent;⁴⁵
2. The establishmnent of metricx has a reflexive effect on researchers, leading them to adopt strategic behaviors that undermine its very capacity for measurement. As Mario Biagioli writes in Nature,⁴⁶

All metrics of scientific evaluation are bound to be abused. Goodhart’s law (named after the British economist who may have been the first to announce it) states that when a feature of the economy is picked as an indicator of the economy, then it inexorably ceases to function as that indicator because people start to game it.⁴⁷

The reflexivity that turns a measure into a target is the uncertainty principle of the social sciences: quotations can only be the currency of science if they are the currency of nothing else. In this respect, state evaluation is not only practically despotic, it is also theoretically retrograde, in more than one respect.

6. Rule of law, rule of men

A reader of Bonaccorsi might think that this conclusion – state evaluation is a practically despotic and theoretically retrograde operation – does not do justice to his argumentative effort. Indeed, our analysis has mostly treated it as irrelevant, because it has addressed the preliminary question of the practical and theoretical legitimacy of a state evaluation based on the enforcement of the results, absolutized, of mere empirical generalizations. The reader may wonder, however, whether it is not our very concession in hypothesi that lends his claims a dogmatic arrogance that goes beyond the author’s intentions. Actually, Bonaccorsi seems to present the evaluation of research as a process based on equality and friendship: the very competition for recognition of the priority of a discovery transcends individual egoism because it is necessary for social certification, even when scientists like Darwin felt that it contradicted the spirit of disinterested research.

And so, Merton concludes, we need institutions that remove scientists from these dilemmas: “Peers and friends in the scientific community do what the tormented Darwin will not do for himself.”⁴⁸

In the passage quoted by Bonaccorsi, Merton was referring to a famous episode: Darwin was working on his theory of evolution when he received an unpublished article from the younger Alfred Russel Wallace with conclusions so similar to his own that he considered abandoning publication. Charles Lyell and Joseph D. Hooker – the “peers and friends” of Bonaccorsi’s quote – stepped in to ensure that credit was given to both by organizing, at the Linnean Society in London, the joint public reading of the two naturalists’ papers that sparked the Darwinian revolution. The year was 1858.

Darwin’s ‘peers and friends’, however, were not state evaluators of research. Their pressure to publish was a friendly insistence, very different from the publish-or-perish of today’s bureaucratized and normalized science, which did not spare even future Nobel laureates.⁴⁹ The Victorian gentlemen involved belonged to a small community that shared an informal ethos, in a network of interpersonal relationships and face-to-face discussions, far removed both from the fetishism of ‘publication’ in commercial journals that brand scholarship and from administrative coercion. For a 21st-century state evaluator, to imagine himself as Sir Charles Lyell or Sir J.D. Hooker is not only innocently deceptive: it is, as we shall see, legally and politically dangerous.

The reader may object that however much the research evaluator of the twenty-first century may have lost his genteel demeanor, it would still be wrong to treat him as a despotic doctrinaire. This would be demonstrated by Bonaccorsi’s reference to Charles S. Peirce, whom he sees as a proponent of a fallibilist sociological theory of knowledge.

But only the scientific method guarantees the formation of correct beliefs in the face of confrontation with facts. The guarantee of correctness is not internal to reasoning, but derives from confrontation with an “external permanence” that can “affect every man” This external permanence is science, understood both as the body of knowledge available at any given time and as the competent community that produces and disseminates that knowledge.

…there is no philosophical reason to conclude that, given enough time, it is not possible to find a solution to the scientific problem that will lead to agreement among the currently available beliefs. If we keep looking, in a few hundred years, ten thousand, a million, or a billion years, there is no reason to believe that the method of scientific inquiry will fail. This is, of course, a paradoxical response that could be translated in terms of a regulative ideal. Indeed, what is postulated is a community of competent people, of indefinite breadth, capable of carrying out research for an infinite time. One can speak of the construction of an asymptotic consensus.⁵⁰.

Research evaluation, if we apply to it the canons that Bonaccorsi believes can be derived from Peirce, would be the result of a process whose procedures can reach an intersubjective consensus among those who are recognized as authoritative in the relevant scientific community. This agreement has as its ideal horizon that of a possible truth that a community of competent researchers of indefinite size will arrive at in an equally indefinite time.⁵¹

But the article “How to Make Our Ideas Clear,”⁵² which Bonaccorsi quotes, is enough to show that Peirce’s position is not the one he ascribes to him.

The opinion which is fated to be ultimately agreed to by all who investigate, is what we mean by the truth, and the object represented in this opinion is the real. That is the way I would explain reality.

But it may be said that this view is directly opposed to the abstract definition which we have given of reality, inasmuch as it makes the characters of the real depend on what is ultimately thought about them. But the answer to this is that, on the one hand, reality is independent, not necessarily of thought in general, but only of what you or I or any finite number of men may think about it; and that, on the other hand, though the object of the final opinion depends on what that opinion is, yet what that opinion is does not depend on what you or I or any man thinks.

Peirce defines reality as “that whose properties are independent of what anyone can think they are.”⁵³ Reality is, therefore, an external permanence in principle different from science, which is just a cumulative collection of thoughts. We, however, can only know real things through their effects: hence, only through our beliefs. How, then, do we measure their truth?

Peirce uses the measure of intersubjective agreement: truth is “The opinion which is fated to be ultimately agreed to by all who investigate” and the object it represents is the real. Can we conclude, then, that reality is science? This is Bonaccorsi’s conclusion, but not Peirce’s.

According to Peirce, the impossibility of bypassing the mediation of thought does not mean that reality, which he defined as independent of our thinking, is what “we” think: “reality is independent, not necessarily of thought in general, but only of what you or I or any finite number of men may think about it.” Therefore, the opinions of a particular existing scientific community, as the opinions of a finite number of people, are not “reality. “⁵⁴ If the state evaluator were to identify the most widely held opinions, either through bibliometrics or through panels of appointed experts, and to adopt them as the standard for evaluation, he would be transforming the currently dominant opinions into norms. But such a method would be similar to what Peirce called the “method of authority”.⁵⁵ For its actual reference would not be an ideal “community of experts of indeterminate breadth and infinite horizon,” but rather a group of limited breadth and finite horizon whose tentative beliefs would be elevated to the standard of judgment by the decree of some “decision-maker”.

The confusion between, on the one hand, the prevailing views within certain scientific communities subject to the power of the state and, on the other hand, the regulatory ideal of endless research in the asymptotic horizon of “thinking in general” is not without consequences. Because of this confusion, Bonaccorsi presents the evaluators appointed by the government not as government officials, but as researchers open to discussion, working among others in the vineyard of science. Again, this representation is not just an innocent deception: it is politically and legally dangerous. The distinction between the government of men and the government of laws of Plato’s Statesman (293a-296a) will help to understand why.

Plato wrote that in an ideal situation, where there are statesmen equipped with science, a government without laws is preferable. The law, in fact, is like an ignorant and stubborn man: it does not allow its commands to be transgressed and it does not accept being questioned, even if someone has come up with something better than what has been established.

In the world of politics, where science is lacking, the rigidity of the law is not a flaw because it keeps the exercise of power in check. The concepts and procedures developed over the centuries by legal tradition and the rule of law have gradually improved its functionality for this purpose.

In the dialogue Protagoras, Socrates cannot agree with his interlocutor on the method of discussion. The sophist Hippias suggests a conventional solution: elect a chairman who will regulate the debate by controlling the timing of each speech. Socrates objects (338b-338e), for a reason that Gabriele Giannantoni explains as follows:

Socrates rejects this idea, arguing that since such a person cannot be inferior to the disputants (because he would not be up to the task) or equal to them (because he would be useless), he must be superior: but who is “wiser” than Protagoras or superior to him? Indeed, an arbiter is useless in a free and common inquiry.⁵⁶

Being scientific is not a question of merit or product, but of method or process: stiffening research procedures with rules and presidents, even if democratically elected, would mean freezing them and removing them from scientific discussion. Scientific communities whose orientation depends on a horizon of possible truth, such as that of Plato’s ideas or even Peirce’s reality, do not need a rigid law that crystallizes immutable criteria of quality, but an ethos that is only informally shared and can therefore easily evolve.⁵⁷

If the state evaluator sees himself as a peer and a friend, participating, like Socrates, in an informal ethos, and forgets that his authority derives not from community recognition but from ministerial appointment, he will be inclined to practice the informality of the government of men at the expense of the formality of the government of laws. Peers and friends can afford to change their minds by changing the standard by which they judge their colleagues: but an administrative authority that ‘changed its mind’ about the criteria for evaluating research would be awarding rewards and punishments according to retroactive rules. A researcher can afford to change a text published on the web overnight; an administrative authority cannot change documents that publish its rules in the middle of the night without undermining legal certainty.

These examples are taken from the small and incomplete gallery of legal horrors collected by Roberto Caso⁵⁸ while studying Italian evaluation practice. These horrors are not accidental: the theoretical root of the ANVUR’s difficulty in remaining within the bounds of the rule of law⁵⁹ is a deception, or perhaps an illusion: The fact that the state evaluator believe that shis authority depends not on his appointment by the government but on an alleged informal recognition of all researchers as his peers.⁶⁰ In short, he believes that he enjoys the scientific title to exercise the government of men when he should be subject to the limits of the government of laws – assuming, for the sake of the argument, that state interference in the exercise of the public use of reason is justifiable.

7. With moles’ eyes

The fourth chapter of Bonaccorsi’s book aims to “use the tools of the social sciences to test whether there is large-scale empirical evidence to support the alarmism about valuation, or even weak signals that may point to imminent and important risks. “⁶¹ His arguments are of marginal relevance to our critique, which portrays state evaluation as despotic and retrograde even while conceding in hypothesi that Bonaccorsi’s empirical generalizations⁶² are true for the past. However, his chapter deserves to be considered, at least in part,⁶³ in order to suggest some not-so-original reflections on the use of experience in research governance.

In 1931, of the more than 1200 professors teaching in Italian universities, just over twelve refused to swear allegiance to the Fascist state. One of the regime’s newspapers could thus contemptuously describe them as “sublimated to one per thousand”.⁶⁴ The rebels were indeed few by the standards of normal science. Similarly, the number of Jewish university professors dismissed under the racial laws in 1938 was negligible, though somewhat larger. A Bonaccorsi living in 1939 might have repeated that the chemical-political process by which a minority of professors went from solid to gas was so small that its significance was merely anecdotal.⁶⁵ But Giorgio Israel⁶⁶ shows that its impact on Italian research was profound and lasting.

Even before the racial laws were promulgated, the zealous minister had initiated a census of Jewish teachers, delivering race determination forms to all school and university authorities. On the eve of the laws’ promulgation, Tevere and Vita universitaria published lists of Jewish university professors and lecturers, demanding that they lose their chairs. Vita universitaria also published a list of school textbooks written by Jewish authors, the use of which was to be prohibited. The Italian-Jewish community numbered less than 50,000, plus another 10,000 foreign Jews who had been living and working in Italy for many years. Nearly 4,000 of them, including professors, members of the armed forces, public-and private-sector office workers, members of the liberal professions, and businessmen were stripped of all civil rights; about 6,000 students were expelled from schools and universities. Some 174 upper-secondary-school teachers were dismissed, along with 99 full professors -about 7% of all full professors in the country. The disciplines most affected were: medicine, 18; mathematics, physics and chemistry, 17; law,23; the arts and philosophy, 20. Of course, this disproportion (7% of full professors, when Jews accounted for only 0.1% of the Italian population) sparked the customary attempts to account for the Jewish pre-eminence in culture and science. Some described it as the result of a plot; others, as demonstrating that science had taken a turn for the worse and become an expression of the typically Jewish mindset, inclined towards abstraction and formalized theories far removed from intuition. It should be noted, however, that the clash between “Jewish science” and “Aryan science,” a classic theme in German racism, did not have a strong following in Italy, except in a series of articles published in “La difesa della razza” and “La vita italiana”: its main theorists were Julius Evola and Guido Landra.

A rapid perusal of the names of the university professors dismissed gives some idea of thecultural devastation caused by the racial laws.

The Italian school of physics, (the “Via Panisperna boys”), renowned the world over for its pioneering research in nuclear physics, was wiped out. Scientists of the caliber of Bruno Rossi (the founder of cosmic-ray theory), Enrico Fermi (whose wife was Jewish), Emilio Segrè, Ugo Fano, and Eugenio Fubini emigrated to the United States. Franco Rasetti, although an “Aryan,” emigrated to Canada rather than remain in a country responsible forsuch disgraceful behavior. Giulio Racah moved to the Hebrew University of Jerusalem, Leo Pincherle emigrated to England, and Sergio De Benedetti and Bruno Pontecorvo to Paris (the latter ultimately to the USSR).

In mathematics, the quality of the loss was more significant than the actual numbers. The list begins with Vito Volterra, nicknamed “Mr. Italian Science,” and Tullio Levi-Civita, perhaps the greatest Italian mathematician of the time, author of fundamental contributions io the mathematical aspects of relativity theory, and includes Beppo Levi, Guido Fubini (the creator of differential projective geometry), Guido Castelnuovo, and Federigo Enriques (the leaders ofthe Italian school of algebraic geometry).

Another casualty was the Turin school of biology, founded by the distinguished histologist Giuseppe Levi. In addition to Levi himself, two of his students and future Nobel laureates were lost to Italian science: Salvatore (Salvador Edward) Luria and Rita Levi Montalcini. The biomedicai sciences lost Maurizio Mosé Ascoli, Camillo Artom, Mario Camis, Amedeo Herlitzka, and Mario Donati, the greatest Italian surgeon of the day. Chemistry lost two of its main protagonists in the industrial sector: Giacomo Mario Levi and Giorgio Renato Levi. In the field of statistics, mention must be made of Giorgio Mortara – the only researcher in statistics and demography who could match theprestige of Corrado Gini – and Roberto Bachi, who emigrated to Palestine where he later founded the Israeli Central Bureau of Statistics. Perhaps the most ironic loss was of several of the regime’s most authoritative theoreticians in the field of “corporative” law: Gino Arias, Giorgio del Vecchio, and Guido Tedeschi (who emigrated to Palestine); as well as the economists Bruno Foà, Gustavo del Vecchio,and Marco Fano. Other distinguished names included the eminent geographer Roberto Almagià- all the maps hanging in schools and offices had to be replaced, because they bore his name -the literary historian Attilio Momigliano, and the philosopher Rodolfo Mondolfo.⁶⁷

Today it is not difficult to see the devastating effect of those “small numbers” whose names were Vito Volterra, Enrico Fermi or Rita Levi Montalcini. And it is easy to see, in retrospect, that a university that expels Piero Martinetti and destroys the careers of Aldo Capitini, is a more hospitable ground, even in terms of academic reproduction, ⁶⁸ for petty clerks of research than for free scholars capable of spreading “the spirit of a rational valuing of one’s own worth and of the calling of each human being to think for himself.”⁶⁹. Removing one research worker from the university and replacing him with another is numerically irrelevant. But removing Volterra, Fermi,⁷⁰ Martinetti and Capitini and replacing them with zealous conformists causes scientific, cultural and spiritual damage that is difficult to calculate in the medium and long term. Researchers – people – are not numbers.

Generalizations from experience are fragile. The task of empirically verifying the statement ‘all swans are white’ is endless. To falsify it, however, is very easy: all it takes is to find at least one swan that is not white to nullify the verifying power of thousands or millions of white swans. Even if it were only one, the dark swan could not be dismissed as “anecdotal”. What is true of swans is even more true of researchers: the presence or absence of an Enrico Fermi and a Vito Volterra or, on the other hand, of a Diederik Stapel and a Paolo Macchiarini is not ‘anecdotal’. A research evaluation system that would exclude or marginalize the former and elevate the latter to academic stars should be seriously reconsidered.⁷¹

Moreover, a naive reliance on experience makes it easier to believe in false generalizations. It does not follow from “almost all mafiosi are Italians” that “almost all Italians are mafiosi”: the same ANVUR agents would feel unfairly discriminated against if they were treated as mafiosi when traveling abroad simply because of their passports. ⁷² What, then, about a state evaluation that inferred from the hypothetical “almost all good researchers publish in top journals” that “almost all those who publish in top journals are good researchers” and rewarded the latter accordingly? Descriptively, such a conclusion is just a logical fallacy; practically, however, it produces obvious injustices.

Treating being Italian as a proxy for being a Mafioso, and therefore discriminating against Italians in the criminal law, is logically equivalent to treating publications in a journal belonging to a certain list as a mark of excellence and discriminating positively in favor of its authors. The unjustified negative or positive discrimination does not depend on the falsity of the premise – it may well be true that almost all mafiosi are Italians and that almost all good researchers publish in the journals included in a certain list – but on a gross, albeit widespread, error of inference. From the fact that some Italians are part of Cosa Nostra, or that some researchers who are considered good publish in certain journals, it does not logically follow that the majority of Italians are mafiosi because they are Italian, or that the majority of those who publish in those journals are excellent because of that mere fact. A similar argument applies to citations: from the fact that some (or even almost all) good researchers are highly cited, it does not follow that almost all those who are highly cited are good.⁷³

As Nicola De Bellis brilliantly argues, “the fact that the best scholars always publish in the same journals is not seen for what it is, i.e., the manifestation of a social dynamic of self-enclosure of academic élites on which commercial publishers can build their fortunes, but as an indicator of intrinsic quality. Almost as if a Platonic-Aristotelian essence of “top-journality” were embodied in these journals, promoting their bibliometric indices as much as their ability to attract the best works.”⁷⁴ This is not empiricism: it is a form of magical thinking that would be no less so if it were shared or attributed by the scientific community as a whole.⁷⁵

8. Black swans: a small (and incomplete) experiment in citation analysis

According to Bonaccorsi, “citation is not a rhetorical device: it is a condition of access to scientific communication: a paper that does not correctly cite the authors who have worked on the same topic will not be accepted by scientific journals and will simply not see the light of day”.⁷⁶ If, for strategic or rhetorical reasons, citations were to be inaccurate or misleading, they would become useless for any bibliometric evaluation of research, although they would still be of some use for the descriptive purposes of sociology of science. We therefore expect Bonaccorsi’s citations to be a model of accuracy: who, if not him, is more interested in adhering strictly to his own theory? Let’s try to learn from him.

Roberto Caso has already dealt with Bonaccorsi’s misleading and instrumental reading of Merton’s sociology of science. We have already shown how La valutazione possibile selectively appealed to the authority of Desrosières, while quietly omitting the very points that brought Desrosières closer to Supiot’s positions. We could charitably⁷⁷ categorize this practice as part of the game of interpreting and partially reusing the work of others for one’s own purposes. But to what extent is this rather self-serving game of citations – and avoidance of citations – compatible with Bonaccorsi’s allegedly “objective” nature of bibliometrics as a research evaluation tool?

To answer this question, we will analyze how Bonaccorsi uses his art of citation in dealing with (8.1) various sociological accounts of centralized systems of research assessment and the use of bibliometrics; (8.2) the issue of “particularism” in science, which might challenge the idea that scientific communities are structured according to an objective hierarchy that is out there and only needs to be elicited by state evaluators; (8.3) the alleged compatibility between democracy and government by numbers.

8.1. ‘Strong’ research evaluation systems and amateur bibliometrics

Bonaccorsi includes “authors such as Whitley and Gläser” among the “critics of evaluation” since they see it as “a parallel and overlapping principle of authority with respect to the self-regulation of scientific communities because it provides standardized indicators that are used by administrations.”⁷⁸ The works to which he refers, without citing any of their passages, are two books⁷⁹ edited by the sociologists of science Richard Whitley and Jochen Gläser, which collect studies of “weak” and “strong” national research evaluation systems. The systems they call “strong” regularly conduct institutionalized and public evaluations according to formalized rules and procedures to produce rankings that influence funding decisions.⁸⁰.

In “strong” systems, evaluation is materially delegated to academic elites who gain a privileged position.⁸¹ “Strong” systems can have consequences whose intensity is inversely proportional to the degree of autonomy, including financial, of universities.

1. Researchers become more aware of the need to compete for the recognition of the evaluating élite.
2. Because evaluation is centralized, the need to develop criteria of quality and relevance for entire disciplinary fields also centralizes judgments about researchers, universities, and research institutions;
3. Centralization and normalization reduce the diversity of research goals and methods, especially those that challenge the dominant orthodoxy;
4. The reinforcement of disciplinary norms and goals makes it more difficult to develop new research goals and fields, and makes innovation more dangerous for researchers.
5. The normalization, formalization, and publication of quality rankings make the university more stratified – that is, more hierarchical.

When evaluation is in the hands of a single centralized authority that imposes its criteria, normalization and hierarchy are hard to avoid, unless other peripheral variables come into play, as the two sociologists rightly point out.⁸² Are these really “criticisms,” or are they not rather almost obvious observations? A centralized evaluation system can certainly avoid using predetermined and uniform criteria, either by adopting the practice of evaluating works extemporaneously and idiosyncratically, or by constantly changing the parameters,⁸³ even ex post: in this way, however, it will inevitably enrich the collection of legal horrors edited by Roberto Caso.

How to overcome such dangers? Bonaccorsi says nothing about this, as if it were enough for him that the decisions of the central evaluation agency are or appear to be “rationally” justified,⁸⁴ and even pointing out some obvious consequences of centralization can be reduced to a criticism unworthy of refutation.

Jochen Gläser is also co-author, with Grit Laudel, of The Social Construction of Bibliometric Evaluations.⁸⁵ The essay does not express a radical doubt about bibliometrics: it only focuses on its social perceptions and uses, to ask why research evaluation so often applies it in an amateurish version. But according to Bonaccorsi, this is enough to label this approach as “constructivist“.

Citations are only one aspect of research quality.⁸⁶ When we cite someone else’s work, we do so not arbitrarily, but because we assign it some importance in the economy of our discourse, for reasons that may be either “Mertonian” recognition of others’ credit, or “constructivist” persuasion of readers who may also be bibliometric evaluators. In order to use bibliometrics in a scientific way, we must be aware of some well-known caveats. (1) Its statistics can only be meaningful for large numbers, i.e. for a large amount of citation data, which, a fortiori, should be complete, especially when considering the performance of individual research units. (2) In many areas of the natural sciences, an article reaches its peak in citations three years after publication, which means that the most recent reliable bibliometric data concern articles from three years ago. (3) Publication and citation habits differ between disciplines: citation data can therefore only be compared if they are normalized to field-specific reference values, which in turn are influenced by the way the fields themselves are classified.⁸⁷

For a long time, however, there was only one reference citation database, that of the former Institute for Scientific Information, now renamed Clarivate Analytics. This monopoly, in the hands of a commercial company, has been much criticized for its lack of control, accuracy, completeness and transparency, and for the short time window of its most successful product, the Journal Impact Factor, which is calculated over two years. But the marketing of the ISI has succeeded in oscuring these limitations in the eyes of most. ⁸⁸

Meanwhile, the corporatization of university administration had made bibliometrics attractive because it appeared to be more scalable and cheaper than peer review, and gave a greater impression of objectivity because it was based on the aggregation of a much larger number of opinions. Above all, its magic numbers seemed to offer politicians and administrators a way to evaluate research without having to rely on the mediation of researchers. In this situation, it became easy to ignore all the caveats of scientific bibliometrics.⁸⁹

Moreover, the position of bibliometric science was weak: ⁹⁰ it was fragmented, because the data it works with are private and expensive, to the extent that many analyses that depend on direct access to the raw data of the former ISI cannot be replicated; isolated from the rest of the social sciences; poorly institutionalized; lacking common criteria; and subject to commercial influences, i.e. the temptation to put satisfying the demands of paying customers before scientific rigor. The combination of weaknesses and conflicts of interest with the managerial need for a quick and seemingly objective system for evaluating research has led to an epidemic of ‘amateur bibliometrics“, to which research itself has become a victim.⁹¹ In other words, the flaw of do-it-yourself bibliometrics is its very low scientific reliability, not its “standardization”.

According to Jochen Gläser and Grit Laudel, bibliometrics can only become less amateurish and less prone to conflicts of interest if it emancipates itself from its customers and data suppliers and creates a citation database that is public and controllable by everyone. On the other hand, according to Bonaccorsi, the dependence of bibliometric evaluation on proprietary, monopolistic and commercial databases seems to be only a meaningless historical accident, unworthy of further discussion.

A statistical system was built on these conventions, which was taken over by commercial enterprises while waiting for a public initiative comparable to that of the national statistical offices.⁹²

8.2 A citation experiment: Stephen Cole and “particularism” in science

Bonaccorsi’s cavalier attitude toward those who fear that “evaluation contributes to the death of the university by stabilizing power relations by (falsely) legitimizing them from a scientific standpoint and thus producing conformity to academic power”⁹³ depends on his belief that such fears have no empirical basis. Among the American literature he claims to be referring to, the most recent text, written by Merton’s student Stephen Cole, dates from 1992.

Validation has been painstakingly studied, especially by examining hiring and promotion procedures in American universities and comparing the weight of strictly scholarly criteria with that of particularistic criteria,⁹⁴ in particular age, gender, and the prestige of home departments. In an important book summarizing this literature, Cole concludes:

The research which has been conducted at an aggregate level has not produced any conclusive evidence that particularism plays more than a small role in the way in which scientists and scientific work are evaluated.⁹⁵

Bonaccorsi’s quote from Cole is an excerpt from a larger paragraph that deserves to be quoted in full:

The above represent only a very small unsystematic sample of cases of particularism I have come across. Many readers of this chapter, I am sure, can add cases from their own research or personal experience. Such cases pose an important paradox. The research which has been conducted at an aggregate level has not produced any conclusive evidence that particularism plays more than a small role in the way in which scientists and scientific work are evaluated. Yet at the day-to-day individual level, science, like other institutions, seems to be riddled with particularism.⁹⁶

Cole’s aggregate analysis certainly suggests that the ways in which scientists and scientific work are evaluated are hardly affected by particularism. Cole adds, however, that such findings are at odds with everyday experience. In the passage above, he mentions a “small unsystematic sample” selected from cases that have come to his attention “in twenty-five years of participatory observation in academic science:”⁹⁷ biased conclusions from discordant peer reviews, implicit exchanges of favors, academic vendettas, discriminatory hiring – a brilliant male researcher with an abrasive temperament seems more acceptable than a brilliant but similarly abrasive female researcher -and so on. Cole takes the experience too seriously to dismiss all these cases as “anecdotal”: instead, he asks why his aggregate analysis has failed to capture situations familiar not only to sociologists of science but to anyone who has had even fleeting contact with an academic career.

The empirical studies that began in the late 1960s of the previous century were devoted mainly to the U.S. scientific community and considered only a few grounds for discrimination – age, religion, gender, race, rank of Ph.D. department, rank of current department, past receipt of scientific rewards – both because they were easier to handle statistically and because of the interests of the time. ⁹⁸ It was precisely this selection, Cole explains, that led to the neglect of the most important reasons for particularism: “the positive and negative feelings of individuals toward other individuals and the embeddedness of scientists in networks of interpersonal social relations.”⁹⁹ Contrary to Bonaccorsi’s artificially limited interpretation, Cole concludes that much of the evaluation in science must be classified as particularistic.

In analyzing the difficulty in determining whether to classify a particular evaluation as being based upon particularistic or universalistic criteria, I have concluded that most evaluation in science must be classified as particularistic, but that at least analytically it is possible to distinguish several different bases for particularistic judgments. These bases would include both irrelevant nonscientific statuses and scientific affiliations of the scientist being evaluated as well as persona valences based upon a wide array of determinants, which range from cognitive evaluations of the scientist’s work to the scientist’s personality, political views, or institutional connections with the evaluator.¹⁰⁰

If we take the trouble to read it rather than just quote from it selectively, Cole’s book is not a summary of empirical research that is more or less outdated and of little relevance to a discussion of state evaluation of research:¹⁰¹ it is a serious confrontation with constructivism, resulting in a realist-constructivist theory for which science is partly socially constructed, in the laboratory and in society at large, and partly dependent on empirical facts.¹⁰²

Cole characterizes constructivism as (1) a challenge to the idea that science is a purely rational activity, (2) a relativistic epistemology that treats solutions to scientific problems as underdetermined, and (3) a reduction of the cognitive content of the natural sciences to the results of social variables and processes.¹⁰³
Against positivism, Cole recognizes that science is socially influenced: empirical facts do not speak for themselves, but can be selected and evaluated by biases and interests within research communities. However, radical constructivism cannot explain why, among so many ideas and theories, all socially influenced, some gain acceptance in scientific communities and others do not.¹⁰⁴ An articulate sociology of science should be able to distinguish, at least analytically, between its cognitive and its social components. ¹⁰⁵ Otherwise, its result would be reduced to a useless tautology, according to which science, insofar as it is socially determined, is socially determined, ¹⁰⁶ which would miss the specificity of science as a social institution, as in Hegel’s night in which all cows are black.

In other words, Cole’s data analysis is part of an articulate theoretical reflection:¹⁰⁷in contrast to Bonaccorsi’s cut-and-paste, he does not claim to have empirically verified the absence of particularism in the part of the American academy he has studied. Rather, in the face of conflicting experiences, he questions the effectiveness of his statistical tools and tries to improve them to the point of recognizing the existence of forms of particularism in much of scholarly evaluation, related to interpersonal relationships and social network positions, that might be attenuated in decentralized evaluation systems,¹⁰⁸ only because scholars can always seek their fortunes elsewhere.

On citations, let’s read what the constructivist Bruno Latour writes:

There is something worse, however, than being criticised by other articles; it is being
misquoted. If the context of citations is as I have described, then this misfortune must happen
quite often! Since each article adapts the former literature to suit its needs, all deformations are fair. A given paper may be cited by others for completely different reasons in a manner far from its own interests. It may be cited without being read, that is perfunctorily; or to support a claim which is exactly the opposite of what its author intended; or for technical details so minute that they escaped their author’s attention; or because of intentions attributed to the authors but not explicitly stated in the text; or for many other reasons. We cannot say that these deformations are unfair and that each paper should be read honestly as it is; these deformations are simply a consequence of what I called the activity of the papers on the literature; they all manage to do the same carving out of the literature to put their claims into as favourable as possible a state. If any of these operations is taken up and accepted by the others as a fact, then that’s it; it is a fact and not a deformation, however much the author may protest.¹⁰⁹

Cole quotes this passage to criticize it: Latour describes the rhetorical strategies of scientists, but does not explain what determines the success of some and the failure of others. Exclusively rhetoric, or something else as well? ¹¹⁰ In our own small way, we will also experience this by measuring the fate of a book like La valutazione possibile, which at some not negligible argumentative junctures theorizes in a Mertonian and cites a constructivist way.¹¹¹

8.3 Democracy and numbers

For Bonaccorsi, research evaluation is democratic because it defends the autonomy of science and its procedures are rationally justified. ¹¹² However, if democracy were based only on the above principles, it could function without elections: in some areas, citizens’ preferences would be calculated by algorithms based on data, proprietary ot not, and in others, some groups of experts, under the control of a board appointed by the government, would make decisions on behalf of citizens. And since the only constraint would be the obligation to provide rational justification, dissenters who were not emotional, dim-witted or dishonest could still use pen and paper to persuade the ‘decision-makers’ to revise their deliberations. But such a regime, in which the kratos would be more visible than the demos, would be better described as an enlightened despotism or a technocracy. Why does Bonaccorsi call it “democracy”?

The mathematician Alessandro Figà Talamanca¹¹³ reported how the journal impact factor was used within Italian biomedicine to challenge local academic power and make way for a generation of researchers with a more international vocation. Pitting ‘objective’ numbers against the claims of academic big wigs might seem a ‘democratic’ move, in such a situation. But Bonaccorsi goes beyond this contingent context.

To my mind, this thesis must be overturned and statistics must be given a fundamental role not only in scientific progress but also in the advancement of democracy. The history of social statistics shows how the extension of measurement served to combat the weight of tradition and the old social order, which provided classifications based on traditional forms of knowledge, anchored in unexplained origins.¹¹⁴

His footnotes, which appeal to the authority of voluminous tomes without indicating which of the hundreds of pages support such a thesis, arouse the reader’s curiosity. Among the texts he cites,¹¹⁵ the one with the broadest historical perspective is Trust in Numbers by T.M. Porter. Porter¹¹⁶ Is it a chronicle of the triumph of democracy, enlightened by numbers, over the obscurantism of tradition? If Ravetz’s review of it is to be believed,¹¹⁷ not exactly.

Porter wonders where the prestige of quantitative methods has come from, and refuses to see its cause in the success of modern science, preferring to go backwards and look for its origins in society rather than in science. Without denying their possible validity and usefulness, numbers, graphs and formulae are studied primarily as communication strategies.¹¹⁸

Quantification is generally a way of maintaining distance, of minimizing the need for personal trust and in-depth knowledge.¹¹⁹ Which systems, which sectors, which disciplines have had and need this depersonalization, and why? The two most widespread historiographical theses – based on two rival and incompatible preconceptions – produce the usual nights in which all cows are black: the first narrates quantification as the affirmation of increasingly powerful research methods, the second as the ideological imposition of a power system.¹²⁰

The positive value of the word “objectivity” is rarely called into question. Even if it is considered metaphysical to understand it as conformity to the external object as a thing in itself, it is still valued as disciplinary objectivity or, preferably, as mechanical objectivity. Disciplinary objectivity – think of that of particle physicists – is based on assumptions and procedures that sound esoteric to the uninitiated: positivists and laypeople therefore value more mechanical objectivity, which consists of personal restraint and following rules. ¹²¹ This is also what the rule of law (rule of law) would require: but the very presence of judges, lawyers, jurists, trials, and appeals indicates that legal rules, variously modulated by the most skilled interpreters, cannot be applied mechanically. ¹²² Instead, when we are dealing with quantitative social scientists, mechanicism looks much more immediate and seems to make the judgment of experts more trustworthy, even if their expertise remains in reality at least partly esoteric and linked to informal knowledge and practices that are difficult to make explicit.

This is why a faith in objectivity tends to be associated with political democracy, or at least with systems in which bureaucratic actors are highly vulnerable to outsiders. The capacity to yield predictions or policy recommendations that seem to be vindicated by subsequent experience doubtless counts in favor of a method or procedure, but quantitative estimates sometimes are given considerable weight even when nobody defends their validity with real conviction. The appeal of numbers is especially compelling to bureaucratic officials who lack the mandate of a popular election, or divine right. Arbitrariness and bias are the most usual grounds upon which such officials are criticized. A decision made by the numbers (or by explicit rules of some other sort) has at least the appearance of being fair and impersonal. Scientific objectivity thus provides an answer to a moral demand for impartiality and fairness. Quantification is a way of making decisions without seeming to decide. Objectivity lends authority to officials who have very little of their own.¹²³

Porter does indeed see a connection between the practice of quantification and contemporary democracy. This connection, however, does not concern its essential character, the self-determination of the sovereign people, but rather its bureaucratic-administrative function, i.e., not surprisingly, what it has in common with technocracy. With one difference: in a democracy, numbers are, at least in principle, a fallback justification for the use of bureaucrats who lack authority and political legitimacy; in a technocracy, on the other hand, they might as well be the main or only legitimacy.¹²⁴

Porter’s book, which puts this general thesis to the test by comparing it with more than one historical case, would also deserve to be read and not just quoted. This time, however, Bonaccorsi is right: if democracy is reduced to a bureaucracy without legitimacy, its connection with numbers – which allows it to make decisions without appearing to do so – is indeed clear. Nor is it surprising that state evaluation, which claims to democratize research, instead strives to bureaucratize it, establishing bibliometrics, medians, thresholds, and parameters that, because they are quantitative, seem objective but in fact say very little about what is being researched and written.

9. For the public use of reason

Of late we can observe distinctly that the German universities in the broad fields of science develop in the direction of the American system. The large institutes of medicine or natural science are ‘state capitalist’ enterprises, which cannot be managed without very considerable funds. Here we encounter the same condition that is found wherever capitalist enterprise comes into operation: the ‘separation of the worker from his means of production.’ The worker, that is, the assistant, is dependent upon the implements that the state puts at his disposal; hence he is just as dependent upon the head of the institute as is the employee in a factory upon the management. For, subjectively and in good faith, the director believes that this institute is ‘his,’ and he manages its affairs. Thus the assistant’s position is often as precarious as is that of any ‘quasi-proletarian’ existence and just as precarious as the position of the assistant in the American university.

In very important respects German university life is being Americanized, as is German life in general. This development, I am convinced, will engulf those disciplines in which the craftsman personally owns the tools, essentially the library, as is still the case to a large extent in my own field. This development corresponds entirely to what happened to the artisan of the past and it is now fully under way.

As with all capitalist and at the same time bureaucratized enterprises, there are indubitable advantages in all this. But the ‘spirit’ that rules in these affairs is different from the historical atmosphere of the German university. An extraordinarily wide gulf, externally and internally, exists between the chief of these large, capitalist, university enterprises and the usual full professor of the old style. This contrast also holds for the inner attitude, a matter that I shall not go into here. Inwardly as well as externally, the old university constitution has become fictitious. What has remained and what has been essentially increased is a factor peculiar to the university career: the question whether or not such a Privatdozent, and still more an assistant, will ever succeed in moving into the position of a full professor or even become the head of an institute. That is simply a hazard. Certainly, chance does not rule alone, but it rules to an unusually high degree. I know of hardly any career on earth where chance plays such a role.¹²⁵

As early as 1917, Max Weber spoke about us: the state evaluation of research is only the last act in the transformation of the university into a state-capitalist enterprise.¹²⁶ Bureaucracy and the material and spiritual precarization of researchers – like the transformation of students into clients – are part of this process. Like Marx’s proletarian, the researcher is not master of his or her own means of production, which in the humanities and social sciences is essentially the library, and lets others determine the meaning of his or her work. Weber, who died in 1920, could foresee our alienation, both material and spiritual: today our works can have a value only if they are given away to an oligopoly of publishers and journals whose “excellence“, in Italy, is decreed by an agency appointed by the government.

The randomness of academic careers is one of the arguments used to support the oft-repeated claim that “any evaluation is better than no evaluation at all.”¹²⁷ With Weber, we could respond that traditional, internal evaluation of research remains preferable to state-based, external evaluation.

Only where parliaments, as in some countries, or monarchs, as in Germany thus far (both work out in the same way), or revolutionary power-holders, as in Germany now, intervene for political reasons in academic selections, can one be certain that convenient mediocrities or strainers will have the opportunities all to themselves.¹²⁸

We might also recall that the Humboldtian university – the German model whose crisis Weber announced – is also affected by chance, because it is inspired by a principle that distinguishes it from the school.

Moreover, it is a peculiarity of the higher scientific institutions that they always treat science as a problem that has still not been fully resolved and therefore remain constantly engaged in research, whereas the school deals with and teaches only finished and agreed-upon bits of knowledge.¹²⁹

Frontier research, precisely because it goes beyond what is established and accepted, is subject to a degree of subjectivity and even abuse in its evaluation, which can perhaps be mitigated in decentralized systems, but never eliminated. ¹³⁰ Weber, -Professor at the German University, wrote that young people interested in academic careers must be asked “Do you in all conscience believe that you can stand seeing mediocrity after mediocrity, year after year, climb beyond you, without becoming embittered and inwardly corrupt?”¹³¹

Nonetheless, we might eristically conclude that no external evaluation can be more effective and more respectful of research freedom than that exercised within the scientific community, among peers:

One of the ways of controlling scientific judgements is the reputational censorship that can be exercised by a vigilant and open scientific community. A necessary condition for this censorship to be exercised is that there is a clear attribution of responsibility.¹³²

But how can a scientific community be and remain vigilant and open? How can responsibility be clearly assigned? It is not 1858: the university is no longer the preserve of an élite but is open to the masses; the scientific community has become, with a few exceptions, much larger and more disorganized than the one that gathered in the London headquarters of the Linnean Society to listen to the memoirs of Darwin and Wallace; the oligopolists of commercial publishing and citation databases dominate the academic game with their interests in profit and power.¹³³

As Max Weber foresaw and Jerome Ravetz recognizes, the industrialization of research and the transformation of the university into a state-capitalist enterprise has fractured and disempowered the community on which the ethos of science was based. Such a community was made up of researcher-craftsmen who were free to choose what to study without too much economic or competitive pressure, and who were bound together by an informal network of consensual knowledge and norms; moreover, its small size made it possible to build reputations based on personal acquaintance. Nowadays, a comparatively solid quality control has yet to be found.¹³⁴ Bonaccorsi himself tried to justify the state evaluation by presenting it as the distillation of the self-evaluation of the same scientific community whose autonomy he had helped to dismantle as a member of the ANVUR board.

If the distributed and informal quality control of the scientific community is replaced by bibliometrics and journal fetishism, fraud becomes easier and more attractive. ¹³⁵ According to Telmo Pievani ¹³⁶ the causes are structural.

Scientists who cheat are not isolated cases, but the product of degenerative mechanisms that facilitate misconduct and that have become more acute in recent years. These include the excessive pressure to publish; the fierce competition in certain fields; the frenetic pace of scientific paper production (about two million papers published each year); the need to keep the media visibility of one’s results high at all times in order to obtain funding (increasingly turning the communication of science into marketing); the business of paid scientific journals and pirated journals; the sense of impunity resulting from poor controls; the defensive instincts of the scientific communities themselves; the voracious search for citations to boost their bibliometric indices; the allure of some scientific stories that are too good to be true, and so succeed for a while despite their shaky foundations (even the scientific community has its confirmation biases, which sooner or later collapse in the face of observations that refute them).

If the structures are so strong, it is in vain to invoke ethics as a deus ex machina descending from a heaven far removed from the earth and its customs, or to replace it with a state evaluation, to whose agents government appointment would confer a superior morality. What is collapsing is the very system of relationships that produces, as Pievani puts it, “a free community of equals who learn from their mistakes and constantly self-correct”.

As for the attribution of responsibility, the use of bibliometrics to make decisions without seeming to decide¹³⁷ and anonymous peer review are designed to avoid it. Let’s quote Georgio Israel again:

The anonymity of the reviewer, on the other hand, is a silly and scandalous idea. Those who have to sign a review, and thus put their reputations on the line, are quite careful about what they write, whereas-and there are many examples of this-an anonymous reviewer can afford the luxury of making hasty, superficial judgments, or even blatantly false statements, with the most diverse of intentions, and without having to pay any price for it. The proliferation of selection procedures using anonymous evaluators, far from guaranteeing the seriousness and objectivity of the judgment – it is argued that the anonymous evaluator would be free to express himself without the restraint dictated by his possible relations of acquaintance or friendship with the evaluated or the fear of retaliation – induces unethical, if not downright improper, behavior. Why is anonymity necessary? A person belonging to the world of research and academia should be able to conform to criteria of “science and conscience” and not be afraid to defend decisions made on that basis. Instead, anonymity risks providing cover for intellectually shallow or ethically incorrect behavior.

Anonymity transforms what could be – and has been¹³⁸ – a public discussion among scholars into an opaque exercise of power without interaction with the authors. Understanding editorial publication as a mark of scholarly quality and diligence may have made some sense in the age of printing, whose technological and economic limitations forced a pre-selection of texts destined to see the light of day. Now, however, it would be neither costly nor difficult to treat publication as a trivial act, and to postpone peer review to a later conversation open to all.¹³⁹

In Italy, the main obstacle to this simple revolution is a despotic and retrograde evaluation of research, which does not promote publicness but privatization.

I suggest that all shortcomings in the current publication system are rooted in the fact that it has drifted away from Science ethics, **with publication – peer review, evaluation and dissemination – being privatized. A process whose rationale is to be open, transparent, and community-wide has become trapped in editors’ mailboxes**. **The validity and value of a scientific work are both decided once and for all time, by two or three people in a process that is confidential, private, anonymous, undocumented, and with short deadlines.** Here, I use the term “privatization” not mean that the process is conducted by private companies, but to imply it concentrated in a few hands. Whilst some may consider that private publishers charge exorbitant (and unaffordable) prices for their journals, my arguments still stand if the current system was entirely run by public institutions, learned societies or any non-profit organization.

Understood in the sense explained by Michaël Bon when presenting his alternative, privatization is much more than the outsourcing of data and research evaluation processes into the corporate hands of oligopolists animated by commercial interests: it is the dispossession of that publicly accessible and non-ancillary space for discussion that the scientific community had been able to invent and defend around the first modern scientific journal, the Philosophical Transactions, even if only as a club-good.¹⁴¹ of the Royal Society.

To grasp the difference, which is quite obvious, between Renaissance magic and modern science, it is necessary to reflect not only on content and methods, but also on the images of knowledge and the images of the knower. There are certainly many mysteries in our world, and many theorists and practitioners of the arcana imperii live in it. There are also many and often not “honest” dissimulations. Even in the history of science there have been dissimulators. It should be emphasized, however, that after the first scientific revolution, in the scientific literature and in the literature about science, there is and can be no eulogy or positive assessment of dissimulation, in contrast to what has largely happened and is happening in the world of politics. Dissimulation, not making one’s views public, is simply cheating or betraying. Scientists, as a community, can be forced into secrecy, but they must be forced. When such coercion occurs, they protest in various ways or even, as has been the case even in this century, resolutely rebel against it. The particle of in the linguistic expression “Kepler’s laws” does not indicate any property at all: it only serves to perpetuate the memory of a great personality. Secrecy, for science and within science, has become a disvalue.¹⁴²

In state evaluation, bibliometrics, as the supposed science of science, depends on proprietary data; the authority of the evaluators rests not on the judgment of scientists but on government appointment; the names of the reviewers are secret; and public discussion is a waste of time for those engaged in producing “research products”. In 1957, Merton wrote that outright fraud – such as cheating and falsifying data – is relatively rare in the scientific world because “personal honesty is supported by the public and testable character of science”.¹⁴³ Would he hold the same view today?

Roberto Caso reports on an article written by Bonaccorsi in collaboration with others, available in open access and offered for open peer review on the F1000 platform. The fact that the names of the reviewers were public made it possible to verify that the authors were employed by ANVUR at the time the article was published, while three of the reviewers were involved in research projects funded by the same agency, and one of them subsequently published a positive review of La valutazione possibile. However, the publicness and openness of the comments on F1000 allowed an independent reader, at his own risk, to point out some weaknesses in the paper.¹⁴⁴ If the paper had been published in a subscription journal after a secret peer review, no one would have known that the reviewers and authors belonged to the same institutional and social network; the independent commenter, if he could afford to read the article, would have been able to submit his comments only separately and not necessarily with the same visibility.

Despite these problems, between 2000 and 2010, while the number of scientific articles published increased by 40 percent, there was a tenfold increase in retractions, i.e., the withdrawal of articles already published in closed peer-reviewed journals because they were found ex post to be affected by plagiarism, fabrication or falsification of data, and other misconduct. Is this a sign of good health? According to Ivan Oransky and Adam Marcus, who report these figures, not necessarily:¹⁴⁵ a system based on publication fetishism encourages honing the art of cheating and fails to reward those who read texts carefully and discover their weaknesses. The publish or perish turns publication into the agonistic gesture of those who must win at all costs, and ex post criticism into an antagonistic gesture, because the space of public discussion, in which each researcher presents in good faith the best he has managed to achieve and others help him correct his mistakes, has again been privatized and thus suppressed. We must ask if and how it can be reconstructed.

Technically, it would be enough to use the¹⁴⁶ tools we already have. The open World Wide Web¹⁴⁷ was designed for the research world to allow anyone to share documents with anyone. In Europe, we could even now bypass commercial publishers and use overlay journals, which report on and review texts already deposited in open access institutional and disciplinary repositories by subjecting them to closed or, preferably, open peer review.¹⁴⁸ All that now remains in the dark, with an esoteric aura that has nothing to do with public scholarship, would be said in the light.

In particular, putting research online in all its stages would show something that the print-based world has kept hidden: the process, complex and never completed,¹⁴⁹ that leads to the – tentative – conviction that a theory has some value.

When science was a type of publishing, it aimed at producing knowledge that was – like a publication -broken off from its source because it was embodied in a physical thing with a life of its own. The new issue of Nature arrives on the desk of the scientist, and she sighs in relief. Her research is out there at last. If, heaven forbid, a truck were to hit her this morning, the knowledge wouldn’t die with her. It now has a life of its own that can be tracked and weighed. But now that science is becoming a network, knowledge is not something that gets pumped out of the system as its product. The hyperlinking of science not only links knowledge back to its sources. It also links knowledge into the human contexts and processes that produced it and that use it, debate it, and make sense of it. The final product of networked science is not knowledge embodied in self-standing publications. Indeed, the final product of science is now neither final nor a product. It is the network itself—the seamless connection of scientists, data, methodologies, hypotheses, theories, facts, speculations, instruments, readings, ambitions, controversies, schools of thought, textbooks, faculties, collaborations, and disagreements that used to struggle to print a relative handful of articles in a relative handful of journals..¹⁵⁰

According to David Weinberger, networked science is more like science as experienced by researchers than as reported by the mass media and,¹⁵¹ we might add, by state evaluators who reduce research to “products. The Royal Society was a forum for discussion, although only an élite participated in its philosophical transactions. The open World Wide Web could be just as much, but under the eyes of everyone. On one condition: that researchers were encouraged to make their work public, and rewarded for sharing texts and data and taking responsibility for showing that science is a problem not yet fully solved, instead of kneeling before a fetishistic celebration of those who privatize science and hide it behind anonymity, usually in the service of the publishing oligopolies or the state.

These are not new ideas: in his essay on the Enlightenment (Ak, VIII, 37), Immanuel Kant distinguished between a public use of reason – made by every human being, regardless of his or her profession, when speaking as a scholar to the society of the citizens of the world – and a private and mechanical use of reason, to which those who act as officials within certain collective organizations are bound. While the private use of reason can be restricted, its public use must be left free: people who speak as scholars must be allowed to speak to everyone and say what they think, as they think it, as they want to say it.

This freedom is culturally and socially important because it can spread “the spirit of a rational valuing of one’s own worth and of the calling of each human being to think for himself” (Ak, VIII, 36). Access to and participation in the debate on problems that have not yet been fully resolved helps those who are constrained by the mechanisms of the state or the corporation to understand that what seems predetermined and unquestionable is not at all so, that they themselves can be more than their obedience to orders, that their subjugation can be overcome, that the world can be changed.¹⁵²

Certainly, a society that defends the freedom of the public use of reason and accepts to pay scholars who are free and critical thinkers rather than functionaries must accept that efficiency is not its only and main problem. It must therefore be able to recognize that, as a society, it is imperfect and still collectively engaged in research, even in the seemingly technical question of how to disseminate and discuss science. From this point of view, a structurally despotic and retrograde state evaluation does not simply hurt the privileges of a minority, but contributes to the construction of a general model of society that is equally despotic and retrograde. The digital revolution has only made clearer what was already clear in Kant’s time and well known to the authors of the Italian Constitution: freedom of research and culture is not only a freedom for scholars: it is a freedom for everyone, because everyone can enjoy and participate in reason when its use is truly and not rhetorically public.